diff --git a/blog/2023-03-29-troubleshooting-common-issues-and-solutions-to-mongodb-etl-errors.mdx b/blog/2023-03-29-troubleshooting-common-issues-and-solutions-to-mongodb-etl-errors.mdx index 9ca72f15d..be659322c 100644 --- a/blog/2023-03-29-troubleshooting-common-issues-and-solutions-to-mongodb-etl-errors.mdx +++ b/blog/2023-03-29-troubleshooting-common-issues-and-solutions-to-mongodb-etl-errors.mdx @@ -336,48 +336,67 @@ Are you interested in unlocking the full potential of your data without the need With features like data ingestion from 150+ sources including MongoDB connectors, data warehousing, data analytics, and data transformation solutions, Datazip can help you make fast, data-driven decisions. ## FAQs -### Q1. What are the most common MongoDB ETL errors and how do you diagnose them? -- **Connection timeout errors** — Check network connectivity, firewall/security group rules blocking port 27017, and MongoDB authentication credentials -- **Schema validation failures** — Caused by polymorphic fields or missing required fields across documents in the same collection -- **Data type mismatch errors** — Where the source field type differs from the target column type -- **Socket timeout (`socketTimeoutMS`) exhaustion during large collection scans** — Occurs when MongoDB takes longer than the configured `socketTimeoutMS` to respond to a query, common during unoptimized aggregate queries or large full-collection reads. Increase `socketTimeoutMS` in your connection settings and ensure queries are properly indexed to avoid full collection scans. - -### Q2. How should I set up MongoDB for ETL to minimize pipeline errors? -Best practices for an ETL-ready MongoDB setup include: - -- **Enable Read Preference on secondary nodes** to offload ETL reads from the primary and avoid impacting operational performance -- **Create indexes on user-defined timestamp fields** (such as an application-managed `updated_at` field) that are used for cursor-based incremental sync — note this is not a built-in MongoDB field and must be maintained by your application -- **Set `socketTimeoutMS` and `serverSelectionTimeoutMS`** appropriately per operation for long-running collection reads, keeping in mind these are per-operation settings, not session-level configurations in most drivers -- **Configure oplog retention** to cover at least 24–48 hours of changes to ensure CDC consumers do not fall behind the retention window -- **Ensure a replica set is configured:** this is a hard requirement for change streams and oplog-based CDC; standalone MongoDB instances do not support these features - -### Q3. What causes connection timeout errors in MongoDB ETL pipelines and how do I fix them? -Connection timeouts typically occur due to: - -- **Network/firewall issues:** Firewall or security group rules blocking the ETL tool's IP from reaching MongoDB on port 27017 -- **Authentication failures:** Wrong credentials, incorrect `authSource` database, or the user lacking required permissions -- **Connection pool exhaustion:** Too many concurrent ETL workers exceeding the `maxPoolSize` setting, or connection leaks in application code causing "server selection timed out" errors -- **SSL/TLS configuration mismatches:** The ETL tool lacking the correct CA certificate to validate the MongoDB server's SSL certificate - -**Recommended debug approach:** Test connectivity directly with the MongoDB shell (`mongosh`) using the same connection string first. If that succeeds, the issue is in your ETL tool's configuration — verify credentials, SSL settings, and connection string parameters. If the shell also fails, the issue is at the network or DNS level. - -### Q4. How do I handle schema validation errors when MongoDB documents have inconsistent structures? -Schema validation errors occur because MongoDB allows polymorphic data — documents with varying structures or different data types for the same field — within a single collection. Solutions include: - -- **Use schema inference with adequate sampling** — Increase the sample size when inferring the schema so the ETL tool captures the full range of field variations, rather than relying on a small, potentially unrepresentative subset -- **Mark fields as nullable/optional** for fields that may be absent in some documents -- **Apply type coercion rules** to handle polymorphic fields by enforcing a consistent target type during ingestion -- **Filter or quarantine malformed documents** using pre-ingestion validation rules — MongoDB also supports `validationAction: "warn"` mode, which logs invalid documents without rejecting them, making it a useful diagnostic tool during ETL pipeline development -- **Use a compatible ETL tool** that natively supports MongoDB's BSON types (including `Decimal128`, `ObjectID`) and flexible schema evolution - -### Q5. What are best practices for MongoDB ETL setup in production environments? -For production MongoDB ETL pipelines: - -- **Use a dedicated read-only ETL user** with the minimum permissions required — typically `read` on source collections and `clusterMonitor` for oplog access -- **Connect to a replica set secondary** to avoid adding read load to the primary node -- **Implement checkpointing using resume tokens** so failed syncs resume from the last successfully processed oplog position rather than restarting from scratch — store the resume token durably and pass it back on reconnection -- **Monitor oplog lag actively** — a small oplog (e.g., 1GB on a high-throughput cluster) may only retain a few hours of changes; if your CDC consumer falls behind the retention window, you will need to trigger a full resync -- **Test oplog partial-update handling in staging** before deploying to production — MongoDB's `$set` update operator produces partial update events in the oplog (not full document replacements), and many ETL tools handle these differently; validate that your tool correctly reconstructs the full document from partial oplog events before going live - + +
  • Connection timeout errors. The job fails before reading any data. Diagnose by checking network connectivity to the host, firewall or security group rules blocking port 27017, and your MongoDB authentication credentials.
    • +
  • Schema validation failures. Records get rejected at the destination. Diagnose by sampling documents in the collection and looking for polymorphic fields or required fields missing across documents.
    • +
  • Data type mismatch errors. A column fails to write or values get coerced unexpectedly. Diagnose by comparing the source field type against the target column type for the field named in the error.
    • +
  • Socket timeout (socketTimeoutMS) exhaustion during large collection scans. A query runs longer than the configured socketTimeoutMS and the connection drops mid-scan. Diagnose by comparing query duration against socketTimeoutMS and checking whether the query uses an index. Raise socketTimeoutMS and add indexes so reads avoid full collection scans.
    • + + }, + { + question: "Q2. How should I set up MongoDB for ETL to minimize pipeline errors?", + answer:
    +

    Best practices for an ETL-ready MongoDB setup include:

    + +
    + }, + { + question: "Q3. What causes connection timeout errors in MongoDB ETL pipelines and how do I fix them?", + answer:
    +

    Connection timeouts typically occur due to:

    + +

    Recommended debug approach: Test connectivity directly with the MongoDB shell (mongosh) using the same connection string first. If that succeeds, the issue is in your ETL tool's configuration. If the shell also fails, the issue is at the network or DNS level.

    +
    + }, + { + question: "Q4. How do I handle schema validation errors when MongoDB documents have inconsistent structures?", + answer:
    +

    Schema validation errors occur because MongoDB allows polymorphic data within a single collection. Solutions include:

    + +
    + }, + { + question: "Q5. What are best practices for MongoDB ETL setup in production environments?", + answer:
    +

    For production MongoDB ETL pipelines:

    + +
    + }, +]} /> \ No newline at end of file diff --git a/blog/2024-09-16-mongodb-etl-challenges.mdx b/blog/2024-09-16-mongodb-etl-challenges.mdx index 704f3af08..1fdfed1b9 100644 --- a/blog/2024-09-16-mongodb-etl-challenges.mdx +++ b/blog/2024-09-16-mongodb-etl-challenges.mdx @@ -563,7 +563,70 @@ By following best practices, such as using CDC, batching, and data validation, c *4. MongoDB Documentation, "Working with Nested Data," MongoDB.* - +## FAQs + +

    The four key challenges are:

    +
      +
    1. Schema flexibility: MongoDB's schema-less design creates inconsistent field structures that clash with rigid warehouse schemas
      2. +
    2. Large initial loads: Must be parallelized and checkpointed to handle terabyte-scale collections reliably
      3. +
    3. Changing data types (polymorphic keys): The same field can appear as different types across documents (e.g., age as an integer in one document and a string in another)
      4. +
    4. Complex nested fields and arrays: Must be transformed into a flat relational format without causing row explosion or data duplication
      5. +
    + + }, + { + question: "Q2. How does MongoDB's schema flexibility create problems during ETL to structured systems?", + answer:
    +

    MongoDB allows any document to omit fields or use different types for the same field across documents. When moving this data to relational warehouses or Iceberg tables that require consistent schemas, you encounter:

    +
      +
    • Type mismatches: A field that is an integer in some documents and a string in others
      • +
    • Missing values: Sparse fields that exist in only a subset of documents require NULL-filling across the rest
      • +
    • Inconsistent nesting structures: The same logical field may appear as a simple string in one document and a nested object in another
      • +
    +

    ETL pipelines must detect and resolve these variations without silently dropping or corrupting data.

    +
    + }, + { + question: "Q3. What is the best approach for handling the first full load of a large MongoDB collection?", + answer:
    +

    For large collections, parallelize the initial load using _id-based range queries or bucket-based partitioning to split the collection into independent read ranges, then load those ranges concurrently across multiple worker threads.

    +

    Key practices to follow:

    +
      +
    • Implement checkpointing so that if the load fails mid-way, it resumes from the last successfully completed chunk rather than restarting from scratch
      • +
    • Read from a replica set secondary to avoid load on the primary during the bulk read
      • +
    • Record the oplog timestamp or resume token at the start of the full load so that incremental CDC replication can pick up exactly from that point once the snapshot completes
      • +
    +

    After the full load completes, switch to change streams or oplog-based CDC for ongoing incremental replication.

    +
    + }, + { + question: "Q4. How should you handle array fields when doing MongoDB ETL to a relational target?", + answer:
    +

    Arrays in MongoDB documents should generally be exploded into separate child tables with foreign key references to the parent document. Strategies by array type:

    +
      +
    • Arrays of simple values: Create a child table with a parent_id column and a value column. Each array element becomes one row.
      • +
    • Arrays of complex objects: Each object in the array becomes a full row in the child table, with all object fields mapped to columns and a foreign key back to the parent.
      • +
    +

    Avoid flattening arrays inline: this causes row explosion and massive data duplication, making the final dataset several times larger than the original.

    +

    For arrays you do not need for analytics, consider skipping them entirely during extraction rather than flattening unnecessarily.

    +
    + }, + { + question: "Q5. How do you handle polymorphic data types in MongoDB ETL pipelines?", + answer:
    +

    Polymorphic fields, where the same key holds values of different types across documents, can be addressed using one of these strategies:

    +
      +
    • Type promotion. Promote all values to the most permissive compatible type (e.g., convert both integers and strings to string). Use numeric type promotions where safe (e.g., int to long, float to double). Best when the types are compatible and you want a single clean column. Avoid it when promoting to string would lose type information you need downstream.
      • +
    • Separate typed columns. Create distinct columns per data type (e.g., age_int and age_string). Older data stays in the original column; new data with a different type populates the new column. Best when you need to preserve every value losslessly and downstream consumers can handle querying across columns. The tradeoff is a wider, sparser table.
      • +
    • Schema inference with sampling. Run a sampling step across the collection before defining your pipeline schema, to determine the dominant type for each field and surface polymorphic fields early. Best as a first step before any pipeline run, so you choose the right handling strategy instead of discovering type conflicts mid-sync.
      • +
    • JSON/variant column. Store the field as a semi-structured column and handle parsing in downstream transformations. Best when types are unpredictable or change often and you want to defer parsing. Only available when your target warehouse natively supports it, such as Snowflake's VARIANT, BigQuery's JSON, or Redshift's SUPER.
      • +
    +
    + }, +]} /> I’d love to hear your thoughts about this, so feel free to reach out to me on [LinkedIn](https://www.linkedin.com/in/zriyansh/). diff --git a/blog/2024-09-24-querying-json-in-snowflake.mdx b/blog/2024-09-24-querying-json-in-snowflake.mdx index e27d1840c..eae23befb 100644 --- a/blog/2024-09-24-querying-json-in-snowflake.mdx +++ b/blog/2024-09-24-querying-json-in-snowflake.mdx @@ -1449,104 +1449,126 @@ That’s when they realized why their queries have been taking longer and longer Other than that, the only thing holding you to run your queries fast is the size of your warehouse and the amount you wish to spend on Snowflake. -### FAQs - -**1. What file formats does Snowflake support for loading semi-structured data?** - -Snowflake supports several file formats for loading semi-structured data. These include: - -* JSON (classic for handling key-value pairs) - -* Avro (used in data pipelines) - -* Parquet (for analytics) - -* ORC (efficient storage for large datasets) - -* XML (even though it's messy, it works) - -* CSV (yup, plain old text files too) - - -**2. Which data formats are supported by Snowflake when unloading semi-structured data?** - -When you’re unloading semi-structured data, Snowflake supports: - -* JSON - -* Parquet - -* CSV - - -So if you're looking to move your data out in a semi-structured format, those are your go-to options. Parquet is especially handy when you need something lightweight for analytics. - -**3. Does Snowflake charge a premium for storing semi-structured data?** - -No, Snowflake doesn’t charge extra for storing semi-structured data. The costs are based on the compressed storage size, so whether it’s JSON or Parquet, you're not paying a premium just because it’s not structured. - -**4. Why is Snowflake good for working with JSON and semi-structured data?** - -Snowflake handles JSON and semi-structured data well. Here’s why: - -* **Native VARIANT type**: Snowflake stores JSON as a VARIANT data type, which can handle nested and flexible schemas. - -* **Built-in Functions**: You get functions like `FLATTEN()` and JSON path expressions, making it easy to query nested data. - -* **Columnar Storage**: Even though it's JSON, Snowflake stores it in a way that still supports fast analytics (without slow JSON parsing). - -* **Schema-less Storage**: You don’t have to worry about rigid schemas. - - -**5. What is the recommended Snowflake data type to store semi-structured data like JSON?** - -The recommended data type is VARIANT. It’s Snowflake’s magic box for semi-structured data. You can throw JSON, Avro, Parquet, etc., into this column, and Snowflake will handle it all. - -**6. How does Snowflake support semi-structured data such as JSON and Parquet?** - -Snowflake has a native semi-structured data architecture, which means you can: - -* **Ingest data**: Load JSON, Parquet, Avro, or whatever into VARIANT columns. - -* **Query it**: Use SQL (and JSON path expressions) to extract the exact data you need. - -* **Analyze it**: Snowflake's columnar storage makes querying this data fast, even with complex, nested structures. - - -**7. How does Snowflake handle semi-structured JSON data for fast analytics?** - -Snowflake uses columnar storage even for semi-structured data like JSON. This means: - -* It stores JSON efficiently by breaking it into columns behind the scenes. - -* When you run queries, it doesn’t need to parse through the entire JSON structure — it can just take the parts you need. - - -So, even though it's semi-structured, you still get analytics speeds similar to structured data! - -**8. Why is JSON considered semi-structured data?** - -JSON is considered semi-structured because: - -* It doesn’t have a rigid schema like a relational database. - -* The structure can vary between records (you can have optional fields, nested objects, etc.). - -* It’s more flexible than CSVs or relational tables but still has some structure — it’s not a random string of text, after all! - - -That’s why we call it "semi-structured" — it’s structured, but loosely. - -**9. Can I ingest BSON or JSONB or XML directly into Snowflake?** - -You can ingest `JSON` and XML directly into Snowflake, no problem. As for `BSON` and `JSONB`, you’d need to convert those into regular `JSON` before loading them. Snowflake likes its `JSONs` pure and simple. - -**10. What is the difference between flatten and lateral flatten in Snowflake?** - -`FLATTEN` handles basic unnesting of arrays or objects. `LATERAL FLATTEN` lets you dig deeper, applying flattening row by row in more complex scenarios. - +I’d love to hear your thoughts about this, so feel free to reach out to me on [LinkedIn](https://www.linkedin.com/in/zriyansh/). +## FAQs + +

    JSON sits between fully structured and unstructured data:

    +
      +
    • It has no rigid schema the way a relational table does.
      • +
    • The structure can vary between records, with optional fields and nested objects.
      • +
    • It is more flexible than CSV or relational tables but still carries structure through its keys and values, so it is not free-form text.
      • +
    + + }, + { + question: "Q2. What file formats does Snowflake support for loading semi-structured data?", + answer:
    +

    Snowflake can load semi-structured data from:

    +
      +
    • JSON for key-value and nested structures
      • +
    • Avro commonly used in data pipelines
      • +
    • Parquet for analytics workloads
      • +
    • ORC for efficient storage of large datasets
      • +
    • XML
      • +
    • CSV
      • +
    +
    + }, + { + question: "Q3. Which data formats does Snowflake support when unloading semi-structured data?", + answer:
    +

    When unloading semi-structured data, Snowflake supports JSON, Parquet, and CSV. Parquet is the lightest option when the output feeds analytics.

    +
    + }, + { + question: "Q4. Can I ingest BSON, JSONB, or XML directly into Snowflake?", + answer:
    +

    You can ingest JSON and XML directly. BSON and JSONB are not supported natively, so convert them to standard JSON before loading.

    +
    + }, + { + question: "Q5. Does Snowflake charge a premium for storing semi-structured data?", + answer:
    +

    No. Storage cost is based on compressed size, so semi-structured data like JSON or Parquet is billed the same way as structured data, with no premium for being semi-structured.

    +
    + }, + { + question: "Q6. What is the recommended Snowflake data type for storing semi-structured data like JSON?", + answer:
    +

    Use VARIANT. It accepts JSON, Avro, Parquet, and similar formats in a single column, and Snowflake manages the internal representation for you.

    +
    + }, + { + question: "Q7. Why is Snowflake good for working with JSON and semi-structured data?", + answer:
    +

    Snowflake has native semi-structured support built into its architecture:

    +
      +
    • Native VARIANT type stores nested, flexible structures without a fixed schema, so ingestion is schema-less.
      • +
    • Built-in functions like FLATTEN() and JSON path expressions let you query nested data with standard SQL.
      • +
    • Columnar storage applies even to JSON. Snowflake shreds the structure into columns at load time, so queries read only the parts they need instead of parsing the full document on every SELECT.
      • +
    • One workflow to ingest, query, and analyze JSON, Parquet, and Avro through the same VARIANT columns.
      • +
    +
    + }, + { + question: "Q8. Why does Snowflake sometimes fail to extract VARIANT elements into columnar form, and how does it affect performance?", + answer:
    +

    Snowflake tries to extract VARIANT fields into columnar form at load time for faster querying, but two cases block this:

    +
      +
    • Elements that contain even a single "null" value are not extracted into a column.
      • +
    • Elements that contain multiple data types are also not extracted into a column.
      • +
    +

    When a field is not extracted into a column, the execution engine has to scan the entire JSON structure and traverse it row by row, which directly slows queries. To avoid this:

    +
      +
    • Extract semi-structured elements containing null values into relational columns before loading.
      • +
    • Or set the file format option STRIP_NULL_VALUES = TRUE when loading, which removes array or object elements containing null values.
      • +
    +
    + }, + { + question: "Q9. When should I prefer OBJECT or ARRAY types over VARIANT for storing semi-structured data?", + answer:
    +
      +
    • Use typed ARRAY / OBJECT when the nested schema is stable and you want strict type validation and predictable query performance. These structured types suit production tables with repeated, well-known nested fields.
      • +
    • Keep using VARIANT when the nested shape is highly variable, evolving, or ingestion is heterogeneous. VARIANT remains the best option for exploratory, ingestion-first workflows.
      • +
    +

    Important: Typed ARRAY and OBJECT columns are only supported in standard Snowflake-managed tables. They cannot be used in dynamic, hybrid, or external tables, and defining them in those table types raises an error.

    +
    + }, + { + question: "Q10. When should I use LATERAL FLATTEN vs. [] bracket notation to access array elements?", + answer:
    +

    These serve different purposes:

    +
      +
    • LATERAL FLATTEN(...). Use it when you need every element of an array as its own row, or when the array size varies across records. It is also required when chaining multiple FLATTEN calls on nested structures, since each FLATTEN can reference the output of the previous one.
      • +
    • Bracket [] notation. The right tool when you already know the exact index of the element you need. It avoids exploding the entire array into rows and also handles keys containing hyphens, spaces, or other characters that dot notation cannot.
      • +
    +
    + }, + { + question: "Q11. Can PARSE_JSON and LATERAL FLATTEN be used together to flatten an array stored as a plain string?", + answer:
    +

    Yes. If your array is stored as a raw string rather than a VARIANT, first convert it with PARSE_JSON (or TRY_PARSE_JSON for safety with untrusted or dynamic data, since it returns NULL instead of throwing an error on invalid JSON), then pass the result into LATERAL FLATTEN.

    +

    Best practice: Use PARSE_JSON in your COPY INTO or INSERT statement so the data lands as VARIANT at load time, rather than storing it as VARCHAR and parsing it at query time on every SELECT. Parsing at query time is an avoidable, repeated performance cost.

    +

    Note: If your column is already VARIANT, do not call PARSE_JSON on it again. That is unnecessary work and adds overhead on every query.

    +
    + }, + { + question: "Q12. What are the real performance trade-offs of using LATERAL FLATTEN on large datasets?", + answer:
    +

    FLATTEN should be used sparingly. Flattening large arrays expands rows rapidly and consumes more compute. Best practices:

    +
      +
    • Pre-aggregate or stage the data before flattening where possible.
      • +
    • For very large arrays, filter first or flatten incrementally in smaller steps.
      • +
    • Avoid running LATERAL FLATTEN across entire dbt projects at scale when only a subset of the nested structure is needed. Flattening the whole JSON structure when only a single column's history is required causes queries to grow significantly as data volumes increase.
      • +
    +
    + }, +]} /> -I’d love to hear your thoughts about this, so feel free to reach out to me on [LinkedIn](https://www.linkedin.com/in/zriyansh/). \ No newline at end of file diff --git a/blog/2024-10-10-handling-changing-data-type-during-semi-structured-data-ingestion.mdx b/blog/2024-10-10-handling-changing-data-type-during-semi-structured-data-ingestion.mdx index 876742fbd..3440506cf 100644 --- a/blog/2024-10-10-handling-changing-data-type-during-semi-structured-data-ingestion.mdx +++ b/blog/2024-10-10-handling-changing-data-type-during-semi-structured-data-ingestion.mdx @@ -925,5 +925,61 @@ If set to `ALLOW_ALL`, Fivetran will sync all newly detected columns and tables, I’d love to hear your thoughts about this, so feel free to reach out to me on [LinkedIn](https://www.linkedin.com/in/zriyansh/). +## FAQs + + +

    Polymorphic keys are fields in semi-structured data (such as MongoDB documents or JSON payloads) where the same field can hold values of different types across records, for example a field that is sometimes an integer and sometimes a string.

    +

    This creates serious problems when loading data into typed systems like data warehouses or Iceberg tables, which require columns to have a single consistent data type.

    + + }, + { + question: "Q2. What is the best strategy for handling polymorphic data types during ingestion?", + answer:
    +

    The most common strategies are:

    +
      +
    1. Separate columns per data type: Create distinct columns per type (e.g. age_int and age_string), keeping values in their original type
      2. +
    2. Type promotion: Promote all values to the most flexible compatible type (e.g. coerce all values to string)
      3. +
    3. Dynamic typing: Apply dynamic typing using a VARIANT or JSON column
      4. +
    4. Continuous schema inference: Re-sample incoming data at regular intervals to update the schema automatically as new types appear
      5. +
    +

    The best choice depends on whether downstream queries need type-correct arithmetic or just text access.

    +
    + }, + { + question: "Q3. What is type promotion in data ingestion and when should I use it?", + answer:
    +

    Type promotion means automatically casting a column from a narrow type to a wider compatible type when new data requires it for example, upgrading an integer column to bigint when a value exceeds integer bounds, or promoting a numeric column to string when a text value arrives.

    +

    Use type promotion when you want a single unified column with the most permissive compatible type and downstream consumers can handle the promoted type without breaking their queries.

    +

    Important: In Iceberg-backed lakehouses, type promotion is strictly widening-only (e.g. int → long, float → double). Narrowing changes such as BIGINT → INT are not supported and will result in an error. Type promotion in Iceberg is a one-way operation you cannot downgrade a column type after promoting it.

    +
    + }, + { + question: "Q4. How does continuous schema inference help manage changing data types at scale?", + answer:
    +

    Continuous schema inference re-samples incoming data at regular intervals to detect new fields or type changes, then updates the target table schema automatically. This avoids the need to manually intervene when source schemas evolve.

    +

    The trade-off is computational overhead from ongoing schema analysis. Regarding query disruption:

    +
      +
    • In traditional warehouses (e.g. Redshift, Snowflake), some schema updates that require type changes can involve table migrations that disrupt queries. Safe widenings like INT to BIGINT are usually metadata-only, but narrowing or incompatible type changes are the ones that force a rewrite.
      • +
    • In Iceberg-backed lakehouses, most schema updates are metadata-only operations and do not affect existing data files, meaning there is no data migration and minimal query disruption in most cases.
      • +
    +
    + }, + { + question: "Q5. How does OLake Go handle polymorphic keys when replicating MongoDB to Apache Iceberg?", + answer:
    +

    OLake Go uses Iceberg's native schema evolution to handle type changes. When a field that was previously an integer begins arriving as a string, OLake Go detects the incompatibility and applies the safest resolution available today:

    +
      +
    • For compatible widening changes (e.g. int → long, float → double), OLake Go automatically promotes the column type via a metadata-only Iceberg schema update, no existing data files are rewritten.
      • +
    • For incompatible type changes not supported by Iceberg v2 (e.g. INT → STRING), OLake Go converts the value directly to the new type at write time rather than rejecting or queuing it.
      • +
    +

    Coming soon: Routing values that can't be safely converted to a Dead Letter Queue (DLQ) column, so they're preserved without blocking the sync, is on the roadmap but not yet available. Check the OLake documentation for current status.

    +
    + }, +]} /> + + \ No newline at end of file diff --git a/blog/2024-10-18-flatten-array.mdx b/blog/2024-10-18-flatten-array.mdx index eaa3b2b0f..b8367ecd5 100644 --- a/blog/2024-10-18-flatten-array.mdx +++ b/blog/2024-10-18-flatten-array.mdx @@ -728,4 +728,45 @@ The key is to understand your data and choose the method that makes your life ea I’d love to hear your thoughts about this, so feel free to reach out to me on [LinkedIn](https://www.linkedin.com/in/zriyansh/). +## FAQs + +

    The main approaches are:

    +
      +
    1. Flatten only first-level keys. Map simple first-level fields into columns and keep nested objects or arrays as JSON strings for later processing.
      2. +
    2. Recursive flattening. Recursively flatten all nested keys into dot-notation columns (e.g. user.address.city becomes user_address_city).
      3. +
    3. Separate tables for arrays. Explode array fields into separate related tables with foreign keys back to the parent.
      4. +
    4. Tool-based flattening. For moderate-scale or exploratory work, pandas json_normalize handles flattening in Python. For large-scale distributed workloads, PySpark with spark.read.json() and explode() for arrays does the same at scale.
      5. +
    +

    Each method has different trade-offs in complexity, query ease, and storage efficiency.

    + + }, + { + question: "Q2. What is the best strategy for handling array fields in nested JSON flattening?", + answer:
    +

    The recommended approach for array fields is to create separate child tables that reference the parent via a foreign key, rather than flattening arrays inline. This prevents data explosion, where every array element duplicates all parent-level columns, and keeps the resulting tables at a manageable row count.

    +

    Note that not every tool implements this approach. OLake Go, for example, stores the array field as a stringified value, so the entire array lands in a single string column rather than being exploded into a child table. If you need arrays in separate related tables, you handle that in a downstream transformation after load.

    +
    + }, + { + question: "Q3. How do ETL tools like Airbyte and Fivetran handle nested JSON flattening?", + answer:
    +

    Airbyte previously offered Basic Normalization, which flattened one level of nesting and created separate tables for arrays. That feature is now deprecated. Current Airbyte uses Typing and Deduping, which maps each stream to a single table with typed top-level columns and does not unnest nested objects or arrays into separate tables. Handling deeper nesting is left to downstream SQL transformations or dbt models. Fivetran automatically unpacks the top level of nested JSON into columns and leaves deeper structures as JSON for you to flatten in downstream transformations.

    +
    + }, + { + question: "Q4. When should I use recursive JSON flattening versus creating separate tables for arrays?", + answer:
    +
      +
    • Use recursive flattening when nested objects are shallow (1–2 levels) and arrays are small or rarely queried directly. This keeps everything in a single table for simple queries.
      • +
    • Create separate tables for arrays when arrays are large, deeply nested, or frequently joined with other data. Separate tables prevent the cartesian explosion problem, where flattening large arrays multiplies every parent row by the array length, creating massive and redundant result sets.
      • +
    +

    A useful rule of thumb: if flattening an array would increase your row count by more than 2–3x on average, it belongs in a separate child table.

    +
    + }, +]} /> + + \ No newline at end of file diff --git a/blog/2024-11-05-mongodb-synchronization-strategies.mdx b/blog/2024-11-05-mongodb-synchronization-strategies.mdx index 74bd2b3ad..500c10e4e 100644 --- a/blog/2024-11-05-mongodb-synchronization-strategies.mdx +++ b/blog/2024-11-05-mongodb-synchronization-strategies.mdx @@ -169,6 +169,70 @@ Achieving real-time data synchronization in MongoDB presents a unique set of cha 3. **Data Consistency Challenges**: Ensuring data consistency across distributed systems is complex, particularly when multiple sources are writing to the same MongoDB instance. **Solution**: Above mentioned strategies can solve this problem as per MongoDB docs there are no chances of missing the data by using any of the above strategies properly. - + +## FAQs + + +

    The three strategies are:

    +
      +
    1. Incremental sync: Uses an updated_at timestamp cursor to pull only records changed since the last run. Simple to implement but cannot track deletes and misses concurrent updates where two records share the same timestamp.
      2. +
    2. Oplog-based sync: Reads MongoDB's internal operation log (local.oplog.rs) to capture every insert, update, and delete as a stream of events. Captures all changes without querying collections directly, avoiding load on the source database.
      3. +
    3. Change Streams: MongoDB's higher-level API built on top of the oplog that provides a cleaner, more durable event stream with resume tokens for fault-tolerant consumption.
      4. +
    + + }, + { + question: "Q2. What is the MongoDB oplog and how does it enable real-time data sync?", + answer:
    +

    The oplog (operation log) is a special capped collection in the local database of every MongoDB replica set that records every successful data modification in chronological order. It was originally designed for replica set replication to keep secondary nodes in sync with the primary.

    +

    CDC tools use the oplog by tailing it as a stream of insert, update, and delete events, capturing every change without querying the collections directly, which avoids putting read load on the source database.

    +

    Note: Failed or no-op write operations do not create oplog entries. The oplog only reflects successful data modifications, so CDC pipelines should not assume the oplog is a complete record of all attempted writes.

    +
    + }, + { + question: "Q3. What are the limitations of incremental cursor-based MongoDB sync?", + answer:
    +

    Incremental sync using an updated_at column has three key limitations:

    +
      +
    1. Cannot detect deletes: Rows removed from the source simply disappear from the change stream with no trace for a timestamp-cursor query to find
      2. +
    2. Can miss updates: If two records are updated simultaneously with the same timestamp, one update may be skipped on the next sync run
      3. +
    3. Performance degrades without proper indexing: If the updated_at field is not indexed, the query requires a full collection scan on every sync run. With a proper index on updated_at, incremental queries remain efficient as data grows.
      4. +
    +

    For production pipelines requiring delete tracking or high accuracy, oplog or change stream strategies are necessary.

    +
    + }, + { + question: "Q4. How do MongoDB change streams differ from reading the oplog directly?", + answer:
    +

    Change streams are a higher-level MongoDB API built on top of the oplog that provides several improvements:

    +
      +
    • Resume tokens: Allow a consumer to restart from exactly where it left off after a failure, by passing the token back when reopening the cursor
      • +
    • Cleaner event filtering: Events are structured and easier to consume than raw oplog entries
      • +
    • Sharded cluster support: Change streams work across sharded clusters when issued via mongos
      • +
    +

    However, change streams on sharded clusters have important caveats:

    +
      +
    • They must be opened from the mongos, not individual shards
      • +
    • Shards with little or no activity (cold shards) can introduce latency in the event stream
      • +
    • A shard removal event can close an open change stream cursor, and the closed cursor may not be fully resumable
      • +
    +

    Direct oplog reading requires replica set access and manual resume position management, making change streams the preferred approach for modern CDC implementations, but with the sharded cluster caveats above factored in.

    +
    + }, + { + question: "Q5. What are the best practices for handling deletes in MongoDB real-time sync pipelines?", + answer:
    +
      +
    • Use oplog-based or change stream sync if deletes must propagate to the destination, since incremental sync cannot detect deletes at all
      • +
    • Ensure your MongoDB deployment is a replica set: oplog and change streams do not work on standalone MongoDB instances. Attempting to use watch() on a standalone server returns: MongoServerError: The $changeStream stage is only supported on replica sets
      • +
    • Set oplog retention long enough to cover the maximum lag your pipeline may experience. MongoDB only removes an oplog entry if both the oplog has reached its maximum configured size and the entry is older than the configured minimum retention hours
      • +
    • Implement delete logic at the destination based on the operation type field in the CDC event (op: "d") captured from the oplog or change stream: use this to apply either soft-delete (flag the row) or hard-delete (remove the row) depending on your downstream requirements
      • +
    +
    + }, +]} /> \ No newline at end of file diff --git a/blog/2024-11-11-mongodb-cdc-using-debezium-and-kafka.mdx b/blog/2024-11-11-mongodb-cdc-using-debezium-and-kafka.mdx index 53c1a340d..03f212f04 100644 --- a/blog/2024-11-11-mongodb-cdc-using-debezium-and-kafka.mdx +++ b/blog/2024-11-11-mongodb-cdc-using-debezium-and-kafka.mdx @@ -950,60 +950,7 @@ Setting up a real-time data streaming pipeline using **Debezium** and **Apache K - -## FAQs - -### 1. Does Debezium use Kafka Connect? - -**Yes, Debezium uses Kafka Connect.** - -Debezium operates as a **Kafka Connect connector**. Kafka Connect is a framework that simplifies the integration of Apache Kafka with other data systems like databases, key-value stores, and more. By using Kafka Connect, Debezium can easily capture changes from your databases and stream them into Kafka topics without requiring you to write custom code. - -**Example:** - -* **Debezium Connector:** Acts as a bridge between your MongoDB database and Kafka. - -* **Kafka Connect Framework:** Manages the connector, handling tasks like starting, stopping, and scaling the Debezium connector as needed. - - -### 2. Is Kafka and Kafka Connect different? - -**Yes, Kafka and Kafka Connect are different components, but they work together.** - -* **Apache Kafka:** - - * **What It Is:** A powerful distributed streaming platform. - - * **Main Functions:** It handles publishing and subscribing to streams of records, storing them, and processing them in real-time. - - * **Use Cases:** Messaging system, real-time analytics, log aggregation, event sourcing, and more. - -* **Kafka Connect:** - - * **What It Is:** A tool within the Kafka ecosystem. - - * **Main Functions:** Simplifies the process of connecting Kafka with external systems like databases, file systems, or other data sources and sinks. - - * **Use Cases:** Importing data into Kafka from a database (source connector) or exporting data from Kafka to another system (sink connector). - - -**How They Work Together:** - -* **Kafka:** Manages the data streams. - -* **Kafka Connect:** Facilitates the movement of data between Kafka and other systems using connectors like Debezium. - - -**Example:** - -* **Without Kafka Connect:** - - * You would need to write custom code to move data from MongoDB to Kafka. - -* **With Kafka Connect:** - - * Use Debezium as a Kafka Connect connector to automatically capture and stream changes from MongoDB to Kafka without writing additional code. - + ## Additional Resources @@ -1023,5 +970,63 @@ To further explore Debezium, Kafka, and their integrations with MongoDB, here ar * **Monitoring Kafka with Prometheus and Grafana:** https://grafana.com/docs/grafana/latest/datasources/prometheus/ - +## FAQs + + +

    Only at initialization, not for ongoing change capture. When the connector first detects a replica set, it reads the oplog directly to obtain the last recorded transaction, which it uses as the starting position. It then takes a snapshot of the databases and collections, and opens a change stream beginning from that oplog position.

    +

    For ongoing change capture after that, the connector does not tail the oplog directly. It delegates capture and decoding to MongoDB's Change Streams feature, which abstracts oplog access into a clean event stream API. This is the recommended approach since MongoDB 4.x and avoids dealing with the raw oplog format for streaming.

    + +}, + { + question: "Q2. What happens when the MongoDB primary fails and a new primary is elected?", + answer:
    +

    Debezium handles this automatically. When it detects a primary change, it stops streaming from the old primary, connects to the new primary, and resumes from the same oplog position. It uses exponential backoff when reconnecting to avoid overwhelming the replica set during transitions.

    +

    No manual intervention is typically needed for normal failovers. However, if reconnection attempts exceed the configured maximum (connect.max.attempts), the connector will fail and require a manual restart.

    +
    + }, + { + question: "Q3. Why is Debezium slow at processing changes for multiple collections simultaneously?", + answer:
    +

    Debezium can parallelize the initial snapshot across collections using snapshot.max.threads, so the full-load phase is not strictly single-threaded. The slowness shows up in two other places:

    +
      +
    • Within a single collection. A snapshot of one collection is copied by a single thread, so one very large collection becomes a bottleneck even when other collections finish quickly.
      • +
    • During streaming. A connector task consumes the change stream sequentially, so ongoing change capture is processed as a single ordered stream rather than in parallel across collections.
      • +
    +
    + }, + { + question: "Q4. What MongoDB permissions does the Debezium user need?", + answer:
    +

    The MongoDB user needs:

    +
      +
    • Read access to the admin database: where the oplog lives
      • +
    • Read access to the config database: required for sharded clusters
      • +
    • listDatabases privilege: to enumerate available databases
      • +
    • Cluster-wide find and changeStream privilege actions: required when using Change Streams (the default mode since MongoDB 4.x)
      • +
    +

    Always create a dedicated, least-privilege user for Debezium rather than using a superuser account.

    +
    + }, + { + question: "Q5. What causes silent ingestion failures in Debezium, and how can I prevent them?", + answer:
    +

    Silent failures often occur from:

    +
      +
    • Schema evolution: A new field or renamed collection that the connector isn't configured to handle
      • +
    • Oplog overrun: The cursor position was lost because MongoDB purged the oplog while the connector was inactive, creating an offset mismatch. The connector may appear to be running while actually not processing any changes
      • +
    • Kafka topic mismatches: Events are emitted to unexpected topics, causing downstream consumers to silently miss data
      • +
    +

    To prevent them:

    +
      +
    • Enable heartbeat messages: Debezium will emit periodic heartbeats even when no changes are captured, so you can detect stalled pipelines. This is especially important when only non-captured collections are being written to, which would otherwise allow the oplog to rotate without the connector noticing
      • +
    • Use a schema registry: for schema evolution to handle new fields and type changes without breaking the pipeline
      • +
    • Set monitoring alerts: on connector lag and restart counts to catch issues before they become data gaps
      • +
    +
    + }, +]} /> + \ No newline at end of file diff --git a/blog/2024-11-21-issues-debezium-kafka.mdx b/blog/2024-11-21-issues-debezium-kafka.mdx index df0d50e6a..97d19d0b7 100644 --- a/blog/2024-11-21-issues-debezium-kafka.mdx +++ b/blog/2024-11-21-issues-debezium-kafka.mdx @@ -1,7 +1,7 @@ --- slug: issues-debezium-kafka -title: "Debezium Kafka Challenges & How OLake Solves Them: Explained" -description: "Explore common Debezium and Kafka CDC challenges including setup complexity, overhead, schema changes, and why OLake is a faster, simpler alternative." +title: "Debezium Kafka Challenges & How OLake Go Solves Them: Explained" +description: "Explore common Debezium and Kafka CDC challenges including setup complexity, overhead, schema changes, and why OLake Go is a faster, simpler alternative." image: /img/blog/cover/issues-debezium-kafka-cover.webp authors: [priyansh] tags: [debezium] @@ -256,7 +256,7 @@ Debezium streams data into Kafka topics, but moving that data into target system 2. **Handle Schema Evolution**: Implement logic to map Debezium's change event structure to Iceberg's schema (or other lakehouse format), including handling schema changes. -3. **Maintain the Pipeline**: Monitor and update the consumer application as schemas evolve and data volumes grow. A tool like OLake is needed in such scenarios. +3. **Maintain the Pipeline**: Monitor and update the consumer application as schemas evolve and data volumes grow. A tool like OLake Go is needed in such scenarios. ## 7. Kafka Dependency and Resource Consumption @@ -451,5 +451,51 @@ Some users on the internet mentioned issues with Debezium (or due to its complex Another user on reddit said “When a large number of updates occur, Debezium is unable to keep up with the throughput, resulting in untimely downstream data.” [Source](https://www.reddit.com/r/dataengineering/comments/1fv186f/is_there_an_alternative_to_debezium_kafka/) +## FAQs + + +

    Debezium with Kafka requires installing and configuring multiple interdependent components: Kafka brokers, ZooKeeper (deprecated since Kafka 3.5 and fully removed in Kafka 4.0; KRaft is now the only supported metadata mode), Kafka Connect workers, Debezium source connectors, and sink connectors. Each component needs version-compatible configuration, separate scaling strategies, and monitoring.

    +

    Implementing custom transformations requires writing Java classes (Java 17 or later) and deploying them to the Kafka Connect classpath. This complexity demands deep expertise in distributed systems before a single CDC event is captured.

    + + }, + { + question: "Q2. How does Debezium handle schema changes in source databases?", + answer:
    +

    Debezium handles simple schema changes automatically: it tracks schema history in a dedicated Kafka topic and embeds schema information in each event, keeping events self-contained. However, complex schema changes require careful configuration and often manual intervention:

    +
      +
    • Column type changes can cause compatibility issues between Debezium's captured schema and the target system's schema, often requiring custom Single Message Transforms (SMTs)
      • +
    • Primary key changes (add, remove, rename) can cause brief periods of desynchronization. The recommended approach is to make primary key changes when the system is in read-only mode, allow all events to be processed, stop Debezium, apply the changes, then restart
      • +
    • Connector configuration updates may be required after schema changes, and in some cases connectors must be paused and restarted
      • +
    +
    + }, + { + question: "Q3. What are the performance limitations of Debezium plus Kafka for large table snapshots?", + answer:
    +

    Debezium's initial snapshotting is single-threaded by default for some connectors, but snapshot.max.threads can be configured to enable parallel snapshotting. Additionally, Debezium's incremental snapshot mode (available since version 1.6) allows snapshotting to run concurrently with CDC streaming without blocking ongoing change capture, and is resumable after connector restarts.

    +

    During snapshotting in some configurations (such as MySQL), the source table may be locked or unavailable for writes for the duration of the snapshot lock. For very large tables this window can be significant.

    +

    Additionally, very large Kafka topics from high-volume CDC streams can become expensive to manage in terms of storage retention and consumer lag monitoring.

    +
    + }, + { + question: "Q4. Why might a data team choose an alternative to Debezium plus Kafka for CDC?", + answer:
    +

    Teams choose alternatives when the operational overhead outweighs the benefits: maintaining Kafka infrastructure, managing connector versions, debugging complex SMT transformations, and handling schema drift all require dedicated engineering effort.

    +

    Organizations wanting a simpler, lower-maintenance CDC solution that handles the full pipeline from source to data lakehouse without Kafka expertise or infrastructure often migrate to purpose-built tools like OLake Go. For scenarios where a full Kafka cluster is not warranted, Debezium Server is also available as a lightweight standalone alternative.

    +
    + }, + { + question: "Q5. How does Debezium handle data deduplication when consumers restart or fail?", + answer:
    +

    Debezium uses Kafka Connect's internal offset storage to track its position in the source database log (oplog position or LSN), not Kafka consumer group offsets, which are managed by downstream sink connectors or application consumers.

    +

    After a failure, the connector resumes from the last committed offset position in the source log.

    +

    Exactly-once semantics: As of Debezium 3.3, exactly-once delivery is natively supported for all core connectors (MariaDB, MongoDB, MySQL, Oracle, PostgreSQL, and SQL Server), built on top of Kafka's transaction support. This means events are delivered and written to a Kafka topic exactly once without duplicates, and manual configuration of idempotent producers and transactional consumers is no longer required for supported versions.

    +

    For deployments on older Debezium versions, at-least-once delivery is the default, meaning duplicate events can appear downstream and require deduplication logic in the sink or transformation layer.

    +
    + }, +]} /> \ No newline at end of file diff --git a/blog/2024-11-22-debezium-vs-olake.mdx b/blog/2024-11-22-debezium-vs-olake.mdx index d86ff7399..8be782cae 100644 --- a/blog/2024-11-22-debezium-vs-olake.mdx +++ b/blog/2024-11-22-debezium-vs-olake.mdx @@ -1,13 +1,13 @@ --- slug: debezium-vs-olake -title: "OLake vs Debezium + Kafka | CDC Performance & Ease of Use" -description: "Compare OLake and Debezium+Kafka CDC tools on speed, setup ease, scalability, and cost. OLake offers faster, simpler, and cheaper real-time replication." +title: "OLake Go vs Debezium + Kafka | CDC Performance & Ease of Use" +description: "Compare OLake Go and Debezium+Kafka CDC tools on speed, setup ease, scalability, and cost. OLake Go offers faster, simpler, and cheaper real-time replication." image: /img/blog/cover/debezium-vs-olake-cover.webp authors: [priyansh] tags: [debezium] --- -# Problems with Debezium and How we (OLake, Open-Source) solve it? +# Problems with Debezium and How we (OLake Go, Open-Source) solve it? ![OLake platform: Change data from MySQL, MongoDB, PostgreSQL flows to OLake, processed and stored in S3 and Iceberg](/img/blog/cover/debezium-vs-olake-cover.webp) @@ -20,22 +20,22 @@ However, while this combination offers powerful capabilities, it also comes with In the previous article we discussed in details about all the common challenges with Debezium, and now here, we present you an better open source alternative to debezium + kafka setup. -In this article we will see how OLake solves most of the problems that exist with your current Debezium + Kafka Connect setup. +In this article we will see how OLake Go solves most of the problems that exist with your current Debezium + Kafka Connect setup. -## Introducing OLake (Open Source Alternative to Debezium + Kafka Setup) +## Introducing OLake Go (Open Source Alternative to Debezium + Kafka Setup) ## 1. Purpose and Overview -| **Feature** | **OLake** | **Debezium + Kafka** | +| **Feature** | **OLake Go** | **Debezium + Kafka** | | --- | --- | --- | -| **Description** | Fastest open-source platform for real-time ingestion and normalization of tables, and collections ( MongoDB, currently) data into data Lakeshouses (*Iceberg on S3*) with minimal coding and single component architecture. | An open-source CDC tool that streams changes from MongoDB to Kafka for real-time data processing. | -| **Best For** | Data teams seeking fault-proof, streamlined pipelines to replicate huge data into a lakehouse for analytics use cases. | Organizations with existing Kafka-based architectures needing robust CDC for MongoDB **without** built-in transformation capabilities. | +| **Description** | Fastest open-source platform for CDC (Change Data Capture) ingestion and normalization of data from PostgreSQL, MySQL, MongoDB, MSSQL, DB2 LUW, Kafka, S3, and Oracle into data lakehouses Apache Iceberg on S3 and S3 Parquet files, with minimal coding and a single-component architecture. | An open-source CDC tool that captures database changes and streams them to Kafka for real-time processing; loading into lakehouses or file destinations typically requires additional Kafka Connect sink connectors. | +| **Best For** | Data teams seeking fault-proof, streamlined pipelines to replicate data from multiple sources into a lakehouse or S3 for analytics use cases. | Organizations with existing Kafka-based architectures needing robust CDC **without** built-in load or transformation capabilities. | ## 2. Installation and Setup ### Installation Options and Ease of Setup -| **Feature** | **OLake** | **Debezium + Kafka** | +| **Feature** | **OLake Go** | **Debezium + Kafka** | | --- | --- | --- | | **Installation Options** | - Open-source & **Self-Hosted**. Deploy on standalone machines with flexibility. | **-** Open-Source | | **Ease of Setup** | - User-friendly, code-free interface with Docker support for quick deployment. Minimal configuration required. | Complex setup requiring detailed knowledge of Kafka clusters, Debezium connectors, and potential DevOps support. **NO UI.** | @@ -43,57 +43,51 @@ In this article we will see how OLake solves most of the problems that exist wit ## 3. Data Processing -### ELT Capabilities and Transformation - -| **Feature** | **OLake** | **Debezium + Kafka** | +| **Feature** | **OLake Go** | **Debezium + Kafka** | | --- | --- | --- | -| **ELT Functionality** | Fully automated ELT: extraction, data merging, schema and table creation, data type conversion. | Only handles data extraction (CDC). Loading and transformation require custom coding or additional tools. | -| **Data Transformation** | Automated parsing, extraction, flattening, and transformation of nested `JSON` (semi-structured data) into relational streams / tables. | Requires Kafka Streams or KSQL for transformations, necessitating additional setup and coding. | -| **JSON Array Flattening** | Arrays are exploded into separate tables with primary and foreign keys relationship (Flatbread by Datazip). | No native support, outside of Debezium product | +| **EL Functionality** | Automated EL: extraction, data merging, schema and table creation, data type conversion, with JSON support limited to Level-0 flattening via the Normalization feature. | Only handles data extraction (CDC). Loading requires custom coding or additional tools. | +| **JSON Array Flattening** | Supports Level-0 JSON flattening only that is top-level JSON objects into separate columns via Normalization. | No native support, outside of Debezium product | | **Schema Handling** | Automatically manages schema changes and complex polymorphic (changing key types, `INT`-> `STRING`) data without manual intervention. | Minimal schema change support; complex changes may require manual intervention and risk schema drift. | -| **Historical Data Handling** | Automatically handles both historical and real-time data seamlessly with incremental snapshotting. | Manual backfill processes needed for historical data; requires maintaining large Kafka log retention or manual snapshotting. | +| **Historical Data Handling** | Automatically handles both historical and CDC data seamlessly with incremental snapshotting. | Manual backfill processes needed for historical data; requires maintaining large Kafka log retention or manual snapshotting. | | **CDC Impact** | Change Data Capture (CDC) utilizes a log-based approach, enabling replication through parallel and multithreaded loading, as well as partitioning and compression techniques. | Snapshotting is incremental but not concurrent, meaning large tables with millions of rows may take hours to complete. During this process, tables remain inaccessible until snapshotting finishes. | ## 4. Performance, Scalability and Rich Feature -| **Feature** | **OLake** | **Debezium + Kafka** | +All performance comparisons in this section are considering **PostgreSQL** is used as the source database. + +| **Feature** | **OLake Go** | **Debezium + Kafka** | | --- | --- | --- | -| **Full Data Sync Speed** | - Up to **15x faster** full data sync compared to Debezium. | Dependent on Debezium snapshotKafka performance. | -| **Incremental Sync Speed** | **- 27.3x faster** incremental synchronization, enabling rapid updates and real-time data availability. | Dependent on Debezium CDC performance; JVM and processing CDC in sequential way affects severely. | -| **Scalability** | - Automatically handles large datasets with parallel processing by breaking large collections into chunks. | Requires manual scaling of Kafka brokers and connectors, which can be complex and resource-intensive. | +| **Full Data Sync Speed** | Up to **39x faster** full data sync compared to Debezium. | Dependent on Debezium snapshot and Kafka performance. | +| **CDC Sync Speed** | **4x faster** CDC synchronization, enabling rapid updates. | Dependent on Debezium CDC performance; JVM and processing CDC in sequential way affects severely. | +| **Scalability** | Automatically handles large datasets with parallel processing by breaking large collections into chunks. | Requires manual scaling of Kafka brokers and connectors, which can be complex and resource-intensive. | | **Snapshotting** | Incremental and automatic snapshotting, enabling seamless historical data integration. | Manual and time-consuming snapshots, and potentially causing lag to read large tables. | -| **Backpressure and Resource Limitations** | OLake monitors data flow and automatically scales resources to handle high volumes, preventing backpressure and ensuring consistent performance | - Possibility of data loss from Consumer lag | -| **Latency** | - **Near real-time** replication (1-minute benchmark). | - Real-time streaming with low latency possible. | +| **Backpressure and Resource Limitations** | OLake Go monitors data flow and automatically scales resources to handle high volumes, preventing backpressure and ensuring consistent performance | Possibility of data loss from Consumer lag | +| **Latency** | **Near real-time** replication, with CDC jobs that can be scheduled as frequently as every second. | Real-time streaming with low latency possible. | | **Data Type Support** | Supports handling of most commonly used data types (natively supported of each source) including BLOB types. | Certain Debezium connectors have limitations with specific data types; for example, the Oracle connector has restrictions when handling the BLOB data type. | ## 5. Cost Efficiency -### Cost Comparison - -| **Feature** | **OLake** | **Debezium + Kafka** | +| **Feature** | **OLake Go** | **Debezium + Kafka** | | --- | --- | --- | -| **Pricing Model** | **Open-Source**: Free to use with no licensing fees. | **Open-Source**: Free to use, but incurs costs related to infrastructure and maintenance. | -| **Cost Comparison** | - **20x cheaper than Fivetran**. | - **Infrastructure Costs**: Maintaining Kafka clusters can be expensive, especially at scale. | -| **Example** | A medium-sized company can implement OLake for approximately $300/month, saving significantly compared to managed alternatives. | A similar company might spend $900 - $1200 monthly on cloud resources and DevOps labor to maintain a Debezium + Kafka setup. | +| **Pricing Model** | **Open-Source**: Free to use, but incurs cost related to infrastructure. | **Open-Source**: Free to use, but incurs costs related to infrastructure. | +| **Example** | A medium-sized company can implement OLake Go for approximately $300/month, saving significantly compared to managed alternatives. | A similar company might spend $900 - $1200 monthly on cloud resources and DevOps labor to maintain a Debezium + Kafka setup. | -More on Cost. [See Benchmarks](https://olake.io/docs/benchmarks?tab=mongodb) - -![Comparison table of monthly sync costs: Olake vs Fivetran, Airbyte, and Debezium MSK, highlighting Olake's lower price](/img/blog/2024/11/debezium-vs-olake-1.webp) +These cost comparisons are based on benchmarks with **Postgres** as the source; for detailed numbers and methodology, see the [PostgreSQL benchmarks](https://olake.io/docs/benchmarks?tab=postgres). ## 6. Integration and Monitoring -### Integration with Destinations and Monitoring +In this section, we look at how OLake Go and Debezium plug into your downstream systems and how they help to keep an eye on pipeline health. -| **Feature** | **OLake** | **Debezium + Kafka** | +| **Feature** | **OLake Go** | **Debezium + Kafka** | | --- | --- | --- | -| **Integration with Destinations** | - **Native Support**: Apache Iceberg, Amazon S3, | - **Kafka Topics**: Streams data to Kafka topics, requiring manual setup of sink connectors for specific destinations. | -| **Monitoring & Alerts** | **Built-In**: Real-time monitoring with configurable alerts for pipeline status and schema changes. | **External Tools**: Relies on Kafka monitoring tools like Kafka Manager or Prometheus for monitoring and alerts, requiring additional setup. | +| **Integration with Destinations** | **Native Support**: Apache Iceberg, S3 Parquet files. | **Kafka Topics**: Streams data to Kafka topics, requiring manual setup of sink connectors for specific destinations. | +| **Monitoring & Alerts** | **Built-In**: Real-time monitoring with configurable webhook alerts that notify you in Slack, Microsoft Teams, or any webhook-compatible tool whenever a job fails, including job details and error messages. | **External Tools**: Relies on Kafka monitoring tools like Kafka Manager or Prometheus for monitoring and alerts, requiring additional setup. | ## 7. Ease of Use ### User Interface and Learning Curve -| **Feature** | **OLake** | **Debezium + Kafka** | +| **Feature** | **OLake Go** | **Debezium + Kafka** | | --- | --- | --- | | **User Interface** | **User-Friendly**: Designed with a code-free interface, making it accessible for non-technical users. | **Technical**: Command-line based, requiring familiarity with Kafka and Debezium configurations. | | **Implementation** | Quick to implement and manage, tailored for data teams with minimal coding requirements. | Requires significant technical expertise to set up, configure, and manage Kafka clusters and Debezium connectors. | @@ -103,7 +97,7 @@ More on Cost. [See Benchmarks](https://olake.io/docs/benchmarks?tab=mongodb) ### Support Channels and Documentation -| **Feature** | **OLake** | **Debezium + Kafka** | +| **Feature** | **OLake Go** | **Debezium + Kafka** | | --- | --- | --- | | **Support Channels** | **Dedicated Support**: Access to OLake and Datazip communities, with opportunities for partnerships and technical advisory roles. | **Community-Driven**: Supported by a large open-source community with extensive documentation and active forums. | | **Documentation** | Comprehensive documentation tailored to OLake’s features and integrations. [Docs here.](https://olake.io/docs) | Extensive and long Kafka and Debezium documentation, but may require piecing together information for specific use cases. | @@ -113,7 +107,7 @@ More on Cost. [See Benchmarks](https://olake.io/docs/benchmarks?tab=mongodb) ### Advanced Capabilities and Extensibility -| **Feature** | **OLake** | **Debezium + Kafka** | +| **Feature** | **OLake Go** | **Debezium + Kafka** | | --- | --- | --- | | **Advanced Capabilities** | - Automatic Handling of Complex Data Types and Schema Drift. Complexity will only grow as data grows. | - Minimal Schema Drift Handling functionality | | **Extensibility** | NA, input required | Highly flexible, enabling deep customization through Kafka Streams and the development of custom connectors as needed. | @@ -122,7 +116,7 @@ More on Cost. [See Benchmarks](https://olake.io/docs/benchmarks?tab=mongodb) While Debezium with Kafka provides powerful capabilities for CDC, it comes with significant challenges that can impact your organization's resources and efficiency. The complexity of setup, high operational overhead, and the need for specialized expertise make it a solution that requires careful consideration. -Organizations must weigh the benefits against the drawbacks, considering factors like existing infrastructure, team expertise, and long-term maintenance costs. With the emergence of new tools and managed services like OLake, Fivetran, and others, companies now have more options to achieve real-time data replication with reduced complexity and operational burden. +Organizations must weigh the benefits against the drawbacks, considering factors like existing infrastructure, team expertise, and long-term maintenance costs. With the emergence of new tools and managed services like OLake Go, Fivetran, and others, companies now have more options to achieve real-time data replication with reduced complexity and operational burden. **Key Takeaways:** @@ -132,5 +126,40 @@ Organizations must weigh the benefits against the drawbacks, considering factors As data volumes grow and architectures evolve, choose a solution that can adapt to your organization's changing needs. - +## FAQs + +

    Setting up Debezium with Kafka requires deploying and managing multiple components: Kafka brokers, ZooKeeper (deprecated since Kafka 3.5 and fully removed in Kafka 4.0; KRaft is now the only supported metadata mode), Kafka Connect, Debezium connectors, and sink connectors. Each needs its own configuration, version compatibility management, and scaling strategy.

    +

    Historical data backfills require manual processes, schema changes need custom handling, and the entire stack requires deep expertise in distributed Java systems to operate reliably.

    + + }, + { + question: "Q2. How is OLake Go fundamentally different from Debezium plus Kafka for Change Data Capture?", + answer:
    +

    OLake Go is a single-component, open-source CDC tool with a web UI that replaces the entire Debezium plus Kafka plus sink connector stack. It handles extraction, schema inference, type conversion, flattening of nested JSON, and writing directly to Iceberg on S3, all without Kafka or ZooKeeper. Setup takes minutes instead of days, and no Java expertise is required (OLake is written in Go).

    +
    + }, + { + question: "Q3. Does OLake Go handle MongoDB JSON array flattening that Debezium lacks natively?", + answer:
    +

    Partially. OLake Go supports Level-0 JSON flattening through its Normalization feature, which expands top-level JSON objects into separate columns during ingestion. Flattening stops at that level: anything nested deeper, including arrays, is stored as a stringified value in a single column. Arrays are not exploded into separate child tables, so if you need arrays in separate related tables, that happens in a downstream transformation after load.

    +

    Debezium has no equivalent built-in normalization step. Its Single Message Transforms (SMTs) can handle simple field-level extractions inline, but they cannot perform the row multiplication required for array-to-table flattening. That requires Kafka Streams, ksqlDB, or a downstream transformation step.

    +
    + }, + { + question: "Q4. What databases does OLake Go support for CDC compared to Debezium?", + answer:
    +

    OLake Go supports CDC for PostgreSQL, MySQL, MongoDB and Microsoft SQL Server. Debezium also offers CDC across a broad range of databases, but requires separate connector configurations and Kafka infrastructure for each one, whereas OLake Go handles multi-source ingestion through a unified UI and a consistent configuration model across all supported databases.

    +
    + }, + { + question: "Q5. How does OLake Go handle the first full load of large tables compared to Debezium?", + answer:
    +

    OLake Go performs the initial full load using parallel chunking: splitting large tables into segments and processing them concurrently across multiple threads. It uses checkpointing to track progress so a failed sync can resume from where it left off rather than starting over.

    +

    For Debezium, snapshotting is single-threaded by default but can be parallelized via snapshot.max.threads. Debezium's incremental snapshot mode (available since v1.6) allows snapshotting to run alongside CDC streaming without blocking ongoing change capture and is resumable after restarts. However, some connectors like MySQL may still briefly lock tables during the initial snapshot.

    +
    + }, +]} /> \ No newline at end of file diff --git a/blog/2025-01-07-olake-architecture.mdx b/blog/2025-01-07-olake-architecture.mdx index 24f408361..9256b85cd 100644 --- a/blog/2025-01-07-olake-architecture.mdx +++ b/blog/2025-01-07-olake-architecture.mdx @@ -15,19 +15,19 @@ update: [18.02.2025] 1. We support S3 data partitioning - refer docs [here](https://olake.io/docs/writers/parquet/partitioning) 2. Support of Iceberg coming next week. -When building [OLake](https://olake.io/), our goal was simple: *Fastest DB to Data LakeHouse (Apache Iceberg to start) data pipeline.* +When building [OLake Go](https://olake.io/), our goal was simple: *Fastest DB to Data LakeHouse (Apache Iceberg to start) data pipeline.* Checkout GtiHub repository for OLake - [https://github.com/datazip-inc/olake](https://github.com/datazip-inc/olake) Over time, many of us who’ve worked with data pipelines have dealt with the toil of building one-off ETL scripts, battling performance bottlenecks, or worrying about vendor lock-in. -With OLake, we wanted a clean, open-source solution that solves these problems in a straightforward, high-performing manner. +With OLake Go, we wanted a clean, open-source solution that solves these problems in a straightforward, high-performing manner. -In this blog, I’m going to walk you through the architecture of OLake—how we capture data from MongoDB, push it into S3 in Apache Iceberg format or other data Lakehouse formats, and handle everything from schema evolution to high-volume parallel loads. +In this blog, I’m going to walk you through the architecture of OLake Go—how we capture data from MongoDB, push it into S3 in Apache Iceberg format or other data Lakehouse formats, and handle everything from schema evolution to high-volume parallel loads. ## Architectural Overview -At a high level, OLake reads from source systems (currently focused on MongoDB) and writes directly to configurable destinations such as local Parquet, S3 Iceberg Parquet. Our architecture is built around four main components: +At a high level, OLake Go reads from source systems (currently focused on MongoDB) and writes directly to configurable destinations such as local Parquet, S3 Iceberg Parquet. Our architecture is built around four main components: **Components** @@ -54,7 +54,7 @@ At a high level, OLake reads from source systems (currently focused on MongoDB) * **SDK** -We wanted to keep these components as modular as possible so each one can excel at its responsibilities. Here’s a simplified conceptual diagram showing how data flows through OLake. +We wanted to keep these components as modular as possible so each one can excel at its responsibilities. Here’s a simplified conceptual diagram showing how data flows through OLake Go. ![Architecture of a data platform: UI, connectors and logic, Colake framework modules, writing to a lakehouse](/img/blog/2025/01/olake-architecture-1.webp) @@ -62,11 +62,11 @@ Here’s how it all ties together: 1. **Initial Snapshot**: We run a full collection read from MongoDB. That happens via query firing. -2. **Change Data Capture (CDC)**: After completing the snapshot, OLake sets up MongoDB change streams (built on oplogs,a way for change data capture, CDC) for near real-time updates. For any record changes that occurred during the snapshot process, CDC reflects those changes as well just as initial snapshot finishes and from then on it continues to capture any other changes that occur in your source database (depending on your scheduled time). +2. **Change Data Capture (CDC)**: After completing the snapshot, OLake Go sets up MongoDB change streams (built on oplogs,a way for change data capture, CDC) for near real-time updates. For any record changes that occurred during the snapshot process, CDC reflects those changes as well just as initial snapshot finishes and from then on it continues to capture any other changes that occur in your source database (depending on your scheduled time). 3. The users have an option to set the number of parallel running threads so you can decide what’s the optimal configuration for you in terms of **speed vs load** on your mongodb cluster. -4. **Transformation & Normalization**: We flatten or extrapolate complex, semi-structured fields into relational streams. Level 0 flattening support in OLake and nested level 2,3,4 flattening of JSON coming soon. +4. **Transformation & Normalization**: We flatten or extrapolate complex, semi-structured fields into relational streams. Level 0 flattening support in OLake Go and nested level 2,3,4 flattening of JSON coming soon. 5. **Integrated Writes**: We write the transformed data into destinations with minimal overhead. @@ -144,11 +144,11 @@ The Core is the foundation on which OLake Drivers and Destinations rest. It give ### Handling Schema Evolution & Error Recovery -When working with a schema-less store like MongoDB, we often encounter evolving document structures. OLake detects such changes in the source schema and raises alerts. We take care of new column creation or any non-breaking data type changes. +When working with a schema-less store like MongoDB, we often encounter evolving document structures. OLake Go detects such changes in the source schema and raises alerts. We take care of new column creation or any non-breaking data type changes. -## OLake’s Role in a Lakehouse Ecosystem +## OLake Go’s Role in a Lakehouse Ecosystem -We emphasised early on that we wanted to avoid vendor lock-in. Hence, OLake stores data in open formats—like `parquet`—using a table format like *Apache Iceberg*. +We emphasised early on that we wanted to avoid vendor lock-in. Hence, OLake Go stores data in open formats—like `parquet`—using a table format like *Apache Iceberg*. This opens up the possibility for various query engines, including *Spark*, *Trino*, *Flink*, or even Snowflake external tables. By decoupling how data is written and how it’s consumed, we make sure you can leverage the best analytical tools without rewriting or re-ingesting the data. @@ -161,19 +161,19 @@ This opens up the possibility for various query engines, including *Spark*, *Tri ## Performance Benchmarks -We designed OLake from the ground up to focus on throughput, especially for high-volume data. We tested OLake on a `Standard_D64as_v5` machine (64 `vCPUs`, 256 GiB RAM, 250 GB shared storage) connected to a three-node MongoDB replica set. Here’s a summary of the results: +We designed OLake Go from the ground up to focus on throughput, especially for high-volume data. We tested OLake Go on a `Standard_D64as_v5` machine (64 `vCPUs`, 256 GiB RAM, 250 GB shared storage) connected to a three-node MongoDB replica set. Here’s a summary of the results: 1. **Full Load** * **230 million rows** (~664.81 GB) of Twitter data. - * OLake completed this full load in **46 minutes**, compared to several hours or more with other tools. + * OLake Go completed this full load in **46 minutes**, compared to several hours or more with other tools. 2. **Incremental Sync** * We measured ~50 million rows per month in ongoing incremental loads. - * OLake handled this in **28.3 seconds**, roughly **35,694 records/second**, which was multiple times faster than the competition. + * OLake Go handled this in **28.3 seconds**, roughly **35,694 records/second**, which was multiple times faster than the competition. These results prove that with chunk-based parallel loading and direct Writer integration, we can handle hefty datasets much more efficiently than many existing solutions. @@ -232,6 +232,69 @@ In the future iterations we plan to: 11. **Multiple cloud support** including GCP and Azure, with multiple catalog support for enhanced flexibility and organization +## FAQs + +

    OLake Go is built around four modular components:

    +
      +
    • CLI and UI: Two distinct interfaces (command-line and web UI) that share the same underlying core framework
      • +
    • Core Framework : Orchestrates the entire data flow pipeline, handling state management, configuration validation, logging, and type detection
      • +
    • Connectors / Drivers : Source-specific plugins for MongoDB, PostgreSQL, MySQL, and other supported databases
      • +
    • Writers : Destination plugins for Apache Iceberg on S3 and local Parquet files
      • +
    +

    Each component has a single responsibility and is independently extensible. Adding a new source or destination does not require changes to the core framework. Each connector is also autonomous, with its own dependencies kept separate to minimize binary size.

    + + }, + { + question: "Q2. How does OLake Go perform the initial historical data load for large tables?", + answer:
    +

    OLake Go splits large tables into parallel chunks and processes them concurrently across multiple worker threads. Chunking strategies vary by source:

    +
      +
    • MongoDB : Uses a split-vector strategy (via MongoDB's splitVector command) by default, falling back to a timestamp-based strategy that generates chunk boundaries from the _id field's embedded timestamp when split-vector isn't available
      • +
    • PostgreSQL : Uses CTID ranges, batch-size splits, and next-query paging
      • +
    +

    Each chunk is independently processed by a dedicated thread, enabling concurrent extraction that can be configured via the max_threads setting in the source configuration. OLake Go also uses checkpointing to track chunk-level progress so a failed sync resumes from the last completed chunk rather than restarting the entire load.

    +
    + }, + { + question: "Q3. What destinations does OLake Go currently support for writing replicated data?", + answer:
    +

    OLake Go supports:

    +
      +
    • Apache Iceberg on S3-compatible object storage (AWS S3, MinIO, GCS, Azure Blob) as its primary lakehouse destination
      • +
    • Local Parquet files for development and testing, writing to a local directory or uploading directly to an S3 bucket
      • +
    +

    When writing to Iceberg, OLake Go registers tables with any supported Iceberg catalog including REST catalogs (Lakekeeper), AWS Glue, Hive Metastore, Nessie, Polaris, and Unity Catalog.

    +
    + }, + { + question: "Q4. How does OLake Go handle schema evolution during replication without data loss?", + answer:
    +

    When source table schemas change (new columns added, types changed), OLake Go detects the mismatch against the current Iceberg table schema and applies Iceberg's native schema evolution:

    +
      +
    • New columns are added via metadata-only operations without rewriting existing data files
      • +
    • Compatible type promotions (int to bigint, float to double) are applied automatically using Iceberg v2 widening promotion rules
      • +
    • Incompatible type changes (e.g. INT to STRING) are converted directly to the new type at write time rather than rejected, so the sync continues without data loss
      • +
    +

    Coming soon: Routing values that can't be safely converted to a Dead Letter Queue (DLQ) column, so they're preserved without blocking the sync, is on the roadmap but not yet available. Check the OLake documentation for current status.

    +
    + }, + { + question: "Q5. What makes OLake Go faster than traditional ETL tools for database replication?", + answer:
    +

    OLake Go achieves higher throughput through several architectural decisions:

    +
      +
    • Parallel chunking : Multiple table segments are processed concurrently rather than sequentially
      • +
    • Direct Parquet writes : Data flows directly from the driver into the destination, eliminating unnecessary read-and-write cycles to local disk and intermediate formats
      • +
    • Apache Arrow as the columnar memory format : Arrow data is already in columnar layout, so writing to Parquet requires only encoding and compression with no restructuring cost and no memory copying
      • +
    • CDC for ongoing replication : Only changed rows are read after the initial load, dramatically reducing ongoing data transfer
      • +
    +
    + }, +]} /> + **Further Reading & References** * [OLake MongoDB Benchmark](https://olake.io/docs/benchmarks?tab=mongodb) @@ -239,6 +302,6 @@ In the future iterations we plan to: Feel free to explore the official OLake documentation or check out our open-source repository to see how you can incorporate it into your own data pipelines. -By keeping performance, schema fidelity, and modular design at the forefront, OLake stands poised to simplify real-time data replication for modern analytics. +By keeping performance, schema fidelity, and modular design at the forefront, OLake Go stands poised to simplify real-time data replication for modern analytics. \ No newline at end of file diff --git a/blog/2025-03-18-binlogs.mdx b/blog/2025-03-18-binlogs.mdx index cc09de02b..11b5e9340 100644 --- a/blog/2025-03-18-binlogs.mdx +++ b/blog/2025-03-18-binlogs.mdx @@ -202,4 +202,63 @@ Let’s dive into some lesser-known but crucial aspects of MySQL binlogs that ev ### Final Thought: Understanding these nuances gives you more control over how MySQL behaves, especially in complex environments where replication and recovery are critical. Binlogs are powerful, but like any powerful tool, they need to be managed carefully to avoid pitfalls and get the most out of them. +## FAQs + + +

    MySQL binary logs (binlogs) are files that record every data-modification operation (INSERT, UPDATE, and DELETE statements) in chronological order. They do not log read-only SELECT queries.

    +

    Binlogs can store information in three formats:

    +
      +
    • Statement-based. Records the actual SQL statement.
      • +
    • Row-based. Records row-level changes rather than the statement. What gets captured depends on the operation: an INSERT logs the new row (after image), a DELETE logs the removed row (before image), and an UPDATE logs both. How many columns are written is controlled by binlog_row_image: the default FULL writes all columns, while MINIMAL writes only the primary key plus changed columns.
      • +
    • Mixed. Uses statement-based by default, switching to row-based for queries that could produce inconsistent results.
      • +
    +

    Note: binlog_format is deprecated as of MySQL 8.0.34 and is subject to removal in a future version. Row-based logging is now the default and the only recommended format for new MySQL replication and CDC setups.

    +

    Binlogs are used for replication, point-in-time recovery, auditing, and Change Data Capture pipelines.

    + + }, + { + question: "Q2. How do MySQL binlogs enable Change Data Capture (CDC) for data pipelines?", + answer:
    +

    CDC tools like OLake Go read MySQL binlogs to capture every data change without querying the source tables directly. The binlog records every insert, update, and delete as a stream of events. A CDC tool connects to MySQL as a replica, reads this stream, and forwards the changes to downstream systems like Apache Iceberg or data warehouses.

    +

    This approach adds zero query load to the source database and delivers near-real-time data replication.

    +

    Requirement: For CDC to work correctly, binlog_format must be set to ROW. Statement-based or mixed format binlogs do not provide the exact before/after row values that CDC tools require to reconstruct changes reliably.

    +
    + }, + { + question: "Q3. What is the difference between statement-based, row-based, and mixed binlog format?", + answer:
    +
      +
    • Statement-based logging: Records the actual SQL statement (e.g. UPDATE users SET age=30 WHERE id=5). Uses less storage but risks inconsistency if the same query produces different results on a replica (e.g. queries using UUID(), USER(), or AUTO_INCREMENT with triggers). Only recommended when the binary log must be kept as small as possible and all functions are guaranteed deterministic.
      • +
    • Row-based logging: Records row-level changes rather than the statement. The column scope depends on binlog_row_image: the default FULL writes all columns for before and after images, while MINIMAL writes only the primary key plus changed columns. What is captured also varies by operation, since an INSERT has only an after image, a DELETE only a before image, and an UPDATE both. Safer and more storage-intensive than statement-based, and the only format recommended for new replication setups.
      • +
    • Mixed logging: Uses statement-based by default and automatically switches to row-based for specific unsafe operations, including queries using UUID(), AUTO_INCREMENT columns updated with triggers, USER(), and CURRENT_USER().
      • +
    +

    Deprecation note: binlog_format is deprecated as of MySQL 8.0.34. Statement-based and mixed formats are being phased out. Row-based logging is the only format recommended for all new MySQL setups.

    +
    + }, + { + question: "Q4. How can MySQL binlogs be used for point-in-time database recovery?", + answer:
    +

    If your database is corrupted or data is accidentally deleted, you can restore a backup from before the incident and then replay binlogs up to the exact moment just before the problem occurred.

    +

    MySQL's mysqlbinlog utility extracts SQL or row events from binlog files by time range or log position. This gives you surgical recovery to any specific point in time rather than being forced to restore the entire backup from hours or days earlier.

    +
    + }, + { + question: "Q5. What binlog retention settings should I configure for CDC pipelines?", + answer:
    +

    For CDC pipelines to work reliably, binlog retention must be longer than the maximum expected gap between syncs.

    +

    Recommended minimum: 7 days (604800 seconds)

    +
      +
    • Use binlog_expire_logs_seconds on MySQL 8.0+ (takes precedence over expire_logs_days if both are set). The default is 2592000 seconds (30 days). Verify this has not been reduced in your environment.
      • +
    • Set to at least 604800 seconds (7 days), which is the consistent recommendation across CDC tool documentation and Percona's operational guidelines.
      • +
    +

    If binlogs are deleted before a CDC tool catches up, such as after lag during maintenance or an outage, the tool cannot bridge the gap and must perform a full re-snapshot of the source tables, which can take hours for large datasets.

    +

    Amazon RDS users: The default binlog retention hours on RDS for MySQL is NULL, meaning binary logs are not retained at all unless explicitly configured. Set retention using:

    + 
    {`CALL mysql.rds_set_configuration('binlog retention hours', 168);`}
    +

    This sets the maximum of 168 hours (7 days) on RDS.

    +
    + }, +]} /> \ No newline at end of file diff --git a/blog/2025-03-18-data-lake-vs-delta-lake.mdx b/blog/2025-03-18-data-lake-vs-delta-lake.mdx index 646a202c4..3ca8c6544 100644 --- a/blog/2025-03-18-data-lake-vs-delta-lake.mdx +++ b/blog/2025-03-18-data-lake-vs-delta-lake.mdx @@ -32,5 +32,64 @@ tags: [lake, tools] - **Data Lakes** are flexible and scalable storage repositories that can handle large volumes of diverse data types but often lack data management, consistency, and performance optimizations. - **Delta Lakes** enhance traditional data lakes by adding ACID transactions, data integrity, performance optimizations, and more, making them suitable for more critical and complex use cases. +## FAQs + +
      +
    • Data Lake: A general architectural pattern: a centralized repository storing raw structured and unstructured data at scale on object storage or HDFS, without guaranteed consistency or transactional guarantees.
      • +
    • Delta Lake: A specific open table format built on top of a Data Lake that adds ACID transactions, schema enforcement, time travel, and data versioning via a transaction log layered over Parquet files. It sits in the same category as Apache Iceberg and Apache Hudi.
      • +
    +

    Delta Lake transforms a fragile file dump into a reliable, manageable table store while keeping data on the same low-cost object storage infrastructure.

    + + }, + { + question: "Q2. Why was Delta Lake created on top of existing Data Lakes?", + answer:
    +

    Data Lakes built on bare file systems lacked transactional guarantees:

    +
      +
    • Concurrent writes caused data corruption
      • +
    • Failed jobs left partial files that polluted the dataset
      • +
    • Schema changes required full rewrites of existing data files
      • +
    +

    Delta Lake was created to solve these reliability problems by adding a transaction log that records all changes atomically. This enables rollbacks, prevents dirty reads, and supports UPDATE, DELETE, and MERGE operations that raw Parquet-on-S3 cannot provide.

    +
    + }, + { + question: "Q3. Does Delta Lake require Apache Spark to work?", + answer:
    +

    Delta Lake was originally designed for Apache Spark and has the deepest integration there, but it is no longer Spark-only. Delta supports a broad range of compute engines including Trino (native read/write support since Trino v373), PrestoDB, Flink, Hive, and APIs for Scala, Java, Rust, Ruby, and Python.

    +

    However, the most complete feature support, including Deletion Vectors, Liquid Clustering, and Databricks-native optimizations, remains in the Spark and Databricks ecosystem.

    +

    Protocol compatibility warning: Enabling Deletion Vectors or Liquid Clustering on a Delta table triggers a protocol upgrade (writer version 7 / reader version 3 for Liquid Clustering). After this upgrade, clients that do not support the upgraded protocol will be unable to read the table at all, not just miss the optimization. Before enabling these features, verify that all engines in your pipeline support the required protocol versions.

    +
    + }, + { + question: "Q4. What ACID transaction guarantees does Delta Lake provide?", + answer:
    +

    Delta Lake provides full ACID compliance enforced through an append-only transaction log (_delta_log):

    +
      +
    • Atomicity: A write either fully succeeds or is completely rolled back; no partial writes are visible
      • +
    • Consistency: The table is always in a valid state; schema enforcement prevents invalid data from landing
      • +
    • Isolation: Concurrent readers and writers do not see each other's in-progress changes through snapshot isolation
      • +
    • Durability: Once committed, data survives system failures; the transaction log is the authoritative record of all table state
      • +
    +
    + }, + { + question: "Q5. When should I choose Delta Lake over a raw Data Lake?", + answer:
    +

    Choose Delta Lake over a raw Data Lake when you need:

    +
      +
    • Reliable data quality guarantees: Schema enforcement prevents silent type mismatches and malformed records
      • +
    • UPDATE and DELETE operations: Raw Parquet-on-S3 is effectively immutable; Delta handles row-level mutations
      • +
    • Time travel: Query historical snapshots of data for auditing, debugging, or reproducibility
      • +
    • Concurrent write safety: Multiple pipelines writing simultaneously without corrupting the dataset
      • +
    • Schema evolution: Add or rename columns without rewriting existing data files
      • +
    +
    + }, +]} /> + diff --git a/blog/2025-03-18-json-vs-bson-vs-jsonb.mdx b/blog/2025-03-18-json-vs-bson-vs-jsonb.mdx index 361da907a..8cdac5f12 100644 --- a/blog/2025-03-18-json-vs-bson-vs-jsonb.mdx +++ b/blog/2025-03-18-json-vs-bson-vs-jsonb.mdx @@ -123,5 +123,63 @@ The evolution from JSON to BSON and JSONB illustrates the ongoing efforts to bal Choosing between these formats depends on your specific needs: the nature of your data, the size of your dataset, the performance requirements, and the underlying database. Understanding the differences between JSON, BSON, and JSONB helps ensure that you're using the right tool for the job, maximizing performance while minimizing storage overhead. - +## FAQs + +
      +
    • JSON: A human-readable, text-based format for data interchange, widely used in APIs and configuration files
      • +
    • BSON (Binary JSON): MongoDB's binary-encoded superset of JSON that supports additional data types like dates, binary data, int32, and int64. BSON is not universally more compact than JSON: it includes length prefixes and explicit type metadata per field, which can make it slightly larger than equivalent JSON. However, this structure makes BSON significantly faster to parse than text-based JSON.
      • +
    • JSONB: PostgreSQL's binary-stored JSON type that decomposes and pre-parses JSON into a fast-indexable internal format during insert, enabling efficient querying and indexing unlike plain JSON which is stored as raw text.
      • +
    + + }, + { + question: "Q2. Why does MongoDB use BSON instead of plain JSON?", + answer:
    +

    MongoDB uses BSON because it supports data types that JSON does not natively:

    +
      +
    • Native Date type (UTC datetime)
      • +
    • Binary data (BinData)
      • +
    • Distinct integer types (int32, int64, Decimal128)
      • +
    • ObjectId and other MongoDB-specific types
      • +
    +

    BSON also stores length information alongside each field, which allows MongoDB to skip directly to the fields it needs without parsing the entire document. Type information is stored inline with each value, so MongoDB knows the exact type at read time without runtime inference.

    +
    + }, + { + question: "Q3. When should you use JSONB over JSON in PostgreSQL?", + answer:
    +

    Use JSONB when you need to query, index, or filter on JSON data in PostgreSQL. Key points:

    +
      +
    • Pre-parsed binary storage: JSONB decomposes and parses JSON at insert time, making all queries faster than plain JSON even before adding any indexes
      • +
    • GIN index support: GIN indexes on JSONB accelerate containment queries using the @> operator and key-existence queries using ?, ?|, and ?&. Note that GIN indexes activate only for these specific operator-based queries; plain equality or expression queries on JSONB fields may still need expression B-tree indexes
      • +
    • Plain JSON stores the text as-is and requires full parsing on every query, with no index acceleration available
      • +
    +

    JSONB has slightly higher write overhead since every record is fully parsed during insert, but delivers significantly better read and query performance for analytics on JSON fields.

    +
    + }, + { + question: "Q4. What are the performance differences between JSON, BSON, and JSONB for analytical queries?", + answer:
    +
      +
    • JSONB: Fastest for analytical queries in PostgreSQL. Pre-parsed binary storage speeds up all queries, and GIN indexes further accelerate containment (@>) and key-existence (?) queries
      • +
    • BSON: Enables fast reads in MongoDB because length information and type metadata are stored inline per field, allowing MongoDB's query engine to skip directly to needed fields without parsing the entire document
      • +
    • Plain JSON: Slowest for queries. The entire text must be parsed and types inferred on every read operation, with no index support available
      • +
    +
    + }, + { + question: "Q5. Which format should I choose for storing data in an operational database?", + answer:
    +
      +
    • Use BSON if you are storing documents in MongoDB. It is automatic and optimized for MongoDB's workloads. When you insert JSON via a MongoDB driver, the database converts it to BSON behind the scenes with no extra work required
      • +
    • Use JSONB if you are in PostgreSQL and need to query, filter, or index JSON fields frequently
      • +
    • Use plain JSON only for storing data that you will retrieve and display as-is, without complex server-side querying
      • +
    • For large-scale analytics, convert any of these formats to Parquet-based columnar storage (e.g. Apache Iceberg) for maximum query performance. Row-oriented formats like JSON, BSON, and JSONB are not optimized for the scan-heavy workloads typical of analytical queries
      • +
    +
    + }, +]} /> \ No newline at end of file diff --git a/blog/2025-04-22-olake-architecture-deep-dive.mdx b/blog/2025-04-22-olake-architecture-deep-dive.mdx index 11569062f..96541572d 100644 --- a/blog/2025-04-22-olake-architecture-deep-dive.mdx +++ b/blog/2025-04-22-olake-architecture-deep-dive.mdx @@ -1,26 +1,26 @@ --- slug: olake-architecture-deep-dive -title: "Deep Dive into OLake Architecture & Data Replication" -description: "Explore OLake's modular architecture for real-time database replication, parallel processing, and scalable data lakehouse ingestion for faster analytics" +title: "Deep Dive into OLake Go Architecture & Data Replication" +description: "Explore OLake Go's modular architecture for real-time database replication, parallel processing, and scalable data lakehouse ingestion for faster analytics" image: /img/blog/cover/olake-architecture-deep-dive-cover.webp authors: [sandeep] tags: [olake] --- -# A Deep Dive into OLake Architecture and Inner Workings +# A Deep Dive into OLake Go Architecture and Inner Workings -![Inside OLake's Architecture with schematic database, data flow, and network icon illustrations on a dark blue background](/img/blog/cover/olake-architecture-deep-dive-cover.webp) +![Inside OLake Go's Architecture with schematic database, data flow, and network icon illustrations on a dark blue background](/img/blog/cover/olake-architecture-deep-dive-cover.webp) -OLake is an open-source tool designed for efficiently replicating databases to data lakes in Open table formats like Apache Iceberg. It provides a high-performance, scalable solution for data ingestion, enabling real-time analytics by efficiently moving data from operational databases to analytical storage systems. +OLake Go is an open-source tool designed for efficiently replicating databases to data lakes in Open table formats like Apache Iceberg. It provides a high-performance, scalable solution for data ingestion, enabling real-time analytics by efficiently moving data from operational databases to analytical storage systems. -With OLake, organizations can seamlessly bridge their operational databases with analytics platforms, unlocking deeper insights faster than traditional methods. +With OLake Go, organizations can seamlessly bridge their operational databases with analytics platforms, unlocking deeper insights faster than traditional methods. -In this blog post, we'll dive deep into OLake's architecture and explain how it works under the hood. +In this blog post, we'll dive deep into OLake Go's architecture and explain how it works under the hood. -## The Problem OLake Solves +## The Problem OLake Go Solves -Before diving into the architecture, let's understand the problem OLake addresses. Organizations often need to: +Before diving into the architecture, let's understand the problem OLake Go addresses. Organizations often need to: 1. Replicate data from operational databases ([MongoDB](https://olake.io/docs/connectors/mongodb/overview), [PostgreSQL](https://olake.io/docs/connectors/postgres/overview), [MySQL](https://olake.io/docs/connectors/mysql/overview)) to data Lakehouses ([Apache Iceberg](https://olake.io/docs/writers/iceberg/overview)), with supoort of sources like AWS S3 and Kafka comming soon. 2. Supports backfills (full data refreshes) and real-time change data capture (CDC). @@ -30,7 +30,7 @@ Before diving into the architecture, let's understand the problem OLake addresse Traditional ETL tools often struggle with these requirements, especially when dealing with real-time data changes and large volumes. They typically introduce latency, schema compatibility issues, and scalability challenges. -OLake was built specifically to address these challenges. OLake natively supports replicating advanced data types, giving better performance like resulting in 4x to 10x faster large data loads. See detailed benchmarks [here](https://olake.io/docs/benchmarks?tab=postgres). +OLake Go was built specifically to address these challenges. OLake Go natively supports replicating advanced data types, giving better performance like resulting in 4x to 10x faster large data loads. See detailed benchmarks [here](https://olake.io/docs/benchmarks?tab=postgres). Its tailored optimizations and resilience mechanisms ensure stable and efficient operations even under high-load conditions. @@ -46,7 +46,7 @@ Its tailored optimizations and resilience mechanisms ensure stable and efficient style={{ width: '80%', maxWidth: '500px', margin: '2rem auto', display: 'block' }} /> -OLake follows a modular, plugin-based architecture with clear separation of concerns. At its core, OLake consists of: +OLake Go follows a modular, plugin-based architecture with clear separation of concerns. At its core, OLake Go consists of: 1. **Core Framework:** The central component that orchestrates the entire data flow 2. **Drivers (Sources)**: Connectors to source databases like MongoDB, PostgreSQL, and MySQL @@ -68,7 +68,7 @@ Let's explore each of these components in detail. /> -The core framework is the heart of OLake, providing: +The core framework is the heart of OLake Go, providing: * **Modular Architecture**: Each component (drivers, writers) is designed to be modular and extensible. * **Command-Line Interface**: Built using Cobra, it exposes commands like `spec`, `check`, `discover`, and `sync` * **Configuration Management**: Handles source, destination, streams, and state configurations @@ -81,7 +81,7 @@ This modularity allows developers to quickly introduce new sources or destinatio ### Protocol Layer -The protocol layer defines critical interfaces ensuring OLake's modularity and ease of extension. This design simplifies maintenance by clearly defining interaction points and responsibilities across components. Key interfaces include the Connector, Driver, and Writer interfaces, which standardize operations across sources and destinations. +The protocol layer defines critical interfaces ensuring OLake Go's modularity and ease of extension. This design simplifies maintenance by clearly defining interaction points and responsibilities across components. Key interfaces include the Connector, Driver, and Writer interfaces, which standardize operations across sources and destinations. ```go @@ -124,7 +124,7 @@ These interfaces ensure that all components adhere to a consistent contract, mak ### Drivers (Sources) -OLake supports multiple database sources through its driver architecture. Each driver implements the `Driver` interface and provides specific functionality for its database type. +OLake Go supports multiple database sources through its driver architecture. Each driver implements the `Driver` interface and provides specific functionality for its database type. #### MongoDB Driver @@ -136,7 +136,7 @@ The MongoDB driver connects to MongoDB databases and supports: * **Parallel Processing**: Splits large collections into chunks for parallel processing * **Schema Detection**: Automatically detects and adapts to MongoDB's flexible schema -The MongoDB driver handles MongoDB-specific data types and converts them to OLake's internal representation. For CDC, it uses MongoDB's change streams API to capture inserts, updates, and deletes in real-time. +The MongoDB driver handles MongoDB-specific data types and converts them to OLake Go's internal representation. For CDC, it uses MongoDB's change streams API to capture inserts, updates, and deletes in real-time. Refer -> [MongoDB documentation](https://olake.io/docs/connectors/mongodb/overview) for more details. @@ -192,7 +192,7 @@ The PostgreSQL driver supports: * **Full Refresh Mode**: Reads all data from tables in batches * **CDC Mode**: Uses PostgreSQL's logical replication to capture changes * **Parallel Processing**: Splits tables into chunks based on primary keys or CTIDs -* **Schema Detection**: Maps PostgreSQL data types to OLake's type system +* **Schema Detection**: Maps PostgreSQL data types to OLake Go's type system As already mentioned, for CDC, the PostgreSQL driver uses the `pglogrepl` library to connect to PostgreSQL's logical replication slots and process WAL (Write-Ahead Log) entries using the native pgoutput protocol, scoped by a PostgreSQL publication. @@ -237,7 +237,7 @@ The MySQL driver supports: * Full Refresh Mode: Reads all data from tables in batches * CDC Mode: Uses MySQL's binlog to capture changes * Parallel Processing: Splits tables into chunks for parallel processing -* Schema Detection: Maps MySQL data types to OLake's type system +* Schema Detection: Maps MySQL data types to OLake Go's type system For CDC, the MySQL driver uses the `go-mysql` library to connect to MySQL and process change events from binlogs. @@ -245,7 +245,7 @@ Refer -> [MySQL documentation](https://olake.io/docs/connectors/mysql/overview) ## Writers (Destinations) -OLake supports multiple destination (S3 and Apache Iceberg) types through its writer architecture. Each writer implements the `Writer` interface and provides specific functionality for its destination type. +OLake Go supports multiple destination (S3 and Apache Iceberg) types through its writer architecture. Each writer implements the `Writer` interface and provides specific functionality for its destination type. ### Parquet Writer @@ -257,7 +257,7 @@ The Parquet writer writes data to Parquet files, either locally or on AWS S3: * **Local File System**: Writes Parquet files to a local directory. * **S3 Integration**: Uploads Parquet files to an S3 bucket * **Partitioning**: Supports partitioning data based on record fields. -* **Normalization**: OLake supports L1 (level 1) normalization +* **Normalization**: OLake Go supports L1 (level 1) normalization * **Schema Evolution**: Handles schema changes by creating new `.parquet` files (for S3 and local dump only, not to Iceberg Database) @@ -296,7 +296,7 @@ func (p *Parquet) Write(_ context.Context, record types.RawRecord) error { ### Iceberg Writer -The Iceberg writer leverages Java-based gRPC components to fully support Iceberg operations, including handling complex transactions and schema evolution. By using Apache Iceberg, OLake provides robust features for time-travel queries and data consistency guarantees essential for analytics workloads. +The Iceberg writer leverages Java-based gRPC components to fully support Iceberg operations, including handling complex transactions and schema evolution. By using Apache Iceberg, OLake Go provides robust features for time-travel queries and data consistency guarantees essential for analytics workloads. The Iceberg writer writes data to Apache Iceberg tables and offers the following: @@ -333,11 +333,11 @@ func (i *Iceberg) Write(_ context.Context, record types.RawRecord) error { } ``` -## Data Flow in OLake +## Data Flow in OLake Go -Now that we understand the components, let's look at how data flows through OLake: +Now that we understand the components, let's look at how data flows through OLake Go: -1. **Configuration**: The user provides configuration files for the source (`source.json`) and destination (`destination.json`) and OLake generates `streams.json`, and `state.json` file. +1. **Configuration**: The user provides configuration files for the source (`source.json`) and destination (`destination.json`) and OLake Go generates `streams.json`, and `state.json` file. 2. **Discovery**: The driver discovers available streams (tables/collections) along with their detailed metadata like data types, indexed columns, etc 3. **Sync Execution**: * **For full refresh**: Executes a complete data replication, optimized through chunking and parallel processing. @@ -351,25 +351,25 @@ Let's walk through a typical sync operation: ### Full Refresh Sync 1. The user runs the `sync` command (without passing the `--state` argument for the first time). -2. OLake initializes the driver and writer based on the configurations. +2. OLake Go initializes the driver and writer based on the configurations. 3. The driver discovers available streams and matches them with the streams. -4. For each selected stream, OLake: +4. For each selected stream, OLake Go: * Splits the data into chunks for parallel processing * Creates writer threads for the destination * Reads data from the source in batches - * Converts the data type to OLake's supported Data types. + * Converts the data type to OLake Go's supported Data types. * Writes the data to the destination * Updates the state to track progress in `state.json` file and `stats.json` file. -5. Once all streams are processed, OLake finalizes the sync and reports statistics. +5. Once all streams are processed, OLake Go finalizes the sync and reports statistics. 6. It automatically generates a state file which can be used later to initial incremental or CDC sync. So that it can only process the newly arrived data. ### CDC Sync 1. The user runs the `sync` command with appropriate configuration files with `--state` flag and state file (that stored the last sync state / pointer). -2. OLake initializes the driver and writer based on the configurations. +2. OLake Go initializes the driver and writer based on the configurations. 3. The driver discovers available streams (with new column or table addition / modifications for schema evolution or schema data type changes) and matches them with the catalog. -4. For each selected stream with CDC mode, OLake: +4. For each selected stream with CDC mode, OLake Go: * Checks if a full refresh is needed (first run or incomplete previous run) * If needed, performs a full refresh first (in case a new table is added or schema changes) * Sets up a change stream connection to the source database @@ -377,12 +377,12 @@ Let's walk through a typical sync operation: * Captures change events (inserts, updates, deletes) from the last processed state (as noted by the state file cursor or pointer) * Writes the events to the destination * Updates the state to track progress -5. The CDC process ends if there are no new updates. Use Airflow on ec2 or k8s to schedule the sync process periodically. Related [docs](https://olake.io/docs/install/kubernetes) and blogs on running OLake on [AWS EC2](https://olake.io/blog/olake-airflow-on-ec2) and [Kubernetes](https://olake.io/blog/olake-airflow). +5. The CDC process ends if there are no new updates. Use Airflow on ec2 or k8s to schedule the sync process periodically. Related [docs](https://olake.io/docs/install/kubernetes) and blogs on running OLake Go on [AWS EC2](https://olake.io/blog/olake-airflow-on-ec2) and [Kubernetes](https://olake.io/blog/olake-airflow). -## Sync Execution in OLake: Advanced Chunking Strategies and Parallel Processing +## Sync Execution in OLake Go: Advanced Chunking Strategies and Parallel Processing -To process large amounts of data efficiently, OLake employs DB specific chunking strategies. These strategies are critical for optimizing performance during Full Refresh (Backfill Mode) operations. +To process large amounts of data efficiently, OLake Go employs DB specific chunking strategies. These strategies are critical for optimizing performance during Full Refresh (Backfill Mode) operations. ### Database-Specific Chunking Strategies @@ -704,7 +704,7 @@ MySQL implements efficient chunking strategies tailored to its storage engine ch /> -Once chunks are created, OLake processes them in parallel: +Once chunks are created, OLake Go processes them in parallel: The parallel processing system includes: @@ -739,7 +739,7 @@ MaxThreads, processChunk) /> -OLake's chunking and parallel processing approach includes several key optimizations: +OLake Go's chunking and parallel processing approach includes several key optimizations: 1. Memory-Aware Chunk Sizing * Adjusts chunk sizes based on available system memory @@ -843,11 +843,11 @@ This chunking and parallel processing approach significantly improves performanc * Enabling efficient use of multi-core systems * Provides resumable full-load by saving state of each chunk -This parallel chunking strategy is a key factor in OLake's high-performance data replication capabilities, especially for large datasets. +This parallel chunking strategy is a key factor in OLake Go's high-performance data replication capabilities, especially for large datasets. -## Concurrency Model in OLake +## Concurrency Model in OLake Go -OLake implements a multi-level concurrency model that adapts to different sync modes and database types, optimizing performance while maintaining system stability. +OLake Go implements a multi-level concurrency model that adapts to different sync modes and database types, optimizing performance while maintaining system stability. ### Core Concurrency Architecture @@ -860,7 +860,7 @@ OLake implements a multi-level concurrency model that adapts to different sync m /> -OLake's concurrency model strategically manages resources across three levels—Global, Stream-Level, and Writer Pool—to optimize throughput without overwhelming systems. +OLake Go's concurrency model strategically manages resources across three levels—Global, Stream-Level, and Writer Pool—to optimize throughput without overwhelming systems. #### 1. Global Concurrency @@ -939,12 +939,12 @@ WithIdentifier(fmt.Sprintf("thread-%d", threadID))) /> -OLake's concurrency model adapts to different sync modes, optimizing performance for each scenario: +OLake Go's concurrency model adapts to different sync modes, optimizing performance for each scenario: #### Backfill (Full Refresh) Concurrency -For backfill operations, OLake implements a chunking-based concurrency model: +For backfill operations, OLake Go implements a chunking-based concurrency model: @@ -1106,7 +1106,7 @@ streams ...protocol.Stream) error { #### Adaptive Concurrency Control -OLake implements adaptive concurrency that responds to system conditions: +OLake Go implements adaptive concurrency that responds to system conditions: @@ -1144,7 +1144,7 @@ go func() { #### Hybrid Sync Mode Coordination -For streams that require both backfill and CDC, OLake coordinates the transition: +For streams that require both backfill and CDC, OLake Go coordinates the transition: ```go @@ -1242,7 +1242,7 @@ error) { /> -OLake uses a state management system to track sync progress and enable resumable operations: +OLake Go uses a state management system to track sync progress and enable resumable operations: 1. **Stream-Level State**: Tracks progress for individual streams @@ -1283,12 +1283,12 @@ type State interface { ``` -This state management system ensures that OLake can resume operations after interruptions, making it resilient to failures. +This state management system ensures that OLake Go can resume operations after interruptions, making it resilient to failures. ## Key Design Principles -OLake's architecture is guided by several key design principles: +OLake Go's architecture is guided by several key design principles: @@ -1298,16 +1298,81 @@ OLake's architecture is guided by several key design principles: 4. **Modularity**: Clear separation of concerns through well-defined interfaces 5. **Extensibility**: Easy to add new drivers and writers through the plugin architecture -These principles ensure that OLake remains efficient, maintainable, and extensible as it evolves. +These principles ensure that OLake Go remains efficient, maintainable, and extensible as it evolves. ## Conclusion -OLake provides a powerful, high-performance solution for replicating databases to data lakehouses. Its modular architecture, support for both full refresh and CDC modes, and efficient concurrency model make it an excellent choice for organizations looking to enable real-time analytics. Also, OLake includes several performance optimizations like parallel processing, adaptive batch sizing to achieve high throughput even with large datasets. - -By clearly understanding OLake's internal workings and principles, developers and organizations can better leverage its capabilities to drive insightful analytics and informed decision-making. Whether using OLake for MongoDB, PostgreSQL, or MySQL, and writing to Parquet files or Apache Iceberg tables, the system's consistent design principles ensure a reliable and efficient data replication experience. - -If this excites you, check out OLake, check out the [GitHub repository](https://github.com/datazip-inc/olake) and join the [Slack community](https://join.slack.com/t/getolake/shared_invite/zt-2usyz3i6r-8I8c9MtfcQUINQbR7vNtCQ) to get started. - +OLake Go provides a powerful, high-performance solution for replicating databases to data lakehouses. Its modular architecture, support for both full refresh and CDC modes, and efficient concurrency model make it an excellent choice for organizations looking to enable real-time analytics. Also, OLake Go includes several performance optimizations like parallel processing, adaptive batch sizing to achieve high throughput even with large datasets. + +By clearly understanding OLake Go's internal workings and principles, developers and organizations can better leverage its capabilities to drive insightful analytics and informed decision-making. Whether using OLake Go for MongoDB, PostgreSQL, or MySQL, and writing to Parquet files or Apache Iceberg tables, the system's consistent design principles ensure a reliable and efficient data replication experience. + +If this excites you, check out OLake, check out the [GitHub repository](https://github.com/datazip-inc/olake) and join the [Slack community](https://olake.io/slack) to get started. +## FAQs + +

    OLake Go's architecture is built around five main components:

    +
      +
    1. Core Framework: The central orchestrator that coordinates the entire data pipeline lifecycle, including command-line interface, configuration management, concurrency management, state management, and monitoring.
      2. +
    2. Drivers (Sources): Database-specific connectors for MongoDB, PostgreSQL, MySQL, and other supported sources. Each driver is autonomous with its own dependencies, keeping the overall binary size minimal.
      3. +
    3. Writers (Destinations): Components that write data to OLake Go's two supported destinations: Apache Iceberg and Parquet files on S3.
      4. +
    4. Protocol Layer: Defines the interfaces and abstractions so sources and destinations remain interchangeable.
      5. +
    5. Type System: Handles data type conversions and schema management across different database and lakehouse type systems.
      6. +
    + + }, + { + question: "Q2. How does OLake Go's CDC (Change Data Capture) mechanism work for ongoing replication?", + answer:
    +

    After the initial full snapshot, OLake Go switches to CDC mode to capture ongoing changes using the native replication mechanism of each source:

    +
      +
    • PostgreSQL: Uses logical replication slots and WAL events via the pgoutput protocol, scoped by a PostgreSQL publication. A single WAL reader thread distributes messages to multiple dedicated writer threads, one per stream.
      • +
    • MySQL: Reads the binary log (binlog) using a single-reader, multi-writer pattern: one thread reads and tracks its exact binlog position for resumability, while multiple writer threads process the events concurrently.
      • +
    • Microsoft SQL Server: Uses SQL Server's native CDC feature, which records every insert, update, and delete into change tables derived from the transaction log. OLake Go reads these change tables and applies the changes downstream so the destination stays aligned with the source. CDC must be enabled on the database and on each table you want to sync.
      • +
    • MongoDB: After the snapshot completes, OLake Go watches MongoDB's change stream, which reports every insert, update, and delete as it happens, to keep the destination current in near real time. Change streams are MongoDB's built-in API over the oplog, so OLake Go does not read the raw oplog directly.
      • +
    +

    Each captured event (insert, update, delete) is written to the destination as an Iceberg snapshot with atomic commit semantics, so no partial writes reach the destination. When normalization is enabled, events are also processed by the Type System and normalized to the target schema before they are written; with normalization turned off, the data is written without that schema normalization step.

    +
    + }, + { + question: "Q3. What is parallel chunking in OLake Go and how does it speed up large data loads?", + answer:
    +

    Parallel chunking splits a large source table into smaller, non-overlapping segments and processes several of them at once across separate worker threads, instead of reading the table top to bottom in a single pass. Each chunk is read, transformed, and written to the destination concurrently, which is what makes large loads fast.

    +

    A table that would take 4 hours to copy sequentially might finish in around 30 minutes with 8 parallel threads. You control the thread count per pipeline with max_threads. OLake Go also adjusts how many threads actually run at once so it doesn't overload the source database or the machine's CPU, memory, and network.

    +

    OLake Go uses different strategies to decide where to split each source (for example, primary key ranges, CTID ranges, or row-count based splits). The chunking strategies are covered in detail here: What makes OLake Go fast.

    +
    + }, + { + question: "Q4. How does OLake Go ensure data consistency during parallel writes to Apache Iceberg?", + answer:
    +

    OLake Go follows Iceberg's ACID commit protocol:

    +
      +
    1. Each worker thread is assigned a chunk of data and writes its Parquet data files to object storage independently and concurrently
      2. +
    2. After all workers complete their file writes, OLake Go performs a single atomic metadata commit that registers all new Parquet files under the Iceberg table format via an AddFiles (REGISTER) operation in one operation
      3. +
    +

    This ensures readers either see the complete batch or nothing. There are no partial states visible to concurrent readers during a bulk load. If an ingestion job fails midway, there is zero risk of a downstream consumer reading a partial or corrupted dataset.

    +
    + }, + { + question: "Q5. What databases and destinations does OLake Go currently support?", + answer:
    +

    Sources:

    +
      +
    • PostgreSQL, MySQL, MongoDB, Oracle, IBM Db2, Microsoft SQL Server (MSSQL), Apache Kafka, and Amazon S3.
      • +
    • CDC is supported for PostgreSQL, MySQL, MongoDB, and MSSQL using each source's native replication mechanism (pgoutput/WAL, binlogs, oplogs, change tables).
      • +
    • Apache Kafka uses latest offset bounded incremental sync, ingesting new messages from the event stream up to the latest offset at the time of each run.
      • +
    • Oracle, IBM Db2, and Amazon S3 support Full Refresh and Full Refresh + Incremental, but not CDC. Full CDC mode for Oracle is currently work-in-progress; check OLake's documentation for the latest status.
      • +
    +

    Destinations:

    +
      +
    • Apache Iceberg on S3-compatible object storage (AWS S3, GCS, Azure Blob, MinIO). Supported catalog types are AWS Glue, REST, JDBC/SQL, and Hive. The REST catalog covers Generic REST, Lakekeeper, Nessie, S3 Tables, Unity Catalog, and Apache Polaris.
      • +
    • Parquet files on S3, plus local Parquet for development and testing.
      • +
    +

    OLake Go is actively expanding its source and destination coverage. Check the official documentation for the latest supported connectors and CDC availability per source.

    +
    + }, +]} /> \ No newline at end of file diff --git a/blog/2025-04-23-how-to-set-up-postgresql-cdc-on-aws-rds.mdx b/blog/2025-04-23-how-to-set-up-postgresql-cdc-on-aws-rds.mdx index 2cd042f2b..5490f061c 100644 --- a/blog/2025-04-23-how-to-set-up-postgresql-cdc-on-aws-rds.mdx +++ b/blog/2025-04-23-how-to-set-up-postgresql-cdc-on-aws-rds.mdx @@ -122,13 +122,13 @@ If you are wondering what you have done, let’s get into details: ## **Configure above parameter group & VPC Security Group** -Everything on RDS runs within virtual private networks, which means we need to configure accessibility to our DB instance from OLake. Configure inbound and outbound access routes. +Everything on RDS runs within virtual private networks, which means we need to configure accessibility to our DB instance from OLake Go. Configure inbound and outbound access routes. * Navigate to RDS → Databases → Our Postgres DB → Modify (Top right corner) * Choose the new parameter group created in the parameter group section. -* The VPC security group you have selected, should have connectivity from OLake setup. +* The VPC security group you have selected, should have connectivity from OLake Go setup. * In Additional configuration, specify the DB parameter group to use the group we just created for PostgreSQL CDC. @@ -202,63 +202,131 @@ select pg_drop_replication_slot(''); SELECT * from pg_publication; ``` -Now you are ready to configure the ETL/ELT sync using OLake to dump the data into Apache Iceberg or Parquets for downstream analytics or machine-learning. +Now you are ready to configure the ETL/ELT sync using OLake Go to dump the data into Apache Iceberg or Parquets for downstream analytics or machine-learning. This setup can be used for other downstream processing as well, example PG --> Debezium --> Kafka --> Other backend-services consuming the CDC. -## **Common FAQs** -### **How does OLake handle PostgreSQL CDC data extraction and what are the performance considerations?** -OLake uses PostgreSQL's logical replication to capture change data in real-time. The tool connects to your RDS instance using the configured replication slot and extracts INSERT, UPDATE, and DELETE operations. Performance impact is minimal when properly configured - OLake only reads WAL (Write-Ahead Log) entries, not the actual data files. - -### **What are the best practices for configuring OLake with PostgreSQL CDC on AWS RDS?** - -Key best practices include: -- Set `rds.logical_replication = 1` and configure appropriate `max_replication_slots` -- Use dedicated database users with minimal required privileges -- Create replication slots with `pgoutput` plugin (currently supported by OLake) -- Monitor replication lag and adjust `max_wal_senders` as needed -- Use separate replication slots for different OLake jobs to avoid conflicts - -### **How does OLake transform and load PostgreSQL CDC data into data lake-house (Apache-iceberg)?** - -OLake processes the CDC data through its ELT engine, which can handle schema evolution, data type conversions, and business logic transformations (Coming soon). The tool supports multiple destination formats including Apache Iceberg, Parquet (On top of all popular cloud providers). OLake automatically handles the orchestration, logging of ETL pipeline runs from PostgreSQL CDC with near-real-time latency. - -### **What are the common troubleshooting steps for OLake PostgreSQL CDC integration?** - -Common issues and solutions: -- **Replication lag**: Check `max_wal_senders` and `max_replication_slots` settings -- **Connection failures**: Verify VPC security groups and network connectivity -- **Permission errors**: Ensure the database user has `rds_replication` role and proper schema permissions -- **Slot conflicts**: Use unique replication slot names for different OLake jobs -- **Data consistency**: Monitor OLake's checkpoint mechanism to ensure no data loss (Coming soon in monitoring dashboard feature) - -### **How does OLake compare to other ETL tools for PostgreSQL CDC workflows?** - -OLake offers several advantages for PostgreSQL CDC: -- **Real-time processing**: Native support for streaming CDC data as low as 1 minute latency (continuous-batching coming soon) -- **Schema evolution**: Automatic handling of table schema changes during replication -- **Multi-destination support**: Write to multiple formats (Iceberg, Parquet, etc.) -- **Built-in monitoring & alerting**: Comprehensive metrics, alerting and logging for CDC pipeline health (Coming soon) -- **Cloud-native**: Optimized for AWS RDS and other cloud database services - -### **What monitoring and alerting should be set up for OLake PostgreSQL CDC pipelines?** - -Essential monitoring includes: -- Replication lag metrics from OLake dashboard (Coming soon) -- Alerting for sync failures (Coming soon) -- WAL generation rate and replication slot status (Coming soon) -- Destination write performance and error rates (Coming soon) -- Set up alerts for replication lag exceeding thresholds and connection failures (Coming soon) - -### **Does OLake take care of Full-historical snapshot/replication before CDC? How fast is it?** - -OLake has fastest optimised historical load: -- OLake has Historical-load + CDC mode for this -- Tables are chunked into smaller pieces to make it parallel and recoverable from failures -- Any new table additions is also taken care of automatically. - -For more detailed information about OLake's PostgreSQL CDC capabilities, visit [olake.io](https://olake.io) and [olake.io/docs](https://olake.io/docs). +## FAQs + + +

    OLake Go uses PostgreSQL's logical replication to capture change data in real-time. The tool connects to your RDS instance using the configured replication slot and extracts INSERT, UPDATE, and DELETE operations. Performance impact is minimal when properly configured - OLake Go only reads WAL (Write-Ahead Log) entries, not the actual data files.

    + + }, + { + question: "Q2. What are the best practices for configuring OLake Go with PostgreSQL CDC on AWS RDS?", + answer:
    +

    Key best practices include:

    +
      +
    • Set rds.logical_replication = 1 and configure appropriate max_replication_slots
      • +
    • Use dedicated database users with minimal required privileges
      • +
    • Create replication slots with pgoutput plugin (currently supported by OLake Go)
      • +
    • Monitor replication lag and adjust max_wal_senders as needed
      • +
    • Use separate replication slots for different OLake Go jobs to avoid conflicts
      • +
    +
    + }, + { + question: "Q3. How does OLake Go load PostgreSQL CDC data into a data lakehouse (Apache Iceberg)?", + answer:
    +

    OLake Go processes PostgreSQL CDC data through its EL (Extract-Load) engine, which handles schema evolution and data type conversions as it writes. It supports two destination formats: Apache Iceberg tables and Parquet files on Amazon S3. OLake Go automatically handles orchestration, state management, and logging of pipeline runs, keeping the destination current with near real-time latency.

    +
    + }, + { + question: "Q4. What are the common troubleshooting steps for OLake Go PostgreSQL CDC integration?", + answer:
    +

    Common issues and solutions:

    +
      +
    • Replication lag: Check max_wal_senders and max_replication_slots settings
      • +
    • Connection failures: Verify VPC security groups and network connectivity
      • +
    • Permission errors: Ensure the database user has rds_replication role and proper schema permissions
      • +
    • Slot conflicts: Use unique replication slot names for different OLake Go jobs
      • +
    • Data consistency: Monitor OLake Go's checkpoint mechanism to ensure no data loss via 2PC protocol
      • +
    +
    + }, + { + question: "Q5. How does OLake Go compare to other ETL tools for PostgreSQL CDC workflows?", + answer:
    +

    OLake Go is an EL tool built specifically for replicating databases into open lakehouse formats, which shapes its advantages for PostgreSQL CDC:

    +
      +
    • Real-time processing: Native support for streaming CDC data as low as 1 minute latency
      • +
    • Schema evolution: Automatic handling of table schema changes during replication, without breaking downstream jobs.
      • +
    • Two native destinations: Apache Iceberg tables and Parquet files on S3, written directly with no intermediate systems.
      • +
    • Source-level filtering: SQL-style filter conditions applied at ingestion, so only the rows you need are read and replicated.
      • +
    • No duplicate rows: In Upsert mode, OLake Go deduplicates records using the source table's primary key, so each row appears exactly once in the destination.
      • +
    • Resumable, stateful syncs: Parallel chunking splits large tables for fast initial loads, and checkpointing lets interrupted syncs resume from the last checkpoint instead of restarting.
      • +
    • Alerts and notifications: Configurable alerts for sync events and failures on any OLake Go job.
      • +
    +
    +}, + { + question: "Q6. What monitoring and alerting should be set up for OLake Go PostgreSQL CDC pipelines?", + answer:
    +

    Monitor the pipeline from two sides:

    +
      +
    • On the PostgreSQL side, watch replication slot health and lag with pg_replication_slots (check active status and the size of retained WAL), and keep an eye on WAL generation so a lagging slot doesn't fill the disk.
      • +
    • On the OLake Go side, use OLake Go's alerts and notifications to get notified of sync events and failures. See the alerts and notifications documentation for setup.
      • +
    +

    Because you are using separate replication slots per job, monitoring each slot individually makes it easy to isolate which pipeline is falling behind.

    +
    + }, + { + question: "Q7. Does OLake Go take care of Full-historical snapshot/replication before CDC? How fast is it?", + answer:
    +

    OLake Go has fastest optimised historical load:

    +
      +
    • OLake Go has Historical-load + CDC mode for this
      • +
    • Tables are chunked into smaller pieces to make it parallel and recoverable from failures
      • +
    • Any new table additions is also taken care of automatically.
      • +
    +

    For more detailed information about OLake Go's PostgreSQL CDC capabilities, visit the Postgres connector documentation.

    +
    + }, + { + question: "Q8. What is the difference between pgoutput and wal2json plugins, and which should I use with OLake Go?", + answer:
    +

    Both plugins receive the same core change information from the WAL - the difference is how they output it:

    +
      +
    • pgoutput: Encodes changes in PostgreSQL's native binary logical replication protocol. It is the default and highest-performance option for most CDC workloads and is supported on all managed services (AWS RDS, Aurora PostgreSQL, Google Cloud SQL, Azure).
      • +
    • wal2json: Converts WAL changes into JSON format, which is easier to parse in any programming language but has higher overhead than pgoutput.
      • +
    +

    OLake Go uses pgoutput via a publication, which is the recommended approach for both RDS PostgreSQL and Aurora PostgreSQL.

    +

    Important: pgoutput does not emit change events for tables that lack a primary key unless REPLICA IDENTITY FULL is explicitly set on those tables. If you are replicating tables without primary keys, run ALTER TABLE <table_name> REPLICA IDENTITY FULL before starting CDC.

    +
    + }, + { + question: "Q9. Why should the CDC database user have minimal privileges rather than using the RDS master user?", + answer:
    +

    While you can use the AWS master user account for CDC setup since it already has the rds_superuser and rds_replication roles, best practice is to create a dedicated account with only the minimum required permissions. This limits the blast radius of a potential credential leak.

    +

    The dedicated user only needs:

    +
      +
    • USAGE on the relevant schemas
      • +
    • SELECT on the tables being replicated
      • +
    • The rds_replication role
      • +
    +

    Nothing more. Always use a purpose-built, least-privilege CDC user rather than a shared superuser account.

    +
    + }, + { + question: "Q10. Can you use a single replication slot for multiple OLake Go sync jobs?", + answer:
    +

    This is not recommended. A replication slot acts as an anchor - PostgreSQL keeps all WAL files needed by the slowest consumer on that slot, regardless of how far ahead other consumers have progressed. Using a shared slot across multiple jobs means one slow or stalled job blocks WAL cleanup for all consumers.

    +

    Best practice: Each independent OLake Go pipeline should use its own uniquely named replication slot to isolate failures and allow independent progress tracking. If one pipeline stalls, it does not hold back WAL cleanup for the others.

    +
    + }, + { + question: "Q11. Does this CDC setup work with RDS Aurora PostgreSQL as well?", + answer:
    +

    Yes. Aurora PostgreSQL supports CDC through PostgreSQL logical decoding using the native pgoutput plugin, and OLake Go has a dedicated Aurora PostgreSQL setup guide.

    +

    Logical replication on Aurora is enabled through the cluster parameter group (by setting rds.logical_replication = 1) rather than an instance parameter group, and changes are streamed via pgoutput with a PostgreSQL publication.

    +

    Note: The configuration differs from RDS PostgreSQL in a few areas. Aurora uses a cluster parameter group, requires a writer-node reboot to apply the change, and replication slot behavior can differ in a multi-AZ cluster. Follow OLake's Aurora PostgreSQL setup guide rather than the standard RDS PostgreSQL guide to avoid misconfiguration.

    +
    + }, +]} /> diff --git a/blog/2025-04-30-olake-airflow.mdx b/blog/2025-04-30-olake-airflow.mdx index 0730ce4bb..31de89389 100644 --- a/blog/2025-04-30-olake-airflow.mdx +++ b/blog/2025-04-30-olake-airflow.mdx @@ -11,9 +11,9 @@ tags: [olake] ![Apache Airflow logo with OLake logo, illustrating Airflow and OLake integration](/img/blog/cover/olake-airflow-cover.webp) -At OLake, we're building tools to make data integration seamless. Today, we're excited to show you how to leverage your existing Apache Airflow setup to automate OLake data synchronization tasks directly on your Kubernetes cluster! +At OLake Go, we're building tools to make data integration seamless. Today, we're excited to show you how to leverage your existing Apache Airflow setup to automate OLake Go data synchronization tasks directly on your Kubernetes cluster! -OLake is designed to efficiently sync data from various sources to your chosen destinations. This guide provides an Airflow DAG (Directed Acyclic Graph) that orchestrates the OLake sync command, handling dependencies like persistent storage and configuration management within Kubernetes. +OLake Go is designed to efficiently sync data from various sources to your chosen destinations. This guide provides an Airflow DAG (Directed Acyclic Graph) that orchestrates the OLake Go sync command, handling dependencies like persistent storage and configuration management within Kubernetes. This post assumes you already have: @@ -47,7 +47,7 @@ Before deploying the DAG, ensure the following are in place: Crucially, your Airflow instance must be configured to use the `LocalExecutor` or `SequentialExecutor`. ::: -Our DAG utilizes Airflow's `PythonOperator` to dynamically check for and create Kubernetes resources (like PersistentVolumeClaims) before running the main OLake task. +Our DAG utilizes Airflow's `PythonOperator` to dynamically check for and create Kubernetes resources (like PersistentVolumeClaims) before running the main OLake Go task. This Python code runs directly on the machine where the Airflow scheduler is running, which is how `LocalExecutor` and `SequentialExecutor` operate. @@ -82,7 +82,7 @@ Choose or create a namespace in your Kubernetes cluster where OLake pods and rel ### Kubernetes ConfigMaps -The OLake DAG relies on Kubernetes ConfigMaps to inject configuration. You must create three ConfigMaps in the target namespace **before** running the DAG: +The OLake Go DAG relies on Kubernetes ConfigMaps to inject configuration. You must create three ConfigMaps in the target namespace **before** running the DAG: 1. `olake-source-config`: Containing your source configuration in a key named source.json. @@ -109,7 +109,7 @@ curl -Lo cm_olake-streams-config.yaml https://raw.githubusercontent.com/datazip- ``` :::info -This setup requires a `streams.json` generated beforehand using the OLake `discover` command against your source database. +This setup requires a `streams.json` generated beforehand using the OLake Go `discover` command against your source database. * Streams Generation Guides: * [Streams config](https://olake.io/docs/install/docker-cli#streams-config) * The content of this file will be placed within the `cm_olake-streams-config.yaml` ConfigMap. @@ -117,10 +117,10 @@ This setup requires a `streams.json` generated beforehand using the OLake `disco ## Kubernetes StorageClass -OLake uses a state file to keep track of sync progress. To persist this state between runs, the DAG creates a `PersistentVolumeClaim` (PVC). You need a `StorageClass` in your Kubernetes cluster that supports `ReadWriteMany` (RWX) access mode. +OLake Go uses a state file to keep track of sync progress. To persist this state between runs, the DAG creates a `PersistentVolumeClaim` (PVC). You need a `StorageClass` in your Kubernetes cluster that supports `ReadWriteMany` (RWX) access mode. :::note -At start of the sync, the state file just contains `{}`. OLake will update this file as it processes data. +At start of the sync, the state file just contains `{}`. OLake Go will update this file as it processes data. ::: **Examples**: `azurefile` on AKS, Google Cloud Filestore provisioner on GKE, AWS EFS provisioner on EKS, or an NFS provisioner. @@ -194,7 +194,7 @@ STREAMS_CONFIG_MAP_NAME = "olake-streams-config" 3. Airflow automatically scans this folder. Wait a minute or two, and the DAG named `olake_sync_from_source` should appear in the Airflow UI. You might need to unpause it (toggle button on the left) if it loads in a paused state. -## Running Your OLake Sync +## Running Your OLake Go Sync 1. Navigate to the Airflow UI. @@ -210,11 +210,11 @@ STREAMS_CONFIG_MAP_NAME = "olake-streams-config" 6. Once that succeeds, the `sync_data` task will start. This task launches the actual OLake pod on your Kubernetes cluster. -7. You can click on the `sync_data` task instance and view its logs to see the output from the OLake process itself. +7. You can click on the `sync_data` task instance and view its logs to see the output from the OLake Go process itself. -## How to set up OLake cron job scheduler +## How to set up OLake Go cron job scheduler Modify the line #52 from the `olake_sync_from_source.py` DAG file (`schedule=None`) with the frequency you wish to setup. @@ -224,13 +224,73 @@ with DAG( start_date=dag_start_date, schedule="@daily", # SAMPLE VALUE catchup=False, - ... + ...) ``` - For more information on how to add a schedule, refer [Cron & Time](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/cron.html#cron-time-intervals) Intervals docs. -That's it! You've successfully configured and run an OLake sync task orchestrated by your existing Airflow instance, leveraging the power of Kubernetes for execution. This setup provides a robust and automated way to manage your data synchronization pipelines. +That's it! You've successfully configured and run an OLake Go sync task orchestrated by your existing Airflow instance, leveraging the power of Kubernetes for execution. This setup provides a robust and automated way to manage your data synchronization pipelines. Happy Syncing! +## FAQs + +

    Create an Airflow DAG that uses the KubernetesPodOperator from the apache-airflow-providers-cncf-kubernetes package. The DAG defines the OLake Go sync task as a Kubernetes pod that:

    +
      +
    • Mounts OLake Go configuration from a ConfigMap
      • +
    • Uses a PersistentVolumeClaim for sync state storage across runs
      • +
    +

    Airflow's scheduler triggers the DAG on your defined schedule, dynamically creating OLake pods in the cluster for each sync run and cleaning them up on completion.

    + + }, + { + question: "Q2. What Airflow executor is required to run OLake Go's Kubernetes integration?", + answer:
    +

    This DAG requires the LocalExecutor or SequentialExecutor.

    +

    The reason is the DAG's first task, create_pvc, which uses Airflow's PythonOperator to check for and create Kubernetes resources (such as the PersistentVolumeClaim) before the sync runs. That Python code executes directly on the machine running the Airflow scheduler, which is how the LocalExecutor and SequentialExecutor operate. It will not work correctly out of the box with the CeleryExecutor or KubernetesExecutor without modification.

    +

    You can verify your setting under the [core] section of $AIRFLOW_HOME/airflow/airflow.cfg: it should read executor = LocalExecutor or executor = SequentialExecutor.

    +

    Note that this constraint comes from the DAG's PythonOperator step, not from the KubernetesPodOperator itself, which launches pods via the Kubernetes API regardless of executor.

    +
    +}, + { + question: "Q3. How does OLake Go manage state between Airflow-triggered sync runs on Kubernetes?", + answer:
    +

    OLake Go stores sync state - checkpoints, cursor positions, and schema snapshots - in a PersistentVolumeClaim mounted to each worker pod. The PVC is created once and reused across all sync runs.

    +

    The Airflow DAG includes a pre-task that checks whether the PVC already exists and creates it if not - this makes the setup idempotent and safe to re-run. If a sync fails mid-way, the next Airflow-triggered run resumes from the last checkpoint rather than starting a full reload.

    +
    + }, + { + question: "Q4. What Kubernetes resources does the OLake Go Airflow DAG create and manage?", + answer:
    +

    The DAG manages three types of Kubernetes resources:

    +
      +
    1. ConfigMap - Contains OLake Go's source and destination configuration files, mounted into the pod as a volume
      2. +
    2. PersistentVolumeClaim - Provides durable state storage for sync progress, created once and reused across runs
      3. +
    3. OLake worker pods - Dynamically created by KubernetesPodOperator for each sync execution. The operator passes config via volume mounts, runs the OLake Go sync command, and cleans up the pod on completion (is_delete_operator_pod=True), keeping the cluster tidy between runs
      4. +
    +
    + }, + { + question: "Q5. What are the prerequisites for running OLake Go with Airflow on Kubernetes?", + answer:
    +

    You need the following before setting up OLake Go with Airflow on Kubernetes:

    +
      +
    • Apache Airflow with the Kubernetes provider installed:
      • +
    + 
    {`pip install apache-airflow-providers-cncf-kubernetes`}
    +
      +
    • Kubernetes cluster access with kubectl configured and pointing to your target cluster
      • +
    • Airflow Kubernetes Connection - Create a connection of type Kubernetes Cluster Connection in Airflow's connection management UI, providing your kubeconfig in JSON format with the connection ID k8s_conn
      • +
    • StorageClass that supports PersistentVolumeClaims - Required for OLake Go state persistence between sync runs
      • +
    • OLake Go configuration files for your source database and destination (Iceberg catalog, S3 credentials, etc.)
      • +
    +
    + } +]} /> + + + + \ No newline at end of file diff --git a/blog/2025-05-07-what-makes-olake-fast.mdx b/blog/2025-05-07-what-makes-olake-fast.mdx index 3914069e2..da6d6ee00 100644 --- a/blog/2025-05-07-what-makes-olake-fast.mdx +++ b/blog/2025-05-07-what-makes-olake-fast.mdx @@ -12,18 +12,18 @@ tags: [olake] ![What makes OLake fast? Chunking strategies deep dive title with OLake logo on white background](/img/blog/cover/what-makes-olake-fast-cover.webp) -OLake is engineered for high-throughput ELT workloads, leveraging a combination of adaptive chunking strategies & parallelized execution for historical load, and change data capture (CDC) techniques to optimize data ingestion performance. +OLake Go is engineered for high-throughput ELT workloads, leveraging a combination of adaptive chunking strategies & parallelized execution for historical load, and change data capture (CDC) techniques to optimize data ingestion performance. -Its architectural emphasis on concurrency, intelligent partitioning, and efficient use of system resources enables it to handle large volumes of data with minimal latency. By aligning chunk generation with source tables distribution characteristics and executing data loads in parallel across multiple threads, OLake ensures optimal throughput while maintaining consistency and scalability. +Its architectural emphasis on concurrency, intelligent partitioning, and efficient use of system resources enables it to handle large volumes of data with minimal latency. By aligning chunk generation with source tables distribution characteristics and executing data loads in parallel across multiple threads, OLake Go ensures optimal throughput while maintaining consistency and scalability. To prevent overwhelming the source database, the maximum number of concurrent threads is configurable via the `source.json` file, enabling users to fine-tune parallelism based on workload sensitivity and infrastructure capacity. -## Types of Chunking strategies supported by olake +## Types of Chunking strategies supported by OLake Go -To maximize parallelism and throughput for full/historical load of tables, OLake splits table into manageable fragments called chunks. Each chunk can be processed independently by a dedicated worker thread, enabling concurrent extraction and transformation. Chunking is central to OLake's performance model and is tailored per source system to align with its data distribution patterns and indexing structures. +To maximize parallelism and throughput for full/historical load of tables, OLake Go splits table into manageable fragments called chunks. Each chunk can be processed independently by a dedicated worker thread, enabling concurrent extraction and transformation. Chunking is central to OLake Go's performance model and is tailored per source system to align with its data distribution patterns and indexing structures. -OLake supports multiple chunking strategies, each optimized for a particular database engine. The strategy selection is dynamic and driver-aware—for example: +OLake Go supports multiple chunking strategies, each optimized for a particular database engine. The strategy selection is dynamic and driver-aware—for example: ![Database splitting strategies by type: MongoDB (Split Vector, Bucket Auto, Timestamp), MySQL (Split via next query), Postgres (CTID Ranges, Split via batch size or next query)](/img/blog/2025/05/what-makes-olake-fast-1.webp) @@ -31,7 +31,7 @@ OLake supports multiple chunking strategies, each optimized for a particular dat ### 1. Split Vector Strategy -The Split Vector Strategy utilizes MongoDB's internal `splitVector` command to derive optimal chunk boundaries based on collection size and document distribution. This command is originally designed for MongoDB sharding, where it calculates split points to balance chunk sizes across shards. OLake repurposes this capability to partition data for parallel ingestion. +The Split Vector Strategy utilizes MongoDB's internal `splitVector` command to derive optimal chunk boundaries based on collection size and document distribution. This command is originally designed for MongoDB sharding, where it calculates split points to balance chunk sizes across shards. OLake Go repurposes this capability to partition data for parallel ingestion. In this strategy, chunks are generated such that each contains approximately a fixed amount of data by default, `1024` MB per chunk. The process determines the minimum and maximum _id values in the collection and then uses `splitVector` to compute intermediate `_id` boundaries that divide the collection into chunks of roughly equal size (in MB), while ensuring no document is split across chunks. @@ -50,7 +50,7 @@ db.adminCommand({ A key benefit of this approach is that chunking is driven by the actual data distribution, not just timestamp or primary key ranges. This improves balance across threads and avoids skewed workloads, especially in collections with irregular insert patterns. :::note -The `splitVector` command is a privileged internal operation and is not permitted on MongoDB Atlas or other managed environments with restricted permissions. In such cases, OLake gracefully falls back to the `bucketAutoStrategy`, ensuring compatibility without user intervention +The `splitVector` command is a privileged internal operation and is not permitted on MongoDB Atlas or other managed environments with restricted permissions. In such cases, OLake Go gracefully falls back to the `bucketAutoStrategy`, ensuring compatibility without user intervention ::: @@ -63,7 +63,7 @@ The [Bucket Auto Strategy](https://www.mongodb.com/docs/manual/reference/operato -OLake constructs an aggregation pipeline using `$sort` and `$bucketAuto`, where the number of buckets is set to `4` times the configured maximum number of threads. Each bucket returned by MongoDB contains a range (`min`, `max`) of `_id` values and a document count. These ranges are then used for parallel processing. +OLake Go constructs an aggregation pipeline using `$sort` and `$bucketAuto`, where the number of buckets is set to `4` times the configured maximum number of threads. Each bucket returned by MongoDB contains a range (`min`, `max`) of `_id` values and a document count. These ranges are then used for parallel processing. @@ -85,13 +85,13 @@ The Timestamp Strategy leverages the time-encoded nature of MongoDB’s `ObjectI #### How It Works #### 1. Determine Temporal Bounds: -The strategy begins by identifying the earliest and latest timestamps in the collection using the `_id` field’s embedded timestamp. Specifically, the first 4 bytes of an `ObjectId` represent the Unix time (in seconds) at which the document was created. This structure makes ObjectIds naturally increase over time, allowing them to be used as proxies for document creation time. By extracting the timestamp from the smallest and largest ObjectIds in the collection, OLake determines the temporal range of the dataset. +The strategy begins by identifying the earliest and latest timestamps in the collection using the `_id` field’s embedded timestamp. Specifically, the first 4 bytes of an `ObjectId` represent the Unix time (in seconds) at which the document was created. This structure makes ObjectIds naturally increase over time, allowing them to be used as proxies for document creation time. By extracting the timestamp from the smallest and largest ObjectIds in the collection, OLake Go determines the temporal range of the dataset. ![OLake batching and compression pipeline diagram reducing network and storage overhead](/img/blog/2025/05/what-makes-olake-fast-2.webp) #### 2. Define Time Granularity (Density): -OLake divides the total time range between the first and last record into 6 segments to estimate how data is distributed over time. Based on this, it calculates a dynamic chunk interval (`density`) by multiplying each segment by 10 seconds. This creates finer chunks if data is dense, and coarser chunks if it’s sparse — with a minimum granularity of 10 seconds per chunk. This approach helps balance the number of records in each chunk, even if the data insertion pattern is uneven. +OLake Go divides the total time range between the first and last record into 6 segments to estimate how data is distributed over time. Based on this, it calculates a dynamic chunk interval (`density`) by multiplying each segment by 10 seconds. This creates finer chunks if data is dense, and coarser chunks if it’s sparse — with a minimum granularity of 10 seconds per chunk. This approach helps balance the number of records in each chunk, even if the data insertion pattern is uneven. ```py timeDiff := last.Sub(first).Hours() / 6 @@ -143,7 +143,7 @@ The algorithm slides the window forward by the density until it spans the entire ## MySql -In MySQL, OLake chunks data based on the primary key of the source table. +In MySQL, OLake Go chunks data based on the primary key of the source table. ```sql SELECT MIN(id) AS min_value, MAX(id) AS max_value @@ -151,7 +151,7 @@ FROM .; ``` -To begin, OLake queries the minimum and maximum primary key values in the table. It then estimates the optimal chunk size by dividing the total row count by 8 times the number of threads configured in `source.json`. This `8×` multiplier ensures there are more chunks than threads, allowing for better parallelism and dynamic load balancing—especially useful in cases of uneven record sizes or skewed key distributions. +To begin, OLake Go queries the minimum and maximum primary key values in the table. It then estimates the optimal chunk size by dividing the total row count by 8 times the number of threads configured in `source.json`. This `8×` multiplier ensures there are more chunks than threads, allowing for better parallelism and dynamic load balancing—especially useful in cases of uneven record sizes or skewed key distributions. ```py func calculate_chunk_size(table_name, max_threads){ @@ -188,7 +188,7 @@ The `8×` multiplier used to calculate the chunk size is a design decision aimed While this multiplier is based on previous performance benchmarks, it is adjustable and may be fine-tuned through further testing to match specific workloads, optimizing throughput across different environments. ::: -Using this chunk size, OLake repeatedly executes a query to fetch the next chunk boundary using a sliding window over the primary key. +Using this chunk size, OLake Go repeatedly executes a query to fetch the next chunk boundary using a sliding window over the primary key. ```sql SELECT MAX(id) @@ -241,12 +241,12 @@ for{ ### 1. CTID Ranges -The `CTID` Ranges strategy is employed when a specific column for splitting the rows isn't defined. In such cases, OLake uses [`CTID`](https://www.postgresql.org/docs/current/ddl-system-columns.html#DDL-SYSTEM-COLUMNS-CTID), a system column in PostgreSQL that uniquely identifies rows within a table. This method partitions data based on relational page numbers, which represent blocks of rows in the underlying storage. +The `CTID` Ranges strategy is employed when a specific column for splitting the rows isn't defined. In such cases, OLake Go uses [`CTID`](https://www.postgresql.org/docs/current/ddl-system-columns.html#DDL-SYSTEM-COLUMNS-CTID), a system column in PostgreSQL that uniquely identifies rows within a table. This method partitions data based on relational page numbers, which represent blocks of rows in the underlying storage. Here’s how it works: #### 1. Calculate the Number of Pages: -OLake starts by querying the total number of pages in the table (using PostgreSQL's `relPages`), which essentially represents blocks of rows stored in the table. This is done through a specialized query that counts the number of pages. +OLake Go starts by querying the total number of pages in the table (using PostgreSQL's `relPages`), which essentially represents blocks of rows stored in the table. This is done through a specialized query that counts the number of pages. ```sql @@ -409,7 +409,72 @@ for { current_value = next_value } ``` - +## FAQs + +

    OLake Go dynamically selects chunking strategies per database engine:

    +
      +
    • MongoDB: Three strategies are available: +
        +
      • Split Vector: Calls MongoDB's internal splitVector command to compute chunk boundaries based on actual _id data distribution. Only works when all _id fields are ObjectIDs - collections with mixed or non-ObjectID _id types (UUIDs, integers, strings) will cause this strategy to fail and should use Bucket Auto instead
        • +
      • Bucket Auto: Uses MongoDB's $bucketAuto aggregation stage to divide a collection into balanced buckets. Used as a fallback when splitVector is unavailable (e.g. MongoDB Atlas) or when _id fields are non-ObjectID types
        • +
      • Timestamp-based: Generates chunk boundaries from time ranges derived from the _id field's embedded timestamp
        • +
      +
      • +
    • PostgreSQL - Uses CTID range chunking based on physical storage page ranges, plus batch-size splits and next-query paging
      • +
    • MySQL - Uses primary key range splitting
      • +
    +

    Each strategy ensures chunks are balanced to avoid skewed workloads across threads.

    + + }, + { + question: "Q2. How does OLake Go's Split Vector strategy work for MongoDB parallel loading?", + answer:
    +

    The Split Vector strategy calls MongoDB's internal splitVector command on the target collection, specifying a maximum chunk size (default 1024MB):

    + 
    {`db.adminCommand({
+  splitVector: "your_collection",
+  keyPattern: { "_id": 1 },
+  maxChunkSize: 1024
+})`}
    +

    MongoDB calculates split points based on the collection's actual data distribution and _id values, returning balanced chunk boundaries. OLake Go uses these boundaries to create independent read ranges, each processed by a separate worker thread simultaneously.

    +

    This data-distribution-aware approach avoids the skew that occurs with simple range or time-based splitting.

    +

    Limitation: Split Vector assumes all documents have ObjectID _id fields. Collections where _id fields are strings, integers, or UUIDs will cause this strategy to fail - use Bucket Auto for those collections instead.

    +
    + }, + { + question: "Q3. What is the Bucket Auto chunking strategy in OLake Go and when is it used?", + answer:
    +

    Bucket Auto uses MongoDB's $bucketAuto aggregation stage to automatically divide a collection into a specified number of equal-sized buckets based on document distribution.

    +

    OLake Go uses Bucket Auto in two scenarios:

    +
      +
    1. When splitVector is unavailable: for example, on MongoDB Atlas, which restricts the splitVector command due to its privileged admin access requirement
      2. +
    2. When _id fields are non-ObjectID types: collections with UUID, integer, or string _id fields are incompatible with Split Vector and should use Bucket Auto instead
      3. +
    +

    Bucket Auto provides similar balanced chunking without requiring privileged admin access, using the standard aggregation framework.

    +
    + }, + { + question: "Q4. How does OLake Go's parallel execution model prevent overwhelming the source database?", + answer:
    +

    OLake Go allows configuring the maximum number of concurrent threads via the max_threads setting in source.json. This caps how many parallel reads hit the source database simultaneously, preventing CPU, I/O, and connection pool saturation.

    +

    For sensitive production databases, teams can set a conservative thread count. The chunking ensures each thread reads a non-overlapping portion of the table, so there is no contention between threads. OLake Go caps active readers and writers accordingly to prevent CPU, memory, or network oversubscription.

    +
    + }, + { + question: "Q5. What makes OLake Go's CDC sync more efficient than batch replication for ongoing data pipelines?", + answer:
    +

    After the initial full load, OLake Go switches to CDC mode which reads only the database change log:

    +
      +
    • PostgreSQL: WAL (Write-Ahead Log) via pgoutput
      • +
    • MySQL: Binary log (binlog)
      • +
    • MongoDB: Oplog or change streams
      • +
    +

    Instead of re-scanning the entire table on every sync, OLake Go captures only the rows that changed since the last sync - typically orders of magnitude smaller than the full table. CDC also provides near-real-time latency (seconds instead of hours) and consumes minimal source database resources since it reads from the log, not the primary tables.

    +
    + } +]} /> \ No newline at end of file diff --git a/blog/2025-05-08-olake-airflow-on-ec2.mdx b/blog/2025-05-08-olake-airflow-on-ec2.mdx index d2025fdf3..22740f328 100644 --- a/blog/2025-05-08-olake-airflow-on-ec2.mdx +++ b/blog/2025-05-08-olake-airflow-on-ec2.mdx @@ -12,9 +12,9 @@ tags: [olake] ![Apache Airflow on AWS EC2 workflow integration with OLake platform logo](/img/blog/cover/olake-airflow-on-ec2-cover.webp) -At OLake, we're building tools to make data integration seamless. Today, we're excited to show you how to leverage your existing Apache Airflow setup to automate OLake data synchronization tasks directly on your EC2 Server! +At OLake Go, we're building tools to make data integration seamless. Today, we're excited to show you how to leverage your existing Apache Airflow setup to automate OLake Go data synchronization tasks directly on your EC2 Server! -Olake is designed to efficiently sync data from various sources to your chosen destinations. This guide provides an Airflow DAG (Directed Acyclic Graph) that orchestrates the Olake sync command by provisioning a dedicated EC2 instance, executing Olake within a Docker container and handling configuration and state persistence through Amazon S3. +Olake Go is designed to efficiently sync data from various sources to your chosen destinations. This guide provides an Airflow DAG (Directed Acyclic Graph) that orchestrates the Olake Go sync command by provisioning a dedicated EC2 instance, executing Olake Go within a Docker container and handling configuration and state persistence through Amazon S3. This post assumes you already have: @@ -132,7 +132,7 @@ Before deploying the DAG, ensure the following are in place: ``` -* **SSH Connection (`SSH_CONNECTION_ID` in the DAG):** This connection allows Airflow to securely connect to the dynamically created EC2 instance to execute the Olake setup and run commands. +* **SSH Connection (`SSH_CONNECTION_ID` in the DAG):** This connection allows Airflow to securely connect to the dynamically created EC2 instance to execute the Olake Go setup and run commands. * Still in the Airflow UI (`Admin` -> `Connections`), click the `+` icon to add another new record. * Set the **Connection Type** to **SSH**. * Enter a **Connection Id** (e.g., `ssh_ec2_olake`). This exact ID will be used for the `SSH_CONNECTION_ID` variable in your DAG. @@ -172,22 +172,22 @@ Before deploying the DAG, ensure the following are in place: -#### 3. **Amazon S3 Setup for Olake Configurations and State:** -* **S3 Bucket (`S3_BUCKET_NAME` in the DAG):** Create an S3 bucket where Olake's configuration files and persistent state file will be stored. -* **S3 Prefix for Configurations (`S3_PREFIX` in the DAG):** Decide on a "folder" (S3 prefix) within your bucket where your Olake configuration files will reside (e.g., `olake/projectA/configs/`). +#### 3. **Amazon S3 Setup for Olake Go Configurations and State:** +* **S3 Bucket (`S3_BUCKET_NAME` in the DAG):** Create an S3 bucket where Olake Go's configuration files and persistent state file will be stored. +* **S3 Prefix for Configurations (`S3_PREFIX` in the DAG):** Decide on a "folder" (S3 prefix) within your bucket where your Olake Go configuration files will reside (e.g., `olake/projectA/configs/`). -* **Upload Olake Configuration Files:** Before running the DAG, you must upload your Olake `source.json`, `streams.json`, and `destination.json` files to the S3 bucket under the prefix you defined. The DAG's SSH script will sync these files to the EC2 instance. Please visit[ OLake Docs](https://olake.io/docs) website to learn how the[ source](https://olake.io/docs/connectors/overview) and[ destinations](https://olake.io/docs/writers/overview) can be set up. +* **Upload Olake Go Configuration Files:** Before running the DAG, you must upload your Olake Go `source.json`, `streams.json`, and `destination.json` files to the S3 bucket under the prefix you defined. The DAG's SSH script will sync these files to the EC2 instance. Please visit[ OLake Docs](https://olake.io/docs) website to learn how the[ source](https://olake.io/docs/connectors/overview) and[ destinations](https://olake.io/docs/writers/overview) can be set up. -We need to generate `streams.json` beforehand using the OLake `discover` command against your source database. +We need to generate `streams.json` beforehand using the OLake Go `discover` command against your source database. * Streams Generation Guides: * [Streams config](https://olake.io/docs/install/docker-cli#streams-config) * The content of this file will be placed within the `streams.json` file. #### 4. **EC2 Instance IAM Role (`IAM_ROLE_NAME` in the DAG):** -The EC2 instances launched by Airflow (which will act as the worker nodes for Olake) need their own set of permissions to perform their tasks. This is achieved by assigning them an IAM Instance Profile. This instance profile must have an attached IAM policy granting permissions to: -* Access Amazon S3 to download Olake configuration files. -* Access Amazon S3 to read and write the Olake state file. +The EC2 instances launched by Airflow (which will act as the worker nodes for Olake Go) need their own set of permissions to perform their tasks. This is achieved by assigning them an IAM Instance Profile. This instance profile must have an attached IAM policy granting permissions to: +* Access Amazon S3 to download Olake Go configuration files. +* Access Amazon S3 to read and write the Olake Go state file. ```json # s3_access_policy.json @@ -290,7 +290,7 @@ OLAKE_IMAGE = "DOCKER_IMAGE_NAME" ## Recap of Values to Change: -To ensure the DAG runs correctly in your environment, you **must** update the following placeholder variables in the `olake_sync_from_source_ec2.py` (or your DAG file name) with your specific AWS and Olake details: +To ensure the DAG runs correctly in your environment, you **must** update the following placeholder variables in the `olake_sync_from_source_ec2.py` (or your DAG file name) with your specific AWS and Olake Go details: @@ -303,7 +303,7 @@ To ensure the DAG runs correctly in your environment, you **must** update the fo ### **EC2 Instance Configuration:** * `AMI_ID`: Replace with the actual AMI ID of a container-ready image (with Docker/containerd, aws-cli, jq) in your chosen `AWS_REGION_NAME`. -* `INSTANCE_TYPE`: (Optional) Select an appropriate EC2 instance type based on your Olake workload's resource needs (e.g., `t3.medium`, `m5.large`, or an ARM equivalent like `t4g.medium`). \ +* `INSTANCE_TYPE`: (Optional) Select an appropriate EC2 instance type based on your Olake Go workload's resource needs (e.g., `t3.medium`, `m5.large`, or an ARM equivalent like `t4g.medium`). \ The AMI tag we have hardcoded is EKS supported Ubuntu image with containerd and aws-cli pre-installed which are very crucial for the DAG to work. Another point to note is that since Graviton powered machines are cheaper compared to x86 machines, so the AMI already uses ARM architecture AMI. * `KEY_NAME`: Enter the name of the EC2 Key Pair you want to associate with the launched instances. This is the same key we have used while setting up the SSH Connection. * `SUBNET_ID`: Provide the ID of the VPC subnet where the EC2 instance should be launched. @@ -311,10 +311,10 @@ The AMI tag we have hardcoded is EKS supported Ubuntu image with containerd and * `IAM_ROLE_NAME`: Enter the **name** (not the ARN) of the IAM Instance Profile that grants the EC2 instance necessary permissions (primarily S3 access). * `DEFAULT_EC2_USER`: Change this if the default SSH username for your chosen `AMI_ID` is different from `ubuntu` (e.g., `ec2-user` for Amazon Linux). -### **ETL Configuration (S3 & Olake):** +### **ETL Configuration (S3 & Olake Go):** -* `S3_BUCKET_NAME`: The name of your S3 bucket where Olake configurations and state will be stored. -* `S3_BUCKET_PREFIX`: The "folder" path (prefix) within your S3 bucket for Olake files (e.g., `olake/projectA/configs/`). Remember the trailing slash if it's part of your intended structure. +* `S3_BUCKET_NAME`: The name of your S3 bucket where Olake Go configurations and state will be stored. +* `S3_BUCKET_PREFIX`: The "folder" path (prefix) within your S3 bucket for Olake Go files (e.g., `olake/projectA/configs/`). Remember the trailing slash if it's part of your intended structure. * `OLAKE_IMAGE`: The full name of the Olake Docker image you want to use (e.g., `olakego/source-postgres:latest`, `olakego/source-mysql:latest`, `olakego/source-mongodb:latest`). ### Deploying the DAG to Airflow @@ -325,7 +325,7 @@ The AMI tag we have hardcoded is EKS supported Ubuntu image with containerd and 2. Place the file into the `dags` folder recognized by your Airflow instance. The location of this folder depends on your Airflow setup. 3. Airflow automatically scans this folder. Wait a minute or two, and the DAG named `olake_sync_from_source` should appear in the Airflow UI. You might need to unpause it (toggle button on the left) if it loads in a paused state. -### Running Your Dynamic Olake Sync on EC2 +### Running Your Dynamic Olake Go Sync on EC2 1. **Access Airflow UI:** Navigate to your Airflow web UI. 2. **Find and Unpause DAG:** Locate the DAG, likely named `olake_sync_from_source` (or whatever `dag_id` you've set). If it's paused, click the toggle to unpause it. @@ -333,21 +333,21 @@ The AMI tag we have hardcoded is EKS supported Ubuntu image with containerd and 4. **Monitor the Run:** Click on the DAG run instance to view its progress in the Graph, Gantt, or Tree view. You will see the following sequence of tasks: * `create_ec2_instance_task`: This task will begin first, using the AWS connection to launch a new EC2 instance according to your DAG's configuration (AMI, instance type, networking, IAM role). Airflow will wait for this instance to be in a 'running' state. * `get_instance_ip_task`: Once the instance is running, this Python task will execute. It queries AWS to get the IP address or DNS name of the new EC2 instance, making it available for the next task. It also includes a pause to allow the SSH service on the new instance to become fully available. - * `run_olake_docker_task`: This is the core task where Olake runs. It will: + * `run_olake_docker_task`: This is the core task where Olake Go runs. It will: * Connect to the newly created EC2 instance via SSH using the configured SSH connection. * Execute the shell commands defined in `olake_ssh_command` within your DAG. This script prepares the EC2 instance by: * Creating necessary directories. - * Downloading your Olake configuration files and the latest state file from S3. + * Downloading your Olake Go configuration files and the latest state file from S3. * Pulling the specified Olake Docker image using `ctr image pull`. - * Running the Olake `sync` process inside a Docker container using `ctr run ... /home/olake sync ...`. + * Running the Olake Go `sync` process inside a Docker container using `ctr run ... /home/olake sync ...`. * Uploading the updated state file back to S3 upon successful completion. - * You can click on this task instance in the Airflow UI and view its logs. These logs will contain the **real-time STDOUT and STDERR** from the SSH session on the EC2 instance, including the output from the Olake Docker container. This is where you'll see Olake's synchronization progress and any potential errors from the Olake process itself. + * You can click on this task instance in the Airflow UI and view its logs. These logs will contain the **real-time STDOUT and STDERR** from the SSH session on the EC2 instance, including the output from the Olake Docker container. This is where you'll see Olake Go's synchronization progress and any potential errors from the Olake Go process itself. * `terminate_ec2_instance_task`: After the `run_olake_docker_task` completes (whether it succeeds or fails, due to `trigger_rule=TriggerRule.ALL_DONE`), this final task will execute. It securely terminates the EC2 instance that was launched for this DAG run, ensuring you don't incur unnecessary AWS charges. ![Apache Airflow DAG olake_sync_ec2 graph view showing EC2 instance creation, IP retrieval, OLake Docker task, and termination steps, all completed successfully](/img/blog/2025/05/olake-airflow-on-ec2-3.webp) -### How to set up OLake cron job scheduler +### How to set up OLake Go cron job scheduler Modify the line #52 from the `olake_sync_from_source.py` DAG file (`schedule=None`) with the frequency you wish to setup. @@ -359,14 +359,81 @@ with DAG( catchup=False, ... ``` - - For more information on how to add a schedule, refer[ Cron & Time](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/cron.html#cron-time-intervals) Intervals docs. -That's it! You've successfully configured and run an OLake sync task orchestrated by your existing Airflow instance, leveraging the power of Kubernetes for execution. This setup provides a robust and automated way to manage your data synchronization pipelines. +That's it! You've successfully configured and run an OLake Go sync task orchestrated by your existing Airflow instance, leveraging the power of Kubernetes for execution. This setup provides a robust and automated way to manage your data synchronization pipelines. Happy Syncing! +## FAQs + + +

    Create an Airflow DAG that uses the AWS provider to dynamically provision an EC2 instance, SSH into it, and run OLake Go inside a Docker container. The DAG handles the full EC2 lifecycle:

    +
      +
    1. Launch - EC2CreateInstanceOperator provisions the instance
      2. +
    2. Wait - EC2InstanceStateSensor polls until the instance is ready
      3. +
    3. Execute - SSHOperator SSHes into the instance and runs the OLake Go sync command
      4. +
    4. Terminate - EC2TerminateInstanceOperator shuts down the instance on completion
      5. +
    +

    OLake Go configuration files are stored in S3 and downloaded to the EC2 instance at runtime.

    + + }, + { + question: "Q2. What AWS IAM permissions are required to run the OLake Go Airflow EC2 DAG?", + answer:
    +

    The Airflow AWS connection needs IAM permissions to:

    +
      +
    • Create and terminate EC2 instances - ec2:RunInstances, ec2:TerminateInstances
      • +
    • Describe EC2 instance status - ec2:DescribeInstances, ec2:DescribeInstanceStatus
      • +
    • Pass an IAM role to the EC2 instance - iam:PassRole (required security check at launch time to verify the caller is allowed to associate a specific role with the new instance)
      • +
    • Retrieve the role being passed - iam:GetRole (recommended best practice alongside iam:PassRole)
      • +
    • List instance profiles - iam:ListInstanceProfiles (required when using the EC2 console to assign instance profiles)
      • +
    +

    The EC2 instance itself needs an IAM role with permissions to:

    +
      +
    • Read OLake Go configuration files from the S3 config bucket
      • +
    • Read and write OLake Go state files to the S3 state path
      • +
    • Write Iceberg data files to the destination S3 bucket
      • +
    +

    Using IAM roles attached to instances is preferred over embedding access keys in configuration files.

    +
    + }, + { + question: "Q3. How does OLake Go use S3 for state persistence when running on ephemeral EC2 instances?", + answer:
    +

    Since EC2 instances are terminated after each sync run, OLake Go's state (checkpoints, cursor positions) cannot be stored on the instance's local disk. Instead, the DAG configures OLake Go to read and write its state files to a dedicated S3 path:

    +
      +
    • At the start of each run - state is downloaded from S3 to the instance
      • +
    • At the end of a successful sync - the updated state is uploaded back to S3
      • +
    +

    This enables incremental sync across stateless ephemeral instances - each run picks up exactly where the last one left off without needing a persistent server.

    +
    + }, + { + question: "Q4. What is the benefit of dynamically provisioning EC2 instances for each OLake Go sync run?", + answer:
    +

    Dynamic provisioning means you only pay for compute when syncs are actually running - the instance is terminated immediately after each job completes. Key benefits:

    +
      +
    • Cost efficiency - Dramatically more cost-effective than running a persistent server for low-frequency syncs (e.g. hourly or daily). You are billed only for the minutes the sync runs.
      • +
    • Clean state - Each sync starts on a fresh instance with no risk of accumulated state corruption from previous runs
      • +
    • Right-sizing - You can choose the optimal EC2 instance type and size for each pipeline independently, rather than over-provisioning a shared persistent server
      • +
    +
    + }, + { + question: "Q5. How do I configure Airflow connections for AWS and SSH to enable the OLake Go EC2 DAG?", + answer:
    +

    AWS connection: In the Airflow UI, create an Amazon Web Services connection with your AWS credentials, or configure IAM role-based authentication if your Airflow instance runs on AWS (MWAA or EC2) and has an attached IAM role.

    +

    SSH connection: The DAG dynamically creates a temporary SSH connection at runtime using the EC2 instance's public IP and your key pair, retrieved from the AWS API response after launch. The private key file should be stored in the Airflow DAGs directory on S3.

    +

    Security note: Dynamic SSH connections to freshly provisioned EC2 instances cannot verify host keys, which produces a warning: "No Host Key Verification. This won't protect against Man-In-The-Middle attacks." For production deployments, consider using AWS Systems Manager Session Manager as an alternative to direct SSH - this avoids exposing port 22 entirely and uses IAM-based access control instead of key pairs.

    +
    + } +]} /> + + \ No newline at end of file diff --git a/blog/2025-07-29-enhancing-data-ingestion-with-filter-feature.mdx b/blog/2025-07-29-enhancing-data-ingestion-with-filter-feature.mdx index 6e43890be..a3d151119 100644 --- a/blog/2025-07-29-enhancing-data-ingestion-with-filter-feature.mdx +++ b/blog/2025-07-29-enhancing-data-ingestion-with-filter-feature.mdx @@ -1,7 +1,7 @@ --- slug: olake-ingestion-filters-explained -title: "OLake Ingestion Filters: Smart SQL-Style Data Filtering Guide" -description: "Learn how OLake's ingestion filters optimize data pipelines with SQL-style WHERE clauses for Postgres, MySQL, and MongoDB for efficient ingestion." +title: "OLake Go Ingestion Filters: Smart SQL-Style Data Filtering Guide" +description: "Learn how OLake Go's ingestion filters optimize data pipelines with SQL-style WHERE clauses for Postgres, MySQL, and MongoDB for efficient ingestion." image: /img/blog/cover/filter-ingestion-cover.webp authors: [duke] tags: [olake, data-ingestion] @@ -146,7 +146,7 @@ In data ingestion, large datasets are split into smaller chunks for efficient pr 2. **Chunk Processing**: Fetching and processing the data within each chunk. -For a comprehensive understanding of how OLake's chunking strategies work across different databases, read our detailed explanation [here](https://olake.io/blog/what-makes-olake-fast#types-of-chunking-strategies-supported-by-olake). This section covers all the various chunking approaches we use for MongoDB, MySQL, and PostgreSQL. +For a comprehensive understanding of how OLake Go's chunking strategies work across different databases, read our detailed explanation [here](https://olake.io/blog/what-makes-olake-fast#types-of-chunking-strategies-supported-by-olake). This section covers all the various chunking approaches we use for MongoDB, MySQL, and PostgreSQL. ### Why Apply Filters in Both Stages? @@ -281,8 +281,92 @@ The reality is that data volumes will continue to grow, and the cost of processi Our filter feature significantly enhances our data ingestion system by enabling selective data processing, improving efficiency, and optimizing resource use. Its robust implementation across Postgres, MySQL, and MongoDB ensures flexibility while addressing edge cases through strategic application during both chunk generation and processing. Although limitations exist in certain chunking strategies, the system gracefully adapts by applying filters during processing, ensuring data relevance and pipeline performance. + Whether you're just starting your career in data engineering or you're a seasoned professional optimizing complex pipelines, effective filtering is one of those foundational skills that pays dividends across every project you work on. **Start filtering smarter, and watch your pipelines become faster, cheaper, and more reliable.** - +## FAQs + +

    OLake Go's filter feature applies SQL-style WHERE conditions at the source database level before data is extracted, so only matching rows are transferred and written to the lakehouse. This reduces the volume of data ingested, cutting:

    +
      +
    • Storage costs. Fewer files written to object storage.
      • +
    • Network transfer fees. Less data moved from source to destination.
      • +
    • Downstream processing time. Smaller datasets for transformation and querying.
      • +
    +

    The size of the improvement depends on how selective your filter is: the more irrelevant rows you exclude at the source, the larger the reduction in ingestion time and cost.

    +

    Note: Filtering requires Normalization to be enabled on the stream ("normalization": true in streams.json). Without it, the filter is not applied.

    + + }, + { + question: "Q2. What SQL-style filter syntax does OLake Go support?", + answer:
    +

    OLake Go supports filter strings in the format column operator value with standard comparison operators:

    + + + + + + + + + + + + + + + +
    OperatorMeaning
    =Equal
    !=Not equal
    >Greater than
    <Less than
    >=Greater than or equal
    <=Less than or equal
    +

    Multiple conditions can be combined with AND or OR logical operators (see FAQ 4 for limits).

    +

    Examples:

    +
      +
    • Single numeric condition: age > 18
      • +
    • Single string condition: country = "USA"
      • +
    • Two conditions combined: age > 18 and country = "USA"
      • +
    +

    String quoting: String values in OLake Go filter expressions must use double quotes (e.g. country = "USA"), not single quotes. The driver automatically translates these to the correct quote style for each target database: single quotes for PostgreSQL, backtick-quoted columns for MySQL.

    +

    The filter is specified per stream in the streams.json configuration, alongside "normalization": true, which must be enabled for the filter to take effect.

    +
    + }, + { + question: "Q3. How are OLake Go filters applied differently for SQL databases versus MongoDB?", + answer:
    +

    PostgreSQL and MySQL - OLake Go translates the filter string into a SQL WHERE clause appended to the SELECT query, letting the database engine apply the filter efficiently using its indexes:

    +
      +
    • PostgreSQL: "age" > 18 AND "country" = 'USA'
      • +
    • MySQL: `age` > 18 AND `country` = 'USA'
      • +
    +

    MongoDB - The filter is converted into a MongoDB query document (BSON filter) passed to the find() operation, allowing MongoDB to use its indexes for efficient filtering.

    +

    Both approaches push filtering to the source engine for maximum efficiency, and both require "normalization": true on the stream for the filter to be applied.

    +

    Oracle exception: Oracle uses DBMS_PARALLEL_EXECUTE.CREATE_CHUNKS_BY_ROWID for chunking, which does not accept user-defined filters. As a result, filter conditions cannot influence chunk boundary generation for Oracle - filters are applied only during chunk processing, not chunk generation. This makes Oracle filtering less efficient than PostgreSQL, MySQL, or MongoDB filtering.

    +
    + }, + { + question: "Q4. Can I combine multiple filter conditions using AND and OR in OLake Go?", + answer:
    +

    Yes, with one important limit: OLake Go currently supports a maximum of two conditions per filter string, combined with a single logical operator (AND or OR).

    +

    Valid examples:

    +
      +
    • age > 18 AND country = "USA"
      • +
    • status = "active" OR country = "USA"
      • +
    +

    Three-condition expressions (e.g. age > 18 AND country = "USA" OR status = "active") are not valid OLake Go filter syntax - the parser accepts exactly one logical operator joining exactly two conditions. Expressions with more than two conditions will be rejected or produce undefined behavior.

    +

    The filter parser uses a regular expression to extract both conditions and the logical operator, then generates the appropriate SQL WHERE clause or MongoDB query filter for the target database. As with all filtering, "normalization": true must be set on the stream for this to run.

    +
    + }, + { + question: "Q5. Does source-level filtering in OLake Go work during both full refresh and CDC sync modes?", + answer:
    +

    Yes. Filters are applied during both sync modes, provided "normalization": true is set on the stream, filtering does not run otherwise:

    +
      +
    • Full refresh - Only rows matching the filter are read from the source during the initial historical snapshot. The filter is applied at both chunk generation and chunk processing stages, so non-matching rows are never read.
      • +
    • CDC incremental sync - OLake Go reads change events from the database log (WAL for PostgreSQL, binlog for MySQL, oplog for MongoDB) and applies the filter before writing to the destination. Change events for filtered-out rows are discarded without consuming storage or processing resources.
      • +
    +

    This means the filter is consistently enforced end-to-end across both the initial load and all ongoing incremental updates.

    +
    + }, +]} /> \ No newline at end of file diff --git a/blog/2025-07-29-next-gen-lakehouse.mdx b/blog/2025-07-29-next-gen-lakehouse.mdx index e220f7975..4586ec1c9 100644 --- a/blog/2025-07-29-next-gen-lakehouse.mdx +++ b/blog/2025-07-29-next-gen-lakehouse.mdx @@ -1,381 +1,484 @@ ---- - -slug: building-modern-data-lakehouse-with-olake-iceberg-lakekeeper-trino -title: Building Modern Lakehouse with Iceberg, OLake, Lakekeeper & Trino -description: 'Iceberg is the storage "brain," OLake is the real-time "pipeline," and Trino is the fast "question-answering" engine. Together they turn raw object-storage files into a governed, low-latency analytics platform.' -image: /img/blog/cover/next-gen-lakehouse-cover.webp -authors: [akshay] -tags: [olake, iceberg] ---- - -![The Next Gen Lakehouse: Iceberg, OLake, Lakekeeper, and Trino integration logos on a white background](/img/blog/cover/next-gen-lakehouse-cover.webp) - -# The Next-Gen Lakehouse — Iceberg, OLake, Lakekeeper and Trino - -Quick on that here : -**Iceberg** is the storage "brain", **OLake** is the real-time "pipeline", and **Trino** is the fast "question-answering" engine. - -Together they turn raw object-storage files into a governed, low-latency analytics platform. -Data engineering nowadays has swung toward the **lakehouse paradigm** where we are mixing data lake scale with warehouse reliability and this blog walks through a hands-on stack of open-source tools that make a Lakehouse work. - -We'll see how Apache Iceberg (an open table format), OLake (fast DB→lake loader), Lakekeeper (Iceberg's REST catalog), and Trino (distributed SQL engine) fit together. Think of it as a friendly tour of each component (what it is, why it matters and most importantly how it all connects) - -## What Is Apache Iceberg? - -Well to understand it a bit simpler you can picture a giant library where every book (data file) sits on random floors with no catalog. Finding the Harry Potter series would be a chaos. Iceberg acts as the modern **Dewey Decimal system**: it tags every file with rich metadata, tracks versions, and lets you time-travel to yesterday's shelves all without moving the books themselves . Thus, now you can read books and data engineers can query data better . - -**Well what's the value of it ?** - - -Firstly , you get database-style "all-or-nothing" writes on cheap object storage, e.g. S3, GCS, Azure blobs or Minio. You can add, rename, or drop columns and partitions without rewriting terabytes. - - -Query your data "as of" last Tuesday for audits or bug fixes. - - -**Hidden Partitioning** — Iceberg tracks which files hold which dates, so queries auto-skip irrelevant chunks, no brittle dt='2025-07-21' filters required. - - -Most importantly it is **engine-agnostic** you might have heard this term a lot and here it gets a meaning iceberg supports [Spark](/iceberg/query-engine/spark), [Trino](/iceberg/query-engine/trino), Flink, [DuckDB](/iceberg/query-engine/duckdb), Dremio, and Snowflake all speak to the tables natively - - -Before Iceberg, data lakes were basically digital junkyards. You'd dump data files into cloud storage (like Amazon S3), and finding anything useful was like looking for a specific needle in a haystack . - -## Understanding Iceberg Catalogs: The Foundation Layer - -Think of the catalog as the librarian of your data lakehouse it keeps track of which tables exist, where they are, and which version is current. - -### The Catalog Challenge in Iceberg - -Every time you make changes to an Iceberg table whether adding rows, updating schemas, or modifying partitions Iceberg creates a new metadata.json file (like v1.metadata.json, v2.metadata.json). Over time, you accumulate hundreds of these files. - -The critical question becomes: how do query engines like Trino, Spark, or DuckDB know **which metadata file represents the "current" state of the table?** - -This is where the catalog becomes essential. The catalog serves two fundamental purposes: - - -1. Maintains a registry of existing Iceberg tables like a phone book for your data assets - -2. Tracks pointers to the current metadata.json file ensuring everyone sees the same version of truth - - -There are different type of data catalogs file based and service based and today, we're taking a closer look at Lakekeeper a type of data catalog that resolves these challenges so we will cover what it is, how it stands out, and exactly what category of catalog it belongs to for Apache Iceberg. - -## What is Lakekeeper? - -#### (Empowering Object Storage with Iceberg and Enterprise-Grade Governance) - -Lakekeeper turns ordinary object storage (like S3) into a fully governed Apache Iceberg lakehouse. Out of the box, S3 is just a bunch of files with basic access controls. Lakekeeper manages all the Iceberg metadata (schemas, snapshots, pointers) so your storage now supports transactions, time travel(going back at a particular commit ) and consistent views across all engines (Trino, Spark, DuckDB, you name it). - -What really makes Lakekeeper stand out is how it hooks into your organization's identity, access control, and policy tools (like **OPA** and **OpenFGA**). This means you get table, column, even row-level permissions and governance applied no matter who's querying or how they access the data. - -**In short**: By combining open table formats (Iceberg) and centralized, policy-driven control, Lakekeeper upgrades your object store from "just a bucket" to a secure, compliant, analytics-ready platform. That's the key to making S3 truly enterprise and audit—friendly. - -### Why Lakekeeper Isn't Just Another Catalog - -It fully implements the Apache Iceberg REST Catalog API, making it compatible with any query engine that supports the Iceberg standard REST interface. - - -Written in Rust, it's lightning-fast and extremely lightweight; just one binary to deploy, but strong enough for enterprise-grade operations - -### How it suits modern pipelines? - -What really sets Lakekeeper apart is how it bridges the gap between raw object storage and truly managed, governed Iceberg tables. Most basic catalogs just keep a list of where the data is, but Lakekeeper acts as a smart, active coordinator that actually turns cloud buckets into a secure, production-ready analytical layer. - -Instead of relying on the old approach where you had to manage clunky file-based pointers or deal with slow, legacy services, Lakekeeper brings the agility, real-time response, and open standards you'd expect from a modern data stack. - -## What is OLake? - -Now that you understand how Apache Iceberg organizes your data lake into a well-structured, reliable system, you're probably wondering: **"That's great, but how do I actually get my operational data INTO Iceberg format?"** - -This is where the rubber meets the road in building a modern lakehouse. - -Traditional approaches involve complex setups with tools like **Debezium + Kafka**, or using generic ETL platforms that weren't designed for the specific demands of lakehouse architectures. These solutions often struggle with scale, reliability, and the unique requirements of Apache Iceberg. That's exactly the **gap** that OLake was built to fill it's like having a specialized highway that moves data from your **operational databases** directly into the **Iceberg ecosystem**, fast and reliably. - -**Still confused ?** -Think of it this way: if your operational database is like a busy restaurant kitchen constantly taking orders and making food, OLake is like having a super-efficient catering service that continuously packages up perfect copies of those meals and delivers them to your analytics "dining hall" (the data lakehouse) without disrupting the kitchen operations. - -### How Lakekeeper and OLake Work Together for Ingestion - -#### (And Why It's a Game-Changer for Real-Time, Production-Grade Data Stacks) - -So you've set up your data lakehouse and you want fresh, reliable data to land there fast, secure, and always discoverable by everyone who needs it. That's where the tag team of **OLake** (for high-speed database replication) and **Lakekeeper** (as the brains of your Iceberg metadata) steps in. Let's break down exactly how these two connect, and what happens under the hood when new data is ingested. - -### The Big Picture Flow - -OLake handles pulling change data from your databases (like MongoDB, PostgreSQL, MySQL), breaks it into efficient chunks, and writes it directly into Apache Iceberg tables. But it needs to coordinate all metadata table versions, schema, file locations to make sure what it writes is instantly queryable, consistent, and secure. - -Lakekeeper is the REST catalog that acts as the "metadata authority" where OLake checks what tables exist it is the place OLake registers new data and schema changes while also acting as the guardrail for access, audit, and governance - -**In short**: OLake does the heavy lifting, but Lakekeeper makes sure every lift is recorded, governed, and easy to discover in real-time by analytical engines like **Trino**. - -![OLake and Lakekeeper ingesting data from MongoDB, PostgreSQL, and MySQL into Apache Iceberg data lakehouse architecture](/img/blog/2025/07/olake-lakekeeper.webp) - -## What is trino? : Distributed SQL on Everything -![Trino logo featuring a cute astronaut bunny icon with black 'trino' text](/img/blog/2025/07/trino-logo.webp) - - -With storage (Iceberg) and metadata (Lakekeeper) in place, we need a query engine. Trino fits the bill for that . It's an open-source distributed SQL engine designed for interactive analytics on big data lakes. - -It lets you ask questions in standard SQL across data sitting anywhere, from object-storage **"data lakes"** to traditional databases, without copying or moving that data. Think of it as a super-fast interpreter that speaks SQL, fans work out to many servers in parallel, and hands you results in seconds, even at petabyte scale. - -### Quick Definition -Trino is a "query engine," not a database. - -It stores no data itself. Instead, it connects to many systems, splits your SQL into small tasks, runs them in parallel on worker nodes, and merges the answers back to you - -### Core Concept -1. It stores no data itself and basically purely acts as compute layer while also being able to connect to multiple systems (S3, MySQL, Kafka, etc.) and make them look like SQL tables. - -2. It also supports massive parallelism where it breaks queries into small tasks, runs them across worker nodes, and merges results. Thus, making things more efficient as compared to traditional approach - -### Why Trino Exists? -1. **Speed over Hive & MapReduce**: Facebook's data analysts needed sub-minute answers on multi-petabyte Hadoop clusters; Hive's MapReduce jobs took hours. The first Presto prototype (Trino's original name) appeared in 2012 to fix that. - -2. **SQL for Everything**: Teams hated learning four different APIs. Trino made any data source look like an ANSI-SQL table. - -3. **Data Stays Put**: Copying data is slow, risky, and costly. Trino federates live sources so analytics teams can join S3 logs with MySQL customer tables in one query. - -4. **Open Governance**: After leaving Facebook, the creators forked PrestoSQL and rebranded it as Trino in 2020 to keep development community-driven. - -So if you are wondering why would you even consider trino let me get you in on some benefits on using that - -### Key Benefits — Why it's so powerful? -1. Trino is like that powerhouse for the organizations that want to save time on analysis here we get answers in seconds on terabyte datasets vs. hours with traditional tools like presto . - -2. Same familiar SQL syntax across all data sources so you don't have to learn multiple ways of querying that data - -3. If you are organization scaling is the **NEED of the hour** well it goes ahead and supports elastic scaling where you can add workers to cut runtime nearly linearly - -4. Most importantly it doesn't make you dependent on one vendor, might have heard of vendor lock in right? Yup, sorts it out aswell . - - -## Putting It All Together with Docker Compose 🐋 - -Here's a minimal Docker Compose snippet that wires up these services. It uses **MinIO** as S3 storage, **Lakekeeper** as the catalog, **OLake** as an ingestion service, and **Trino** as the query engine. (In practice you'd supply actual config files or env vars as needed.) - -For those wondering what even is MinIO is it's a simple, S3-compatible object storage service that runs locally, making it incredibly useful for experimenting with what you're learning before moving to a full cloud setup. - -
    -Click to expand Docker Compose YAML - -```yaml -version: '3.8' - -services: - minio: - image: quay.io/minio/minio:latest - container_name: minio - volumes: - - minio-data:/data - environment: - MINIO_ROOT_USER: minio - MINIO_ROOT_PASSWORD: minio123 - command: server /data --console-address ":9001" - ports: - - "9000:9000" # API port - - "9001:9001" # Console port - healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] - interval: 30s - timeout: 20s - retries: 3 - - lakekeeper: - image: ${LAKEKEEPER__SERVER_IMAGE:-quay.io/lakekeeper/catalog:v0.11.1} - pull_policy: always - environment: - - LAKEKEEPER__PG_ENCRYPTION_KEY=This-is-NOT-Secure! - - LAKEKEEPER__PG_DATABASE_URL_READ=postgresql://postgres:postgres@db:5432/postgres - - LAKEKEEPER__PG_DATABASE_URL_WRITE=postgresql://postgres:postgres@db:5432/postgres - - LAKEKEEPER__AUTHZ_BACKEND=allowall - # Externally taken from environment variables if set - - LAKEKEEPER__OPENID_PROVIDER_URI - - LAKEKEEPER__OPENID_AUDIENCE - - LAKEKEEPER__OPENID_ADDITIONAL_ISSUERS - - LAKEKEEPER__UI__OPENID_CLIENT_ID - - LAKEKEEPER__UI__OPENID_SCOPE - command: [ "serve" ] - healthcheck: - test: [ "CMD", "/home/nonroot/lakekeeper", "healthcheck" ] - interval: 1s - timeout: 10s - retries: 3 - start_period: 3s - depends_on: - migrate: - condition: service_completed_successfully - ports: - - "8181:8181" - restart: unless-stopped - - migrate: - image: ${LAKEKEEPER__SERVER_IMAGE:-quay.io/lakekeeper/catalog:v0.11.1} - pull_policy: always - environment: - - LAKEKEEPER__PG_ENCRYPTION_KEY=This-is-NOT-Secure! - - LAKEKEEPER__PG_DATABASE_URL_READ=postgresql://postgres:postgres@db:5432/postgres - - LAKEKEEPER__PG_DATABASE_URL_WRITE=postgresql://postgres:postgres@db:5432/postgres - - LAKEKEEPER__AUTHZ_BACKEND=allowall - # Externally taken from environment variables if set - - LAKEKEEPER__OPENID_PROVIDER_URI - - LAKEKEEPER__OPENID_AUDIENCE - - LAKEKEEPER__OPENID_ADDITIONAL_ISSUERS - - LAKEKEEPER__UI__OPENID_CLIENT_ID - - LAKEKEEPER__UI__OPENID_SCOPE - restart: "no" - command: [ "migrate" ] - depends_on: - db: - condition: service_healthy - - db: - image: bitnami/postgresql:16.6.0 - container_name: db - environment: - - POSTGRESQL_USERNAME=postgres - - POSTGRESQL_PASSWORD=postgres - - POSTGRESQL_DATABASE=postgres - healthcheck: - test: [ "CMD-SHELL", "pg_isready -U postgres -p 5432 -d postgres" ] - interval: 2s - timeout: 10s - retries: 2 - start_period: 10s - volumes: - - volume-lakekeeper:/bitnami/postgresql - - trino: - image: trinodb/trino:latest - container_name: trino - ports: - - "8082:8080" - depends_on: - - lakekeeper - - minio - volumes: - - ./trino/etc:/etc/trino:ro - environment: - # Optional: Add any Trino-specific environment variables - TRINO_ENVIRONMENT: development - restart: unless-stopped - # Note: Ensure ./trino/etc contains proper catalog configuration: - # File: ./trino/etc/catalog/iceberg.properties - # connector.name=iceberg - # iceberg.catalog.type=rest - # iceberg.rest-catalog.uri=http://lakekeeper:8181 - # iceberg.rest-catalog.warehouse=warehouse - # fs.s3.aws-access-key=minio - # fs.s3.aws-secret-key=minio123 - # fs.s3.endpoint=http://minio:9000 - # fs.s3.path-style-access=true - - # OLake service - commented out as the image may not be publicly available - # olake: - # image: datazipinc/olake:latest - # container_name: olake - # depends_on: - # - minio - # - lakekeeper - # environment: - # # Add required OLake configuration - # OLAKE_S3_ENDPOINT: http://minio:9000 - # OLAKE_S3_ACCESS_KEY: minio - # OLAKE_S3_SECRET_KEY: minio123 - # OLAKE_CATALOG_URI: http://lakekeeper:8181 - # restart: unless-stopped - # # Add ports if OLake exposes a web interface - # # ports: - # # - "8082:8082" - -volumes: - minio-data: - driver: local - volume-lakekeeper: - -networks: - default: - name: lakehouse-network -``` - -
    - -With that up (docker compose up), you'd have MinIO as object storage, Lakekeeper listening on port **8181**, and Trino on **8080**. - -In Trino's catalog config (shown above as comments), we create an Iceberg catalog of type rest pointing to Lakekeeper. This tells Trino to use Lakekeeper as the metadata store. and OLake would be running too, ready to sync data from your databases into Iceberg tables on MinIO. - -**Note**: The exact configs (region, bucket, etc.) depend on your setup. You'll also need to create a warehouse in Lakekeeper (via its UI or API) and tell Trino which warehouse to use. Check [Lakekeeper docs](https://docs.lakekeeper.io/) and [Trino's Iceberg connector docs](https://trino.io/docs/current/connector/iceberg.html) for full details - -## Example Trino Query - -Once everything is running, you can query your Iceberg tables with Trino like any SQL database. Here's a simple example of a business query joining two tables: - -```sql -SELECT c.country, SUM(o.amount) AS total_sales -FROM customers AS c -JOIN orders AS o - ON c.customer_id = o.customer_id -WHERE o.order_date >= DATE '2025-01-01' -GROUP BY c.country -ORDER BY total_sales DESC; -``` - -In this query, customers and orders are Iceberg tables managed by Lakekeeper. Trino will translate the SQL into distributed file scans: reading Parquet files in S3/MinIO, filtering, grouping, etc because Iceberg tracks partitions and snapshots, Trino can push down predicates (e.g. on order_date) and only read relevant files. - -Even updates or deletes you made via Iceberg's **MERGE/DELETE** commands will be handled correctly under the hood (Iceberg's metadata ensures consistency). - -This simple example shows the power of the stack: you define your tables in Iceberg, OLake or other pipelines load them, Lakekeeper keeps metadata, and Trino lets you run normal SQL against the data. You could swap in DuckDB for ad-hoc local queries on the same data, or add Spark to the mix for large ETL jobs the data format remains the same. - -## Trino x Lakekeeper - -Now that Trino is ready to run your queries, Lakekeeper steps in as its metadata guardian ensuring every Iceberg table is tracked, secured, and instantly discoverable. - -Lakekeeper is a service-based Apache Iceberg REST catalog written in Rust. Point Trino's Iceberg connector to Lakekeeper: - -```properties -connector.name=iceberg -iceberg.catalog.type=rest -iceberg.rest-catalog.uri=http://lakekeeper:8181/catalog -``` - -From there you can get Consistent Snapshots: All engines (Trino, Spark, DuckDB) see the same table version no stale or conflicting metadata. - -Security stays one of the important concerns but lakekeeper enforces OIDC authentication plus OpenFGA/OPA policies for table, column, and row-level access automatically on every Trino query if you want to read more about them you can do so [here](https://openfga.dev/) - -**In short**, Lakekeeper transforms raw object storage into a governed, high-performance Iceberg layer that makes Trino queries reliable, secure, and effortlessly current. - -## How Iceberg, OLake,Lakekeeper and Trino Mesh -![Lakekeeper Iceberg REST Catalog architecture with object storage, Trino, Open Policy Agent, and OpenFGA-based authorization](/img/blog/2025/07/mesh-iceberg-trino-olake-lakekeeper.webp) - -### End-to-End Flow - -1. Ingest – OLake ingests CDC streams and commits Iceberg snapshots via Lakekeeper API. -2. Discover – Trino's Iceberg connector points to Lakekeeper, instantly seeing new tables and versions. -3. Secure & Govern – Lakekeeper checks OpenFGA policies for each Trino user before handing back metadata. -4. Query – Trino executes federated SQL joins across fresh Iceberg data, legacy MySQL tables, and even Kafka streams—all through one engine. - - -## What You Gain - -| Benefit | Iceberg | OLake | Trino | -|---------|---------|-------|-------| -| Low-cost object storage | ✓ | - | - | -| Transactional writes | ✓ | ✓ (via Iceberg commits) | - | -| Real-time ingestion | - | ✓ | - | -| Snapshot time travel | ✓ | - | ✓ (select snapshot) | -| Interactive SQL | - | - | ✓ | -| Federated joins | - | - | ✓ | -| Centralized auth | - | - | ✓ (via Lakekeeper/OPA) | -| Engine-agnostic metadata | ✓ | ✓ | ✓ | - -## Conclusion - -We've covered a modern open-source lakehouse setup: **Iceberg for storage**, **OLake for loading data**, **Lakekeeper for metadata**, and **Trino for querying**. - -Each piece is designed for scale and flexibility. For example, Iceberg's features mean you can evolve schemas without downtime and "time travel" in your data. Lakekeeper adds security and standardization for those Iceberg tables. OLake takes care of the heavy lifting of moving data into the lake. And Trino glues it together by giving you a familiar SQL interface. - -All of these tools play nicely with Docker (as shown) or Kubernetes, so you can spin them up for testing or production. If you're already familiar with Docker, you should have no trouble experimenting: try loading some sample data and running queries. The best way to learn is to dive in! - -**Happy building and welcome to the lakehouse club!** - - +--- + +slug: building-modern-data-lakehouse-with-olake-iceberg-lakekeeper-trino +title: Building Modern Lakehouse with Iceberg, OLake, Lakekeeper & Trino +description: 'Iceberg is the storage "brain," OLake is the real-time "pipeline," and Trino is the fast "question-answering" engine. Together they turn raw object-storage files into a governed, low-latency analytics platform.' +image: /img/blog/cover/next-gen-lakehouse-cover.webp +authors: [akshay] +tags: [olake, iceberg] +--- + +![The Next Gen Lakehouse: Iceberg, OLake, Lakekeeper, and Trino integration logos on a white background](/img/blog/cover/next-gen-lakehouse-cover.webp) + +# The Next-Gen Lakehouse — Iceberg, OLake Go, Lakekeeper and Trino + +Quick on that here : +**Iceberg** is the storage "brain", **OLake Go** is the real-time "pipeline", and **Trino** is the fast "question-answering" engine. + +Together they turn raw object-storage files into a governed, low-latency analytics platform. +Data engineering nowadays has swung toward the **lakehouse paradigm** where we are mixing data lake scale with warehouse reliability and this blog walks through a hands-on stack of open-source tools that make a Lakehouse work. + +We'll see how Apache Iceberg (an open table format), OLake Go (fast DB→lake loader), Lakekeeper (Iceberg's REST catalog), and Trino (distributed SQL engine) fit together. Think of it as a friendly tour of each component (what it is, why it matters and most importantly how it all connects) + +## What Is Apache Iceberg? + +Well to understand it a bit simpler you can picture a giant library where every book (data file) sits on random floors with no catalog. Finding the Harry Potter series would be a chaos. Iceberg acts as the modern **Dewey Decimal system**: it tags every file with rich metadata, tracks versions, and lets you time-travel to yesterday's shelves all without moving the books themselves . Thus, now you can read books and data engineers can query data better . + +**Well what's the value of it ?** + + +Firstly , you get database-style "all-or-nothing" writes on cheap object storage, e.g. S3, GCS, Azure blobs or Minio. You can add, rename, or drop columns and partitions without rewriting terabytes. + + +Query your data "as of" last Tuesday for audits or bug fixes. + + +**Hidden Partitioning** — Iceberg tracks which files hold which dates, so queries auto-skip irrelevant chunks, no brittle dt='2025-07-21' filters required. + + +Most importantly it is **engine-agnostic** you might have heard this term a lot and here it gets a meaning iceberg supports [Spark](/iceberg/query-engine/spark), [Trino](/iceberg/query-engine/trino), Flink, [DuckDB](/iceberg/query-engine/duckdb), Dremio, and Snowflake all speak to the tables natively + + +Before Iceberg, data lakes were basically digital junkyards. You'd dump data files into cloud storage (like Amazon S3), and finding anything useful was like looking for a specific needle in a haystack . + +## Understanding Iceberg Catalogs: The Foundation Layer + +Think of the catalog as the librarian of your data lakehouse it keeps track of which tables exist, where they are, and which version is current. + +### The Catalog Challenge in Iceberg + +Every time you make changes to an Iceberg table whether adding rows, updating schemas, or modifying partitions Iceberg creates a new metadata.json file (like v1.metadata.json, v2.metadata.json). Over time, you accumulate hundreds of these files. + +The critical question becomes: how do query engines like Trino, Spark, or DuckDB know **which metadata file represents the "current" state of the table?** + +This is where the catalog becomes essential. The catalog serves two fundamental purposes: + + +1. Maintains a registry of existing Iceberg tables like a phone book for your data assets + +2. Tracks pointers to the current metadata.json file ensuring everyone sees the same version of truth + + +There are different type of data catalogs file based and service based and today, we're taking a closer look at Lakekeeper a type of data catalog that resolves these challenges so we will cover what it is, how it stands out, and exactly what category of catalog it belongs to for Apache Iceberg. + +## What is Lakekeeper? + +#### (Empowering Object Storage with Iceberg and Enterprise-Grade Governance) + +Lakekeeper turns ordinary object storage (like S3) into a fully governed Apache Iceberg lakehouse. Out of the box, S3 is just a bunch of files with basic access controls. Lakekeeper manages all the Iceberg metadata (schemas, snapshots, pointers) so your storage now supports transactions, time travel(going back at a particular commit ) and consistent views across all engines (Trino, Spark, DuckDB, you name it). + +What really makes Lakekeeper stand out is how it hooks into your organization's identity, access control, and policy tools (like **OPA** and **OpenFGA**). This means you get table, column, even row-level permissions and governance applied no matter who's querying or how they access the data. + +**In short**: By combining open table formats (Iceberg) and centralized, policy-driven control, Lakekeeper upgrades your object store from "just a bucket" to a secure, compliant, analytics-ready platform. That's the key to making S3 truly enterprise and audit—friendly. + +### Why Lakekeeper Isn't Just Another Catalog + +It fully implements the Apache Iceberg REST Catalog API, making it compatible with any query engine that supports the Iceberg standard REST interface. + + +Written in Rust, it's lightning-fast and extremely lightweight; just one binary to deploy, but strong enough for enterprise-grade operations + +### How it suits modern pipelines? + +What really sets Lakekeeper apart is how it bridges the gap between raw object storage and truly managed, governed Iceberg tables. Most basic catalogs just keep a list of where the data is, but Lakekeeper acts as a smart, active coordinator that actually turns cloud buckets into a secure, production-ready analytical layer. + +Instead of relying on the old approach where you had to manage clunky file-based pointers or deal with slow, legacy services, Lakekeeper brings the agility, real-time response, and open standards you'd expect from a modern data stack. + +## What is OLake Go? + +Now that you understand how Apache Iceberg organizes your data lake into a well-structured, reliable system, you're probably wondering: **"That's great, but how do I actually get my operational data INTO Iceberg format?"** + +This is where the rubber meets the road in building a modern lakehouse. + +Traditional approaches involve complex setups with tools like **Debezium + Kafka**, or using generic ETL platforms that weren't designed for the specific demands of lakehouse architectures. These solutions often struggle with scale, reliability, and the unique requirements of Apache Iceberg. That's exactly the **gap** that OLake Go was built to fill it's like having a specialized highway that moves data from your **operational databases** directly into the **Iceberg ecosystem**, fast and reliably. + +**Still confused ?** +Think of it this way: if your operational database is like a busy restaurant kitchen constantly taking orders and making food, OLake Go is like having a super-efficient catering service that continuously packages up perfect copies of those meals and delivers them to your analytics "dining hall" (the data lakehouse) without disrupting the kitchen operations. + +### How Lakekeeper and OLake Go Work Together for Ingestion + +#### (And Why It's a Game-Changer for Real-Time, Production-Grade Data Stacks) + +So you've set up your data lakehouse and you want fresh, reliable data to land there fast, secure, and always discoverable by everyone who needs it. That's where the tag team of **OLake Go** (for high-speed database replication) and **Lakekeeper** (as the brains of your Iceberg metadata) steps in. Let's break down exactly how these two connect, and what happens under the hood when new data is ingested. + +### The Big Picture Flow + +OLake Go handles pulling change data from your databases (like MongoDB, PostgreSQL, MySQL), breaks it into efficient chunks, and writes it directly into Apache Iceberg tables. But it needs to coordinate all metadata table versions, schema, file locations to make sure what it writes is instantly queryable, consistent, and secure. + +Lakekeeper is the REST catalog that acts as the "metadata authority" where OLake Go checks what tables exist it is the place OLake Go registers new data and schema changes while also acting as the guardrail for access, audit, and governance + +**In short**: OLake Go does the heavy lifting, but Lakekeeper makes sure every lift is recorded, governed, and easy to discover in real-time by analytical engines like **Trino**. + +![OLake and Lakekeeper ingesting data from MongoDB, PostgreSQL, and MySQL into Apache Iceberg data lakehouse architecture](/img/blog/2025/07/olake-lakekeeper.webp) + +## What is trino? : Distributed SQL on Everything +![Trino logo featuring a cute astronaut bunny icon with black 'trino' text](/img/blog/2025/07/trino-logo.webp) + + +With storage (Iceberg) and metadata (Lakekeeper) in place, we need a query engine. Trino fits the bill for that . It's an open-source distributed SQL engine designed for interactive analytics on big data lakes. + +It lets you ask questions in standard SQL across data sitting anywhere, from object-storage **"data lakes"** to traditional databases, without copying or moving that data. Think of it as a super-fast interpreter that speaks SQL, fans work out to many servers in parallel, and hands you results in seconds, even at petabyte scale. + +### Quick Definition +Trino is a "query engine," not a database. + +It stores no data itself. Instead, it connects to many systems, splits your SQL into small tasks, runs them in parallel on worker nodes, and merges the answers back to you + +### Core Concept +1. It stores no data itself and basically purely acts as compute layer while also being able to connect to multiple systems (S3, MySQL, Kafka, etc.) and make them look like SQL tables. + +2. It also supports massive parallelism where it breaks queries into small tasks, runs them across worker nodes, and merges results. Thus, making things more efficient as compared to traditional approach + +### Why Trino Exists? +1. **Speed over Hive & MapReduce**: Facebook's data analysts needed sub-minute answers on multi-petabyte Hadoop clusters; Hive's MapReduce jobs took hours. The first Presto prototype (Trino's original name) appeared in 2012 to fix that. + +2. **SQL for Everything**: Teams hated learning four different APIs. Trino made any data source look like an ANSI-SQL table. + +3. **Data Stays Put**: Copying data is slow, risky, and costly. Trino federates live sources so analytics teams can join S3 logs with MySQL customer tables in one query. + +4. **Open Governance**: After leaving Facebook, the creators forked PrestoSQL and rebranded it as Trino in 2020 to keep development community-driven. + +So if you are wondering why would you even consider trino let me get you in on some benefits on using that + +### Key Benefits — Why it's so powerful? +1. Trino is like that powerhouse for the organizations that want to save time on analysis here we get answers in seconds on terabyte datasets vs. hours with traditional tools like presto . + +2. Same familiar SQL syntax across all data sources so you don't have to learn multiple ways of querying that data + +3. If you are organization scaling is the **NEED of the hour** well it goes ahead and supports elastic scaling where you can add workers to cut runtime nearly linearly + +4. Most importantly it doesn't make you dependent on one vendor, might have heard of vendor lock in right? Yup, sorts it out aswell . + + +## Putting It All Together with Docker Compose 🐋 + +Here's a minimal Docker Compose snippet that wires up these services. It uses **MinIO** as S3 storage, **Lakekeeper** as the catalog, **OLake Go** as an ingestion service, and **Trino** as the query engine. (In practice you'd supply actual config files or env vars as needed.) + +For those wondering what even is MinIO is it's a simple, S3-compatible object storage service that runs locally, making it incredibly useful for experimenting with what you're learning before moving to a full cloud setup. + +
    +Click to expand Docker Compose YAML + +```yaml +version: '3.8' + +services: + minio: + image: quay.io/minio/minio:latest + container_name: minio + volumes: + - minio-data:/data + environment: + MINIO_ROOT_USER: minio + MINIO_ROOT_PASSWORD: minio123 + command: server /data --console-address ":9001" + ports: + - "9000:9000" # API port + - "9001:9001" # Console port + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] + interval: 30s + timeout: 20s + retries: 3 + + lakekeeper: + image: ${LAKEKEEPER__SERVER_IMAGE:-quay.io/lakekeeper/catalog:v0.11.1} + pull_policy: always + environment: + - LAKEKEEPER__PG_ENCRYPTION_KEY=This-is-NOT-Secure! + - LAKEKEEPER__PG_DATABASE_URL_READ=postgresql://postgres:postgres@db:5432/postgres + - LAKEKEEPER__PG_DATABASE_URL_WRITE=postgresql://postgres:postgres@db:5432/postgres + - LAKEKEEPER__AUTHZ_BACKEND=allowall + # Externally taken from environment variables if set + - LAKEKEEPER__OPENID_PROVIDER_URI + - LAKEKEEPER__OPENID_AUDIENCE + - LAKEKEEPER__OPENID_ADDITIONAL_ISSUERS + - LAKEKEEPER__UI__OPENID_CLIENT_ID + - LAKEKEEPER__UI__OPENID_SCOPE + command: [ "serve" ] + healthcheck: + test: [ "CMD", "/home/nonroot/lakekeeper", "healthcheck" ] + interval: 1s + timeout: 10s + retries: 3 + start_period: 3s + depends_on: + migrate: + condition: service_completed_successfully + ports: + - "8181:8181" + restart: unless-stopped + + migrate: + image: ${LAKEKEEPER__SERVER_IMAGE:-quay.io/lakekeeper/catalog:v0.11.1} + pull_policy: always + environment: + - LAKEKEEPER__PG_ENCRYPTION_KEY=This-is-NOT-Secure! + - LAKEKEEPER__PG_DATABASE_URL_READ=postgresql://postgres:postgres@db:5432/postgres + - LAKEKEEPER__PG_DATABASE_URL_WRITE=postgresql://postgres:postgres@db:5432/postgres + - LAKEKEEPER__AUTHZ_BACKEND=allowall + # Externally taken from environment variables if set + - LAKEKEEPER__OPENID_PROVIDER_URI + - LAKEKEEPER__OPENID_AUDIENCE + - LAKEKEEPER__OPENID_ADDITIONAL_ISSUERS + - LAKEKEEPER__UI__OPENID_CLIENT_ID + - LAKEKEEPER__UI__OPENID_SCOPE + restart: "no" + command: [ "migrate" ] + depends_on: + db: + condition: service_healthy + + db: + image: bitnami/postgresql:16.6.0 + container_name: db + environment: + - POSTGRESQL_USERNAME=postgres + - POSTGRESQL_PASSWORD=postgres + - POSTGRESQL_DATABASE=postgres + healthcheck: + test: [ "CMD-SHELL", "pg_isready -U postgres -p 5432 -d postgres" ] + interval: 2s + timeout: 10s + retries: 2 + start_period: 10s + volumes: + - volume-lakekeeper:/bitnami/postgresql + + trino: + image: trinodb/trino:latest + container_name: trino + ports: + - "8082:8080" + depends_on: + - lakekeeper + - minio + volumes: + - ./trino/etc:/etc/trino:ro + environment: + # Optional: Add any Trino-specific environment variables + TRINO_ENVIRONMENT: development + restart: unless-stopped + # Note: Ensure ./trino/etc contains proper catalog configuration: + # File: ./trino/etc/catalog/iceberg.properties + # connector.name=iceberg + # iceberg.catalog.type=rest + # iceberg.rest-catalog.uri=http://lakekeeper:8181 + # iceberg.rest-catalog.warehouse=warehouse + # fs.s3.aws-access-key=minio + # fs.s3.aws-secret-key=minio123 + # fs.s3.endpoint=http://minio:9000 + # fs.s3.path-style-access=true + + # OLake Go service - commented out as the image may not be publicly available + # olake: + # image: datazipinc/olake:latest + # container_name: olake + # depends_on: + # - minio + # - lakekeeper + # environment: + # # Add required OLake Go configuration + # OLAKE_S3_ENDPOINT: http://minio:9000 + # OLAKE_S3_ACCESS_KEY: minio + # OLAKE_S3_SECRET_KEY: minio123 + # OLAKE_CATALOG_URI: http://lakekeeper:8181 + # restart: unless-stopped + # # Add ports if OLake exposes a web interface + # # ports: + # # - "8082:8082" + +volumes: + minio-data: + driver: local + volume-lakekeeper: + +networks: + default: + name: lakehouse-network +``` + +
    + +With that up (docker compose up), you'd have MinIO as object storage, Lakekeeper listening on port **8181**, and Trino on **8080**. + +In Trino's catalog config (shown above as comments), we create an Iceberg catalog of type rest pointing to Lakekeeper. This tells Trino to use Lakekeeper as the metadata store. and OLake Go would be running too, ready to sync data from your databases into Iceberg tables on MinIO. + +**Note**: The exact configs (region, bucket, etc.) depend on your setup. You'll also need to create a warehouse in Lakekeeper (via its UI or API) and tell Trino which warehouse to use. Check [Lakekeeper docs](https://docs.lakekeeper.io/) and [Trino's Iceberg connector docs](https://trino.io/docs/current/connector/iceberg.html) for full details + +## Example Trino Query + +Once everything is running, you can query your Iceberg tables with Trino like any SQL database. Here's a simple example of a business query joining two tables: + +```sql +SELECT c.country, SUM(o.amount) AS total_sales +FROM customers AS c +JOIN orders AS o + ON c.customer_id = o.customer_id +WHERE o.order_date >= DATE '2025-01-01' +GROUP BY c.country +ORDER BY total_sales DESC; +``` + +In this query, customers and orders are Iceberg tables managed by Lakekeeper. Trino will translate the SQL into distributed file scans: reading Parquet files in S3/MinIO, filtering, grouping, etc because Iceberg tracks partitions and snapshots, Trino can push down predicates (e.g. on order_date) and only read relevant files. + +Even updates or deletes you made via Iceberg's **MERGE/DELETE** commands will be handled correctly under the hood (Iceberg's metadata ensures consistency). + +This simple example shows the power of the stack: you define your tables in Iceberg, OLake Go or other pipelines load them, Lakekeeper keeps metadata, and Trino lets you run normal SQL against the data. You could swap in DuckDB for ad-hoc local queries on the same data, or add Spark to the mix for large ETL jobs the data format remains the same. + +## Trino x Lakekeeper + +Now that Trino is ready to run your queries, Lakekeeper steps in as its metadata guardian ensuring every Iceberg table is tracked, secured, and instantly discoverable. + +Lakekeeper is a service-based Apache Iceberg REST catalog written in Rust. Point Trino's Iceberg connector to Lakekeeper: + +```properties +connector.name=iceberg +iceberg.catalog.type=rest +iceberg.rest-catalog.uri=http://lakekeeper:8181/catalog +``` + +From there you can get Consistent Snapshots: All engines (Trino, Spark, DuckDB) see the same table version no stale or conflicting metadata. + +Security stays one of the important concerns but lakekeeper enforces OIDC authentication plus OpenFGA/OPA policies for table, column, and row-level access automatically on every Trino query if you want to read more about them you can do so [here](https://openfga.dev/) + +**In short**, Lakekeeper transforms raw object storage into a governed, high-performance Iceberg layer that makes Trino queries reliable, secure, and effortlessly current. + +## How Iceberg, OLake Go, Lakekeeper and Trino Mesh +![Lakekeeper Iceberg REST Catalog architecture with object storage, Trino, Open Policy Agent, and OpenFGA-based authorization](/img/blog/2025/07/mesh-iceberg-trino-olake-lakekeeper.webp) + +### End-to-End Flow + +1. Ingest – OLake Go ingests CDC streams and commits Iceberg snapshots via Lakekeeper API. +2. Discover – Trino's Iceberg connector points to Lakekeeper, instantly seeing new tables and versions. +3. Secure & Govern – Lakekeeper checks OpenFGA policies for each Trino user before handing back metadata. +4. Query – Trino executes federated SQL joins across fresh Iceberg data, legacy MySQL tables, and even Kafka streams—all through one engine. + + +## What You Gain + +| Benefit | Iceberg | OLake Go | Trino | +|---------|---------|-------|-------| +| Low-cost object storage | ✓ | - | - | +| Transactional writes | ✓ | ✓ (via Iceberg commits) | - | +| Real-time ingestion | - | ✓ | - | +| Snapshot time travel | ✓ | - | ✓ (select snapshot) | +| Interactive SQL | - | - | ✓ | +| Federated joins | - | - | ✓ | +| Centralized auth | - | - | ✓ (via Lakekeeper/OPA) | +| Engine-agnostic metadata | ✓ | ✓ | ✓ | + +## Conclusion + +We've covered a modern open-source lakehouse setup: **Iceberg for storage**, **OLake Go for loading data**, **Lakekeeper for metadata**, and **Trino for querying**. + +Each piece is designed for scale and flexibility. For example, Iceberg's features mean you can evolve schemas without downtime and "time travel" in your data. Lakekeeper adds security and standardization for those Iceberg tables. OLake Go takes care of the heavy lifting of moving data into the lake. And Trino glues it together by giving you a familiar SQL interface. + +All of these tools play nicely with Docker (as shown) or Kubernetes, so you can spin them up for testing or production. If you're already familiar with Docker, you should have no trouble experimenting: try loading some sample data and running queries. The best way to learn is to dive in! + +**Happy building and welcome to the lakehouse club!** + +## FAQs + +

    Lakekeeper is an open-source Apache Iceberg REST Catalog written in Rust, designed to turn ordinary object storage into a fully governed Iceberg lakehouse.

    +

    Key differentiators from Hive Metastore or AWS Glue:

    +
      +
    • No JVM or Python runtime required - ships as a single binary executable for all major platforms
      • +
    • Natively implements the Iceberg REST Catalog API - no adapters or shims needed; compatible with any Iceberg-compatible engine out of the box
      • +
    • Enterprise access control - hooks into OPA and OpenFGA for table, column, and row-level permissions
      • +
    • Cloud-agnostic - works with AWS S3, GCS, Azure Blob, and MinIO
      • +
    +

    Dependency note: While Lakekeeper eliminates the JVM overhead of Hive Metastore, production deployments do require a PostgreSQL persistence backend (currently the only supported metadata database) and a secret store. It is not entirely dependency-free - but the operational footprint is significantly smaller than Hive Metastore.

    + + }, + { + question: "Q2. How do OLake Go and Lakekeeper work together for real-time data ingestion?", + answer:
    +

    OLake Go captures Change Data Capture events from operational databases (MongoDB, PostgreSQL, MySQL) and needs to commit Iceberg snapshots to object storage in a governed, discoverable way. Lakekeeper acts as the REST catalog authority throughout this process:

    +
      +
    1. OLake Go checks what tables exist through Lakekeeper's REST API
      2. +
    2. OLake Go registers new tables and schema changes via Lakekeeper
      3. +
    3. OLake Go commits new Iceberg snapshots through Lakekeeper
      4. +
    +

    This ensures every write is immediately discoverable and governed across all query engines - Trino, Spark, PyIceberg, and others - the moment the commit completes.

    +
    + }, + { + question: "Q3. What is Trino's role in the Iceberg, OLake Go, and Lakekeeper lakehouse stack?", + answer:
    +

    Trino is the distributed SQL query engine that provides the analytics layer. After OLake Go writes data as Iceberg snapshots and Lakekeeper manages the metadata:

    +
      +
    • Trino connects to Lakekeeper's REST catalog using its native Iceberg connector
      • +
    • Trino discovers tables and reads current metadata from Lakekeeper
      • +
    • Trino executes federated SQL queries with massively parallel processing (MPP), delivering sub-second results on large datasets
      • +
    +

    Full Iceberg feature support is available through Trino including time travel queries, partition pruning, and predicate pushdown - all metadata served by Lakekeeper.

    +
    + }, + { + question: "Q4. Why should I choose Lakekeeper over Hive Metastore for Iceberg table management?", + answer:
    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Hive MetastoreLakekeeper
    RuntimeRequires JVMSingle Rust binary, no JVM
    Iceberg supportRequires adapterNative REST Catalog spec
    AuthenticationBasicOIDC + OpenFGA/OPA
    Cloud supportLimitedS3, GCS, Azure Blob, MinIO
    Metadata backendMySQL or PostgreSQLPostgreSQL
    +

    The key practical differences:

    +
      +
    • No JVM - Lakekeeper eliminates the JVM dependency and operational overhead of Hive Metastore
      • +
    • Native Iceberg REST API - no adapter or compatibility layer needed; all Iceberg-compatible engines connect directly
      • +
    • Modern auth - OIDC identity provider integration and fine-grained access control via OpenFGA out of the box
      • +
    +

    Note: Lakekeeper still requires a PostgreSQL instance as its metadata persistence backend - it replaces Hive Metastore's JVM and adapter complexity, not its need for a backing database.

    +
    + }, + { + question: "Q5. What is the complete data flow in the Iceberg, OLake Go, and Lakekeeper stack?", + answer:
    +

    The end-to-end flow across the four components:

    +
      +
    1. OLake Go captures database changes (inserts, updates, deletes) from MongoDB, PostgreSQL, or MySQL via CDC and writes Iceberg snapshots - Parquet data files and metadata - directly to object storage (S3, MinIO, GCS, Azure Blob)
      2. +
    2. Lakekeeper manages all table metadata via its REST Catalog API, tracking which metadata file represents the current state of each table. PostgreSQL serves as Lakekeeper's backing metadata store.
      3. +
    3. Trino connects its Iceberg connector to Lakekeeper, discovers tables and their current metadata, then executes distributed SQL queries by reading Parquet data files from object storage in parallel across worker nodes
      4. +
    +

    All three components operate in real time - a CDC event committed by OLake Go is immediately visible to Trino via Lakekeeper with no additional sync or polling step required.

    +
    + } +]} /> + + + diff --git a/blog/2025-07-31-apache-iceberg-vs-delta-lake-guide.mdx b/blog/2025-07-31-apache-iceberg-vs-delta-lake-guide.mdx index 61280ee6f..0f4cb462b 100644 --- a/blog/2025-07-31-apache-iceberg-vs-delta-lake-guide.mdx +++ b/blog/2025-07-31-apache-iceberg-vs-delta-lake-guide.mdx @@ -1,167 +1,223 @@ ---- -slug: apache-iceberg-vs-delta-lake-guide -title: "Apache Iceberg vs Delta Lake: Ultimate Guide for Data Lakes" -description: "Explore the key differences between Apache Iceberg and Delta Lake for batch analytics, ML pipelines, and cost-effective data lake management." -image: /img/blog/cover/iceberg-delta-comparison-cover.webp -authors: [shubham] -tags: [iceberg, delta-lake, data-lakehouse, analytics, ml] ---- - -![Apache Iceberg vs Delta Lake comparison architecture for batch analytics and ML](/img/blog/cover/iceberg-delta-comparison-cover.webp) - -# Apache Iceberg vs Delta Lake: Comparison for Batch Analytics & ML Pipelines - -You've probably heard the buzz around Apache Iceberg and Delta Lake. If you're deep into big data, picking the right one can be tricky. So let's skip the hype and focus on what really matters for batch analytics and ML pipelines. - -Open table formats are the game-changer for data lakes - they bring ACID transactions, time travel, and schema evolution to cheap object storage. That's warehouse-grade power on a lakehouse budget. - -## What This Comparison Is About - -We're zooming in on use cases like ETL pipelines, feature engineering, and reproducible training runs where consistent, reliable data matters most. - -Here's why formats like Iceberg and Delta shine in these workflows: - -- **Time travel & snapshots**: Train models from a specific version, enabling reproducibility that truly works. -- **ACID operations**: MERGE, UPDATE, DELETE, essential for managing feature tables and late-arriving data. -- **Schema evolution**: Add or update fields without rewriting history, because nobody's got time for that. - -![Lakehouse architecture diagram highlighting ACID transactions, time travel, schema evolution, data ingestion, metadata management, storage, governance, and query engine](/img/blog/2025/08/architecture-comparison.webp) - -## Performance & Scalability: Where the Rubber Meets the Road - -Performance is often the deciding factor and rightly so. The good news? Both formats are built to handle large-scale datasets efficiently, though each has its own strengths depending on the workload. - -### File Layout & Updates - -Delta Lake uses a **copy-on-write** approach by default for the open-source version. When you need to update data, it creates new files and marks the old ones for deletion. The new [**Deletion Vectors (DVs)**](/blog/iceberg-delta-lake-delete-methods-comparison/#how-deletion-vectors-work-in-iceberg-v3) feature is pretty clever, it marks row-level changes without immediately rewriting entire files, which saves you from write amplification headaches. Databricks offers DVs as a default for any Delta tables. - -Iceberg takes a different approach with its **equality** and **position deletes** for V2. The new Format v3 introduces compact binary Deletion Vectors that reduce both read and write amplification, especially helpful for update-heavy tables. - -**OSS Delta Deletion Vectors Are Limited** - -Here's a key detail for folks using open-source Delta: Delta Lake's Deletion Vectors (DVs) are now broadly GA(general availability) on Databricks runtimes, whereas in open-source Delta (OSS) the capability is still "emerging": you can read DVs from 2.3.0 onward and perform DELETE, UPDATE, and MERGE with them as of 3.1, yet many connectors, libraries, and sharing tools still lack full read or write-side support. - -### Query Planning at Scale - -Apache Iceberg leverages a **hierarchical metadata model** using manifest lists, which efficiently summarize partitions and files. This allows the query planner to skip costly object storage scans, enabling fast query planning even across millions of files, a major performance win for large-scale datasets. - -Delta Lake, on the other hand, uses a **transaction log** with **periodic checkpoints**. When maintained properly, it performs well, but planning might slow down for very large tables if checkpointing and optimize/vacuum routines aren't regularly performed. Maintaining a healthy checkpoint cadence is key to ensuring consistent performance at scale. - -### Partitioning, Clustering & Data Skipping - -Partitioning plays a key role in Delta Lake performance. Recently, Delta has introduced **Liquid Clustering** as a smarter alternative to static partitions and Z-ORDER. It allows redefining clustering keys without rewriting existing data, a big win, and it has shown strong performance gains in real-world use cases. - -That said, Iceberg supports clustering too, but with more flexibility. It offers partition transforms and true partition evolution, so you can modify partition strategies over time without rewriting historical data. Additionally, Iceberg supports **Z-ordering** and file sort optimization via the **rewrite_data_files** action. - -![Data file management for big data analytics showing partitioning, clustering, and optimization with indexing and caching](/img/blog/2025/08/partitioning-chart.webp) - -**TAKEAWAY?** - -With proper layout, both formats can saturate compute for large scan‑heavy analytics. Delta often excels in update‑heavy streaming/near‑real‑time scenarios (COW + DVs), while Iceberg shines at massive partition/file counts thanks to its **planning model** and **partition evolution**. - -## Ecosystem Integration: Where the Real Differences Show - -This is probably the most important section for your decision-making, especially when we look at those limitations on support with different engines. - -### Multi-Engine Support & Performance - -Iceberg's multi-engine integration is an industry benchmark - no other format matches its breadth and maturity: - -- **Spark & Flink**: Native readers and writers deliver full support for streaming and batch, time travel, partition evolution, and schema flexibility. -- **Trino/Presto**: Iceberg is perfectly integrated and widely used for federated ad-hoc analytics, supporting advanced features out-of-the-box. -- **ClickHouse & Doris**: Production-grade readers let you tap into Iceberg tables for ultra-fast OLAP and analytics, with support for key features like partition pruning and time travel. -- **Hive/Impala, Dremio, DuckDB, StarRocks, BigQuery, Snowflake, Redshift**: Extensive catalog and metadata support enables you to query and govern your Iceberg tables across almost any environment, cloud, or workflows. - -Delta Lake offers compatibility with engines like Spark (where it excels), Flink (Maturing), Trino (Matured), and read-only for: Presto, Hive, BigQuery, Athena, and others. It's added features such as Delta Standalone Reader and Delta UniForm to enhance cross-platform access. However, advanced features and the smoothest experience (Ex, Deletion vectors, enterprise level governance) are generally still found within Spark and Databricks environments. - -However its cross-query-engine support isn't as universally mature or **engine-neutral** as Iceberg. - -### Cloud Services Integration - -**AWS:** -Athena natively queries (Read-Write-Optimise) Iceberg. Amazon S3 Tables offer **fully managed** Iceberg with automatic optimization, maintenance, and **REST Catalog APIs** for wide tool integration. - -**Azure:** -Synapse Analytics and **Microsoft Fabric** recently started the support for Iceberg. **Azure Data Factory** can write in Iceberg format to Data Lake Storage Gen2. **Azure Databricks** supports managed Iceberg and hybrid scenarios with UniForm (currently at a nascent level). It has out of the box matured support for Delta. - -**Snowflake:** -Provides managed Iceberg tables with choice of internal or external catalogs. Lifecycle maintenance is automated with Snowflake-managed mode, while external catalogs allow custom integration. - -**Google Cloud:** -BigQuery BigLake delivers managed Iceberg tables, with open-query-engine access and full support for **mutations, schema evolution, and streaming**. Also BQ can read external Iceberg tables as well. - -**Databricks:** -Offers managed Delta and Iceberg tables (Nascent append-only & Databricks managed only) across all clouds - strongest on the Databricks platform. - -Check out query-engine support matrix [(here)](https://olake.io/iceberg/query-engine) - -![Iceberg integration services with AWS, Azure, Google Cloud, S3, BigQuery, Microsoft Fabric, Synapse, and Databricks logos](/img/blog/2025/08/cloud-service-image.webp) - -### Catalogs & Governance - -Catalogs are like the brain (metadata-management + ACID) for lakehouses and its ecosystem is evolving fast. [**Apache Polaris**](/blog/apache-polaris-lakehouse) (incubating) now unifies Iceberg and Delta Lake tables in one open-source catalog, delivering vendor-neutral management and robust RBAC governance across major query engines. - -REST-based options like **Polaris, Gravitino, Lakekeeper, and Nessie** make Iceberg highly flexible; you can connect multiple warehouses and tools while maintaining a single table format, making multi-tool architectures easy and future-proof if **vendor neutrality** matters to you (you can avoid being locked-in into one single vendor and take ownership of cost, tools, performance in your own hands.) - -In a recent move, Databricks open-sourced the **Unity Catalog** and its APIs, a major move that now lets you manage Delta (and, increasingly, Iceberg) tables beyond the Databricks platform. It's a step toward making Delta less proprietary and more accessible. However, many top optimizations and enterprise features are still exclusive to the managed Databricks offering, and current Unity Catalog support for Iceberg remains somewhat limited (e.g., append-only operations). - -**Bottom line**: For organizations wanting maximum optionality, Iceberg's open REST catalog ecosystem stands out for governance and seamless integration across any engine. Delta Lake and Unity Catalog deliver the strongest, most turnkey experience within Databricks, but outside the platform, cross-format features and true parity are still catching up. - -## Enterprise Features: The Compliance & Security Story - -Both formats handle the enterprise requirements well, but with different approaches: - -- **ACID & Time Travel**: Both provide snapshot isolation and "as of" queries for audits and reproducibility. This is crucial when you need to prove your ML model was trained on specific data for regulatory compliance. -- **Deletes/Updates for GDPR/CCPA**: Delta uses COW + Deletion Vectors, while Iceberg provides equality/position deletes and v3 Deletion Vectors. Both can handle right-to-be-forgotten requests, though the implementation differs. -- **Schema Evolution**: Iceberg's stable column IDs enable safe rename/reorder/type-widening operations. Delta supports metadata-only renames and drops via column mapping. - -## Real-World Cost Impact: The Numbers Don't Lie - -Here's where things get really interesting from a business perspective. **DoorDash's** migration to Iceberg provides some compelling real-world data: - -- **25-49% storage cost reduction** compared to their original Snowflake architecture using just default ZSTD compression [(ref)](https://www.youtube.com/watch?v=_nnNHC90nMI&feature=youtu.be) -- **40-70% compute cost reduction** compared to fully Snowflake based setup due to better resource utilization & reduction of 1 data layer flink-raw & snowflake (snowpipe) raw and combined into 1. -- **Millions of dollars saved** from migrating their highest volume events to Iceberg format pipelines. - -The cost savings were so significant that they could completely remove expensive Snowflake-native storage and ingestion costs while maintaining query capabilities through External Iceberg tables in Snowflake. - -## The Bottom Line: When to Choose What - -**Go with Apache Iceberg if you want:** -- Maximum vendor neutrality and flexibility to use Spark, Flink, Trino, BigQuery, Snowflake, or Athena over a single table format. -- Cost savings on query and storage (e.g., DoorDash observed up to 40% lower costs after moving from Snowflake-native-storage-format to Iceberg). -- Efficient handling of large datasets with many partitions or evolving access patterns. -- Fine-grained deletes at scale and robust, open governance options. - -**Go with Delta Lake if you want:** -- Turnkey automation for Spark-centric, update-heavy pipelines. -- Deepest integration and fully managed enterprise governance within the Databricks ecosystem. -- The most advanced support for streaming workloads and real-time analytics. - -## Operational Reality Check - -**Setup Complexity:** - -**Iceberg:** -Leverage Hive, AWS Glue, or open REST catalogs like **Polaris** and **Lakekeeper** for seamless multi-engine coordination. Engines connect using standard connectors and interact with tables via standard SQL, making it easy to integrate. Currently, you can easily start by opting for **Ingestion** (OLake, Debezium-Kafka, or Flink), AWS S3 tables (Polaris/Lakekeeper), and any query engine/warehouse (Athena, Trino, BigQuery, Snowflake, or even Databricks-Spark). - -**Delta:** -Basic tables can be used path-based right away with **Spark**. For enterprise features and cross-engine governance, tables are registered in a metastore or catalog such as Unity Catalog, which enables enhanced management and security within supported platforms. The Databricks gives you end-to-end managed Delta tables. - -**Table Maintenance:** - -If you are planning to not use managed offering like S3-tables or Databricks: - -**Iceberg**: Schedule expire_snapshots and rewrite data/delete files jobs based on your workload patterns. Schedule compaction jobs manually based on how often your data lands and how your workloads hit those tables. - -**Delta**: Schedule OPTIMIZE (compaction) and VACUUM (file Garbage Collection) based on your workload patterns; maintain checkpoints. - -**Managed Options:** - -**Iceberg:** -Amazon S3 Tables offer fully managed Iceberg tables with built-in background maintenance (compaction, optimization), REST-catalog APIs, and seamless multi-engine access (Athena, EMR, Trino, and more). **Dremio** is also one of the fully managed platforms and **ryft.io** supports Iceberg compaction and snapshot lifecycle management. Google Cloud now provides managed Iceberg tables in BigQuery BigLake, and Snowflake also supports managed Iceberg; however, both impose notable limitations on external querying/access. - -**Delta:** -Databricks provides fully managed Delta tables with deep integration across its own Spark/Photon compute engine and the Databricks ecosystem; cross-engine access is possible via newer standards, but the richest features and optimizations remain on Databricks itself. - +--- +slug: apache-iceberg-vs-delta-lake-guide +title: "Apache Iceberg vs Delta Lake: Ultimate Guide for Data Lakes" +description: "Explore the key differences between Apache Iceberg and Delta Lake for batch analytics, ML pipelines, and cost-effective data lake management." +image: /img/blog/cover/iceberg-delta-comparison-cover.webp +authors: [shubham] +tags: [iceberg, delta-lake, data-lakehouse, analytics, ml] +--- + +![Apache Iceberg vs Delta Lake comparison architecture for batch analytics and ML](/img/blog/cover/iceberg-delta-comparison-cover.webp) + +# Apache Iceberg vs Delta Lake: Comparison for Batch Analytics & ML Pipelines + +You've probably heard the buzz around Apache Iceberg and Delta Lake. If you're deep into big data, picking the right one can be tricky. So let's skip the hype and focus on what really matters for batch analytics and ML pipelines. + +Open table formats are the game-changer for data lakes - they bring ACID transactions, time travel, and schema evolution to cheap object storage. That's warehouse-grade power on a lakehouse budget. + +## What This Comparison Is About + +We're zooming in on use cases like ETL pipelines, feature engineering, and reproducible training runs where consistent, reliable data matters most. + +Here's why formats like Iceberg and Delta shine in these workflows: + +- **Time travel & snapshots**: Train models from a specific version, enabling reproducibility that truly works. +- **ACID operations**: MERGE, UPDATE, DELETE, essential for managing feature tables and late-arriving data. +- **Schema evolution**: Add or update fields without rewriting history, because nobody's got time for that. + +![Lakehouse architecture diagram highlighting ACID transactions, time travel, schema evolution, data ingestion, metadata management, storage, governance, and query engine](/img/blog/2025/08/architecture-comparison.webp) + +## Performance & Scalability: Where the Rubber Meets the Road + +Performance is often the deciding factor and rightly so. The good news? Both formats are built to handle large-scale datasets efficiently, though each has its own strengths depending on the workload. + +### File Layout & Updates + +Delta Lake uses a **copy-on-write** approach by default for the open-source version. When you need to update data, it creates new files and marks the old ones for deletion. The new [**Deletion Vectors (DVs)**](/blog/iceberg-delta-lake-delete-methods-comparison/#how-deletion-vectors-work-in-iceberg-v3) feature is pretty clever, it marks row-level changes without immediately rewriting entire files, which saves you from write amplification headaches. Databricks offers DVs as a default for any Delta tables. + +Iceberg takes a different approach with its **equality** and **position deletes** for V2. The new Format v3 introduces compact binary Deletion Vectors that reduce both read and write amplification, especially helpful for update-heavy tables. + +**OSS Delta Deletion Vectors Are Limited** + +Here's a key detail for folks using open-source Delta: Delta Lake's Deletion Vectors (DVs) are now broadly GA(general availability) on Databricks runtimes, whereas in open-source Delta (OSS) the capability is still "emerging": you can read DVs from 2.3.0 onward and perform DELETE, UPDATE, and MERGE with them as of 3.1, yet many connectors, libraries, and sharing tools still lack full read or write-side support. + +### Query Planning at Scale + +Apache Iceberg leverages a **hierarchical metadata model** using manifest lists, which efficiently summarize partitions and files. This allows the query planner to skip costly object storage scans, enabling fast query planning even across millions of files, a major performance win for large-scale datasets. + +Delta Lake, on the other hand, uses a **transaction log** with **periodic checkpoints**. When maintained properly, it performs well, but planning might slow down for very large tables if checkpointing and optimize/vacuum routines aren't regularly performed. Maintaining a healthy checkpoint cadence is key to ensuring consistent performance at scale. + +### Partitioning, Clustering & Data Skipping + +Partitioning plays a key role in Delta Lake performance. Recently, Delta has introduced **Liquid Clustering** as a smarter alternative to static partitions and Z-ORDER. It allows redefining clustering keys without rewriting existing data, a big win, and it has shown strong performance gains in real-world use cases. + +That said, Iceberg supports clustering too, but with more flexibility. It offers partition transforms and true partition evolution, so you can modify partition strategies over time without rewriting historical data. Additionally, Iceberg supports **Z-ordering** and file sort optimization via the **rewrite_data_files** action. + +![Data file management for big data analytics showing partitioning, clustering, and optimization with indexing and caching](/img/blog/2025/08/partitioning-chart.webp) + +**TAKEAWAY?** + +With proper layout, both formats can saturate compute for large scan‑heavy analytics. Delta often excels in update‑heavy streaming/near‑real‑time scenarios (COW + DVs), while Iceberg shines at massive partition/file counts thanks to its **planning model** and **partition evolution**. + +## Ecosystem Integration: Where the Real Differences Show + +This is probably the most important section for your decision-making, especially when we look at those limitations on support with different engines. + +### Multi-Engine Support & Performance + +Iceberg's multi-engine integration is an industry benchmark - no other format matches its breadth and maturity: + +- **Spark & Flink**: Native readers and writers deliver full support for streaming and batch, time travel, partition evolution, and schema flexibility. +- **Trino/Presto**: Iceberg is perfectly integrated and widely used for federated ad-hoc analytics, supporting advanced features out-of-the-box. +- **ClickHouse & Doris**: Production-grade readers let you tap into Iceberg tables for ultra-fast OLAP and analytics, with support for key features like partition pruning and time travel. +- **Hive/Impala, Dremio, DuckDB, StarRocks, BigQuery, Snowflake, Redshift**: Extensive catalog and metadata support enables you to query and govern your Iceberg tables across almost any environment, cloud, or workflows. + +Delta Lake offers compatibility with engines like Spark (where it excels), Flink (Maturing), Trino (Matured), and read-only for: Presto, Hive, BigQuery, Athena, and others. It's added features such as Delta Standalone Reader and Delta UniForm to enhance cross-platform access. However, advanced features and the smoothest experience (Ex, Deletion vectors, enterprise level governance) are generally still found within Spark and Databricks environments. + +However its cross-query-engine support isn't as universally mature or **engine-neutral** as Iceberg. + +### Cloud Services Integration + +**AWS:** +Athena natively queries (Read-Write-Optimise) Iceberg. Amazon S3 Tables offer **fully managed** Iceberg with automatic optimization, maintenance, and **REST Catalog APIs** for wide tool integration. + +**Azure:** +Synapse Analytics and **Microsoft Fabric** recently started the support for Iceberg. **Azure Data Factory** can write in Iceberg format to Data Lake Storage Gen2. **Azure Databricks** supports managed Iceberg and hybrid scenarios with UniForm (currently at a nascent level). It has out of the box matured support for Delta. + +**Snowflake:** +Provides managed Iceberg tables with choice of internal or external catalogs. Lifecycle maintenance is automated with Snowflake-managed mode, while external catalogs allow custom integration. + +**Google Cloud:** +BigQuery BigLake delivers managed Iceberg tables, with open-query-engine access and full support for **mutations, schema evolution, and streaming**. Also BQ can read external Iceberg tables as well. + +**Databricks:** +Offers managed Delta and Iceberg tables (Nascent append-only & Databricks managed only) across all clouds - strongest on the Databricks platform. + +Check out query-engine support matrix [(here)](https://olake.io/iceberg/query-engine) + +![Iceberg integration services with AWS, Azure, Google Cloud, S3, BigQuery, Microsoft Fabric, Synapse, and Databricks logos](/img/blog/2025/08/cloud-service-image.webp) + +### Catalogs & Governance + +Catalogs are like the brain (metadata-management + ACID) for lakehouses and its ecosystem is evolving fast. [**Apache Polaris**](/blog/apache-polaris-lakehouse) (incubating) now unifies Iceberg and Delta Lake tables in one open-source catalog, delivering vendor-neutral management and robust RBAC governance across major query engines. + +REST-based options like **Polaris, Gravitino, Lakekeeper, and Nessie** make Iceberg highly flexible; you can connect multiple warehouses and tools while maintaining a single table format, making multi-tool architectures easy and future-proof if **vendor neutrality** matters to you (you can avoid being locked-in into one single vendor and take ownership of cost, tools, performance in your own hands.) + +In a recent move, Databricks open-sourced the **Unity Catalog** and its APIs, a major move that now lets you manage Delta (and, increasingly, Iceberg) tables beyond the Databricks platform. It's a step toward making Delta less proprietary and more accessible. However, many top optimizations and enterprise features are still exclusive to the managed Databricks offering, and current Unity Catalog support for Iceberg remains somewhat limited (e.g., append-only operations). + +**Bottom line**: For organizations wanting maximum optionality, Iceberg's open REST catalog ecosystem stands out for governance and seamless integration across any engine. Delta Lake and Unity Catalog deliver the strongest, most turnkey experience within Databricks, but outside the platform, cross-format features and true parity are still catching up. + +## Enterprise Features: The Compliance & Security Story + +Both formats handle the enterprise requirements well, but with different approaches: + +- **ACID & Time Travel**: Both provide snapshot isolation and "as of" queries for audits and reproducibility. This is crucial when you need to prove your ML model was trained on specific data for regulatory compliance. +- **Deletes/Updates for GDPR/CCPA**: Delta uses COW + Deletion Vectors, while Iceberg provides equality/position deletes and v3 Deletion Vectors. Both can handle right-to-be-forgotten requests, though the implementation differs. +- **Schema Evolution**: Iceberg's stable column IDs enable safe rename/reorder/type-widening operations. Delta supports metadata-only renames and drops via column mapping. + +## Real-World Cost Impact: The Numbers Don't Lie + +Here's where things get really interesting from a business perspective. **DoorDash's** migration to Iceberg provides some compelling real-world data: + +- **25-49% storage cost reduction** compared to their original Snowflake architecture using just default ZSTD compression [(ref)](https://www.youtube.com/watch?v=_nnNHC90nMI&feature=youtu.be) +- **40-70% compute cost reduction** compared to fully Snowflake based setup due to better resource utilization & reduction of 1 data layer flink-raw & snowflake (snowpipe) raw and combined into 1. +- **Millions of dollars saved** from migrating their highest volume events to Iceberg format pipelines. + +The cost savings were so significant that they could completely remove expensive Snowflake-native storage and ingestion costs while maintaining query capabilities through External Iceberg tables in Snowflake. + +## The Bottom Line: When to Choose What + +**Go with Apache Iceberg if you want:** +- Maximum vendor neutrality and flexibility to use Spark, Flink, Trino, BigQuery, Snowflake, or Athena over a single table format. +- Cost savings on query and storage (e.g., DoorDash observed up to 40% lower costs after moving from Snowflake-native-storage-format to Iceberg). +- Efficient handling of large datasets with many partitions or evolving access patterns. +- Fine-grained deletes at scale and robust, open governance options. + +**Go with Delta Lake if you want:** +- Turnkey automation for Spark-centric, update-heavy pipelines. +- Deepest integration and fully managed enterprise governance within the Databricks ecosystem. +- The most advanced support for streaming workloads and real-time analytics. + +## Operational Reality Check + +**Setup Complexity:** + +**Iceberg:** +Leverage Hive, AWS Glue, or open REST catalogs like **Polaris** and **Lakekeeper** for seamless multi-engine coordination. Engines connect using standard connectors and interact with tables via standard SQL, making it easy to integrate. Currently, you can easily start by opting for **Ingestion** (OLake Go, Debezium-Kafka, or Flink), AWS S3 tables (Polaris/Lakekeeper), and any query engine/warehouse (Athena, Trino, BigQuery, Snowflake, or even Databricks-Spark). + +**Delta:** +Basic tables can be used path-based right away with **Spark**. For enterprise features and cross-engine governance, tables are registered in a metastore or catalog such as Unity Catalog, which enables enhanced management and security within supported platforms. The Databricks gives you end-to-end managed Delta tables. + +**Table Maintenance:** + +If you are planning to not use managed offering like S3-tables or Databricks: + +**Iceberg**: Schedule expire_snapshots and rewrite data/delete files jobs based on your workload patterns. Schedule compaction jobs manually based on how often your data lands and how your workloads hit those tables. + +**Delta**: Schedule OPTIMIZE (compaction) and VACUUM (file Garbage Collection) based on your workload patterns; maintain checkpoints. + +**Managed Options:** + +**Iceberg:** +Amazon S3 Tables offer fully managed Iceberg tables with built-in background maintenance (compaction, optimization), REST-catalog APIs, and seamless multi-engine access (Athena, EMR, Trino, and more). **Dremio** is also one of the fully managed platforms and **ryft.io** supports Iceberg compaction and snapshot lifecycle management. Google Cloud now provides managed Iceberg tables in BigQuery BigLake, and Snowflake also supports managed Iceberg; however, both impose notable limitations on external querying/access. + +**Delta:** +Databricks provides fully managed Delta tables with deep integration across its own Spark/Photon compute engine and the Databricks ecosystem; cross-engine access is possible via newer standards, but the richest features and optimizations remain on Databricks itself. + +## FAQs + + +

    Apache Iceberg is an engine-agnostic open table format that works natively with Spark, Trino, Flink, DuckDB, Snowflake, and more without requiring any specific vendor. Delta Lake was created by Databricks with the deepest integration in the Databricks and Spark ecosystem.

    +

    Key architectural differences:

    +
      +
    • Iceberg uses a hierarchical metadata model with manifest lists that scales better for tables with millions of files - query planners skip directly to relevant files without scanning object storage
      • +
    • Delta Lake uses a transaction log with periodic checkpoints that excels in update-heavy Spark streaming workloads when OPTIMIZE and VACUUM routines are maintained regularly
      • +
    + + }, + { + question: "Q2. Which open table format is better for batch analytics and ML pipelines?", + answer:
    +

    Both handle large-scale analytics well, but Iceberg offers advantages for batch and ML workloads:

    +
      +
    • Hierarchical metadata enables fast query planning across millions of files without full log scans
      • +
    • True partition evolution without data rewrites - partition schemes can change without rewriting existing data files
      • +
    • Broader engine compatibility for reproducing ML training datasets consistently across Spark, Trino, and DuckDB
      • +
    +

    Delta Lake's Deletion Vectors and Liquid Clustering are competitive for update-heavy near-real-time scenarios, particularly within the Databricks ecosystem.

    +
    + }, + { + question: "Q3. How does Apache Iceberg's query planning differ from Delta Lake?", + answer:
    +

    Iceberg uses a hierarchical metadata model with manifest lists that summarize partitions and files. Query planners skip object storage scans entirely and jump directly to relevant files using the manifest metadata - this scales efficiently even for tables with billions of rows across millions of files.

    +

    Delta Lake uses a transaction log with periodic checkpoints. This is effective when maintained properly, but query performance can degrade for very large tables if OPTIMIZE and VACUUM routines are not run regularly, as the transaction log grows and checkpoint reads become expensive.

    +
    + }, + { + question: "Q4. What are Deletion Vectors in Delta Lake and how do they compare to Iceberg's delete methods?", + answer:
    +

    Delta Lake Deletion Vectors mark row-level changes without rewriting entire files, available in the Databricks runtime and progressively rolling out to open-source Delta.

    +

    Apache Iceberg handles row-level deletes differently across versions:

    +
      +
    • Iceberg v2 - Uses position delete files (identify deleted rows by file location and row position) and equality delete files (identify rows by column value) for a Merge-on-Read approach
      • +
    • Iceberg v3 - Introduces compact binary Deletion Vectors that reduce both read and write amplification. Importantly, position delete files are deprecated in v3 - tables may retain existing position deletes but must not add new ones. There can be at most one deletion vector per data file in a snapshot.
      • +
    +

    Convergence note: Delta Lake and Iceberg v3 Deletion Vectors use compatible binary encodings - Databricks actively contributed Deletion Vectors to the Iceberg v3 specification specifically for cross-format interoperability. These formats are converging rather than diverging on this feature. Engine support for Iceberg v3 DVs is still rolling out - verify your specific engine's v3 support before adopting v3 tables in production.

    +
    + }, + { + question: "Q5. Which format offers better multi-engine support for open lakehouse architectures?", + answer:
    +

    Apache Iceberg is the stronger choice for multi-engine lakehouses. It is natively supported by Spark, Trino, Flink, DuckDB, Dremio, Snowflake, ClickHouse, Apache Doris, and Presto across Iceberg v1 and v2 features.

    +

    Delta Lake has its strongest support in Databricks and Spark, with other engines relying on connector implementations that may lag behind in feature support.

    +

    v3 engine support caveat: While Iceberg v1 and v2 features enjoy broad engine support across all listed engines, Iceberg v3 features - including Deletion Vectors - are still rolling out and are not yet fully supported by all engines. Verify your specific engine's v3 support status before adopting v3 tables in production pipelines.

    +

    If avoiding vendor lock-in is a priority, Iceberg's fully open governance model makes it the safer long-term choice for multi-engine architectures.

    +
    + } +]} /> + \ No newline at end of file diff --git a/blog/2025-08-12-building-open-data-lakehouse-from-scratch.mdx b/blog/2025-08-12-building-open-data-lakehouse-from-scratch.mdx index 47fb4204b..2364f7633 100644 --- a/blog/2025-08-12-building-open-data-lakehouse-from-scratch.mdx +++ b/blog/2025-08-12-building-open-data-lakehouse-from-scratch.mdx @@ -11,9 +11,9 @@ import YouTubeEmbed from '@site/src/components/webinars/YouTubeEmbed'; ![Build Open Data Lakehouse from Scratch: MySQL, Presto, MinIO, and OLake open source stack logos](/img/blog/cover/open-data-lakehouse-cover.webp) -# Building a Complete Open Data Lakehouse from Scratch with MySQL, OLake, PrestoDB and MinIO +# Building a Complete Open Data Lakehouse from Scratch with MySQL, OLake Go, PrestoDB and MinIO -Well, if you're looking to dive into the exciting world of modern data architecture - the lakehouse, you've come to the right place! Today we're building a complete open data lakehouse from scratch using MySQL, OLake, PrestoDB and MiniO. And the best part? We'll have it running on your local machine in just a few steps. +Well, if you're looking to dive into the exciting world of modern data architecture - the lakehouse, you've come to the right place! Today we're building a complete open data lakehouse from scratch using MySQL, OLake Go, PrestoDB and MiniO. And the best part? We'll have it running on your local machine in just a few steps. Now, you might be wondering, "What exactly is a data lakehouse?" @@ -24,7 +24,7 @@ Well, it's essentially the best of both worlds - combining the flexibility and s For this setup, we're going to orchestrate four key components that work together seamlessly: - **MySQL** - Our source database where all the transactional data lives -- **OLake** - The star of our ETL show, handling data replication +- **OLake Go** - The star of our ETL show, handling data replication - **MinIO** - Our S3-compatible object storage acting as the data lake - [**PrestoDB**](/iceberg/query-engine/presto) - The lightning-fast query engine for analytics @@ -34,11 +34,11 @@ What makes this architecture particularly elegant is how these components commun ## How is the data flowing? -Here's where things get really interesting. Unlike traditional ETL pipelines that require complex Kafka, Debezium setups or batch processing windows, our architecture provides data replication without them. OLake captures changes from MySQL using Change Data Capture (CDC) and streams them directly into Iceberg tables stored in MinIO. Then PrestoDB can query this data instantly with sub-second latency. +Here's where things get really interesting. Unlike traditional ETL pipelines that require complex Kafka, Debezium setups or batch processing windows, our architecture provides data replication without them. OLake Go captures changes from MySQL using Change Data Capture (CDC) and streams them directly into Iceberg tables stored in MinIO. Then PrestoDB can query this data instantly with sub-second latency. -## Step 1: Setting Up OLake - CDC Engine +## Step 1: Setting Up OLake Go - CDC Engine -Olake has one of its unique offerings the OLake UI, which we will be using for our setup. This is a user-friendly control center for managing data pipelines without relying heavily on CLI commands. It allows you to configure sources, destinations, and jobs visually, making the setup more accessible and less error-prone. Many organizations actively use OLake UI to reduce manual CLI work, streamline CDC pipelines, and adopt a no-code-friendly approach. +Olake Go has one of its unique offerings the OLake UI, which we will be using for our setup. This is a user-friendly control center for managing data pipelines without relying heavily on CLI commands. It allows you to configure sources, destinations, and jobs visually, making the setup more accessible and less error-prone. Many organizations actively use OLake UI to reduce manual CLI work, streamline CDC pipelines, and adopt a no-code-friendly approach. For our setup, we will be working with the OLake UI. We'll start by cloning the repository from GitHub and bringing it up using Docker Compose. Once the UI is running, it will serve as our control hub for creating and monitoring all CDC pipelines. @@ -65,7 +65,7 @@ Make sure to run this command in your terminal so it saves your file location fo export PWD=$(pwd) ``` -The OLake UI docker-compose file uses `${PWD}/olake-data` as the host persistence path. This means all your OLake configurations, job states, and metadata will be saved to an `olake-data` folder in your current directory. Well, that's exactly what we want - persistent storage that survives container restarts! +The OLake UI docker-compose file uses `${PWD}/olake-data` as the host persistence path. This means all your OLake Go configurations, job states, and metadata will be saved to an `olake-data` folder in your current directory. Well, that's exactly what we want - persistent storage that survives container restarts! Now let's fire up the OLake UI: @@ -321,7 +321,7 @@ Credentials to login are: Head over to new bucket on the left panel and create a bucket. Name it whatever you want we will go ahead with `mylakehousedata` in this demo. -## Step 6: Configure OLake for Data Replication +## Step 6: Configure OLake Go for Data Replication **Complete walkthrough video below** @@ -364,7 +364,7 @@ After completing step 1 configure the destination: -Final step is to Click 'Next' again to finalize the data ingestion process by assigning a suitable Job name. The beauty of OLake is its simplicity - what used to require complex Debezium configurations now takes just a few clicks through the UI. +Final step is to Click 'Next' again to finalize the data ingestion process by assigning a suitable Job name. The beauty of OLake Go is its simplicity - what used to require complex Debezium configurations now takes just a few clicks through the UI. ### Complete Video Walkthrough @@ -384,9 +384,9 @@ Well, the really exciting part is watching near real-time queries work. Try upda ## The Performance Story -What makes this setup particularly impressive is the performance characteristics we get. OLake's benchmarks show it can process over 46,000 rows per second, which is significantly faster than traditional ETL tools like Airbyte (457 rows/second) or Estuary (3,982 rows/second). +What makes this setup particularly impressive is the performance characteristics we get. OLake Go's benchmarks show it can process over 46,000 rows per second, which is significantly faster than traditional ETL tools like Airbyte (457 rows/second) or Estuary (3,982 rows/second). -This means your analytics workloads get fresher data with lower infrastructure costs. The combination of OLake's efficient CDC with PrestoDB's vectorized execution engine creates a seriously powerful analytics platform. +This means your analytics workloads get fresher data with lower infrastructure costs. The combination of OLake Go's efficient CDC with PrestoDB's vectorized execution engine creates a seriously powerful analytics platform. ## Monitoring and Observability @@ -406,9 +406,11 @@ With your lakehouse up and running, you can start exploring advanced features: - **Partitioning**: Optimize query performance with intelligent data partitioning - **Multiple Sources**: Add PostgreSQL, MongoDB, or other databases to your pipeline + + ## Wrapping Up -Building an open data lakehouse has never been this straightforward. With MySQL as our reliable source, OLake handling the heavy lifting of data replication, MinIO providing scalable storage, and PrestoDB delivering lightning-fast analytics, we've created a modern data platform that can scale with your needs. +Building an open data lakehouse has never been this straightforward. With MySQL as our reliable source, OLake Go handling the heavy lifting of data replication, MinIO providing scalable storage, and PrestoDB delivering lightning-fast analytics, we've created a modern data platform that can scale with your needs. **The best part?** Everything we've built uses open-source tools, giving you complete control over your data architecture without vendor lock-in. Whether you're a startup looking to build your first data platform or an enterprise seeking to modernize legacy systems, this lakehouse pattern provides a solid foundation for data-driven decision making. @@ -416,4 +418,63 @@ Well, there you have it - your very own open data lakehouse, running locally and Otherwise, you'd be stuck with traditional approaches that force you to choose between flexibility and performance. But with this modern lakehouse architecture, you get the best of both worlds and that's pretty exciting if you ask me! +## FAQs + +

    The stack consists of five components:

    +
      +
    1. MySQL - source operational database
      2. +
    2. OLake Go - CDC ingestion engine that captures changes from MySQL and writes them as Apache Iceberg tables
      3. +
    3. MinIO - S3-compatible object storage acting as the data lake, storing Parquet data files and Iceberg metadata
      4. +
    4. Iceberg REST Catalog - metadata layer that tracks table schemas, snapshot locations, and current table state
      5. +
    5. PrestoDB - distributed SQL query engine that connects to the REST catalog and queries data with sub-second latency using standard SQL
      6. +
    +

    OLake Go captures changes from MySQL using Change Data Capture and writes them as Apache Iceberg snapshots into MinIO. PrestoDB connects to the Iceberg REST catalog to discover tables and executes distributed queries directly against the Parquet files in MinIO.

    + + }, + { + question: "Q2. What is MinIO's role in a data lakehouse architecture?", + answer:
    +

    MinIO is an S3-compatible open-source object storage server that can run locally or in the cloud. In a data lakehouse, it serves as the cost-effective storage layer where both Parquet data files and Iceberg metadata are stored.

    +

    Because MinIO is fully S3-compatible, any tool that supports AWS S3 - including OLake Go, Trino, Spark, and DuckDB - works with MinIO out of the box, making it ideal for on-premise or self-hosted lakehouses without requiring cloud vendor dependencies.

    +
    + }, + { + question: "Q3. How does OLake Go's CDC pipeline from MySQL work without needing Kafka?", + answer:
    +

    OLake Go connects directly to MySQL's binary log (binlog) and reads change events without requiring Kafka brokers, ZooKeeper, or Kafka Connect. Changes are written in real time as Apache Iceberg snapshots to MinIO.

    +

    This eliminates the entire Debezium + Kafka + sink connector stack - reducing the number of infrastructure components from 5-6 down to 3 (OLake Go, MinIO, and an Iceberg REST Catalog), while maintaining ACID guarantees through Iceberg's atomic commit model.

    +
    + }, + { + question: "Q4. How do you query Apache Iceberg tables stored in MinIO using PrestoDB?", + answer:
    +

    Configure PrestoDB's Iceberg connector with the REST catalog endpoint URL and MinIO/S3 credentials:

    + 
    +        
+connector.name=iceberg
+iceberg.catalog.type=rest
+iceberg.rest.uri=http://iceberg-rest:8181
+hive.s3.endpoint=http://minio:9000
+        
+
    +

    PrestoDB reads the Iceberg REST catalog to discover table schemas and snapshot locations, then reads Parquet files directly from MinIO using its distributed parallel execution. Standard SQL queries including joins, aggregations, and time-travel syntax work immediately without any data movement or transformation.

    +
    + }, + { + question: "Q5. What advantages does an open-source lakehouse stack offer compared to managed cloud services?", + answer:
    +

    An open-source stack (OLake Go + MinIO + Iceberg REST Catalog + PrestoDB + Apache Iceberg) offers several key advantages:

    +
      +
    • No vendor lock-in - data is stored in open formats accessible by any engine
      • +
    • Lower storage costs - object storage (S3/MinIO) costs a fraction of proprietary data warehouse storage, while Iceberg's efficient metadata keeps query performance high
      • +
    • Component portability - swap any layer without migrating data (replace PrestoDB with Trino, replace MinIO with AWS S3) since all components speak the same open standards
      • +
    • Full governance control - you maintain complete control over data security, compliance, and access policies on your own infrastructure or any cloud
      • +
    +
    + } +]} /> + diff --git a/blog/2025-08-29-deploying-olake-on-kubernetes.mdx b/blog/2025-08-29-deploying-olake-on-kubernetes.mdx index 4a24a93df..33cefcd02 100644 --- a/blog/2025-08-29-deploying-olake-on-kubernetes.mdx +++ b/blog/2025-08-29-deploying-olake-on-kubernetes.mdx @@ -7,6 +7,8 @@ authors: [schitiz] tags: [kubernetes, helm, cdc, data-lakehouse, olake] --- +import CodeBlock from '@theme/CodeBlock'; + # Deploying OLake on Kubernetes with Helm ![Deploying OLake on Kubernetes using Helm chart, with OLake and Helm logos](/img/blog/cover/olake-on-kubernetes.webp) @@ -38,7 +40,7 @@ Six key services are deployed by the chart, which work in concert: ## The Mission: A Live Pipeline in 5 Minutes -First, the power of OLake can be demonstrated with a quick development setup. The goal is for the entire platform to be running in the cluster in the next few minutes. +First, the power of OLake Go can be demonstrated with a quick development setup. The goal is for the entire platform to be running in the cluster in the next few minutes. ### Prerequisites @@ -96,7 +98,7 @@ Now that the platform has been seen in action, some of the key design decisions ### More Than Just Storage: The Role of the Shared Volume in Pipeline Coordination -When OLake was first deployed, an NFS server was automatically provisioned. This is not just about storage—it's the coordination backbone by which the entire pipeline is made to work seamlessly. +When OLake Go was first deployed, an NFS server was automatically provisioned. This is not just about storage—it's the coordination backbone by which the entire pipeline is made to work seamlessly. It can be thought of in this way: job configurations need to be handed off from the OLake UI to workers, discovered schemas must be passed from workers to sync processes, and state needs to be coordinated by everyone. Without shared storage, these components would be like orchestra musicians attempting to play together while situated in different buildings. @@ -106,7 +108,7 @@ For a quick start, this has been made invisible—the built-in NFS server just w In production, the cloud provider's managed storage services should be leveraged. The durability, performance, and availability guarantees that are depended on by data pipelines are provided by these services. AWS EFS, Azure Files, and Google Cloud Filestore are all battle-tested, managed solutions that scale with needs and come with built-in redundancy. -The way OLake would be configured to use AWS EFS in production is shown here: +The way OLake Go would be configured to use AWS EFS in production is shown here: ```yaml # values.yaml @@ -118,9 +120,9 @@ nfsServer: name: "my-efs-pvc" ``` -The same pattern works for Azure Files or Google Cloud Filestore—the managed storage is pointed to, and the rest is handled by OLake. Enterprise-grade storage reliability is thus provided to the pipelines while the same simple operational model is kept. +The same pattern works for Azure Files or Google Cloud Filestore—the managed storage is pointed to, and the rest is handled by OLake Go. Enterprise-grade storage reliability is thus provided to the pipelines while the same simple operational model is kept. -This shared storage design doesn't just solve coordination—powerful operational capabilities are also enabled. If a failed pipeline needs to be debugged, the logs and state files are located right there in the shared volume. If a migration of the OLake deployment to a new cluster is desired, the entire pipeline state is moved with the storage. It is resilience through simplicity. +This shared storage design doesn't just solve coordination—powerful operational capabilities are also enabled. If a failed pipeline needs to be debugged, the logs and state files are located right there in the shared volume. If a migration of the OLake Go deployment to a new cluster is desired, the entire pipeline state is moved with the storage. It is resilience through simplicity. ## Escaping the 'Noisy Neighbor' Problem in Your Data Pipelines @@ -128,30 +130,30 @@ This scenario has been lived through by every data engineer: multiple heavy data This is known as the "noisy neighbor" problem, where resource-intensive workloads are impacted by each other when forced to share the same infrastructure. In traditional data pipeline setups, performance bottlenecks and unreliable sync times are the result, especially when large datasets are being processed. -This problem is solved intelligently by OLake for the operations that matter most: heavy data sync workloads. +This problem is solved intelligently by OLake Go for the operations that matter most: heavy data sync workloads. -When pods are created by OLake for sync operations, `podAntiAffinity` rules are automatically applied, by which these resource-intensive tasks are spread across different nodes in the cluster. It can be thought of as an intelligent traffic controller by which the heaviest data processing jobs are ensured not to interfere with each other. +When pods are created by OLake Go for sync operations, `podAntiAffinity` rules are automatically applied, by which these resource-intensive tasks are spread across different nodes in the cluster. It can be thought of as an intelligent traffic controller by which the heaviest data processing jobs are ensured not to interfere with each other. **What this means in practice:** - Consistent Sync Performance: Dedicated node resources are given to large table syncs, so completion is achieved in predictable timeframes without competition for CPU and memory with other sync operations. - Improved Reliability: A single node is prevented from becoming a bottleneck, so the risk of memory pressure or CPU saturation that could cause sync operations to fail or timeout is reduced. -For lightweight operations like discovery and testing, a different approach is taken by OLake—these quick, low-resource operations can be run anywhere without the need for special scheduling constraints. The system is thus kept efficient while the sophisticated scheduling logic is focused where the most value is provided. +For lightweight operations like discovery and testing, a different approach is taken by OLake Go—these quick, low-resource operations can be run anywhere without the need for special scheduling constraints. The system is thus kept efficient while the sophisticated scheduling logic is focused where the most value is provided. The result is a data pipeline architecture by which performance is automatically optimized, ensuring the most critical sync operations are given the resources needed to succeed. ## Precision Scheduling: A Guide to JobID-Based Node Mapping -Now that the handling of resource isolation by OLake is understood, the topic of precision can be addressed. Not all data operations are created equal—some require massive memory for processing large datasets, others benefit from high-CPU nodes for complex normalizations, and some can be run perfectly fine on cost-effective general-purpose instances. +Now that the handling of resource isolation by OLake Go is understood, the topic of precision can be addressed. Not all data operations are created equal—some require massive memory for processing large datasets, others benefit from high-CPU nodes for complex normalizations, and some can be run perfectly fine on cost-effective general-purpose instances. In traditional Kubernetes scheduling, all pods are treated in the same way, being spread randomly across available nodes. But what if a strategic approach could be taken regarding where different types of data work are actually run? -That is exactly what is delivered by OLake's JobID-based node mapping: the power for specific sync operations to be routed to the exact infrastructure that is needed for them to perform at their best. +That is exactly what is delivered by OLake Go's JobID-based node mapping: the power for specific sync operations to be routed to the exact infrastructure that is needed for them to perform at their best. ### How It Works -The concept is elegantly simple. A JobID is given to every data job in OLake—it can be thought of as a unique fingerprint for that particular operation. Through the Helm configuration, a mapping is created that tells OLake: "When JobID X is seen, it should be scheduled on nodes with these specific labels." +The concept is elegantly simple. A JobID is given to every data job in OLake Go—it can be thought of as a unique fingerprint for that particular operation. Through the Helm configuration, a mapping is created that tells OLake Go: "When JobID X is seen, it should be scheduled on nodes with these specific labels." ```yaml # values-production.yaml @@ -182,7 +184,7 @@ For new systems, no mappings are required. For growing enterprises with complex ### Configuration That Just Works -The beauty lies in the simplicity. A few lines are added to the Helm values, a redeployment is done, and jobs are automatically started to be routed by OLake according to the rules. No code changes, no complex rewrites, no architectural overhauls are needed. The intelligence is built into the platform. +The beauty lies in the simplicity. A few lines are added to the Helm values, a redeployment is done, and jobs are automatically started to be routed by OLake Go according to the rules. No code changes, no complex rewrites, no architectural overhauls are needed. The intelligence is built into the platform. This is precision scheduling in action: the right job, on the right infrastructure, at the right time. @@ -190,11 +192,11 @@ This is precision scheduling in action: the right job, on the right infrastructu It is Monday morning, and 20 different sync jobs have been scheduled to be run simultaneously. Customer data from PostgreSQL, product catalogs from MongoDB, transaction logs from MySQL—all are flowing into the data lake at the same time. In a traditional setup, this would be a recipe for chaos. Jobs would interfere with each other, failed workflows would disappear into the void, and everything would be restarted manually while coffee gets cold. -This complexity is handled by OLake through two powerful components working in perfect harmony: Temporal for rock-solid orchestration and OLake Workers for intelligent concurrency management. +This complexity is handled by OLake Go through two powerful components working in perfect harmony: Temporal for rock-solid orchestration and OLake Workers for intelligent concurrency management. ### Temporal: The Reliability Safety Net -When a data sync is kicked off in OLake, it is not just "run" by Temporal—a durable, recoverable workflow is created that persists through failure. If a sync crashes, it is remembered by Temporal where each job was, and a resumption from that exact point is performed on the next schedule of the job. +When a data sync is kicked off in OLake Go, it is not just "run" by Temporal—a durable, recoverable workflow is created that persists through failure. If a sync crashes, it is remembered by Temporal where each job was, and a resumption from that exact point is performed on the next schedule of the job. But what makes Temporal truly powerful for data pipelines is that complete visibility is provided into what is happening. Every step, every retry, every decision point is tracked. If it is needed to know why a sync job failed three days ago, the full execution history is right there on the OLake UI. @@ -210,5 +212,79 @@ The OLake Helm chart is more than a tool; it's a statement. It is believed that **Happy replicating!** +## FAQs + +

    Add the OLake Helm repository and run helm install to deploy the complete OLake stack to your Kubernetes cluster:

    + {`helm repo add datazip https://datazip-inc.github.io/olake +helm repo update +helm install olake olake/olake`} +

    The chart deploys all six required services: OLake UI, OLake Worker, Temporal workflow orchestrator, PostgreSQL for state storage, Elasticsearch for observability, and a shared NFS persistent volume for coordination between the UI and worker pods.

    +

    Once deployed, verify all pods are running:

    + {`kubectl get pods`} +

    You should see elasticsearch, olake-nfs-server, postgresql, temporal, olake-ui, and olake-worker all in Running state.

    +

    To access the OLake UI, forward the port:

    + {`kubectl port-forward svc/olake-ui 8080:8080 8000:8000`} +

    Then open http://localhost:8000 and log in with admin / password to create your first pipeline.

    + + }, + { + question: "Q2. What services does the OLake Helm chart deploy and what does each do?", + answer:
    +

    The OLake Helm chart deploys six services:

    +
      +
    1. OLake UI - Web interface and backend API for managing pipelines, viewing sync status, and browsing execution history
      2. +
    2. OLake Worker - Kubernetes-native engine that creates dedicated pods for each sync job rather than running jobs within itself
      3. +
    3. Temporal - Workflow orchestrator that guarantees reliable execution with retries, state persistence, and full execution history
      4. +
    4. PostgreSQL - Stores all OLake application state and Temporal workflow state
      5. +
    5. Elasticsearch - Provides search and observability into workflow executions, integrated with Temporal's visibility layer
      6. +
    6. Shared Storage (NFS server) - Persistent volume for coordination between the UI and worker pods. The chart bundles a development-grade NFS server for quickstart - replace with a ReadWriteMany-capable solution for production
      7. +
    +
    + }, + { + question: "Q3. What are the minimum requirements for deploying OLake Go on Kubernetes?", + answer:
    +
      +
    • Kubernetes 1.19 or later
      • +
    • Helm 3.2.0 or later
      • +
    • kubectl configured and connected to your target cluster
      • +
    • Default StorageClass defined in the cluster
      • +
    +

    For production deployments, a ReadWriteMany-capable storage solution (such as NFS, EFS, or Azure Files) is recommended to replace the development-grade NFS server bundled with the chart.

    +

    Temporal production note: Temporal's own team recommends using the Helm chart for templating and generating manifests rather than direct production deployment management. The bundled Elasticsearch, Cassandra, Prometheus, and Grafana in the chart are minimal development configurations. For production, replace the bundled Elasticsearch with an external managed instance and manage Temporal Server configuration separately.

    +
    + }, + { + question: "Q4. How does OLake Go's Kubernetes deployment handle pipeline failures and retries?", + answer:
    +

    OLake Go uses Temporal as its workflow orchestrator, which provides built-in retry logic, state persistence, and guaranteed execution. When a data sync is kicked off in OLake Go, a durable, recoverable workflow is created that persists through failure:

    +
      +
    • If a sync job fails mid-way, Temporal records the failure and automatically retries based on configurable retry policies
      • +
    • OLake Go uses checkpointing to remember progress within a sync, so retries resume from the last successful checkpoint rather than restarting the entire data load
      • +
    • Every step, retry, and decision point is tracked and visible in the OLake UI - full execution history is available for debugging failed jobs
      • +
    +
    + }, + { + question: "Q5. Can the OLake Helm chart be used in production or only for development?", + answer:
    +

    The chart is production-ready when configured correctly, with the following requirements for production hardening:

    +
      +
    • Replace the bundled NFS server with an external ReadWriteMany-capable storage solution (EFS, Azure Files, or a managed NFS)
      • +
    • Replace the bundled Elasticsearch with an external managed Elasticsearch instance - the chart's bundled Elasticsearch is explicitly a development-grade component and must be replaced, not merely configured, for production use
      • +
    • Use external managed PostgreSQL rather than the bundled instance for durable persistence
      • +
    • Set resource requests and limits on all pods to prevent noisy-neighbour issues
      • +
    • Configure Temporal Server separately - Temporal's own team recommends using the Helm chart for manifest generation only; the bundled databases and observability components are not production-grade
      • +
    +

    The Helm values file exposes all these configuration options for production hardening.

    +
    + } +]} /> + + + diff --git a/blog/2025-09-04-creating-job-olake-docker-cli.mdx b/blog/2025-09-04-creating-job-olake-docker-cli.mdx index f673695fe..6efa0b789 100644 --- a/blog/2025-09-04-creating-job-olake-docker-cli.mdx +++ b/blog/2025-09-04-creating-job-olake-docker-cli.mdx @@ -1,6 +1,6 @@ --- -title: "Create OLake Replication Jobs: Postgres to Iceberg Docker CLI" -description: "Step-by-step guide to creating OLake replication jobs via Docker CLI. Configure Postgres sources, Iceberg destinations, CDC, partitioning, and scheduling." +title: "Create OLake Go Replication Jobs: Postgres to Iceberg Docker CLI" +description: "Step-by-step guide to creating OLake Go replication jobs via Docker CLI. Configure Postgres sources, Iceberg destinations, CDC, partitioning, and scheduling." slug: creating-job-olake-docker-cli authors: [akshay, vishal] tags: [iceberg,replication] @@ -15,9 +15,9 @@ Data replication has become one of the most essential building blocks in modern Today, there's no shortage of options—platforms like Fivetran, Airbyte, Debezium, and even custom-built Flink or Spark pipelines are widely used to handle replication. But each of these comes with trade-offs: infrastructure complexity, cost, or lack of flexibility when you want to adapt replication to your specific needs. -That's where OLake comes in. Instead of forcing you into one way of working, OLake focuses on making replication into Apache Iceberg (and other destinations) straightforward, fast, and adaptable. You can choose between a guided UI experience for simplicity or a Docker CLI flow for automation and DevOps-style control. +That's where OLake Go comes in. Instead of forcing you into one way of working, OLake Go focuses on making replication into Apache Iceberg (and other destinations) straightforward, fast, and adaptable. You can choose between a guided UI experience for simplicity or a Docker CLI flow for automation and DevOps-style control. -In this blog, we'll walk through how to set up a replication job in OLake, step by step. We'll start with the UI wizard for those who prefer a visual setup, then move on to the CLI-based workflow for teams that like to keep things in code. By the end, you'll have a job that continuously replicates from [Postgres to Apache Iceberg (Glue Catalog)](/iceberg/postgres-to-iceberg-using-glue) with CDC, normalization, filters, partitioning, and scheduling—all running seamlessly. +In this blog, we'll walk through how to set up a replication job in OLake Go, step by step. We'll start with the UI wizard for those who prefer a visual setup, then move on to the CLI-based workflow for teams that like to keep things in code. By the end, you'll have a job that continuously replicates from [Postgres to Apache Iceberg (Glue Catalog)](/iceberg/postgres-to-iceberg-using-glue) with CDC, normalization, filters, partitioning, and scheduling—all running seamlessly. ## Two Setup Styles (pick what fits you) @@ -344,7 +344,7 @@ Confirm the data in your destination (S3 / Iceberg): ### 6) About the `state.json` (Resumable & CDC-friendly) -When a sync starts, OLake writes a `state.json` that tracks progress and CDC offsets (e.g., Postgres LSN). +When a sync starts, OLake Go writes a `state.json` that tracks progress and CDC offsets (e.g., Postgres LSN). This lets you **resume without duplicates** and continue CDC seamlessly. To resume / keep streaming: @@ -363,22 +363,106 @@ docker run --pull=always \ More details: Check out our Postgres connector documentation for state file configuration. ---- - -## Quick Q&A - -**UI or CLI—how should I choose?** -If you're new to OLake or prefer a guided setup, start with **UI**. -If you're automating, versioning configs, or scripting in CI, use **CLI**. - -**Why "Full Refresh + CDC"?** -You get a baseline snapshot *and* continuous changes—ideal for keeping downstream analytics fresh. -**Can I change partitioning later?** -* **UI**: unselect the stream → save → re-add with updated partitioning/filter/normalization. -* **CLI**: edit `streams.json` and re-run. +## FAQs ---- + +

    If you're new to OLake Go or prefer a guided setup, start with UI.

    +

    If you're automating, versioning configs, or scripting in CI, use CLI.

    + + }, + { + question: "Q2. Why Full Refresh + CDC?", + answer:
    +

    You get a baseline snapshot and continuous changes, ideal for keeping downstream analytics fresh.

    +
    + }, + { + question: "Q3. Can I change partitioning later?", + answer:
    +
      +
    • UI: Edit the partitioning on the stream directly. You can delete or update a partition without unselecting and re-adding the stream.
      • +
    • CLI: Edit streams.json and re-run.
      • +
    +
    + }, + { + question: "Q4. What prerequisites do I need before creating a Postgres-to-Iceberg replication job in OLake Go?", + answer:
    +

    You need:

    +
      +
    • Docker installed and running
      • +
    • PostgreSQL 10 or higher with WAL level set to logical for CDC
      • +
    • An Apache Iceberg catalog - AWS Glue, Hive, REST/Nessie, JDBC, or Polaris
      • +
    • An S3-compatible object store - AWS S3, MinIO, etc. - for data storage
      • +
    +

    For CDC specifically, a Postgres replication slot must exist before the connector starts.

    +

    Important: Each OLake Go job must have its own dedicated replication slot and publication. Never share a replication slot across multiple OLake Go jobs - doing so will result in data loss and inconsistencies.

    +

    If you cannot modify WAL settings (for example, on a read replica), OLake Go also supports JDBC-based Full Refresh and bookmark-based Incremental modes that work with standard credentials and do not require a replication slot.

    +
    + }, + { + question: "Q5. What sync modes does OLake Go support for Postgres replication?", + answer:
    +

    OLake Go supports four modes for Postgres:

    +
      +
    1. Full Refresh - Re-loads the entire table on every run. Best for small, infrequently changing tables.
      2. +
    2. Full Refresh + Incremental - Performs an initial full load of the table, then captures new rows using a cursor column. Which changes get captured depends on the cursor you pick: a monotonically increasing column like id captures inserts only, since updated rows keep their old cursor value and are skipped. For updates to be captured, the cursor must be a column that changes when the row changes, such as updated_at. Deletes are never captured in this mode.
      3. +
    3. Full Refresh + CDC - Performs an initial full backfill, then switches to real-time WAL-based CDC for all subsequent changes (inserts, updates, deletes). This is the most common production mode.
      4. +
    4. Strict CDC - Streams only changes from the current WAL position with no initial full load. Use when you only want forward changes from a specific point in time.
      5. +
    +
    + }, + { + question: "Q6. What does 'Normalization' do in OLake Go, and should I enable it?", + answer:
    +

    Normalization in OLake Go automatically expands level-0 nested JSON fields, meaning top-level JSON objects, into individual columns for easier querying. It preserves all data while simplifying structure and reduces the need for complex JSON parsing in downstream SQL queries. Flattening stops at that first level; anything nested deeper remains as a stringified value.

    +

    Normalization only controls JSON flattening. It does not control whether rows are appended or upserted, that is a separate Append/Upsert mode setting.

    +

    Enable normalization when your source tables contain nested JSON fields that you want to query as flat columns.

    +
    + }, + { + question: "Q7. How do I monitor sync progress when running OLake Go from the CLI?", + answer:
    +

    OLake Go writes a stats.json file alongside your config files during a sync run. It contains real-time metrics:

    + 
    +        
+{`{
+  "Estimated Remaining Time": "1642.00",
+  "Memory": "2228 mb",
+  "Running Threads": 21,
+  "Seconds Elapsed": "186.00",
+  "Speed": "76542.20 rps",
+  "Synced Records": 14236868
+}`}
+        
+
    +

    The stats.json file remains available after sync completion for post-run inspection.

    +

    Detailed logs are also written to:

    + 
    +        
+{`/path/to/config/logs/sync_[YYYY-MM-DD]_[HH-MM-SS]/olake.log`}
+        
+
    +

    For the UI, per-run logs are accessible via Jobs → Job Logs & History → View Logs.

    +
    + }, + { + question: "Q8. How do I change the filter or normalization settings for an existing stream?", + answer:
    +

    Most stream-level settings can be updated directly without removing and re-adding the stream:

    +
      +
    • Normalization: A toggle. Turn it on or off for the stream.
      • +
    • Filter: Edit the filter conditions on the stream directly.
      • +
    • Partitioning: See "Can I change partitioning later?" (Q3) above.
      • +
    +

    For the CLI, update streams.json directly with the new settings and re-run the sync command.

    +
    + }, +]} /> \ No newline at end of file diff --git a/blog/2025-09-04-deletion-formats-deep-dive.mdx b/blog/2025-09-04-deletion-formats-deep-dive.mdx index 527cf1da4..cbddcaf06 100644 --- a/blog/2025-09-04-deletion-formats-deep-dive.mdx +++ b/blog/2025-09-04-deletion-formats-deep-dive.mdx @@ -218,4 +218,93 @@ Let's bring this full circle with the key points you should remember: - **Start Simple, Scale Smart**: Begin with the default approaches (copy-on-write for batch workloads, deletion vectors for high-update scenarios) and optimize based on your actual performance characteristics and operational requirements. The world of data lake deletion formats might seem complex, but it's really about solving a fundamental problem: **how do you efficiently manage changing data at scale?** Apache Iceberg and Delta Lake have both arrived at elegant solutions that make this possible, each with their own strengths and ideal use cases. + +## FAQs + +

    Position delete files store the exact file path and row position (row number) of deleted records in a separate delete file. Equality delete files store the column values of deleted rows for example, all rows where customer_id equals 12345.

    +
      +
    • Position deletes - efficient to read (direct index lookup) but expensive to write if many files are affected
      • +
    • Equality deletes - fast to write during CDC but require scanning every row at read time to apply the deletion criteria
      • +
    +

    v3 deprecation note: Position delete files are a v2-specific mechanism. In Iceberg v3, position delete files must not be added to new tables - v3 tables must use deletion vectors instead. Existing position delete files written under v2 remain valid and readable.

    + + }, + { + question: "Q2. What is the difference between Merge-on-Read and Copy-on-Write deletion strategies?", + answer:
    +

    Copy-on-Write (COW) - when a row is deleted or updated, Iceberg rewrites the entire affected Parquet file with the change applied immediately. Reads are fast since there are no delete files to reconcile, but writes are expensive due to full file rewrites.

    +

    Merge-on-Read (MOR) - deletions are recorded as separate small delete files and merged with data files at query time. Writes are fast and cheap, but reads are slightly slower since the engine must merge delete information on every query.

    + + + + + + + + + + + + + + + + + + + + + + + + + +
    Copy-on-WriteMerge-on-Read
    Write costHigh (full file rewrite)Low (small delete file)
    Read costLow (no delete files)Slightly higher (merge at query time)
    Best forRead-heavy, infrequent updatesWrite-heavy, frequent CDC updates
    +
    + }, + { + question: "Q3. How do deletion vectors in Apache Iceberg v3 improve on v2 delete methods?", + answer:
    +

    Iceberg v3 introduces compact binary deletion vectors (DVs) stored in Puffin sidecar files - small companion files paired with each data file (e.g. file_A.parquet is paired with file_A.puffin). Each Puffin file contains a Roaring bitmap encoding the row positions of all deleted rows in the corresponding data file.

    +

    This is a significant improvement over v2's approach:

    +
      +
    • Eliminates accumulation of many small positional delete files across object storage - a common v2 pain point at scale
      • +
    • Reduces metadata complexity - multiple deletion vectors can be stored in a single Puffin file, lowering file count overhead
      • +
    • Faster scan filtering - bitmap-based lookups allow engines to skip deleted rows much faster than reconciling scattered delete files
      • +
    • Reduces both read and write amplification compared to v2's separate positional or equality delete files
      • +
    +

    Important: Deletion vectors are a v3-only feature. They are not supported in v2 or earlier tables. Once a table is upgraded to use DVs, clients that do not support v3 will be unable to read it.

    +
    + }, + { + question: "Q4. How does Delta Lake handle row-level deletions compared to Apache Iceberg?", + answer:
    +

    Delta Lake defaults to Copy-on-Write for the open-source version - rewriting affected files on DELETE, UPDATE, or MERGE. Databricks-managed Delta introduced Deletion Vectors that mark row-level changes without full file rewrites, similar to Iceberg v3's approach.

    +

    Notably, Iceberg v3 DVs and Delta Lake DVs use compatible binary encodings - Databricks actively contributed deletion vectors to the Iceberg v3 specification specifically for cross-format interoperability. The two formats are converging on this feature rather than diverging.

    +

    Open-source Delta's DV support is still maturing with some connectors lacking full read/write support, whereas Iceberg's delete strategies have broader ecosystem compatibility across Spark, Trino, Flink, and DuckDB.

    +
    + }, + { + question: "Q5. When should I choose Copy-on-Write over Merge-on-Read for Iceberg tables?", + answer:
    +

    Choose Copy-on-Write when:

    +
      +
    • Your workload is read-heavy with infrequent updates
      • +
    • Reports and analytics dashboards query the same data many times, no delete-file overhead at read time means consistently fast queries
      • +
    • Data freshness requirements are low and writes are batched
      • +
    +

    Choose Merge-on-Read (equality or position deletes in v2, deletion vectors in v3) when:

    +
      +
    • Your workload is write-heavy with frequent CDC updates and deletes
      • +
    • You are replicating an OLTP database where rows change constantly
      • +
    • COW's full file rewrites would become a throughput bottleneck at scale
      • +
    +

    For most CDC pipelines (OLake Go, Debezium, Flink CDC), MOR is the correct default, the write volume makes COW impractical, and query engines like Trino handle the merge overhead efficiently.

    +
    + } +]} /> + diff --git a/blog/2025-09-07-how-to-set-up-postgres-apache-iceberg.mdx b/blog/2025-09-07-how-to-set-up-postgres-apache-iceberg.mdx index 0cf00ecb6..a4124d52f 100644 --- a/blog/2025-09-07-how-to-set-up-postgres-apache-iceberg.mdx +++ b/blog/2025-09-07-how-to-set-up-postgres-apache-iceberg.mdx @@ -11,7 +11,7 @@ image: /img/blog/cover/postgres-apache-iceberg.webp Ever wanted to run high-performance analytics on your PostgreSQL data without overloading your production database or breaking your budget? **PostgreSQL to Apache Iceberg replication** is quickly becoming the go-to solution for modern data teams looking to build scalable, cost-effective analytics pipelines. -This comprehensive guide will walk you through everything you need to know about setting up real-time CDC replication from PostgreSQL to Iceberg, including best practices, common pitfalls, and a detailed step-by-step implementation using OLake. Whether you're building a modern data lakehouse architecture or optimizing your existing analytics workflows, this tutorial covers all the essential components. +This comprehensive guide will walk you through everything you need to know about setting up real-time CDC replication from PostgreSQL to Iceberg, including best practices, common pitfalls, and a detailed step-by-step implementation using OLake Go. Whether you're building a modern data lakehouse architecture or optimizing your existing analytics workflows, this tutorial covers all the essential components. ![OLake stream selection UI with Full Refresh + CDC mode for dz-stag-users table](/img/blog/2025/12/lakehouse-image.webp) @@ -21,7 +21,7 @@ This comprehensive guide will walk you through everything you need to know about - **Real-time Logical Replication**: PostgreSQL WAL-based CDC streams changes to Iceberg with sub-second latency for up-to-date analytics - **50-75% Cost Reduction**: Organizations report dramatic savings by moving analytics from expensive PostgreSQL RDS to cost-effective S3 + Iceberg architecture - **Open Format Flexibility**: Store data once and query with any engine (Trino, Spark, DuckDB, Athena) - switch tools without data migration -- **Enterprise-Ready Reliability**: OLake handles schema evolution, CDC recovery, and state management automatically for production deployments +- **Enterprise-Ready Reliability**: OLake Go handles schema evolution, CDC recovery, and state management automatically for production deployments ## Why PostgreSQL to Iceberg Replication is Essential for Modern Data Teams @@ -68,7 +68,7 @@ Establishing reliable PostgreSQL logical replication pipelines, particularly dur Apache Iceberg relies on robust metadata management for query performance optimization. Poor partitioning schemes, missing statistics, or inadequate compaction processes can significantly degrade query performance and increase operational costs. -## Step-by-Step Guide: PostgreSQL to Iceberg Replication with OLake +## Step-by-Step Guide: PostgreSQL to Iceberg Replication with OLake Go ### Prerequisites for Setting Up Your Replication Pipeline @@ -81,11 +81,11 @@ Before beginning your PostgreSQL to [Apache Iceberg](/iceberg/why-iceberg) migra - Docker, PostgreSQL credentials, and AWS S3 access configured - Apache Iceberg and Catalog configuration credentials -For this guide, we'll use AWS Glue catalog for Apache Iceberg and S3 as the primary object store, though OLake supports multiple catalog options including Nessie, Polaris, Hive, and Unity. +For this guide, we'll use AWS Glue catalog for Apache Iceberg and S3 as the primary object store, though OLake Go supports multiple catalog options including Nessie, Polaris, Hive, and Unity. ### Step 1: Configure PostgreSQL for Logical Replication -OLake offers both JDBC-based Full Refresh and Bookmark-based Incremental sync modes, so if you don't have permissions to create replication slots, you can start syncing immediately with standard database credentials. +OLake Go offers both JDBC-based Full Refresh and Bookmark-based Incremental sync modes, so if you don't have permissions to create replication slots, you can start syncing immediately with standard database credentials. However, for real-time CDC capabilities, you'll need to enable logical replication in PostgreSQL using these SQL commands: @@ -109,7 +109,7 @@ SELECT pg_reload_conf(); #### Grant Replication Permissions -If using a dedicated role for OLake (e.g., "olake_user"), ensure proper privileges: +If using a dedicated role for OLake Go (e.g., "olake_user"), ensure proper privileges: ```sql ALTER ROLE olake_user WITH REPLICATION; @@ -118,14 +118,14 @@ ALTER ROLE olake_user WITH REPLICATION; Alternatively, you can use any existing superuser or role with replication permissions. #### Create Publication and Logical Replication Slot -OLake captures changes through a logical replication slot with pgoutput and requires a publication: +OLake Go captures changes through a logical replication slot with pgoutput and requires a publication: ```sql CREATE PUBLICATION olake_publication FOR ALL TABLES WITH (publish = 'insert,update,delete,truncate'); SELECT * FROM pg_create_logical_replication_slot('olake_slot', 'pgoutput'); ``` -This begins tracking changes from the current WAL position. Ensure the publication name matches your OLake source configuration. +This begins tracking changes from the current WAL position. Ensure the publication name matches your OLake Go source configuration. ### Step 2: Deploy and Configure OLake UI @@ -145,7 +145,7 @@ curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-c - Default login credentials: admin / password - Complete setup documentation: [OLake UI Getting Started Guide](https://olake.io/docs/getting-started/olake-ui) -**Alternative**: OLake also provides a configurable CLI for advanced users who prefer command-line operations. CLI documentation is available at: [OLake CLI Guide](https://olake.io/docs/install/docker-cli). +**Alternative**: OLake Go also provides a configurable CLI for advanced users who prefer command-line operations. CLI documentation is available at: [OLake CLI Guide](https://olake.io/docs/install/docker-cli). ### Step 3: Configure PostgreSQL Source Connection @@ -156,7 +156,7 @@ In the OLake UI interface: - Host and port information - Username and password credentials - Database name -3. OLake automatically detects optimal chunking strategies for PostgreSQL (using CTID or batch splits for high-throughput scenarios) +3. OLake Go automatically detects optimal chunking strategies for PostgreSQL (using CTID or batch splits for high-throughput scenarios) ![OLake PostgreSQL source configuration UI with config file sample for replica set connection](/img/blog/2025/12/step-3-image.webp) @@ -170,7 +170,7 @@ Configure your Apache Iceberg destination in the OLake UI: - IAM credentials (optional if your instance has appropriate IAM roles) - S3 bucket selection for Iceberg table storage -OLake supports multiple Iceberg catalog implementations including Glue, Nessie, Polaris, Hive, and Unity Catalog. For detailed configuration of other catalogs, refer to the [Catalog Compatibility Overview](/docs/understanding/compatibility-catalogs). +OLake Go supports multiple Iceberg catalog implementations including Glue, Nessie, Polaris, Hive, and Unity Catalog. For detailed configuration of other catalogs, refer to the [Catalog Compatibility Overview](/docs/understanding/compatibility-catalogs). ![OLake UI create destination screen for Apache Iceberg AWS Glue catalog configuration](/img/blog/2025/12/step-4.webp) @@ -231,7 +231,7 @@ s3://your-bucket/ ![Amazon S3 test-olake-pg bucket UI with folders for olake_usecase_test.db and Unsaved](/img/blog/2025/12/stp-6-1.webp) -By default, OLake stores data files as Parquet format with metadata in JSON and Avro formats, following Apache Iceberg specifications. +By default, OLake Go stores data files as Parquet format with metadata in JSON and Avro formats, following Apache Iceberg specifications. ### Step 7 (Optional): Query Iceberg Tables with AWS Athena @@ -249,7 +249,7 @@ Before deploying your PostgreSQL to Iceberg CDC pipeline to production, implemen ### Ensure Data Integrity with Primary Keys -Primary keys are essential for accurate deduplication and CDC processing. OLake relies on primary keys to differentiate between updates and inserts, which becomes critical when handling out-of-order events or recovery scenarios. +Primary keys are essential for accurate deduplication and CDC processing. OLake Go relies on primary keys to differentiate between updates and inserts, which becomes critical when handling out-of-order events or recovery scenarios. ### Implement Smart Partitioning Strategies @@ -324,7 +324,7 @@ Replicating PostgreSQL to Apache Iceberg provides the foundation for a modern, f - Future-proof data architecture with open, vendor-agnostic formats - Scale analytics capabilities without impacting production systems -With OLake, you gain access to: +With OLake Go, you gain access to: - Seamless full and incremental synchronization with minimal configuration overhead - Comprehensive schema evolution support for multiple tables and data types @@ -333,46 +333,73 @@ With OLake, you gain access to: The combination of PostgreSQL's reliability as an operational database and Apache Iceberg's analytical capabilities creates a powerful foundation for data-driven decision making. Whether you're building real-time dashboards, implementing advanced analytics, or developing machine learning pipelines, this replication strategy provides the scalability and flexibility modern organizations require. -## Frequently Asked Questions -### What's the difference between PostgreSQL and Apache Iceberg? - -PostgreSQL is an OLTP database designed for transactional application workloads with fast row-based operations. Apache Iceberg is an open table format optimized for large-scale analytics with columnar storage, built for data lakes rather than operational databases. - -### How does PostgreSQL logical replication work? - -PostgreSQL writes all changes to a Write-Ahead Log (WAL). Logical replication reads this WAL using replication slots and publications, streaming INSERT, UPDATE, and DELETE operations to downstream systems like Iceberg in real-time without impacting database performance. - -### Do I need PostgreSQL superuser privileges for CDC? - -No! While superuser simplifies setup, you only need specific privileges: REPLICATION permission, and SELECT access on tables you want to replicate. Cloud providers like AWS RDS and Google Cloud SQL support logical replication with limited-privilege accounts. - -### Can I replicate PostgreSQL without enabling logical replication? - -Yes! OLake offers JDBC-based Full Refresh and Bookmark-based Incremental sync modes. If you can't modify WAL settings or create replication slots, you can still replicate data using standard PostgreSQL credentials with timestamp-based incremental updates. - -### How does OLake handle PostgreSQL schema changes? - -OLake automatically detects [schema evolution](/docs/features/?tab=schema-evolution). When you add, drop, or modify columns in PostgreSQL, these changes propagate to Iceberg tables without breaking your pipeline. The state management ensures schema and data stay synchronized. - -### What happens if my PostgreSQL WAL fills up? - -Proper replication slot monitoring is crucial. If OLake falls behind, PostgreSQL retains WAL files until they're consumed. OLake provides lag monitoring and automatic recovery to prevent WAL bloat, but you should set appropriate WAL retention limits. - -### How do I handle large PostgreSQL databases for initial load? - -OLake uses intelligent chunking strategies (CTID-based or batch splits) to load data in parallel without locking tables. A 1TB PostgreSQL database typically loads in 4-8 hours depending on network and storage performance, and the process can be paused/resumed. - -### What query engines work with PostgreSQL-sourced Iceberg tables? - -Any Iceberg-compatible engine: [Apache Spark](https://olake.io/iceberg/query-engine/spark) for batch processing, [Trino](https://olake.io/iceberg/query-engine/trino)/[Presto](https://olake.io/iceberg/query-engine/presto) for interactive queries, [DuckDB](https://olake.io/iceberg/query-engine/duckdb) for fast analytical workloads, [AWS Athena](https://olake.io/iceberg/query-engine/athena) for serverless SQL, [Snowflake](https://olake.io/iceberg/query-engine/snowflake), [Databricks](https://olake.io/iceberg/query-engine/databricks), and many others - all querying the same data. - -### Can I replicate specific PostgreSQL tables or schemas? - -Yes! OLake lets you select specific tables, schemas, or even filter rows using SQL WHERE clauses. This selective replication reduces storage costs and improves query performance by replicating only the data you need for analytics. - -### What's the cost comparison between PostgreSQL RDS and Iceberg on S3? - -PostgreSQL RDS storage costs ~$0.115/GB/month plus compute charges that run 24/7. Iceberg on S3 costs ~$0.023/GB/month (5x cheaper) with compute costs only when querying. Organizations typically save 50-75% on analytics infrastructure. +## FAQs + + +

    PostgreSQL is an OLTP database designed for transactional application workloads with fast row-based operations. Apache Iceberg is an open table format optimized for large-scale analytics with columnar storage, built for data lakes rather than operational databases.

    + + }, + { + question: "Q2. How does PostgreSQL logical replication work?", + answer:
    +

    PostgreSQL writes all changes to a Write-Ahead Log (WAL). Logical replication reads this WAL using replication slots and publications, streaming INSERT, UPDATE, and DELETE operations to downstream systems like Iceberg in real-time without impacting database performance.

    +
    + }, + { + question: "Q3. Do I need PostgreSQL superuser privileges for CDC?", + answer:
    +

    No! While superuser simplifies setup, you only need specific privileges: REPLICATION permission, and SELECT access on tables you want to replicate. Cloud providers like AWS RDS and Google Cloud SQL support logical replication with limited-privilege accounts.

    +
    + }, + { + question: "Q4. Can I replicate PostgreSQL without enabling logical replication?", + answer:
    +

    Yes. Logical replication (WAL slots and pgoutput) is only required for CDC. Without it, OLake Go can still replicate using Full Refresh and Incremental sync modes with standard PostgreSQL credentials, so you don't need to modify WAL settings or create replication slots. Incremental sync tracks changes using a cursor column such as updated_at instead of the WAL.

    +
    + }, + { + question: "Q5. How does OLake Go handle PostgreSQL schema changes?", + answer:
    +

    OLake Go automatically detects schema evolution. When you add, drop, or modify columns in PostgreSQL, these changes propagate to Iceberg tables without breaking your pipeline. The state management ensures schema and data stay synchronized.

    +
    + }, + { + question: "Q6. What happens if my PostgreSQL WAL fills up?", + answer:
    +

    WAL can accumulate when a replication slot's consumer falls behind, because PostgreSQL retains WAL segments until the slot confirms it has processed them. If the WAL grows large enough to fill the disk, PostgreSQL can stop accepting writes, which affects the source database, not just replication.

    +

    To prevent this, cap how much WAL a slot can retain with the max_slot_wal_keep_size setting in PostgreSQL, so a lagging or inactive slot cannot fill the disk. Also remove any replication slots you are no longer using, since an abandoned slot will retain WAL indefinitely.

    +
    + }, + { + question: "Q7. How do I handle large PostgreSQL databases for initial load?", + answer:
    +

    OLake Go uses parallel chunking strategies (CTID-based or batch splits) to load data without locking tables, splitting large tables into virtual chunks processed concurrently. The process can be paused and resumed.

    +

    In OLake Go's PostgreSQL benchmarks, a full load of 4.01 billion rows into Apache Iceberg ran at roughly 580,000 rows per second, so you can estimate load time for your own dataset from its row count. See the OLake Go ingestion benchmarks for full methodology and hardware details.

    +
    + }, + { + question: "Q8. What query engines work with PostgreSQL-sourced Iceberg tables?", + answer:
    +

    Any Iceberg-compatible engine: Apache Spark for batch processing, Trino/Presto for interactive queries, DuckDB for fast analytical workloads, AWS Athena for serverless SQL, Snowflake, Databricks, and many others, all querying the same data.

    +
    + }, + { + question: "Q9. Can I replicate specific PostgreSQL tables or schemas?", + answer:
    +

    Yes! OLake Go lets you select specific tables, schemas, or even filter rows using SQL WHERE clauses. This selective replication reduces storage costs and improves query performance by replicating only the data you need for analytics.

    +
    + }, + { + question: "Q10. What's the cost comparison between PostgreSQL RDS and Iceberg on S3?", + answer:
    +

    Taking AWS us-east-1 (N. Virginia) pricing as a reference: PostgreSQL RDS storage costs about $0.115/GB per month, plus compute charges that run continuously since the instance is always on. Iceberg on S3 Standard costs about $0.023/GB per month, roughly 5x cheaper for storage, with compute charged only when you run queries. Exact prices vary by region, but the ratio between the two stays similar.

    +

    The total savings depend on your workload and query patterns, but moving cold or infrequently queried data to an Iceberg lakehouse on S3 avoids paying for always-on database compute to retain it.

    +
    + }, +]} /> diff --git a/blog/2025-09-09-mysql-to-apache-iceberg-replication.mdx b/blog/2025-09-09-mysql-to-apache-iceberg-replication.mdx index 025d4a80c..fbe6f99f6 100644 --- a/blog/2025-09-09-mysql-to-apache-iceberg-replication.mdx +++ b/blog/2025-09-09-mysql-to-apache-iceberg-replication.mdx @@ -17,7 +17,7 @@ That's where [**Apache Iceberg**](/iceberg/why-iceberg) comes in. By replicating [Apache Iceberg](/iceberg/why-iceberg) is more than an average table format and it's designed for large-scale, cost-effective analytics. With native support for ACID transactions, seamless schema evolution, and compatibility with query engines like Trino, Spark, and DuckDB, it's ideal for modern data lakehouses. -In this comprehensive guide, we'll walk through setting up a real-time pipeline from MySQL to Apache Iceberg using OLake, covering both UI and CLI approaches. We'll explore why companies like Netflix, Natural Intelligence, and Memed have successfully migrated to Iceberg architectures, achieving dramatic performance improvements and cost savings. +In this comprehensive guide, we'll walk through setting up a real-time pipeline from MySQL to Apache Iceberg using OLake Go, covering both UI and CLI approaches. We'll explore why companies like Netflix, Natural Intelligence, and Memed have successfully migrated to Iceberg architectures, achieving dramatic performance improvements and cost savings. ## Key Takeaways @@ -109,33 +109,33 @@ Moving data from MySQL into Iceberg sounds simple, but in practice there are sev - **MySQL Table Structure vs. Analytics**: MySQL tables aren't designed for analytics, so choosing the right Iceberg partitioning strategy makes or breaks query performance. Poor partitioning decisions can result in slow queries and high file scan costs. - **Reliability and Monitoring**: Network hiccups, binlog rotations, or failed writes can quietly push MySQL CDC pipelines out of sync without proper monitoring. Robust state management and recovery procedures are crucial for production deployments. -These challenges are why many DIY approaches get complicated quickly. Tools like OLake smooth over these technical edges while handling CDC configuration, schema evolution, partitioning optimization, and reliability monitoring automatically. +These challenges are why many DIY approaches get complicated quickly. Tools like OLake Go smooth over these technical edges while handling CDC configuration, schema evolution, partitioning optimization, and reliability monitoring automatically. -## Step-by-Step MySQL to Iceberg Migration Workflow with OLake +## Step-by-Step MySQL to Iceberg Migration Workflow with OLake Go ![MySQL binlog CDC with OLake for real-time analytics and target data systems](/img/blog/2025/13/step-by-step.webp) ### How MySQL to Iceberg Replication Works -At a high level, the flow is straightforward: **MySQL → OLake → Iceberg**. Here's what happens behind the scenes to enable real-time MySQL analytics: +At a high level, the flow is straightforward: **MySQL → OLake Go → Iceberg**. Here's what happens behind the scenes to enable real-time MySQL analytics: **Real-Time Change Data Capture Process** -1. **Listen to Changes**: OLake connects to MySQL's binary logs, which record every insert, update, and delete operation in real-time. This approach provides millisecond-latency change detection without impacting production performance. +1. **Listen to Changes**: OLake Go connects to MySQL's binary logs, which record every insert, update, and delete operation in real-time. This approach provides millisecond-latency change detection without impacting production performance. 2. **Capture and Transform**: Those changes are read continuously, normalized, and mapped into Iceberg-compatible data types while preserving data integrity and handling schema evolution automatically. -3. **Write to Iceberg**: OLake writes the data into Iceberg tables in your data lake (S3, HDFS, MinIO, etc.), respecting partition strategies and schema requirements for optimal query performance. +3. **Write to Iceberg**: OLake Go writes the data into Iceberg tables in your data lake (S3, HDFS, MinIO, etc.), respecting partition strategies and schema requirements for optimal query performance. 4. **Stay in Sync**: As new changes flow into MySQL, they automatically propagate to Iceberg, keeping your lakehouse tables fresh and query-ready for real-time analytics. **Automated Optimization and Reliability** -The best part? You don't need to worry about edge cases like schema evolution, or partitioning logic, OLake handles these automatically while ensuring your Iceberg tables remain efficient and consistent. +The best part? You don't need to worry about edge cases like schema evolution, or partitioning logic, OLake Go handles these automatically while ensuring your Iceberg tables remain efficient and consistent. **Advanced Features Include:** - Schema drift detection and handling for seamless evolution - Partition optimization based on query patterns and data volume - State management with recovery capabilities for production reliability -For deeper technical insights into what makes OLake fast and reliable for MySQL-to-Iceberg pipelines, check out the performance optimization guide: [OLake Performance Guide](https://olake.io/blog/what-makes-olake-fast). +For deeper technical insights into what makes OLake Go fast and reliable for MySQL-to-Iceberg pipelines, check out the performance optimization guide: [OLake Performance Guide](https://olake.io/blog/what-makes-olake-fast). ## Step-by-Step Guide: MySQL to Iceberg Migration @@ -143,7 +143,7 @@ For deeper technical insights into what makes OLake fast and reliable for MySQL- Before starting your MySQL to Apache Iceberg replication, ensure you have the following components configured: -**OLake Platform**: UI deployed (or CLI setup) - Complete setup documentation: [OLake Quickstart Guide](https://olake.io/docs/getting-started/quickstart) +**OLake Go Platform**: UI deployed (or CLI setup) - Complete setup documentation: [OLake Go Quickstart Guide](https://olake.io/docs/getting-started/quickstart) **MySQL Instance Requirements:** - Binary logging enabled (binlog_format=ROW) @@ -163,9 +163,9 @@ For AWS Glue catalog quick setup: [Glue Catalog Configuration](/writers/iceberg/ ### Step 1: Configure MySQL for Logical Replication -**Important Note**: OLake offers JDBC-based Full Refresh and Bookmark-based Incremental sync modes, so if you don't have permissions to create replication slots, you can start syncing your MySQL data immediately with standard database credentials. +**Important Note**: OLake Go offers JDBC-based Full Refresh and Bookmark-based Incremental sync modes, so if you don't have permissions to create replication slots, you can start syncing your MySQL data immediately with standard database credentials. -Before OLake can implement real-time MySQL CDC, configure your database for logical replication using these SQL commands: +Before OLake Go can implement real-time MySQL CDC, configure your database for logical replication using these SQL commands: **Prerequisites for MySQL CDC** - **MySQL Version**: 5.7 or higher for optimal compatibility @@ -199,9 +199,9 @@ Restart MySQL after configuration changes to apply binlog settings. For environment-specific MySQL CDC setup (RDS, Cloud SQL, etc.), refer to: [MySQL CDC Configuration Guide](https://olake.io/docs/connectors/mysql#cdc-setup) -### Step 2: Deploy OLake UI +### Step 2: Deploy OLake Go UI -OLake UI provides a web-based interface for managing replication jobs, data sources, destinations, and configurations. It offers an intuitive way to create, edit, and monitor jobs without command-line complexity. +OLake Go UI provides a web-based interface for managing replication jobs, data sources, destinations, and configurations. It offers an intuitive way to create, edit, and monitor jobs without command-line complexity. **Quick Installation with Docker** @@ -217,7 +217,7 @@ curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-c - **Default Login Credentials**: admin / password - **Complete Documentation**: [OLake UI Getting Started](https://olake.io/docs/getting-started/quickstart) -**Alternative Setup**: OLake provides a configurable CLI interface for advanced users preferring command-line operations. CLI documentation: [Docker CLI Installation](https://olake.io/docs/install/docker-cli) +**Alternative Setup**: OLake Go provides a configurable CLI interface for advanced users preferring command-line operations. CLI documentation: [Docker CLI Installation](https://olake.io/docs/install/docker-cli) ### Step 3: Configure MySQL Source Connection @@ -227,9 +227,9 @@ In the OLake UI, navigate to **Sources → Add Source → MySQL**. - **Host and Port**: Your MySQL server endpoint - **Username/Password**: CDC user credentials created in Step 1 - **Database Name**: Source database identifier -- **Advanced Options**: Chunking strategy (OLake automatically detects optimal chunking based on primary keys for high throughput) +- **Advanced Options**: Chunking strategy (OLake Go automatically detects optimal chunking based on primary keys for high throughput) -OLake automatically optimizes data chunking strategies for MySQL, using primary key-based chunking for maximum performance during initial loads and incremental sync operations. +OLake Go automatically optimizes data chunking strategies for MySQL, using primary key-based chunking for maximum performance during initial loads and incremental sync operations. ![OLake platform setup source configuration UI for MongoDB connector](/img/blog/2025/13/step-3.webp) @@ -245,7 +245,7 @@ Configure your Iceberg destination in the OLake UI for seamless lakehouse integr - **S3 Bucket**: Storage location for Iceberg table data - **Catalog Settings**: Additional Glue-specific configurations -**Multi-Catalog Support**: OLake supports multiple catalogs (Glue, Nessie, Polaris, Hive, Unity), providing flexibility for different architectural requirements. +**Multi-Catalog Support**: OLake Go supports multiple catalogs (Glue, Nessie, Polaris, Hive, Unity), providing flexibility for different architectural requirements. **Detailed Configuration Guide**: See AWS Glue Catalog setup in [Glue Catalog documentation](/docs/writers/iceberg/catalog/glue/) **Alternative Catalogs**: For REST catalogs (Lakekeeper, Polaris) and other options: [Catalog Compatibility Overview](/docs/understanding/compatibility-catalogs) @@ -308,11 +308,11 @@ s3://your-bucket/ ![OLake jobs dashboard showing active sync jobs, sources, destinations, and statuses](/img/blog/2025/13/step-6-1.webp) -**Default File Formats**: OLake stores data files as Parquet format with metadata in JSON and Avro formats, following Apache Iceberg specifications for optimal query performance. +**Default File Formats**: OLake Go stores data files as Parquet format with metadata in JSON and Avro formats, following Apache Iceberg specifications for optimal query performance. ![Amazon S3 bucket olake_test_table UI showing data and metadata folders](/img/blog/2025/13/step-6-2.webp) -**Data Organization**: Within the ".db" folder, you'll find tables synced from MySQL source. OLake normalizes column, table, and schema names to ensure compatibility with Glue catalog writing restrictions. +**Data Organization**: Within the ".db" folder, you'll find tables synced from MySQL source. OLake Go normalizes column, table, and schema names to ensure compatibility with Glue catalog writing restrictions. **File Structure**: Each table contains respective data and metadata files organized for efficient querying and maintenance operations. @@ -372,7 +372,7 @@ Poor partitioning directly impacts performance: Improper partition choices resul - Data loss scenarios during disaster recovery - Manual recovery efforts that could introduce data inconsistencies -**Automated Error Handling**: With OLake, many of these operational concerns are handled automatically, but understanding the underlying mechanisms ensures successful production MySQL to Iceberg deployments. +**Automated Error Handling**: With OLake Go, many of these operational concerns are handled automatically, but understanding the underlying mechanisms ensures successful production MySQL to Iceberg deployments. ## Conclusion: Transforming MySQL Analytics with Apache Iceberg @@ -384,7 +384,7 @@ MySQL remains an excellent choice for transactional applications, but it's not b **Cost Optimization**: Natural Intelligence completed migration with zero downtime while establishing a modern, vendor-neutral platform that scales with evolving analytics needs. Organizations typically see 50-75% cost savings compared to traditional warehouse approaches. -**Operational Excellence**: With OLake's automated approach, you get: +**Operational Excellence**: With OLake Go's automated approach, you get: - Full + incremental synchronization (both JDBC and binlog-based) with minimal setup complexity - Comprehensive schema evolution support for multiple tables and data types - Open file format compatibility that integrates seamlessly with your preferred query engines @@ -406,48 +406,73 @@ Start your MySQL to Apache Iceberg migration today and unlock the full analytica As the data landscape continues evolving toward open, cloud-native architectures, organizations embracing Apache Iceberg lakehouse patterns position themselves for scalable growth while maintaining operational excellence. The question isn't whether to migrate from MySQL analytics, it's how quickly you can implement this transformation to stay competitive in today's data-driven economy. -## Frequently Asked Questions - -### What is the difference between MySQL and Apache Iceberg? - -MySQL is an OLTP (Online Transaction Processing) database designed for handling live application transactions with fast reads and writes. Apache Iceberg is an open table format designed for large-scale analytics on data lakes, optimized for complex queries and petabyte-scale data storage. - -### How does CDC (Change Data Capture) work with MySQL? - -CDC tracks changes in MySQL by reading the binary log (binlog), which records every insert, update, and delete operation. OLake connects to the binlog and streams these changes in real-time to your Iceberg tables without impacting production performance. - -### Can I replicate MySQL to Iceberg without CDC? - -Yes! OLake offers JDBC-based Full Refresh and Bookmark-based Incremental sync modes. If you don't have permissions to enable binlogs, you can start syncing immediately with standard MySQL credentials. - -### What happens to my MySQL schema changes? - -OLake automatically handles schema evolution. When you add, drop, or modify columns in MySQL, these changes are detected and propagated to your Iceberg tables without breaking your pipeline. - -### How much does it cost to store data in Iceberg vs MySQL? - -Iceberg storage on S3 costs approximately $0.023 per GB/month, compared to MySQL RDS storage at $0.115 per GB/month - that's 5x cheaper. Plus, you separate compute from storage, so you only pay for queries when you run them. - -### What query engines can I use with Iceberg tables? - -Apache Iceberg is an open format compatible with: [Trino](https://olake.io/iceberg/query-engine/trino), [Presto](https://olake.io/iceberg/query-engine/presto), [Apache Spark](https://olake.io/iceberg/query-engine/spark), [DuckDB](https://olake.io/iceberg/query-engine/duckdb), [AWS Athena](https://olake.io/iceberg/query-engine/athena), [Snowflake](https://olake.io/iceberg/query-engine/snowflake), [Databricks](https://olake.io/iceberg/query-engine/databricks), and many others. You can switch engines anytime without rewriting data. - -### How do I handle partitioning for optimal query performance? - -Choose partition columns based on your query patterns: use timestamp fields (created_at, updated_at) for time-series queries, or dimensional fields (customer_id, region) for lookup queries. OLake supports regex-based partitioning configuration. - -### Is the initial full load safe for large MySQL databases? - -Yes! OLake uses primary key-based chunking to load data in batches without locking your MySQL tables. The process runs in parallel and can be paused/resumed if needed. - -### What happens if my replication pipeline fails? - -OLake maintains a state.json file that tracks replication progress. If the pipeline fails, it automatically resumes from the last successfully processed position, ensuring no data loss. +Happy syncing! 🧊🐘 -### Can I query both MySQL and Iceberg simultaneously? +## FAQs + + +

    MySQL is an OLTP (Online Transaction Processing) database designed for handling live application transactions with fast reads and writes. Apache Iceberg is an open table format designed for large-scale analytics on data lakes, optimized for complex queries and petabyte-scale data storage.

    + + }, + { + question: "Q2. How does CDC (Change Data Capture) work with MySQL?", + answer:
    +

    CDC tracks changes in MySQL by reading the binary log (binlog), which records every insert, update, and delete operation. OLake Go connects to the binlog and streams these changes in real-time to your Iceberg tables without impacting production performance.

    +
    + }, + { + question: "Q3. Can I replicate MySQL to Iceberg without CDC?", + answer:
    +

    Yes. CDC on MySQL requires binlog access. Without it, OLake Go can still replicate using Full Refresh and Incremental sync modes with standard MySQL credentials, so you don't need binlog permissions to start syncing. Incremental sync tracks changes using a cursor column such as updated_at instead of the binlog.

    +
    + }, + { + question: "Q4. What happens to my MySQL schema changes?", + answer:
    +

    OLake Go automatically handles schema evolution. When you add, drop, or modify columns in MySQL, these changes are detected and propagated to your Iceberg tables without breaking your pipeline.

    +
    + }, + { + question: "Q5. How much does it cost to store data in Iceberg vs MySQL?", + answer:
    +

    For storage, Iceberg on S3 costs about $0.023 per GB per month, compared to MySQL on RDS at about $0.115 per GB per month (AWS us-east-1), roughly 5x cheaper for storage. Iceberg also separates compute from storage, so you pay for query compute only when you actually run queries, rather than for an always-on database instance.

    +
    + }, + { + question: "Q6. What query engines can I use with Iceberg tables?", + answer:
    +

    Apache Iceberg is an open format compatible with: Trino, Presto, Apache Spark, DuckDB, AWS Athena, Snowflake, Databricks, and many others. You can switch engines anytime without rewriting data.

    +
    + }, + { + question: "Q7. How do I handle partitioning for optimal query performance?", + answer:
    +

    Choose partition columns based on your query patterns: use timestamp fields (created_at, updated_at) for time-series queries, or dimensional fields (customer_id, region) for lookup queries. OLake Go supports regex-based partitioning configuration.

    +
    + }, + { + question: "Q8. Is the initial full load safe for large MySQL databases?", + answer:
    +

    Yes. OLake Go uses primary key-based chunking to load data in batches without locking your MySQL tables, and the chunks are processed in parallel. Because OLake Go tracks progress in a state file, an interrupted sync can resume from where it left off rather than restarting the whole load.

    +
    + }, + { + question: "Q9. What happens if my replication pipeline fails?", + answer:
    +

    OLake Go maintains a state.json file that tracks replication progress. If the pipeline fails, it automatically resumes from the last successfully processed position, ensuring no data loss.

    +
    + }, + { + question: "Q10. Can I query both MySQL and Iceberg simultaneously?", + answer:
    +

    Absolutely! Your MySQL database continues serving production traffic while Iceberg handles analytics. This separation ensures operational workloads never compete with analytical queries for resources.

    +
    + } +]} /> -Absolutely! Your MySQL database continues serving production traffic while Iceberg handles analytics. This separation ensures operational workloads never compete with analytical queries for resources. -Happy syncing! 🧊🐘 diff --git a/blog/2025-09-10-how-to-set-up-mongodb-apache-iceberg.mdx b/blog/2025-09-10-how-to-set-up-mongodb-apache-iceberg.mdx index b31f39619..c7791b290 100644 --- a/blog/2025-09-10-how-to-set-up-mongodb-apache-iceberg.mdx +++ b/blog/2025-09-10-how-to-set-up-mongodb-apache-iceberg.mdx @@ -1,6 +1,6 @@ --- title: "How to Set Up MongoDB Apache Iceberg Replication Guide" -description: "Learn step-by-step how to replicate MongoDB to Apache Iceberg with OLake for real-time analytics, schema evolution, partitioning, and cost-efficient querying." +description: "Learn step-by-step how to replicate MongoDB to Apache Iceberg with OLake Go for real-time analytics, schema evolution, partitioning, and cost-efficient querying." authors: [rohan] slug: how-to-set-up-mongodb-apache-iceberg tags: [mongodb,iceberg,olake,cdc] @@ -17,13 +17,13 @@ That's where [**Apache Iceberg**](/iceberg/why-iceberg) comes in. By replicating Apache Iceberg is designed for large-scale, cost-effective analytics with native support for ACID transactions, seamless schema evolution, and compatibility with engines like Trino, Spark, and DuckDB. It's the perfect complement to MongoDB's operational strengths. -In this comprehensive guide, we'll walk through setting up a real-time pipeline from MongoDB to Apache Iceberg using OLake, covering both UI and CLI approaches. We'll explore why companies are successfully migrating to Iceberg architectures, achieving dramatic performance improvements and cost savings. +In this comprehensive guide, we'll walk through setting up a real-time pipeline from MongoDB to Apache Iceberg using OLake Go, covering both UI and CLI approaches. We'll explore why companies are successfully migrating to Iceberg architectures, achieving dramatic performance improvements and cost savings. ## Key Takeaways - **Solve MongoDB Analytics Bottlenecks**: Run complex aggregations and joins on Iceberg without slowing down your MongoDB production workloads - **Real-time Change Streams**: MongoDB Change Streams provide millisecond-latency CDC to keep Iceberg tables continuously synchronized -- **Handle Flexible Schemas**: OLake automatically manages MongoDB's dynamic schema evolution, converting BSON documents to Iceberg-compatible structures +- **Handle Flexible Schemas**: OLake Go automatically manages MongoDB's dynamic schema evolution, converting BSON documents to Iceberg-compatible structures - **Petabyte-Scale Analytics**: Query terabytes or petabytes of data using columnar storage on S3, with costs 5x lower than operational MongoDB - **Multi-Engine Freedom**: Access your MongoDB data through [Trino](https://olake.io/iceberg/query-engine/trino), [Spark](https://olake.io/iceberg/query-engine/spark), [DuckDB](https://olake.io/iceberg/query-engine/duckdb), or [Athena](https://olake.io/iceberg/query-engine/athena) using standard SQL - no MongoDB query language required @@ -76,7 +76,7 @@ Replicating MongoDB into Apache Iceberg solves these fundamental problems: -![MongoDB change streams to OLake CDC pipeline transforming data for Apache Iceberg](/img/blog/2025/14/change-streams.webp) +![MongoDB change streams to OLake Go CDC pipeline transforming data for Apache Iceberg](/img/blog/2025/14/change-streams.webp) ### Key Challenges in MongoDB to Iceberg Replication @@ -97,33 +97,33 @@ Moving data from MongoDB into Iceberg sounds simple, but in practice there are s - **MongoDB Collection Structure vs. Analytics**: MongoDB collections aren't designed for analytics, so choosing the right Iceberg partitioning strategy makes or breaks query performance. Poor partitioning decisions can result in slow queries and high file scan costs. - **Reliability and Monitoring**: Network hiccups, oplog rotations, or failed writes can quietly push MongoDB CDC pipelines out of sync without proper monitoring. Robust state management and recovery procedures are crucial for production deployments. -These challenges are why many DIY approaches get complicated quickly. Tools like OLake smooth over these technical edges while handling CDC configuration, schema evolution, partitioning optimization, and reliability monitoring automatically. +These challenges are why many DIY approaches get complicated quickly. Tools like OLake Go smooth over these technical edges while handling CDC configuration, schema evolution, partitioning optimization, and reliability monitoring automatically. -## Step-by-Step MongoDB to Iceberg Migration Workflow with OLake +## Step-by-Step MongoDB to Iceberg Migration Workflow with OLake Go ![MongoDB operational database to Apache Iceberg analytical lakehouse migration](/img/blog/2025/14/architecture.webp) ### How MongoDB to Iceberg Replication Works -At a high level, the flow is straightforward: **MongoDB → OLake → Iceberg**. Here's what happens behind the scenes to enable real-time MongoDB analytics: +At a high level, the flow is straightforward: **MongoDB → OLake Go → Iceberg**. Here's what happens behind the scenes to enable real-time MongoDB analytics: **Real-Time Change Data Capture Process** -1. **Listen to Changes**: OLake connects to MongoDB's Change Streams, which record every insert, update, and delete operation in real-time. This approach provides millisecond-latency change detection without impacting production performance. +1. **Listen to Changes**: OLake Go connects to MongoDB's Change Streams, which record every insert, update, and delete operation in real-time. This approach provides millisecond-latency change detection without impacting production performance. 2. **Capture and Transform**: Those changes are read continuously, normalized, and mapped into Iceberg-compatible data types while preserving data integrity and handling schema evolution automatically. -3. **Write to Iceberg**: OLake writes the data into Iceberg tables in your data lake (S3, HDFS, MinIO, etc.), respecting partition strategies and schema requirements for optimal query performance. +3. **Write to Iceberg**: OLake Go writes the data into Iceberg tables in your data lake (S3, HDFS, MinIO, etc.), respecting partition strategies and schema requirements for optimal query performance. 4. **Stay in Sync**: As new changes flow into MongoDB, they automatically propagate to Iceberg, keeping your lakehouse tables fresh and query-ready for real-time analytics. **Automated Optimization and Reliability** -The best part? You don't need to worry about edge cases like schema evolution, or partitioning logic, OLake handles these automatically while ensuring your Iceberg tables remain efficient and consistent. +The best part? You don't need to worry about edge cases like schema evolution, or partitioning logic, OLake Go handles these automatically while ensuring your Iceberg tables remain efficient and consistent. **Advanced Features Include:** - Schema drift detection and handling for seamless evolution - Partition optimization based on query patterns and data volume - State management with recovery capabilities for production reliability -For deeper technical insights into what makes OLake fast and reliable for MongoDB-to-Iceberg pipelines, check out the performance optimization guide: [OLake Performance Guide](https://olake.io/blog/what-makes-olake-fast). +For deeper technical insights into what makes OLake Go fast and reliable for MongoDB-to-Iceberg pipelines, check out the performance optimization guide: [OLake Performance Guide](https://olake.io/blog/what-makes-olake-fast). ## Step-by-Step Guide: MongoDB to Iceberg Migration @@ -152,10 +152,10 @@ For this guide we will be using AWS Glue catalog for Apache Iceberg and S3 as th ### Step 1: Setting up MongoDB :::caution -OLake does offer a JDBC based Full Refresh and Bookmark based Incremental sync modes, so if you don't have permissions or access to enable oplog, you can directly start synching your MongoDB data with just JDBC credentials +OLake Go does offer a JDBC based Full Refresh and Bookmark based Incremental sync modes, so if you don't have permissions or access to enable oplog, you can directly start synching your MongoDB data with just JDBC credentials ::: -Before OLake can start replicating data from MongoDB to Apache Iceberg, you need to configure the database for logical replication. +Before OLake Go can start replicating data from MongoDB to Apache Iceberg, you need to configure the database for logical replication. :::info For more details on MongoDB CDC setup for your respective environment, you can follow this document: [MongoDB and Atlas CDC Setup | OLake](https://olake.io/docs/connectors/mongodb) @@ -182,7 +182,7 @@ curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-c - **Default Login Credentials**: admin / password - **Complete Documentation**: [OLake UI Getting Started](https://olake.io/docs/getting-started/quickstart) -**Alternative Setup**: OLake provides a configurable CLI interface for advanced users preferring command-line operations. CLI documentation: [Docker CLI Installation](https://olake.io/docs/install/docker-cli) +**Alternative Setup**: OLake Go provides a configurable CLI interface for advanced users preferring command-line operations. CLI documentation: [Docker CLI Installation](https://olake.io/docs/install/docker-cli) ### Step 3: Configure MongoDB Source Connection @@ -194,7 +194,7 @@ In the OLake UI, navigate to **Sources → Add Source → MongoDB**. - **Database Name**: Source database identifier - **Advanced Options**: Collection selection and filtering options -OLake automatically optimizes data processing strategies for MongoDB, using efficient Change Streams processing for maximum performance during incremental sync operations. +OLake Go automatically optimizes data processing strategies for MongoDB, using efficient Change Streams processing for maximum performance during incremental sync operations. ![OLake platform setup source configuration UI for MongoDB connector](/img/blog/2025/14/step-3.webp) @@ -210,7 +210,7 @@ Configure your Iceberg destination in the OLake UI for seamless lakehouse integr - **S3 Bucket**: Storage location for Iceberg table data - **Catalog Settings**: Additional Glue-specific configurations -**Multi-Catalog Support**: OLake supports multiple catalogs (Glue, Nessie, Polaris, Hive, Unity), providing flexibility for different architectural requirements. +**Multi-Catalog Support**: OLake Go supports multiple catalogs (Glue, Nessie, Polaris, Hive, Unity), providing flexibility for different architectural requirements. **Detailed Configuration Guide**: See AWS Glue Catalog setup in [Glue Catalog documentation](/docs/writers/iceberg/catalog/glue/) @@ -265,10 +265,10 @@ Your MongoDB to Iceberg replication creates a structured hierarchy in S3 object -**Default File Formats**: OLake stores data files as Parquet format with metadata in JSON and Avro formats, following Apache Iceberg specifications for optimal query performance. +**Default File Formats**: OLake Go stores data files as Parquet format with metadata in JSON and Avro formats, following Apache Iceberg specifications for optimal query performance. -**Data Organization**: Within the ".db" folder, you'll find collections synced from MongoDB source. OLake normalizes collection and field names to ensure compatibility with Glue catalog writing restrictions. +**Data Organization**: Within the ".db" folder, you'll find collections synced from MongoDB source. OLake Go normalizes collection and field names to ensure compatibility with Glue catalog writing restrictions. ![Amazon S3 bucket olake_test_table UI showing data and metadata folders](/img/blog/2025/14/step-6-2.webp) **File Structure**: Each collection contains respective data and metadata files organized for efficient querying and maintenance operations. @@ -326,7 +326,7 @@ Poor partitioning directly impacts performance: Improper partition choices resul - Data loss scenarios during disaster recovery - Manual recovery efforts that could introduce data inconsistencies -**Automated Error Handling**: With OLake, many of these operational concerns are handled automatically, but understanding the underlying mechanisms ensures successful production MongoDB to Iceberg deployments. +**Automated Error Handling**: With OLake Go, many of these operational concerns are handled automatically, but understanding the underlying mechanisms ensures successful production MongoDB to Iceberg deployments. ## Conclusion: Transforming MongoDB Analytics with Apache Iceberg @@ -338,7 +338,7 @@ MongoDB remains an excellent choice for operational applications, but it's not b **Cost Optimization**: Natural Intelligence completed migration with zero downtime while establishing a modern, vendor-neutral platform that scales with evolving analytics needs. Organizations typically see 50-75% cost savings compared to traditional warehouse approaches. -**Operational Excellence**: With OLake's automated approach, you get: +**Operational Excellence**: With OLake Go's automated approach, you get: - Full + incremental synchronization (both batch and Change Streams-based) with minimal setup complexity - Comprehensive schema evolution support for flexible document structures - Open file format compatibility that integrates seamlessly with your preferred query engines @@ -353,48 +353,80 @@ The combination of MongoDB's operational flexibility and Iceberg's analytical ca As the data landscape continues evolving toward open, cloud-native architectures, organizations embracing Apache Iceberg lakehouse patterns position themselves for scalable growth while maintaining operational excellence. The question isn't whether to migrate from MongoDB analytics, it's how quickly you can implement this transformation to stay competitive in today's data-driven economy. -## Frequently Asked Questions - -### Why can't I just run analytics directly on MongoDB? - -MongoDB is optimized for operational workloads with fast document reads/writes. Complex analytical queries (aggregations, joins, large scans) consume significant resources and slow down production applications. Replicating to Iceberg separates analytics from operations, keeping both performant. - -### How does MongoDB Change Streams work for CDC? - -Change Streams tap into MongoDB's oplog (operation log) to capture every insert, update, and delete in real-time. OLake reads these changes continuously and applies them to Iceberg tables without impacting MongoDB performance or requiring application changes. - -### Do I need a MongoDB replica set for replication? - -For real-time CDC with Change Streams, yes - MongoDB requires replica set mode. However, OLake also offers JDBC-based Full Refresh and Bookmark-based Incremental modes that work with standalone MongoDB instances if you have permission limitations. - -### How does OLake handle MongoDB's flexible schemas? - -MongoDB documents in the same collection can have different fields. OLake automatically detects schema changes and evolves your Iceberg tables accordingly, adding new columns when new fields appear while maintaining backward compatibility. - -### What happens to nested MongoDB documents in Iceberg? - -OLake intelligently flattens nested BSON structures into Iceberg-compatible schemas. Complex nested objects become structured columns in Iceberg tables, making them queryable with standard SQL rather than MongoDB's aggregation framework. - -### Can I filter which MongoDB collections to replicate? - -Yes! OLake allows you to select specific collections and even apply MongoDB aggregation pipeline filters to replicate only the data you need, reducing storage costs and improving query performance. - -### How long does the initial MongoDB to Iceberg load take? - -Initial load time depends on your data volume and MongoDB performance. OLake processes collections in parallel and can be paused/resumed. For example, a 500GB MongoDB database typically loads in 2-4 hours depending on network and storage speed. - -### What's the difference between Change Streams and binlog CDC? - -Change Streams is MongoDB's native change tracking mechanism (similar to MySQL binlogs). It provides a stream of document-level changes that OLake captures and applies to Iceberg tables in real-time. - -### Can I query both MongoDB and Iceberg simultaneously? +Happy syncing! -Absolutely! MongoDB continues serving your application traffic while Iceberg handles analytics. This architecture ensures your operational database never competes with analytical workloads for resources. +## FAQs + + +

    MongoDB is optimized for operational workloads with fast document reads/writes. Complex analytical queries (aggregations, joins, large scans) consume significant resources and slow down production applications. Replicating to Iceberg separates analytics from operations, keeping both performant.

    + + }, + { + question: "Q2. How does MongoDB Change Streams work for CDC?", + answer:
    +

    Change Streams tap into MongoDB's oplog (operation log) to capture every insert, update, and delete in real-time. OLake Go reads these changes continuously and applies them to Iceberg tables without impacting MongoDB performance or requiring application changes.

    +
    + }, + { + question: "Q3. Do I need a MongoDB replica set for replication?", + answer:
    +

    For real-time CDC with Change Streams, yes, MongoDB requires replica set mode. However, OLake Go also offers Full Refresh and Bookmark-based Incremental modes that work with standalone MongoDB instances if you have permission limitations.

    +
    + }, + { + question: "Q4. How does OLake Go handle MongoDB's flexible schemas?", + answer:
    +

    MongoDB documents in the same collection can have different fields. OLake Go automatically detects schema changes and evolves your Iceberg tables accordingly, adding new columns when new fields appear while maintaining backward compatibility.

    +
    + }, + { + question: "Q5. What happens to nested MongoDB documents in Iceberg?", + answer:
    +

    OLake Go intelligently flattens nested BSON structures into Iceberg-compatible schemas. Complex nested objects become structured columns in Iceberg tables, making them queryable with standard SQL rather than MongoDB's aggregation framework.

    +
    + }, + { + question: "Q6. Can I filter which MongoDB collections to replicate?", + answer:
    +

    Yes! OLake Go allows you to select specific collections and even apply MongoDB aggregation pipeline filters to replicate only the data you need, reducing storage costs and improving query performance.

    +
    + }, + { + question: "Q7. How long does the initial MongoDB to Iceberg load take?", + answer:
    +

    Initial load time depends on your data volume and MongoDB performance. OLake Go reads collections in parallel using chunking, and checkpointing means an interrupted load resumes from where it stopped rather than restarting.

    +

    In OLake Go's MongoDB benchmark, a full load of roughly 234 million rows completed in about 46 minutes on a three-node replica set, so you can estimate load time for your own dataset from its row count. See the OLake Go ingestion benchmarks for full methodology and hardware details.

    +
    + }, + { + question: "Q8. What's the difference between Change Streams and binlog CDC?", + answer:
    +

    Both are native change-tracking mechanisms, but they belong to different databases and work differently:

    +
      +
    • Change Streams (MongoDB): A higher-level API built on top of the oplog. It emits document-level change events (insert, update, delete, replace) that applications consume directly, without reading the raw oplog format. OLake Go uses this for MongoDB CDC.
      • +
    • Binlog (MySQL): The binary log records row-level changes at the storage-engine level. Consumers read the binlog directly and decode its events, tracking a binlog position for resumability. OLake Go uses this for MySQL CDC.
      • +
    +

    The practical difference: Change Streams gives you a clean, application-friendly event stream abstracted away from the underlying log, while the binlog is a lower-level record you read and decode yourself. Both let OLake Go capture inserts, updates, and deletes and apply them to Iceberg, but through each database's own native mechanism.

    +
    + }, + { + question: "Q9. Can I query both MongoDB and Iceberg simultaneously?", + answer:
    +

    Absolutely! MongoDB continues serving your application traffic while Iceberg handles analytics. This architecture ensures your operational database never competes with analytical workloads for resources.

    +
    + }, + { + question: "Q10. How much does Iceberg storage cost compared to MongoDB?", + answer:
    +

    Taking AWS us-east-1 (N. Virginia) pricing as a reference: S3 storage for Iceberg costs about $0.023/GB per month, while MongoDB Atlas storage on AWS runs around $0.25/GB per month, roughly 10x more for storage alone. Exact prices vary by region and Atlas cluster tier, but the gap stays similar. Iceberg's columnar format also compresses better, and you pay for compute only when running queries rather than for an always-on cluster.

    +
    + }, +]} /> -### How much does Iceberg storage cost compared to MongoDB? -S3 storage for Iceberg costs ~$0.023/GB/month compared to MongoDB Atlas storage at ~$0.25/GB/month (10x cheaper). Plus, Iceberg's columnar format compresses better, and you only pay for compute when running queries. -Happy syncing! diff --git a/blog/2025-09-15-apache-hive-vs-apache-iceberg-comparison.mdx b/blog/2025-09-15-apache-hive-vs-apache-iceberg-comparison.mdx index f7c65a60b..d74ed8684 100644 --- a/blog/2025-09-15-apache-hive-vs-apache-iceberg-comparison.mdx +++ b/blog/2025-09-15-apache-hive-vs-apache-iceberg-comparison.mdx @@ -11,7 +11,7 @@ image: /img/blog/cover/hive-vs-iceberg.webp ![Apache Hive vs Iceberg Comparison](/img/blog/cover/hive-vs-iceberg.webp) -Apache Hive and Apache Iceberg represent two different generations of the data lake ecosystem. Hive was born in the **Hadoop era** as a SQL abstraction over HDFS, excelling in batch ETL workloads and still valuable for organizations with large Hadoop/ORC footprints. Iceberg, by contrast, emerged in the **cloud-native era** as an [open table format](/iceberg/move-to-iceberg) designed for multi-engine interoperability, [**schema evolution**](/docs/features/?tab=schema-evolution), and features like [**time travel**](/blog/2025/10/03/iceberg-metadata/#63-time-travel-rollback-and-branching). If you are running a legacy Hadoop stack with minimal need for engine diversity, Hive remains a practical choice. If you want a **flexible, future-proof data lakehouse** that supports diverse engines, reliable transactions, and governance at scale, Iceberg is the more strategic investment. +Apache Hive and Apache Iceberg represent two different generations of the data lake ecosystem. Hive was born in the **Hadoop era** as a SQL abstraction over HDFS, excelling in batch ETL workloads and still valuable for organizations with large Hadoop/ORC footprints. Iceberg, by contrast, emerged in the **cloud-native era** as an [open table format](/iceberg/move-to-iceberg) designed for multi-engine interoperability, [**schema evolution**](/docs/features/schema), and features like [**time travel**](/blog/2025/10/03/iceberg-metadata/#63-time-travel-rollback-and-branching). If you are running a legacy Hadoop stack with minimal need for engine diversity, Hive remains a practical choice. If you want a **flexible, future-proof data lakehouse** that supports diverse engines, reliable transactions, and governance at scale, Iceberg is the more strategic investment. ## Hive vs Iceberg — Feature Comparison at a Glance @@ -215,32 +215,6 @@ Cost models also differ sharply. Hive's reliance on compaction and complex parti In short, tuning Hive is about firefighting compaction and partition sprawl to keep systems running. Tuning Iceberg is about unlocking efficiency—using metadata and layout strategies to deliver better performance and lower costs in cloud-first environments. -## FAQ: People-Also-Ask - -**Is Apache Iceberg better than Hive for analytics?** - -Iceberg generally offers stronger advantages for analytics workloads, especially those involving ad hoc queries, BI dashboards, or interactive exploration. Its metadata-driven pruning and hidden partitioning allow query engines to skip irrelevant files, dramatically reducing scan times. Hive, by contrast, relies on directory-based partitioning, which is slower and less flexible. That said, Hive remains perfectly adequate for batch-oriented ETL jobs in legacy Hadoop environments where performance is less critical. - -**Can I use Hive Metastore with Iceberg?** - -Yes. In fact, many organizations start their Iceberg journey this way. Iceberg supports multiple catalogs, including Hive Metastore, AWS Glue, and REST catalogs. Using the Hive Metastore allows incremental adoption—teams can register Iceberg tables alongside Hive tables and gradually migrate workloads. The limitation is that Hive Metastore itself was not designed for high-scale metadata operations, so as adoption grows, some organizations eventually move to more scalable options like Glue or REST-based catalogs. - -**How does Iceberg handle schema changes compared to Hive?** - -Schema evolution is one of Iceberg's standout features. It tracks columns by IDs rather than by name or position, which means you can rename, add, or drop columns without rewriting underlying data. Type changes are also supported in many cases. Hive, on the other hand, handles schema changes less gracefully. Renaming or dropping columns can cause inconsistencies, and type changes often require rewriting the table. For teams working in fast-moving domains, Iceberg's approach provides far more agility. - -**Do I need ORC for Hive ACID?** - -Yes, typically. Hive's ACID compliance relies on ORC files for transactional tables. These tables maintain base and delta files that must be periodically compacted. While Hive also supports other formats like Parquet or Avro for non-transactional tables, ORC remains the default and most reliable choice for ACID operations. This reliance on ORC is one reason why Hive feels more constrained compared to Iceberg, which supports multiple formats more flexibly. - -**Is Iceberg only for the cloud?** - -Not at all. While Iceberg is popular in cloud-native lakehouse architectures, it can also be deployed on-premise. What makes it cloud-friendly is its separation of storage and compute, plus support for object stores like S3, ADLS, and GCS. On-prem deployments often use Iceberg with distributed file systems like HDFS, though the benefits of time travel, schema evolution, and multi-engine compatibility are equally valuable regardless of environment. - -**Can Hive and Iceberg coexist in the same environment?** - -Yes—and in many cases, they do. Organizations often run Hive and Iceberg side by side during migration. Some workloads remain on Hive where stability and legacy integration matter, while new workloads adopt Iceberg for flexibility and performance. Over time, the balance often shifts toward Iceberg, but coexistence provides a practical path to transition without disrupting critical pipelines. - ## Summary / Conclusion The story of Hive and Iceberg is, in many ways, the story of the data ecosystem itself. Hive emerged in the **Hadoop era** to bring SQL-like querying to massive datasets stored across distributed systems. For more than a decade, it powered reporting, compliance, and batch ETL in countless enterprises. Its rigid partitioning, complex ACID model, and reliance on the Hive Metastore weren't design flaws so much as reflections of the constraints of its time. Hive thrived because it solved the problems that mattered most in the early days of big data. @@ -251,4 +225,45 @@ The choice between Hive and Iceberg comes down to context. For organizations wit Looking ahead, Hive will continue to support legacy systems where stability is valued, while Iceberg is poised to become the default open table format for modern data platforms. The real decision isn't about which technology is "better," but which future you want to build. If that future depends on multi-engine analytics, cost efficiency, and architectures that can evolve with the business, Iceberg is the format built to carry you there. +## FAQs + +

    Iceberg generally offers stronger advantages for analytics workloads, especially those involving ad hoc queries, BI dashboards, or interactive exploration. Its metadata-driven pruning and hidden partitioning allow query engines to skip irrelevant files, dramatically reducing scan times. Hive, by contrast, relies on directory-based partitioning, which is slower and less flexible. That said, Hive remains perfectly adequate for batch-oriented ETL jobs in legacy Hadoop environments where performance is less critical.

    + + }, + { + question: "Q2. Can I use Hive Metastore with Iceberg?", + answer:
    +

    Yes. In fact, many organizations start their Iceberg journey this way. Iceberg supports multiple catalogs, including Hive Metastore, AWS Glue, and REST catalogs. Using the Hive Metastore allows incremental adoption, teams can register Iceberg tables alongside Hive tables and gradually migrate workloads. The limitation is that Hive Metastore itself was not designed for high-scale metadata operations, so as adoption grows, some organizations eventually move to more scalable options like Glue or REST-based catalogs.

    +
    + }, + { + question: "Q3. How does Iceberg handle schema changes compared to Hive?", + answer:
    +

    Schema evolution is one of Iceberg's standout features. It tracks columns by IDs rather than by name or position, which means you can rename, add, or drop columns without rewriting underlying data. Type changes are also supported in many cases. Hive, on the other hand, handles schema changes less gracefully. Renaming or dropping columns can cause inconsistencies, and type changes often require rewriting the table. For teams working in fast-moving domains, Iceberg's approach provides far more agility.

    +
    + }, + { + question: "Q4. Do I need ORC for Hive ACID?", + answer:
    +

    Yes, typically. Hive's ACID compliance relies on ORC files for transactional tables. These tables maintain base and delta files that must be periodically compacted. While Hive also supports other formats like Parquet or Avro for non-transactional tables, ORC remains the default and most reliable choice for ACID operations. This reliance on ORC is one reason why Hive feels more constrained compared to Iceberg, which supports multiple formats more flexibly.

    +
    + }, + { + question: "Q5. Is Iceberg only for the cloud?", + answer:
    +

    Not at all. While Iceberg is popular in cloud-native lakehouse architectures, it can also be deployed on-premise. What makes it cloud-friendly is its separation of storage and compute, plus support for object stores like S3, ADLS, and GCS. On-prem deployments often use Iceberg with distributed file systems like HDFS, though the benefits of time travel, schema evolution, and multi-engine compatibility are equally valuable regardless of environment.

    +
    + }, + { + question: "Q6. Can Hive and Iceberg coexist in the same environment?", + answer:
    +

    Yes and in many cases, they do. Organizations often run Hive and Iceberg side by side during migration. Some workloads remain on Hive where stability and legacy integration matter, while new workloads adopt Iceberg for flexibility and performance. Over time, the balance often shifts toward Iceberg, but coexistence provides a practical path to transition without disrupting critical pipelines.

    +
    + } +]} /> + + diff --git a/blog/2025-10-03-iceberg-metadata.mdx b/blog/2025-10-03-iceberg-metadata.mdx index 2249efc0d..c5d1fc9c9 100644 --- a/blog/2025-10-03-iceberg-metadata.mdx +++ b/blog/2025-10-03-iceberg-metadata.mdx @@ -524,4 +524,63 @@ By decoupling the logical table from the physical data layout, Iceberg's metadat Ultimately, Apache Iceberg represents a fundamental shift in how we manage data at scale. It treats metadata not as a necessary evil, but as the primary key to unlocking performance, reliability, and modern data engineering workflows. For any organization looking to build a robust and future-proof data platform, understanding and leveraging this powerful metadata system is no longer just an option—it is the path forward. + +## FAQs + + +

    Apache Iceberg metadata is a multi-layered system of JSON and Avro files, catalog pointer → metadata.json → manifest list → manifest files, that fully describes a table's schema, partitioning, snapshots, and exact data file locations.

    +

    It matters because it replaces slow, costly directory listings with fast, versioned file indexes, enabling ACID transactions, time travel, and concurrent reads/writes on a plain object store like S3.

    + + }, + { + question: "Q2. What files make up Apache Iceberg's metadata layer?", + answer:
    +

    Four distinct layers make up Iceberg's metadata:

    +
      +
    1. Catalog: Stores a pointer to the current metadata file (e.g. version-hint.text). This is the single entry point engines use to find the table.
      2. +
    2. Table Metadata File (metadata.json): Contains the full table schema, partition specs, and complete snapshot history. Every schema change and table operation produces a new version of this file.
      3. +
    3. Manifest List: An Avro file listing all manifest files for a given snapshot, along with partition-level statistics and boundaries used for partition pruning.
      4. +
    4. Manifest Files: List individual data files with column-level statistics including min/max values (serialized to bytes), null counts, and row counts used for file-level pruning during query planning.
      5. +
    +

    Note: A manifest file stores either data files or delete files, not both. Manifests containing delete files are scanned first during query planning.

    +
    + }, + { + question: "Q3. How does Apache Iceberg achieve ACID transactions without a traditional database?", + answer:
    +

    Iceberg uses a compare-and-swap (CAS) atomic operation on the catalog. The process works as follows:

    +
      +
    1. Writers prepare all new data and metadata files independently, without locking the table
      2. +
    2. The writer then attempts to atomically swap the catalog's pointer from the old metadata.json to the new one
      3. +
    3. If this single pointer update succeeds, the transaction is committed and immediately visible to all readers
      4. +
    4. If it fails due to a concurrent write, the table remains unchanged and the operation retries
      5. +
    +

    No locks, no corruption, readers always see a consistent snapshot.

    +

    Important: Iceberg's ACID guarantees are optimized for analytics workloads. They are not designed as a replacement for RDBMS-level concurrent OLTP transactions. Iceberg does not support BEGIN/COMMIT TRANSACTION semantics across multiple statements or tables.

    +
    + }, + { + question: "Q4. What is the difference between Iceberg metadata and Hive Metastore metadata?", + answer:
    +

    The Hive Metastore stores schema and partition directory paths and requires engines to perform costly LIST operations on those directories to find actual files. In cloud environments, listing millions of files just to plan a query is slow and expensive.

    +

    Iceberg metadata, by contrast, tracks individual files with rich column-level statistics. A query engine resolves the full file list entirely through fast metadata reads, it never lists directories. File pruning happens at the manifest level using pre-computed min/max and null count statistics, not at query runtime.

    +

    Additionally, Hive's metastore is deeply involved in every query planning cycle, while Iceberg's catalog is consulted only once to fetch the current metadata.json path, all subsequent planning uses the local metadata files.

    +
    + }, + { + question: "Q5. How do you recover from a bad data pipeline run in Apache Iceberg?", + answer:
    +

    Recovery is a metadata-only operation that takes seconds:

    +
      +
    1. Identify the last known-good snapshot ID from the table's snapshot log (visible in metadata.json or via SELECT * FROM table.snapshots)
      2. +
    2. Execute a rollback command pointing current-snapshot-id back to that snapshot
      3. +
    3. The atomic commit makes the rollback immediately visible to all readers
      4. +
    +

    The corrupt data files from the bad run are now orphaned, no snapshot references them. They will be safely deleted by the next garbage collection run with no manual file cleanup required.

    +
    + } +]} /> \ No newline at end of file diff --git a/blog/2025-10-09-apache-polaris-lakehouse.mdx b/blog/2025-10-09-apache-polaris-lakehouse.mdx index 66d217fba..539703777 100644 --- a/blog/2025-10-09-apache-polaris-lakehouse.mdx +++ b/blog/2025-10-09-apache-polaris-lakehouse.mdx @@ -8,7 +8,7 @@ tags: [iceberg, polaris, trino, olake, lakehouse, cdc] image: /img/blog/cover/polaris-blog.webp --- -# Building a Scalable Lakehouse with Iceberg, Trino, OLake & Apache Polaris +# Building a Scalable Lakehouse with Iceberg, Trino, OLake Go & Apache Polaris ![Building a Scalable Lakehouse with Iceberg, Trino, OLake and Apache Polaris](/img/blog/cover/polaris-blog.webp) @@ -16,7 +16,7 @@ image: /img/blog/cover/polaris-blog.webp Modern data teams are moving toward the lakehouse architecture—combining the reliability of data warehouses with the scale and cost-efficiency of data lakes. But building one from scratch can feel overwhelming with so many moving parts. -This guide walks you through building a production-ready lakehouse using four powerful open-source tools: **Apache Iceberg** (table format), **Apache Polaris** (catalog), [**Trino**](/iceberg/query-engine/trino) (query engine), and **OLake** (data ingestion). We'll show you exactly what each component does, why it matters, and how they work together. +This guide walks you through building a production-ready lakehouse using four powerful open-source tools: **Apache Iceberg** (table format), **Apache Polaris** (catalog), [**Trino**](/iceberg/query-engine/trino) (query engine), and **OLake Go** (data ingestion). We'll show you exactly what each component does, why it matters, and how they work together. ### Understanding Apache Iceberg: The table format that changes everything @@ -60,11 +60,11 @@ Polaris was designed to solve the catalog complexity problem. Traditional catalo - Lightweight architecture that scales without the bloat - Open source with active community support -### OLake: Real-time data ingestion made simple +### OLake Go: Real-time data ingestion made simple -Now that you have Iceberg tables and a Polaris catalog, how do you actually get data into your lakehouse? This is where OLake comes in. +Now that you have Iceberg tables and a Polaris catalog, how do you actually get data into your lakehouse? This is where OLake Go comes in. -OLake is an open-source, high-performance tool specifically built to replicate data from operational databases directly into Iceberg format. It supports: +OLake Go is an open-source, high-performance tool specifically built to replicate data from operational databases directly into Iceberg format. It supports: - **Popular databases**: PostgreSQL, MySQL, MongoDB, Oracle, plus Kafka streams - **Change data capture (CDC)**: Captures every insert, update, and delete in real-time @@ -73,14 +73,14 @@ OLake is an open-source, high-performance tool specifically built to replicate d ### Why OLake over traditional ETL? -Traditional ETL tools like Debezium + Kafka + Spark require complex pipelines with multiple moving parts. OLake simplifies this dramatically: +Traditional ETL tools like Debezium + Kafka + Spark require complex pipelines with multiple moving parts. OLake Go simplifies this dramatically: - **Direct to Iceberg**: No intermediate formats or complex transformations - **Real-time sync**: Changes appear in your lakehouse within seconds - **Catalog-aware**: Automatically registers tables with Polaris - **CLI and UI**: Choose your preferred way to manage pipelines -What this means in practice: your applications keep writing to operational databases (MySQL, Postgres, MongoDB) as usual. OLake continuously captures those changes and writes them to Iceberg tables that are immediately queryable via Trino or any other Iceberg-compatible engine. +What this means in practice: your applications keep writing to operational databases (MySQL, Postgres, MongoDB) as usual. OLake Go continuously captures those changes and writes them to Iceberg tables that are immediately queryable via Trino or any other Iceberg-compatible engine. ### [Trino](/iceberg/query-engine/trino): Your high-performance query engine @@ -100,7 +100,7 @@ Trino is a distributed SQL engine designed for fast, interactive analytics on ma ![OLake CDC architecture with Trino, MySQL, Polaris, MinIO](/img/blog/2025/18/pieices-mesh.webp) -1. **Ingest**: OLake captures CDC from MySQL/Postgres/MongoDB and commits Iceberg snapshots (data + metadata) into object storage. +1. **Ingest**: OLake Go captures CDC from MySQL/Postgres/MongoDB and commits Iceberg snapshots (data + metadata) into object storage. 2. **Catalog**: Polaris exposes those tables through the Iceberg REST API so all engines share the same view of "current." 3. **Query**: Trino points its Iceberg connector at Polaris and runs federated SQL, including time-travel on Iceberg tables. @@ -109,12 +109,12 @@ Trino is a distributed SQL engine designed for fast, interactive analytics on ma We'll spin up: - **Apache Polaris** — REST catalog pointing to S3 - **MySQL** — sample source DB -- **OLake** — CDC ingestion +- **OLake Go** — CDC ingestion - **Trino** — query engine ## Prerequisites -Before deploying OLake on AWS, ensure the following setup is complete: +Before deploying OLake Go on AWS, ensure the following setup is complete: **EC2 Instance** - Must have Docker and Docker Compose installed. @@ -472,11 +472,11 @@ docker exec mysql mysql -u demo_user -pdemo_password demo_db \ *8 orders spanning different customers and dates* -For this simple test, both the source (MySQL) and OLake are running on the same EC2 instance. However, in a real-world scenario, the source can be hosted anywhere. +For this simple test, both the source (MySQL) and OLake Go are running on the same EC2 instance. However, in a real-world scenario, the source can be hosted anywhere. ### OLake :::info -Ensure the instance running **OLake** has AWS permissions equivalent to those attached to the instance hosting **Polaris** (for example, the same S3 access via IAM role/policy). +Ensure the instance running **OLake Go** has AWS permissions equivalent to those attached to the instance hosting **Polaris** (for example, the same S3 access via IAM role/policy). ::: Start OLake UI @@ -484,9 +484,9 @@ Start OLake UI curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose.yml | docker compose -f - up -d ``` -You can access the UI at port **8000**. In case you are running OLake on an EC2 instance, you can port map to your localhost using this command: `ssh -L :localhost: ` +You can access the UI at port **8000**. In case you are running OLake Go on an EC2 instance, you can port map to your localhost using this command: `ssh -L :localhost: ` -For more detailed instructions on how to run your first job using OLake refer to [Create Your First Job Pipeline](/docs/getting-started/creating-first-pipeline) +For more detailed instructions on how to run your first job using OLake Go refer to [Create Your First Job Pipeline](/docs/getting-started/creating-first-pipeline) **Create Source** @@ -614,7 +614,7 @@ SHOW CATALOGS; SHOW SCHEMAS FROM iceberg; ``` -OLake has already created and populated Iceberg tables automatically. Let's verify the data and explore Iceberg's capabilities. +OLake Go has already created and populated Iceberg tables automatically. Let's verify the data and explore Iceberg's capabilities. **Select Table:** @@ -716,7 +716,58 @@ aws s3 ls s3:/// Building a modern lakehouse doesn't have to be complex. With Iceberg + Polaris + Trino, you get warehouse-grade guarantees on low-cost object storage—with open standards and speed to match. + Welcome to the lakehouse era. 🚀 +## FAQs + + +

    Apache Polaris is an open-source, fully-featured REST catalog for Apache Iceberg, originally developed by Snowflake and contributed to the Apache Software Foundation. It manages table metadata, tracks which metadata file represents the current state of each table, and provides role-based access control.

    +

    Polaris implements the Iceberg REST Catalog specification, so any Iceberg-compatible engine (Trino, Spark, DuckDB, Flink, Dremio) works with it out of the box. It is JVM-based and requires a metadata backend (PostgreSQL or in-memory) for production deployments.

    + + }, + { + question: "Q2. How does OLake Go integrate with Apache Polaris for data ingestion?", + answer:
    +

    OLake Go connects to Apache Polaris as its Iceberg REST catalog. When OLake Go captures CDC changes from a source, it writes data as Iceberg snapshots to S3-compatible object storage and automatically registers and updates table metadata through Polaris, making data queryable by any connected engine.

    +
    + }, + { + question: "Q3. What are the advantages of Apache Polaris over Hive Metastore or AWS Glue?", + answer:
    +

    Apache Polaris offers a few advantages as an Iceberg catalog:

    +
      +
    • Cloud-independent: Polaris works with S3, MinIO, GCS, and Azure Blob Storage without requiring AWS, so data and metadata stay portable. AWS Glue is only available on AWS, which creates lock-in for multi-cloud or hybrid setups.
      • +
    • Open REST catalog, vendor-neutral: Polaris implements the Iceberg REST Catalog API, so you can swap the backing catalog without changing engine configuration. This is a portability advantage over a Glue-centric, AWS-tied setup.
      • +
    • Credential vending: Engines receive short-lived credentials scoped to specific tables, rather than needing broad storage access. Hive Metastore has no equivalent built-in mechanism.
      • +
    • Modern authentication: Polaris supports enterprise authentication via OIDC, which Hive Metastore does not offer natively.
      • +
    +

    Infrastructure note: Polaris is not dependency-free. It is JVM-based and requires a metadata backend (PostgreSQL for production, or in-memory for development). Both Polaris and Hive Metastore require a backing database, so the advantage of Polaris is its open REST API and credential vending, not an absence of infrastructure.

    +
    + }, + { + question: "Q4. How do OLake Go, Iceberg, Polaris, and Trino work together in a lakehouse stack?", + answer:
    +

    The four components each play a distinct role:

    +
      +
    1. OLake Go: Captures CDC events from operational databases (MySQL, PostgreSQL, MongoDB) and writes Iceberg tables to object storage
      2. +
    2. Apache Iceberg: The open table format that structures data as versioned Parquet files with rich metadata
      3. +
    3. Apache Polaris: Acts as the REST catalog, tracking table metadata and ensuring all engines see a consistent, up-to-date state
      4. +
    4. Trino: Connects to Polaris via its Iceberg connector and runs fast distributed SQL queries on the data
      5. +
    +

    Together they create a complete real-time analytics pipeline from operational database to query engine.

    +
    + }, + { + question: "Q5. Can Apache Polaris work with multiple query engines simultaneously?", + answer:
    +

    Yes. Because Apache Polaris implements the open Iceberg REST Catalog API, any engine that supports this standard, including Trino, Apache Spark, DuckDB, Apache Flink, Apache Doris, StarRocks, and Dremio, can query the same tables concurrently without conflicts.

    +

    This enables true multi-engine lakehouses where a single copy of data on object storage is accessible by all engines simultaneously, with Polaris ensuring a consistent, versioned view of the metadata across all consumers.

    +
    + } +]} /> diff --git a/blog/2025-10-10-how-olake-becomes-7x-faster.mdx b/blog/2025-10-10-how-olake-becomes-7x-faster.mdx index 1534d5d9d..777db6a11 100644 --- a/blog/2025-10-10-how-olake-becomes-7x-faster.mdx +++ b/blog/2025-10-10-how-olake-becomes-7x-faster.mdx @@ -10,7 +10,7 @@ image: /img/blog/cover/how-olake-becomes-7x-faster-cover.webp ![How OLake became 7x faster](/img/blog/cover/how-olake-becomes-7x-faster-cover.webp) ## Overview -Data ingestion performance is critical when you are writing in Iceberg format to data lake. When your pipeline becomes a bottleneck, it affects everything downstream—from real-time analytics to machine learning workflows. We started with OneStack in Datazip to solve the problem of Data Analytics, but then we were bound to see the problem on Data Ingestion itself. To solve this problem for iceberg native writes we built OLake. At first we created basic Iceberg writer, but then we saw the issues with it (I will be writing about the issues in a new section). Today we have resolved all those bottlenecks. +Data ingestion performance is critical when you are writing in Iceberg format to data lake. When your pipeline becomes a bottleneck, it affects everything downstream—from real-time analytics to machine learning workflows. We started with OneStack in Datazip to solve the problem of Data Analytics, but then we were bound to see the problem on Data Ingestion itself. To solve this problem for iceberg native writes we built OLake Go. At first we created basic Iceberg writer, but then we saw the issues with it (I will be writing about the issues in a new section). Today we have resolved all those bottlenecks. The result? A **7× performance improvement** in Apache Iceberg, without the complexity of background deduplication jobs or eventual consistency mechanisms. @@ -24,7 +24,7 @@ If you've worked with JVM-based data processing systems, you're probably familia Let me share the specific issues we encountered that were creating bottlenecks and preventing us from extending our codebase with new features. You might recognize some of these in your own systems: -1. **RPC Server Format**: In previous implementation we were using Debezium Formats to communicate between OLake and Java Iceberg server, which had large metadata that was mostly unused. +1. **RPC Server Format**: In previous implementation we were using Debezium Formats to communicate between OLake Go and Java Iceberg server, which had large metadata that was mostly unused. 2. **Serialization And Deserialization using JsonSchema**: At multiple places serialization and deserialization were happening, resulting in less throughput, high memory and CPU consumption. @@ -205,7 +205,7 @@ The commit process handles two scenarios: #### 2. Atomic Schema Evolution -Now we know how files are being committed atomically, but you might be asking: "What about schema changes? How does OLake handle type promotions and schema evolution in Iceberg tables?" +Now we know how files are being committed atomically, but you might be asking: "What about schema changes? How does OLake Go handle type promotions and schema evolution in Iceberg tables?" For schema evolution, we reuse the same global table atomic lock, which ensures that for any particular table, only one thread evolves the schema at a time. This prevents race conditions and ensures consistency. @@ -570,8 +570,55 @@ This refactor demonstrates several important principles for building high-perfor The result is a system that is not only faster but also more reliable, maintainable, and operationally friendly. +*OLake Go is an open-source CDC and data ingestion platform for Apache Iceberg. Built for correctness, designed for speed, optimized for operations.* + +## FAQs + + +

    OLake Go's destination refactor raised throughput from about 46,000 to about 320,000 records per second, roughly a 7x gain, by combining several changes: replacing the Debezium message format with a compact typed gRPC contract, removing JSON serialization and type detection overhead, simplifying double buffering into a single configurable batch, and moving concurrency and type handling from Java to Go.

    +

    No single change delivered the 7x on its own. It came from these optimizations compounding together, while also cutting memory usage from over 80GB to around 40GB.

    + + }, + { + question: "Q2. How does OLake Go guarantee atomic commits when writing to Apache Iceberg?", + answer:
    +

    Each writer thread writes to its own data files, then registration into the Iceberg table happens under a single table-level lock. Either the entire batch becomes visible or none of it does, so readers never see partial state, even during concurrent writes or if a commit fails midway.

    +

    The commit path depends on the operation. Pure inserts use AppendFiles to add new data files. CDC updates and deletes use RowDelta (equality-delete Merge-on-Read), which atomically adds both the data files and the delete files in one operation.

    +
    + }, + { + question: "Q3. Why does OLake Go write fewer, larger files instead of many small ones during full refresh?", + answer:
    +

    This applies to full refresh syncs, where OLake Go controls the entire chunk from source read to commit. Small files hurt query performance, because engines have to read metadata from many files per scan. The previous writer flushed buffers on a memory threshold, which produced inconsistent 30 to 50MB files.

    +

    The new writer commits after a full chunk (about 4GB in historical snapshots, compressing to roughly 350MB), targeting consistent 300 to 400MB files. This gives query engines fewer, evenly sized files to scan.

    +

    Note: During CDC, OLake Go cannot guarantee this same file size consistency. Change events arrive continuously rather than in large historical chunks, so CDC commits produce smaller, more variable file sizes. Compaction is what brings CDC-written files up to optimal size. OLake Fusion is a table maintenance tool that automates this compaction for you.

    +
    + }, + { + question: "Q4. How does OLake Go handle Iceberg schema evolution across concurrent writer threads?", + answer:
    +

    OLake Go keeps a per-thread schema and a global table schema on the Go side. Each thread compares its batch against the global schema and only acquires the table-level lock if it detects a real change, which keeps contention low. The first thread to acquire the lock performs the evolution, then all threads refresh their writers to the new schema.

    +

    For safe type promotions like int to long, OLake Go avoids closing and refreshing the writer until a record of the larger type actually arrives, saving unnecessary file churn.

    +
    + }, + { + question: "Q5. Why does OLake Go split its Iceberg writer between Go and Java?", + answer:
    +

    The refactor assigns each language the work it does best. OLake Go handles the data plane: concurrency, batching, type detection, and schema coordination. Java handles Iceberg I/O, since the Java Iceberg library is the most mature implementation and tends to get new features first.

    +

    The two communicate over a typed gRPC contract rather than a JSON envelope, which removes serialization overhead while keeping the Java server as a focused API layer for writing Iceberg files.

    +
    + }, + { + question: "Q6. Is Apache Iceberg suitable for real-time streaming ingestion?", + answer:
    +

    Not for sub-second latency, at least until the small-file problem is solved. Iceberg is designed for batch-oriented writes, and writing tiny files continuously creates the same fragmentation that slows queries. Even tools like Tableflow batch data before writing to Iceberg.

    +

    OLake Go works within this batch model by buffering into larger chunks and committing them atomically, rather than streaming individual records into the table.

    +
    + } +]} /> -*OLake is an open-source CDC and data ingestion platform for Apache Iceberg. Built for correctness, designed for speed, optimized for operations.* - \ No newline at end of file diff --git a/blog/2025-10-16-iceberg-vs-parquet-table-format-vs-file-format.mdx b/blog/2025-10-16-iceberg-vs-parquet-table-format-vs-file-format.mdx index 19ced98c3..f472f8347 100644 --- a/blog/2025-10-16-iceberg-vs-parquet-table-format-vs-file-format.mdx +++ b/blog/2025-10-16-iceberg-vs-parquet-table-format-vs-file-format.mdx @@ -384,55 +384,6 @@ These tables are the primary tool for an architect to validate that compaction a Of course. We are approaching the conclusion of our architectural blueprint. Before the final summary, it is essential to address the common, practical questions that arise during implementation. This section serves as a direct, authoritative reference to clarify key distinctions and operational realities. - -## FAQ: People Also Ask - -### Is Iceberg a replacement for Parquet? - -No. This is the most fundamental misconception. **Iceberg does not replace Parquet**; it organizes it. They operate at two different architectural layers to solve two completely different problems. - -Let's make this concrete. Think of your data lake as a massive digital music library. - -- **Parquet** files are the individual **MP3 files**. Each one is a perfectly encoded, high-fidelity container for the actual music—your data. It is the raw asset. - -- **Iceberg** is the **playlist**. The playlist file itself contains no music. It is a simple metadata file that points to the specific MP3s that constitute your "Workout Mix". It provides the logical grouping, the name, and the order. - -You can add or remove a song from the playlist (a transaction) or see what the playlist looked like last week (time travel) without ever altering the underlying MP3 files. Iceberg is the management layer; Parquet is the storage layer. - -### Can you use Iceberg with other file formats like ORC or Avro? - -Yes, absolutely. The Iceberg specification is **file-format-agnostic**. While it is most commonly used with Apache Parquet for analytical workloads due to Parquet's columnar performance benefits, it is fully capable of managing tables composed of **Apache ORC** or **Apache Avro** files. This flexibility is a core design principle, ensuring that the table format does not lock you into a single storage format. - -### What are the main differences between Iceberg, Delta Lake, and Hudi? - -All three are open table formats designed to solve similar problems (ACID transactions, schema evolution, time travel). The primary differences lie in their design philosophy and underlying implementation. - -- **Apache Iceberg:** Prioritizes a universal, open specification with zero engine dependencies. Its greatest strengths are **fast query planning at massive scale** (via its manifest file indexes) and **guaranteed interoperability**. It is architected to avoid the "list-then-filter" problem that can plague other formats on petabyte-scale tables, making it a robust choice for multi-engine, large-scale data lakehouses. - -- **Delta Lake:** Originated at Databricks and is deeply integrated with the Apache Spark ecosystem. It uses a chronological JSON transaction log (`_delta_log`) to track table state. It is often considered the most straightforward to adopt if your organization is already standardized on Databricks and Spark. - -- **Apache Hudi:** Originated at Uber with a strong focus on low-latency streaming ingest and incremental processing. It offers more granular control over the trade-off between write performance and read performance through its explicit **Copy-on-Write** and **Merge-on-Read** storage types. - -The choice is one of architectural trade-offs. Iceberg is built for interoperability and scale, Delta for deep integration with Spark, and Hudi for fine-grained control over streaming workloads. - -### Does using Iceberg add significant performance overhead? - -On the contrary, for any non-trivial table, Iceberg provides a **significant performance improvement**. - -The perceived "overhead" is the storage of a few extra kilobytes of metadata files. The problem it solves is the primary performance **bottleneck** in cloud data lakes: recursively listing the millions of files that make up a large table. This `LIST` operation is notoriously slow and expensive. - -Iceberg avoids this entirely by using its manifest files as a pre-built index of the table's data files. The query engine reads this small index to find the exact files it needs to scan, transforming a slow file-system operation into a fast metadata lookup. It trades a negligible amount of storage for a massive gain in query planning speed. - -### How does Iceberg handle row-level deletes on Parquet files? - -It's critical to remember that Parquet files are **immutable**. Iceberg never changes an existing Parquet file. Instead, it handles deletes using a metadata-driven, **merge-on-read** approach. - -When a `DELETE` command is issued, Iceberg creates lightweight **delete files**. These files store the path to a data file and the specific row positions within that file that are marked for deletion. At query time, the engine reads both the original Parquet data file and its associated delete file, merging them on the fly to present a view of the data where the deleted rows are filtered out. - -Think of it as an errata slip published for a book. The original book text is not altered, but the slip tells the reader to ignore a specific sentence on a specific page. The process of making this deletion permanent by rewriting the data files is handled by a separate, asynchronous **compaction** job. - - - ## Conclusion We began this discussion by dissecting the broken promise of the first-generation data lake—a system that offered immense scale but was fundamentally **brittle**, unreliable, and operationally expensive to manage. The root of this fragility was its architecture: a simple collection of files in a directory is not a database. It lacks the transactional integrity, the metadata intelligence, and the structural flexibility required for mission-critical work. @@ -447,4 +398,114 @@ Therefore, the architectural conclusion is clear. The question is not **Parquet For any serious data lake initiative that demands reliability, performance, and agility, the choice is no longer *if* you should adopt a modern table format. The only question is how you will leverage a format like Iceberg to unlock the true potential of your data. To build a future-proof data platform, you need both the optimal storage container and the master blueprint, i.e. **Parquet with Iceberg**! +## FAQs + + +

    No. This is the most fundamental misconception. Iceberg does not replace Parquet; it organizes it. They operate at two different architectural layers to solve two completely different problems.

    +

    Let's make this concrete. Think of your data lake as a massive digital music library.

    +
      +
    • Parquet files are the individual MP3 files. Each one is a perfectly encoded, high-fidelity container for the actual music your data. It is the raw asset.
      • +
    • Iceberg is the playlist. The playlist file itself contains no music. It is a simple metadata file that points to the specific MP3s that constitute your "Workout Mix". It provides the logical grouping, the name, and the order.
      • +
    +

    You can add or remove a song from the playlist (a transaction) or see what the playlist looked like last week (time travel) without ever altering the underlying MP3 files. Iceberg is the management layer; Parquet is the storage layer.

    + + }, + { + question: "Q2. Can you use Iceberg with other file formats like ORC or Avro?", + answer:
    +

    Yes, absolutely. The Iceberg specification is file-format-agnostic. While it is most commonly used with Apache Parquet for analytical workloads due to Parquet's columnar performance benefits, it is fully capable of managing tables composed of Apache ORC or Apache Avro files. This flexibility is a core design principle, ensuring that the table format does not lock you into a single storage format.

    +
    + }, + { + question: "Q3. What are the main differences between Iceberg, Delta Lake, and Hudi?", + answer:
    +

    All three are open table formats designed to solve similar problems (ACID transactions, schema evolution, time travel). The primary differences lie in their design philosophy and underlying implementation.

    +
      +
    • Apache Iceberg: Prioritizes an open specification that is independent of any single compute engine. Its hierarchical metadata enables efficient query planning on very large tables without relying on expensive object store directory listings, which makes it a strong choice for multi-engine lakehouses at scale.
      • +
    • Delta Lake: Originated at Databricks and is deeply integrated with the Apache Spark ecosystem. It uses a chronological JSON transaction log (_delta_log) to track table state. It is often considered the most straightforward to adopt if your organization is already standardized on Databricks and Spark.
      • +
    • Apache Hudi: Originated at Uber with a strong focus on low-latency streaming ingest and incremental processing. It offers more granular control over the trade-off between write and read performance through its explicit Copy-on-Write and Merge-on-Read storage types.
      • +
    +

    The choice is one of architectural trade-offs. Iceberg is built for interoperability and scale, Delta for deep integration with Spark, and Hudi for fine-grained control over streaming workloads.

    +
    + }, + { + question: "Q4. Does using Iceberg add significant performance overhead?", + answer:
    +

    On the contrary, for any non-trivial table, Iceberg provides a significant performance improvement.

    +

    The perceived "overhead" is the storage of a few extra kilobytes of metadata files. The problem it solves is the primary performance bottleneck in cloud data lakes: recursively listing the millions of files that make up a large table. This LIST operation is notoriously slow and expensive.

    +

    Iceberg avoids this entirely by using its manifest files as a pre-built index of the table's data files. The query engine reads this small index to find the exact files it needs to scan, transforming a slow file-system operation into a fast metadata lookup. It trades a negligible amount of storage for a massive gain in query planning speed.

    +
    + }, + { + question: "Q5. How does Iceberg handle row-level deletes on Parquet files?", + answer:
    +

    It's critical to remember that Parquet files are immutable. Iceberg never changes an existing Parquet file. Instead, it handles deletes using a metadata-driven, merge-on-read approach.

    +

    When a DELETE command is issued, Iceberg creates lightweight delete files. These files store the path to a data file and the specific row positions within that file that are marked for deletion. At query time, the engine reads both the original Parquet data file and its associated delete file, merging them on the fly to present a view of the data where the deleted rows are filtered out.

    +

    Think of it as an errata slip published for a book. The original book text is not altered, but the slip tells the reader to ignore a specific sentence on a specific page. The process of making this deletion permanent by rewriting the data files is handled by a separate, asynchronous compaction job.

    +
    + }, + { + question: "Q6. What is the difference between Apache Parquet and Apache Iceberg?", + answer:
    +

    Apache Parquet is a columnar file format that physically stores data on disk with efficient compression and fast analytical reads.

    +

    Apache Iceberg is an open table format a metadata and management layer that sits on top of Parquet files. Iceberg does not replace Parquet; it manages collections of Parquet files to add:

    +
      +
    • ACID transactions
      • +
    • Schema evolution
      • +
    • Time travel
      • +
    • Reliable concurrent reads and writes
      • +
    +
    + }, + { + question: "Q7. Can I use Apache Iceberg without Parquet?", + answer:
    +

    Yes. Apache Iceberg supports multiple underlying file formats including Parquet, Avro, and ORC, though Parquet is by far the most widely used due to its columnar efficiency and broad engine support.

    +

    Iceberg's value comes from its metadata layer, which works independently of which file format stores the actual data. You can mix file formats within the same table across different operations.

    +
    + }, + { + question: "Q8. What makes Apache Iceberg better than storing raw Parquet files in S3?", + answer:
    +

    Raw Parquet files in S3 lack transactional guarantees concurrent writes can corrupt data, schema changes require full file rewrites, and there is no native time travel or partition evolution.

    +

    Apache Iceberg adds:

    +
      +
    • Atomic ACID commits writes are either fully visible or not at all
      • +
    • Schema and partition evolution without data rewrites these are metadata-only operations that leave existing files untouched
      • +
    • Full time-travel query support query any historical snapshot by timestamp or snapshot ID
      • +
    • Metadata-driven file pruning query planners skip irrelevant files using pre-computed statistics, dramatically speeding up queries
      • +
    +
    + }, + { + question: "Q9. How does Apache Iceberg provide ACID transactions on object storage like S3?", + answer:
    +

    Iceberg achieves ACID transactions through atomic metadata swaps:

    +
      +
    1. Every write creates new Parquet data files and a new metadata JSON file independently
      2. +
    2. The commit is a single atomic pointer update in the catalog from the old metadata file to the new one
      3. +
    3. If a write fails, the catalog pointer never changes readers always see a consistent state
      4. +
    4. Readers never acquire locks, so concurrent reads are never blocked by in-progress writes
      5. +
    +
    + }, + { + question: "Q10. What is the 'hidden partitioning' feature in Apache Iceberg?", + answer:
    +

    Hidden partitioning means that Iceberg records partition values in its manifest files and metadata layer, so query engines can use these pre-recorded values to skip irrelevant files automatically without users needing to write explicit partition filters in SQL.

    +

    Unlike Hive-style partitioning where users must write WHERE dt = '2024-01-01' to trigger partition pruning, Iceberg's engines handle this automatically by reading partition information from the manifests.

    +

    Key benefits:

    +
      +
    • Prevents user errors queries are correct even without explicit partition filters
      • +
    • Simplifies SQL no need to know how the table is physically partitioned
      • +
    • Enables partition evolution partition strategies can be changed over time without rewriting historical data. Old data written under the previous spec remains unchanged; new data is written using the new layout
      • +
    +
    + } +]} /> + + diff --git a/blog/2025-11-03-olake-bauplan.mdx b/blog/2025-11-03-olake-bauplan.mdx index ce9cf1815..7d822bcaf 100644 --- a/blog/2025-11-03-olake-bauplan.mdx +++ b/blog/2025-11-03-olake-bauplan.mdx @@ -15,9 +15,9 @@ import BlogCTA from '@site/src/components/BlogCTA'; If you've ever tried to build a data lake, you know it rarely feels simple. Data sits across operational systems (PostgreSQL, Oracle, MongoDB) and getting it into a usable analytical format means chaining together multiple tools for ingestion, transformation, orchestration, and governance. Each layer adds cost, complexity, and maintenance overhead. You end up managing clusters, debugging pipelines, and paying for infrastructure that sits idle more often than it runs. -This is where OLake and Bauplan change the game. OLake moves your data from databases to Apache Iceberg seamlessly skipping the headache of developing custom ETL pipelines. Bauplan, on the other hand, lets you build and run your data transformations serverlessly — in Python or SQL, with no provisioning or maintenance. Together, they form a **serverless open data lakehouse**. +This is where OLake and Bauplan change the game. OLake Go moves your data from databases to Apache Iceberg seamlessly skipping the headache of developing custom ETL pipelines. Bauplan, on the other hand, lets you build and run your data transformations serverlessly — in Python or SQL, with no provisioning or maintenance. Together, they form a **serverless open data lakehouse**. -In this blog, I'll show you how **OLake** and **Bauplan** work together with **Apache Iceberg** to create a data platform that actually makes sense - one where your operational data flows seamlessly into your Data Lakehouse, where your data team can work like software engineers with branches and merges. +In this blog, I'll show you how **OLake Go** and **Bauplan** work together with **Apache Iceberg** to create a data platform that actually makes sense - one where your operational data flows seamlessly into your Data Lakehouse, where your data team can work like software engineers with branches and merges. ## What's a Data Lakehouse, Anyway? @@ -27,7 +27,7 @@ But here's the challenge: building a modern lakehouse that's real-time, version- ## The Three Building Blocks -**OLake** is the fastest and most efficient way to replicate your data from databases (like Postgres, MySQL, MongoDB) to a Data Lakehouse. It's an open-source tool that captures changes using CDC (Change Data Capture) and writes them directly as Apache Iceberg tables on object storage. Think of it as a high-speed bridge between your production databases and your data lakehouse. +**OLake Go** is the fastest and most efficient way to replicate your data from databases (like Postgres, MySQL, MongoDB) to a Data Lakehouse. It's an open-source tool that captures changes using CDC (Change Data Capture) and writes them directly as Apache Iceberg tables on object storage. Think of it as a high-speed bridge between your production databases and your data lakehouse. **Bauplan** is a serverless data processing platform built for Apache Iceberg. It automatically runs your SQL queries and Python transformations whenever you need them—no servers to set up, no infrastructure to manage. What makes it special? It works like Git: you can create separate branches to test your data transformations, run queries against branch-specific data, and only merge to production when you're confident everything works. No more accidentally breaking production dashboards while testing. @@ -48,9 +48,9 @@ Here's the complete picture of how data moves through the system: ### The Architecture -**1. OLake** performs a historical-load and CDC of data from Postgres to Iceberg tables stored in an S3 bucket. +**1. OLake Go** performs a historical-load and CDC of data from Postgres to Iceberg tables stored in an S3 bucket. -**2. Iceberg Tables** are written directly to S3 by OLake. Each table consists of data files (Parquet), metadata files, and manifest files that track the table's structure and history. +**2. Iceberg Tables** are written directly to S3 by OLake Go. Each table consists of data files (Parquet), metadata files, and manifest files that track the table's structure and history. **3. Lakekeeper** acts as the Iceberg REST Catalog. It manages table metadata, tracks table versions, and coordinates access across different tools. @@ -90,7 +90,7 @@ Once you have started the services you can access the Lakekeeper UI at: http://l -**Step 2: Set up OLake** +**Step 2: Set up OLake Go** Deploy the OLake UI with a single command. This starts the OLake UI and backend services: @@ -117,7 +117,7 @@ To access OLake UI from your local machine, make sure you set up SSH port forwar **Step 3: Configure OLake Job** -Now let's configure OLake to sync data from your source database to Iceberg. +Now let's configure OLake Go to sync data from your source database to Iceberg. If you're new to OLake, refer to our guide on [creating your first job pipeline](https://olake.io/docs/getting-started/creating-first-pipeline/) for detailed instructions. @@ -274,13 +274,75 @@ You will see the query results like this: -You've just built a complete data lakehouse stack that bridges operational databases and analytics—without vendor lock-in, without proprietary formats, and without complexity. OLake continuously syncs your Postgres data to Iceberg tables, Lakekeeper manages the metadata catalog, and Bauplan gives your team Git-style workflows for safe, collaborative data development. +You've just built a complete data lakehouse stack that bridges operational databases and analytics—without vendor lock-in, without proprietary formats, and without complexity. OLake Go continuously syncs your Postgres data to Iceberg tables, Lakekeeper manages the metadata catalog, and Bauplan gives your team Git-style workflows for safe, collaborative data development. ## Useful Resources -- [OLake Documentation](https://olake.io/docs) - Complete guide to setting up OLake with various sources and destinations +- [OLake Go Documentation](https://olake.io/docs) - Complete guide to setting up OLake Go with various sources and destinations - [Bauplan Documentation](https://docs.bauplanlabs.com) - Learn about branch workflows and data transformations - [Lakekeeper](https://lakekeeper.io) - Open-source Iceberg REST catalog - [Apache Iceberg](https://iceberg.apache.org) - The open table format powering this architecture +## FAQs + +

    Bauplan is a serverless data processing platform built for Apache Iceberg. It runs SQL queries and Python transformations without servers to provision or manage no containerization, Terraform, or Spark clusters required.

    +

    Its key differentiator is a Git-like branching model powered by Project Nessie as its underlying catalog. All tables are stored as Iceberg tables in your own S3 bucket. You can:

    +
      +
    • Create data branches to test transformations safely
      • +
    • Run queries against branch-specific data
      • +
    • Merge atomically to production only when confident
      • +
    +

    This prevents accidental damage to production dashboards and enables safe, iterative data development.

    + + }, + { + question: "Q2. How do OLake Go and Bauplan work together in a serverless Iceberg lakehouse?", + answer:
    +

    OLake Go performs the historical load and real-time CDC replication from operational databases (Postgres, MySQL, MongoDB) directly into Apache Iceberg tables stored in S3. OLake Go uses Lakekeeper (or another Iceberg REST catalog) to register and manage its table metadata.

    +

    Bauplan connects to those Iceberg tables via its own Nessie-based catalog to run serverless SQL and Python transformations. Because Bauplan's catalog is Nessie-based rather than a plain Iceberg REST catalog, catalog interoperability between OLake Go and Bauplan must be explicitly configured they do not share a single catalog endpoint by default.

    +

    OLake Go handles ingestion; Bauplan handles transformation no Spark clusters or shared infrastructure required.

    +
    + }, + { + question: "Q3. What is Lakekeeper's role in the OLake Go and Bauplan stack?", + answer:
    +

    Lakekeeper is the Apache Iceberg REST catalog that serves OLake Go specifically in this stack. OLake Go registers new tables and snapshots through Lakekeeper after each CDC sync, ensuring the latest table state is always tracked and available.

    +

    Bauplan, however, operates from its own separate Nessie-based catalog it does not read from Lakekeeper directly. In an OLake Go + Bauplan architecture, the two tools maintain separate catalog layers:

    +
      +
    • Lakekeeper serves OLake Go's Iceberg table registration and snapshot management
      • +
    • Bauplan's Nessie catalog serves Bauplan's branching, versioning, and transformation workflows
      • +
    +

    For data written by OLake Go to be consumed by Bauplan, the files can be landed in S3 and imported into a Bauplan branch, or writers can be configured to target Bauplan's Nessie endpoint directly.

    +
    + }, + { + question: "Q4. Why use Iceberg as the foundation for a serverless lakehouse?", + answer:
    +

    Apache Iceberg provides the core properties that make a multi-tool serverless environment reliable:

    +
      +
    • ACID transactions writes are atomic and never partially visible
      • +
    • Snapshot isolation readers and writers never interfere with each other, so OLake Go can write new data while Bauplan simultaneously reads and transforms existing snapshots
      • +
    • Time travel query any historical snapshot by timestamp or snapshot ID
      • +
    • Schema evolution add or rename columns without rewriting data files
      • +
    • Cheap object storage all data lives in S3 or S3-compatible storage you own and control
      • +
    +
    + }, + { + question: "Q5. What are the prerequisites for setting up an OLake Go and Bauplan lakehouse?", + answer:
    +

    You need:

    +
      +
    • An S3 bucket in us-east-1 Bauplan requires this region for its managed compute layer
      • +
    • Docker installed for running OLake Go and Lakekeeper locally
      • +
    • A Bauplan account with the Bauplan CLI installed
      • +
    • A source database such as PostgreSQL a local Postgres instance via Docker is sufficient for testing
      • +
    +

    Write access to the S3 bucket is required for both OLake Go(to write Iceberg data files) and Bauplan (to manage its own Iceberg tables in your bucket).

    +
    + } +]} /> \ No newline at end of file diff --git a/blog/2025-11-04-postgres-iceberg-doris-lakehouse-olake.mdx b/blog/2025-11-04-postgres-iceberg-doris-lakehouse-olake.mdx index 6dc68d09f..a464099fb 100644 --- a/blog/2025-11-04-postgres-iceberg-doris-lakehouse-olake.mdx +++ b/blog/2025-11-04-postgres-iceberg-doris-lakehouse-olake.mdx @@ -1,6 +1,6 @@ --- title: "Postgres → Iceberg → Doris: A Smooth Lakehouse Journey Powered by Olake" -description: "Learn how to build a complete lakehouse architecture using PostgreSQL, Apache Iceberg, and Apache Doris for real-time analytics. Step-by-step guide with OLake for seamless data ingestion." +description: "Learn how to build a complete lakehouse architecture using PostgreSQL, Apache Iceberg, and Apache Doris for real-time analytics. Step-by-step guide with OLake Go for seamless data ingestion." slug: postgres-iceberg-doris-lakehouse-olake date: 2025-11-04 authors: [badal] @@ -21,7 +21,7 @@ Building a modern data lakehouse shouldn't require stitching together a dozen to - Captures real-time changes from PostgreSQL using CDC (Change Data Capture) - Stores data in open Apache Iceberg format on object storage - Queries data at lightning speed with Apache Doris -- All orchestrated seamlessly by OLake +- All orchestrated seamlessly by OLake Go By the end, you'll have a running system that syncs database changes in real-time and lets you query — without moving or duplicating data. @@ -65,7 +65,7 @@ Here's how data flows through our lakehouse stack: 1. **Source Database (PostgreSQL)**: Your operational database continues running normally, handling transactional workloads. -2. **OLake CDC Engine**: Captures changes from PostgreSQL using logical replication and writes them directly to Apache Iceberg format. +2. **OLake Go CDC Engine**: Captures changes from PostgreSQL using logical replication and writes them directly to Apache Iceberg format. 3. **Apache Iceberg Tables**: Your data lands in Iceberg tables stored in object storage (MinIO/S3), maintaining full ACID guarantees with snapshot isolation. @@ -77,17 +77,17 @@ Here's how data flows through our lakehouse stack: **No Data Duplication**: Unlike traditional ETL pipelines that copy data multiple times, your source data is written once to Iceberg and queried directly by Doris. -**Real-Time Insights**: Changes in PostgreSQL appear in your analytics, OLake's CDC sync captures inserts, updates, and deletes as they happen. +**Real-Time Insights**: Changes in PostgreSQL appear in your analytics, OLake Go's CDC sync captures inserts, updates, and deletes as they happen. **Cost-Effective Storage**: Object storage (S3/MinIO) costs a fraction of traditional data warehouse storage, while Iceberg's efficient metadata handling keeps query performance high. **Decoupled Compute and Storage**: Scale your query engine (Doris) independently from storage. Need more query power? Add Doris nodes. Need more storage? Just expand your object store. -### About OLake +### About OLake Go -OLake is an open-source CDC tool specifically built for lakehouse architectures. It supports these sources: **PostgreSQL, MySQL, MongoDB, Oracle,** and **Kafka**. You can check out our [official documentation](/docs) for detailed source configurations. +OLake Go is an open-source CDC tool specifically built for lakehouse architectures. It supports these sources: **PostgreSQL, MySQL, MongoDB, Oracle,** and **Kafka**. You can check out our [official documentation](/docs) for detailed source configurations. -What makes OLake different? It writes directly to Apache Iceberg format with proper metadata management, schema evolution support, and automatic handling of CDC operations (inserts, updates, deletes). No need for complex Spark jobs or Kafka pipelines — OLake handles the entire ingestion flow. We support all major Iceberg catalogs. +What makes OLake Go different? It writes directly to Apache Iceberg format with proper metadata management, schema evolution support, and automatic handling of CDC operations (inserts, updates, deletes). No need for complex Spark jobs or Kafka pipelines — OLake Go handles the entire ingestion flow. We support all major Iceberg catalogs. ### Our Demo Setup @@ -182,19 +182,19 @@ bash start_doris_client.sh This opens the Doris SQL client, and you can now run queries against your lakehouse. We'll use this later to query the Iceberg tables. -## Step 2 – Set Up OLake for CDC Ingestion +## Step 2 – Set Up OLake Go for CDC Ingestion -Now we'll configure OLake to capture changes from your PostgreSQL database and write them to Iceberg tables. +Now we'll configure OLake Go to capture changes from your PostgreSQL database and write them to Iceberg tables. ### Start OLake UI -Open a new terminal session on your cloud instance and deploy OLake: +Open a new terminal session on your cloud instance and deploy OLake Go: ```bash curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose.yml | docker compose -f - up -d ``` -This starts the OLake UI and backend services. OLake runs on port 8000, but since it's on your remote cloud instance, you'll need to access it from your local machine. +This starts the OLake UI and backend services. OLake Go runs on port 8000, but since it's on your remote cloud instance, you'll need to access it from your local machine. ### Set Up SSH Port Forwarding @@ -243,7 +243,7 @@ This bucket will hold all your Iceberg table data files (Parquet) and metadata ( ### Configure OLake Job -Now let's configure OLake to sync data from your source database to Iceberg. +Now let's configure OLake Go to sync data from your source database to Iceberg. **Create Source Connection**: 1. In OLake UI, navigate to **Sources** → **Create Source** @@ -276,7 +276,7 @@ You can check out our [official documentation](/docs/getting-started/creating-fi ## Step 3 – Query Your Iceberg Tables with Doris -With OLake continuously syncing data to Iceberg, it's time to query that data using Apache Doris. Let's explore your lakehouse! +With OLake Go continuously syncing data to Iceberg, it's time to query that data using Apache Doris. Let's explore your lakehouse! ### Connect to Doris @@ -308,7 +308,7 @@ The Iceberg catalog might not immediately reflect newly created tables. Refresh REFRESH CATALOG iceberg; ``` -**Why refresh?** Doris caches catalog metadata for performance. When OLake creates new tables or updates schemas, refreshing ensures Doris sees the latest state. +**Why refresh?** Doris caches catalog metadata for performance. When OLake Go creates new tables or updates schemas, refreshing ensures Doris sees the latest state. ### Explore Your Data @@ -354,11 +354,67 @@ sudo sysctl -w vm.max_map_count=2000000 then restart your Doris BE and then run your table query command and it should work fine. -

    -

    - - **Happy Engineering! Happy Iceberg!** +## FAQs + +

    Apache Doris is a real-time analytical database built on MPP (Massively Parallel Processing) architecture that delivers sub-second query latency on large datasets. It queries Apache Iceberg tables directly from object storage without moving or duplicating data, and uses vectorized execution and a smart query optimizer to maximize performance.

    +

    Native Iceberg features supported include:

    +
      +
    • Time travel: query historical snapshots by timestamp or snapshot ID
      • +
    • Equality delete files: full support for CDC-generated delete records
      • +
    • Positional delete files: efficient row-level delete reads
      • +
    • Deletion Vectors (Doris 4.1.0+): compact binary delete format introduced in Iceberg v3
      • +
    + + }, + { + question: "Q2. How does OLake Go sync PostgreSQL data to Apache Iceberg for Doris to query?", + answer:
    +

    OLake Go uses Change Data Capture (CDC) via PostgreSQL logical replication to capture every insert, update, and delete from the source database in real time. It writes changes directly as Apache Iceberg tables to object storage (MinIO or S3) with:

    +
      +
    • Proper metadata management via an Iceberg REST catalog
      • +
    • Schema evolution support for source table changes
      • +
    • ACID commit guarantees through Iceberg's atomic snapshot model
      • +
    +

    Doris then connects to the Iceberg REST catalog to query these tables with no data movement.

    +
    + }, + { + question: "Q3. What makes this Postgres-to-Iceberg-to-Doris architecture cost-effective?", + answer:
    +

    Three factors drive cost efficiency:

    +
      +
    1. Single copy of data: Data is written once to Iceberg on S3-compatible storage and queried directly by Doris. A traditional pipeline copies data again into the query engine's own storage, so you pay to store and move it twice. Here there is only one copy and no separate storage layer to fund for the query engine.
      2. +
    2. Independent scaling: Compute (Doris) and storage scale independently based on actual workload demands, so you are not forced to over-provision one to grow the other.
      3. +
    3. Simplified infrastructure: OLake Go removes the need for complex Kafka and Spark pipeline infrastructure, reducing both infrastructure cost and operational overhead.
      4. +
    +
    + }, + { + question: "Q4. What CDC operations does OLake Go capture for PostgreSQL to Iceberg replication?", + answer:
    +

    OLake Go captures all three DML operations from PostgreSQL:

    +
      +
    • INSERT: new rows written to the source table
      • +
    • UPDATE: changed rows, captured as delete + insert pairs in Iceberg
      • +
    • DELETE: removed rows, written as equality or positional delete files in Iceberg
      • +
    +

    OLake Go uses PostgreSQL's logical replication and WAL (Write-Ahead Log) to track these changes in real time, writing them to Iceberg tables so that Doris queries always reflect the latest state of the source database.

    +
    + }, + { + question: "Q5. Does Apache Doris support querying Iceberg tables with equality delete files created by OLake Go?", + answer:
    +

    Yes. Apache Doris supports both equality delete files and positional delete files in Apache Iceberg the formats OLake Go uses for CDC operations. This support predates the 4.0 release and is available across currently maintained Doris versions.

    +

    For tables using Deletion Vectors (the compact binary delete format introduced in Iceberg v3), Doris 4.1.0 or later is required.

    +

    This makes Doris a fully compatible query engine for OLake Go-generated Iceberg tables without any conversion or compaction required for standard v2 CDC workloads.

    +
    + } +]} /> + diff --git a/blog/2025-11-13-olake-souce-kafka.mdx b/blog/2025-11-13-olake-souce-kafka.mdx index eafa8cf03..f8d4b29ff 100644 --- a/blog/2025-11-13-olake-souce-kafka.mdx +++ b/blog/2025-11-13-olake-souce-kafka.mdx @@ -1,6 +1,6 @@ --- slug: olake-kafka-iceberg -title: "Deep Dive into Kafka as a Source in OLake: Unpacking Sync, Concurrency, and Partition Mastery" +title: "Deep Dive into Kafka as a Source in OLake Go: Unpacking Sync, Concurrency, and Partition Mastery" description: Explore OLake's Kafka source connector—featuring schema discovery, custom group balancing, partition-aware concurrency, and incremental batch sync to Apache Iceberg with exactly-once semantics. tags: [iceberg, olake, kafka] authors: [duke, shubham] @@ -12,13 +12,13 @@ import TabItem from '@theme/TabItem'; ![Kafka blog cover image](/img/blog/2025/11/kafka_blog_cover.webp) -In the world of data pipelines, Kafka has become the backbone of real-time event streaming. That's why we built OLake's Kafka source connector. It's designed to pull data directly from Kafka topics and land it in Apache Iceberg tables - with proper schema evolution, atomic commits, and state management. No external streaming engines, no complex orchestration. Just OLake reading from Kafka and writing to your data lake. +In the world of data pipelines, Kafka has become the backbone of real-time event streaming. That's why we built OLake Go's Kafka source connector. It's designed to pull data directly from Kafka topics and land it in Apache Iceberg tables - with proper schema evolution, atomic commits, and state management. No external streaming engines, no complex orchestration. Just OLake reading from Kafka and writing to your data lake. -If you're familiar with OLake's technology, you’ll see how OLake connects effortlessly with Kafka’s distributed ecosystem through its flexible, pluggable design. This post breaks down how we built it, the design decisions we made, and why certain things work the way they do. If you're running data pipelines from Kafka to a lakehouse, this might save you some pain. +If you're familiar with OLake Go's technology, you’ll see how OLake Go connects effortlessly with Kafka’s distributed ecosystem through its flexible, pluggable design. This post breaks down how we built it, the design decisions we made, and why certain things work the way they do. If you're running data pipelines from Kafka to a lakehouse, this might save you some pain. -## What OLake Does? A Quick Primer +## What OLake Go Does? A Quick Primer -OLake treats sources like Kafka as "streams" of data, where topics become logical streams with inferred schemas (e.g., JSON payloads augmented with Kafka metadata like offsets and partitions). OLake ingests Kafka topics into respective Iceberg tables with atomic commits (to achieve exactly-once semantics) and seamless schema evolution. +OLake Go treats sources like Kafka as "streams" of data, where topics become logical streams with inferred schemas (e.g., JSON payloads augmented with Kafka metadata like offsets and partitions). OLake Go ingests Kafka topics into respective Iceberg tables with atomic commits (to achieve exactly-once semantics) and seamless schema evolution. Key goals: - **Scalability:** Handle hundreds of partitions across multiple streams/Topics. @@ -29,7 +29,7 @@ Under the hood, we use the segmentio/kafka-go library for its Go-native performa ## Configurations: The Dial for Kafka Syncing -Before diving into architecture, let's talk configurations. OLake's Kafka source is declarative, exposing configs which go through strict early validations. Here is the schema: +Before diving into architecture, let's talk configurations. OLake Go's Kafka source is declarative, exposing configs which go through strict early validations. Here is the schema: - **Bootstrap Servers** - Comma-separated broker addresses (e.g., `broker-1:9092,broker-2:9092`). Provide 2+ for high availability; the rest are auto-discovered. @@ -39,7 +39,7 @@ Before diving into architecture, let's talk configurations. OLake's Kafka source - **SASL Mechanism** (when `SASL_*`): `PLAIN` | `SCRAM-SHA-512`. - **SASL JAAS Config** (when `SASL_*`): JAAS credential string, e.g., `org.apache.kafka.common.security.plain.PlainLoginModule required username="user" password="pass";` - **Consumer Group ID** - - Optional. Uses user-provided ID; otherwise OLake generates `olake-consumer-group-{timestamp}` and persists it for future syncs. + - Optional. Uses user-provided ID; otherwise OLake Go generates `olake-consumer-group-{timestamp}` and persists it for future syncs. - **MaxThreads** - Defaults to 3. Enforces a cap concurrent readers/writers. Higher = more throughput, more resources will be utilized. - **RetryCount** @@ -51,7 +51,7 @@ Before diving into architecture, let's talk configurations. OLake's Kafka source ![Kafka to Apache Iceberg Data Ingestion via OLake Driver](/img/blog/2025/11/kafka-to-iceberg-olake-driver.webp) -OLake’s abstraction methods wrap the Kafka-specific resources, handling Kafka-based fields and JSON-message schema discovery and batch-inclined syncing mechanism. +OLake Go’s abstraction methods wrap the Kafka-specific resources, handling Kafka-based fields and JSON-message schema discovery and batch-inclined syncing mechanism. ### Core Design Principles @@ -71,8 +71,8 @@ OLake’s abstraction methods wrap the Kafka-specific resources, handling Kafka- 3. **Configurable Parallelism: Granular Control Over Consumers** - - One OLake thread = one consumer/reader. - - Users set `max_threads`; OLake caps active readers and writers accordingly. + - One OLake Go thread = one consumer/reader. + - Users set `max_threads`; OLake Go caps active readers and writers accordingly. - Balances throughput versus resource use; prevents CPU/memory/network oversubscription. - Without this, you risk under/over-utilization, uneven partition progress, and delayed writes. @@ -100,7 +100,7 @@ OLake’s abstraction methods wrap the Kafka-specific resources, handling Kafka- ### The "Why" Behind Round Robin Group Balancer -Standard balancers (e.g., in segmentio/kafka-go) can assign partitions unevenly for our workload. To align with OLake’s concurrency model, we introduced **OLake’s Round Robin Group Balancer** for even, exclusive assignment. +Standard balancers (e.g., in segmentio/kafka-go) can assign partitions unevenly for our workload. To align with OLake Go’s concurrency model, we introduced **OLake Go’s Round Robin Group Balancer** for even, exclusive assignment. ### How the Process Flows @@ -110,7 +110,7 @@ Standard balancers (e.g., in segmentio/kafka-go) can assign partitions unevenly - The partition is not empty, and - The partition contains new messages pending commit for the assigned consumer group. - For each selected stream, OLake performs a pre-flight check per partition and fetches three metadata points (`Partition Metadata`): first available offset, last available offset, and last committed offset. + For each selected stream, OLake Go performs a pre-flight check per partition and fetches three metadata points (`Partition Metadata`): first available offset, last available offset, and last committed offset. - Create that many consumer group-based reader instances (unique IDs). @@ -152,8 +152,89 @@ During reader initialization, we set: - Graceful generation-end handling and rebalancing - Offset-lag-based rebalance and custom assignment policies -## Wrapping Up: Kafka in OLake, Production-Ready +## Wrapping Up: Kafka in OLake Go, Production-Ready -We've built OLake's Kafka source to tame the complexity of Kafka sync: secure auth, partition-savvy readers, and concurrency that scales as needed—plus an incremental loop that knows when to stop. Decisions like custom balancing and offset filtering come from real pain points: uneven loads, stalled syncs, and wasted resources. +We've built OLake Go's Kafka source to tame the complexity of Kafka sync: secure auth, partition-savvy readers, and concurrency that scales as needed—plus an incremental loop that knows when to stop. Decisions like custom balancing and offset filtering come from real pain points: uneven loads, stalled syncs, and wasted resources. Next steps? Use Docker or deploy via Helm, tweak `max_threads` for your cluster, and monitor offsets with Kafka tools. + +## FAQs + +

    OLake Go's Kafka source connector reads messages from Kafka topics and writes them as Apache Iceberg tables with atomic commits and schema evolution. Each Kafka topic becomes a logical stream, and JSON message payloads are normalized into columnar Iceberg/Parquet format. Commits happen only after all assigned partitions reach their latest offsets, guaranteeing exactly-once semantics.

    +

    Important: OLake Go's Kafka ingestion operates in append-only mode. It does not support UPSERT or DELETE operations from Kafka topics; every message is appended as a new row in the Iceberg table.

    + + }, + { + question: "Q2. How does OLake Go guarantee exactly-once delivery when reading from Kafka?", + answer:
    +

    OLake Go commits to the Iceberg destination first, writing and committing the Parquet data files and Iceberg metadata, and only then commits the Kafka consumer group offsets. This ordering means that:

    +
      +
    • If a write fails, Kafka offsets are not advanced, so the data can be safely re-read and re-written.
      • +
    • If an offset commit fails after a successful Iceberg write, the worst case is a re-read of already-written data, which is safe because Iceberg's atomic snapshot model ensures idempotent commits.
      • +
    +

    This prevents both data loss and duplication in the Iceberg tables.

    +
    + }, + { + question: "Q3. How does OLake Go handle multiple Kafka partitions concurrently?", + answer:
    +

    OLake Go uses a configurable thread pool where each thread acts as a Kafka consumer reader:

    +
      +
    • MaxThreads: caps concurrent readers to prevent CPU, memory, and network oversubscription
      • +
    • ThreadsEqualTotalPartitions: when enabled, allocates one reader per partition for maximum throughput
      • +
    • Custom Round Robin Group Balancer: distributes partitions evenly across readers to avoid uneven workload distribution
      • +
    +

    Users set max_threads in the source configuration; OLake Go caps active readers and writers accordingly to balance throughput against resource use.

    +
    + }, + { + question: "Q4. What security protocols does OLake Go's Kafka connector support?", + answer:
    +

    OLake Go's Kafka source connector supports three security protocols:

    + + + + + + + + + + + + + + + + + + + + + +
    ProtocolDescription
    PLAINTEXTUnencrypted, no authentication
    SASL_PLAINTEXTSASL authentication over unencrypted connection
    SASL_SSLSASL authentication over TLS-encrypted connection
    +

    For SASL-based protocols, OLake Go supports the following mechanisms:

    +
      +
    • PLAIN: username/password authentication
      • +
    • SCRAM-SHA-512: salted challenge-response authentication
      • +
    +

    Credentials are provided via a JAAS configuration string, covering the most common enterprise Kafka authentication setups.

    +
    + }, + { + question: "Q5. How does OLake Go handle schema inference for Kafka JSON messages?", + answer:
    +

    OLake Go automatically infers schemas from Kafka JSON message payloads at level-0 normalization:

    +
      +
    • Primitive types: strings, numbers, and booleans are extracted as individual Iceberg columns
      • +
    • Nested objects and arrays: stored as JSON strings rather than being recursively flattened
      • +
    • Kafka metadata fields: such as partition and offset are automatically added as additional columns alongside the message payload
      • +
    +

    Schema normalization can be disabled per stream in the configuration if the raw JSON format is preferred over column-level extraction.

    +
    + } +]} /> + \ No newline at end of file diff --git a/blog/2025-11-24-data-lake-vs-data-lakehouse.mdx b/blog/2025-11-24-data-lake-vs-data-lakehouse.mdx index fe6f21334..70c30f989 100644 --- a/blog/2025-11-24-data-lake-vs-data-lakehouse.mdx +++ b/blog/2025-11-24-data-lake-vs-data-lakehouse.mdx @@ -401,29 +401,6 @@ Think of your storage bucket like a corporate filing cabinet: Overall, performance is not just about faster code; it is about fewer files. Cost control is not just about cheaper storage; it is about deleting what you don't need. -## 10. Some FAQs - -In every architectural review, there comes a moment when the whiteboard is full, but the stakeholders still have lingering doubts. These are the "Elephants in the Room"—the questions that often go unasked until it is too late. Let's tackle the most common friction points you might encounter while considering Data Lake vs Data Lakehouse. - -### 10.1 Is the Data Lake dead? - -No. The Data Lake is not dead; it has simply been demoted. The era of the Data Lake as the primary serving layer for analytics is over. However, as a landing zone for raw ingestion and a repository for unstructured data (video, audio, logs), it remains unbeatable in terms of cost and throughput. The Lakehouse does not kill the Lake; it wraps a protective layer around it to make it civilized. - -### 10.2 Can I use Snowflake/BigQuery as a Lakehouse? - -Yes, but with caveats. Originally, Snowflake and BigQuery were distinct Data Warehouses that required you to load data into their proprietary storage. Today, both have evolved. They now offer features (like External Tables or BigLake) that allow them to query open formats, like Parquet, sitting in your own S3 buckets. - -**The Difference:** A "Pure" Lakehouse (like Trino/Iceberg stack) is open by default. A "Warehouse-turned-Lakehouse" is often a proprietary engine reaching out to open storage. The architecture is similar, but the vendor lock-in dynamics differ. - -### 10.3 Does Lakehouse replace Data Warehouse and OLAP? - -You must distinguish between "Reporting" and "Serving." - -**Does it replace the Data Warehouse (Reporting)?** Yes, for most use cases. If your goal is internal BI (Tableau/PowerBI) where a query taking 5 seconds is acceptable, the Lakehouse is more than capable. The days of needing a separate Teradata or Redshift instance just for daily reporting are over. However, for customer facing data, data warehouses are still a preferred choice. - -**Does it replace Real-Time OLAP (Serving)?** No. If you are building "User-Facing Analytics" (e.g., a "Who Viewed My Profile" feature on a website) where thousands of concurrent users expect sub-second latency, the Lakehouse is too slow. For this, you still need a specialized Real-Time OLAP engine (like ClickHouse, Apache Pinot, or Apache Druid) reading from the Lakehouse. - -The Lakehouse retires the Warehouse, but it feeds the OLAP engine. ## 11. Conclusion @@ -435,7 +412,110 @@ The path forward is not to tear down your infrastructure, but to evolve it: keep **Stop moving the data, start managing the state!** -Ready to build your Data Lakehouse? [OLake](https://github.com/datazip-inc/olake) helps you replicate data from operational databases directly to Apache Iceberg tables, providing the foundation for a modern lakehouse architecture. Check out the [GitHub repository](https://github.com/datazip-inc/olake) and join the [Slack community](https://join.slack.com/t/getolake/shared_invite/zt-2usyz3i6r-8I8c9MtfcQUINQbR7vNtCQ) to get started. +## FAQs + + +

    No. The Data Lake is not dead; it has simply been demoted. The era of the Data Lake as the primary serving layer for analytics is over. However, as a landing zone for raw ingestion and a repository for unstructured data (video, audio, logs), it remains unbeatable in terms of cost and throughput.

    +

    The Lakehouse does not kill the Lake; it wraps a protective layer around it to make it civilized.

    + + }, + { + question: "Q2. Can I use Snowflake/BigQuery as a Lakehouse?", + answer:
    +

    Yes, but with caveats. Originally, Snowflake and BigQuery were distinct Data Warehouses that required you to load data into their proprietary storage. Today, both have evolved and now offer features (like External Tables or BigLake) that allow them to query open formats like Parquet sitting in your own object storage.

    +

    The Difference: A "Pure" Lakehouse (like a Trino/Iceberg stack) is open by default. A "Warehouse-turned-Lakehouse" is often a proprietary engine reaching out to open storage. The architecture is similar, but the vendor lock-in dynamics differ.

    +
    + }, + { + question: "Q3. Does Lakehouse replace Data Warehouse and OLAP?", + answer:
    +

    You must distinguish between Reporting and Serving.

    +

    Does it replace the Data Warehouse (Reporting)? Yes, for most use cases. If your goal is internal BI (Tableau/PowerBI) where a query taking a few seconds is acceptable, the Lakehouse is more than capable. It can also serve customer-facing analytics when a latency of around a second is acceptable.

    +

    Does it replace Real-Time OLAP (Serving)? No. For high-concurrency, user-facing analytics where thousands of simultaneous users expect sub-second latency, a raw Lakehouse is too slow. That workload still needs a specialized Real-Time OLAP engine (like ClickHouse, Apache Pinot, or Apache Druid) reading from the Lakehouse.

    +

    The Lakehouse retires the Warehouse, but it feeds the OLAP engine.

    +
    +}, + { + question: "Q4. What is the key difference between a Data Lake and a Data Lakehouse?", + answer:
    +

    A Data Lake stores raw files in cloud object storage with no transactional guarantees. It is flexible but prone to becoming a data swamp with inconsistent data quality.

    +

    A Data Lakehouse adds an open table format layer (such as Apache Iceberg or Delta Lake) on top of the same object storage, providing:

    +
      +
    • ACID transactions
      • +
    • Schema enforcement
      • +
    • Time travel
      • +
    • Row-level operations
      • +
    +

    All without moving the data to a separate warehouse.

    +
    + }, + { + question: "Q5. Why do Data Lakes often become data swamps and how does the Lakehouse solve this?", + answer:
    +

    Data Lakes built on bare object storage lack schema enforcement and ACID transactions. Over time this leads to:

    +
      +
    • Partial writes: failed jobs leave incomplete data files with no rollback mechanism
      • +
    • Data corruption: concurrent writes with no isolation can overwrite or corrupt each other
      • +
    • Schema drift: different teams write incompatible schemas to the same storage location
      • +
    +

    The Lakehouse solves this by introducing an open table format metadata layer (Iceberg, Delta Lake, or Hudi) that acts as a transaction manager: every write is atomic, schemas are tracked explicitly, and the catalog always points to a consistent table state.

    +
    + }, + { + question: "Q6. What open table formats power the Data Lakehouse architecture?", + answer:
    +

    The three leading open table formats are Apache Iceberg, Delta Lake, and Apache Hudi. Each injects a metadata layer on top of standard cloud object storage that enables ACID commits, schema versioning, and time travel.

    + + + + + + + + + + + + + + + + + + + + + +
    FormatBest For
    Apache IcebergMulti-engine lakehouses: works natively with Spark, Trino, Flink, DuckDB, Snowflake, and more
    Delta LakeSpark and Databricks-centric workloads
    Apache HudiStreaming ingestion, CDC, and upsert/delete-heavy workloads
    +
    + }, + { + question: "Q7. Does moving to a Data Lakehouse require migrating away from S3 or existing storage?", + answer:
    +

    No. The Data Lakehouse is not a new storage system; it operates on the same cloud object storage (S3, ADLS, GCS) that Data Lakes already use. The Lakehouse simply adds an open table format layer on top.

    +

    Existing raw data can be converted to Iceberg or Delta Lake tables in-place without moving files to a different storage platform.

    +
    + }, + { + question: "Q8. What performance benefits does a Data Lakehouse offer over a traditional Data Lake?", + answer:
    +

    A Data Lakehouse provides several query performance improvements over raw Parquet or Hive tables on the same object storage:

    +
      +
    • Data skipping: manifest files track column-level statistics so query engines skip irrelevant files
      • +
    • Partition pruning: metadata-driven partition elimination replaces expensive object-store directory listings
      • +
    • Z-Ordering and sorted layouts: data is physically organized to improve selective query performance
      • +
    +

    Teams commonly report significant query speed improvements after migrating from raw Parquet/Hive tables to well-organized Iceberg Lakehouse tables on the same storage.

    +
    + } +]} /> + + + + diff --git a/blog/2025-11-27-apache-iceberg-features-benefits.mdx b/blog/2025-11-27-apache-iceberg-features-benefits.mdx index c52560117..9efd28b42 100644 --- a/blog/2025-11-27-apache-iceberg-features-benefits.mdx +++ b/blog/2025-11-27-apache-iceberg-features-benefits.mdx @@ -178,7 +178,98 @@ For experienced data engineers, Iceberg means you no longer have to choose betwe Apache Iceberg is widely adopted for good reason – it brings sanity to big data management. It empowers data engineers to focus on high-value logic rather than babysitting file layouts and recovery scripts. As the open table format ecosystem matures, Iceberg stands out as a future-proof choice that will likely underpin data lakehouses for years to come. If you're evaluating modern table formats, Iceberg's balance of performance, flexibility, and openness makes it a compelling option to take your data lake to the next level. -Ready to build your Data Lakehouse with Apache Iceberg? [OLake](https://github.com/datazip-inc/olake) provides seamless CDC replication from operational databases directly to Iceberg tables, making it easy to create a modern lakehouse architecture. Check out the [GitHub repository](https://github.com/datazip-inc/olake) and join the [Slack community](https://join.slack.com/t/getolake/shared_invite/zt-2usyz3i6r-8I8c9MtfcQUINQbR7vNtCQ) to get started. +## FAQs + + +

    Apache Iceberg's key features include:

    +
      +
    • ACID transactions: snapshot-based atomic commits ensure writes are never partially visible
      • +
    • Schema evolution: add, drop, rename, or reorder columns without rewriting existing data files
      • +
    • Time travel: query historical snapshots by snapshot ID or timestamp
      • +
    • Hidden partitioning: query engines automatically prune irrelevant partitions without explicit user filters
      • +
    • Engine-agnostic design: works natively with Spark, Trino, Flink, DuckDB, Snowflake, and more
      • +
    +

    Together these bring data warehouse reliability to cheap cloud object storage.

    + + }, + { + question: "Q2. How does Apache Iceberg support ACID transactions on a data lake?", + answer:
    +

    Iceberg uses a snapshot-based architecture where every write creates a new metadata file and commits by atomically swapping the catalog pointer from the old metadata to the new one.

    +
      +
    • If a job fails mid-write, the old snapshot remains intact and readers never see partial data
      • +
    • Readers never acquire locks, so concurrent reads are never blocked
      • +
    • Optimistic concurrency control: simultaneous writers are handled safely: if two writers conflict, one commit fails and retries, preventing corrupt data races
      • +
    +
    + }, + { + question: "Q3. What is time travel in Apache Iceberg and how can I use it?", + answer:
    +

    Time travel in Apache Iceberg lets you query data as it existed at any past snapshot or timestamp. Because Iceberg maintains a complete history of snapshots, each pointing to specific manifest and data files, you can query historical states of your data.

    +

    The exact syntax varies by query engine:

    + + + + + + + + + + + + + + + + + + + + + + + + + +
    EngineSnapshot ID syntaxTimestamp syntax
    SparkVERSION AS OF <snapshot_id>TIMESTAMP AS OF '<timestamp>'
    TrinoFOR VERSION AS OF <snapshot_id>FOR TIMESTAMP AS OF TIMESTAMP '<timestamp>'
    AthenaFOR VERSION AS OF <snapshot_id>FOR TIMESTAMP AS OF '<timestamp>'
    +

    Time travel is invaluable for auditing, reproducing historical reports, and recovering from accidental data overwrites.

    +
    + }, + { + question: "Q4. How does Apache Iceberg's hidden partitioning improve query performance?", + answer:
    +

    Traditional Hive partitioning used physical folder names, requiring users to manually filter on partition columns or suffer full table scans. Iceberg's hidden partitioning stores partition values in metadata, so:

    +
      +
    • Query engines automatically skip non-matching partitions even without explicit partition filters in SQL
      • +
    • No user errors from missing or incorrect partition predicates
      • +
    • Partition strategies can evolve over time (e.g., switching from daily to hourly partitioning) without rewriting historical data files
      • +
    +
    + }, + { + question: "Q5. Which query engines are natively compatible with Apache Iceberg?", + answer:
    +

    Apache Iceberg is engine-agnostic and natively supported by virtually every major query engine:

    +
      +
    • Apache Spark: deepest integration, widest feature support
      • +
    • Trino: full read/write support including time travel and DDL
      • +
    • Apache Flink: streaming reads and writes
      • +
    • DuckDB: lightweight local analytics
      • +
    • Dremio: data lakehouse query acceleration
      • +
    • Snowflake: native Iceberg table support
      • +
    • ClickHouse: high-throughput analytics
      • +
    • Apache Doris: real-time MPP analytics
      • +
    • Presto: distributed SQL on object storage
      • +
    +

    Data written by one engine can be immediately read by any other, making Iceberg the ideal foundation for multi-engine lakehouse architectures.

    +
    + } +]} /> diff --git a/blog/2025-11-27-data-warehouse-vs-lakehouse.mdx b/blog/2025-11-27-data-warehouse-vs-lakehouse.mdx index 20b1243a9..42b01fa2c 100644 --- a/blog/2025-11-27-data-warehouse-vs-lakehouse.mdx +++ b/blog/2025-11-27-data-warehouse-vs-lakehouse.mdx @@ -286,28 +286,6 @@ While the LH's storage costs are low, the decoupled nature introduces new avenue **Compute vs. Storage Trade-off:** Recognize that optimizing the LH often means increasing compute time (e.g., running compaction jobs) to save on future, more frequent query compute time and API costs. This is a deliberate, necessary investment: Spend a little compute time upfront on maintenance to save a lot of money on query execution later. -## 9. Some FAQs - -A sophisticated understanding of data architecture requires directly confronting and clarifying the most common misconceptions. We address these frequently asked questions to solidify the mental model for the reader. - -### 9.1 Is the Data Lakehouse a replacement for a Data Warehouse? - -The technical answer is No, not entirely, but it is a formidable challenger to the DW's monopoly on reliability. The LH has achieved feature parity with the DW in terms of ACID transactions, schema management, and governance via open table formats like Iceberg. This makes the LH the superior choice for unifying batch, streaming, and ML/AI workloads at petabyte scale and minimal cost. However, the DW still holds an advantage in a narrow, specific domain: high-concurrency, sub-second BI query serving. For organizations prioritizing only that single, high-SLA workload and willing to pay the premium for simplicity, the DW remains justifiable. For every other analytical need, the LH is the more future-proof and cost-effective architectural foundation. - -### 9.2 Can I use Snowflake/Databricks/BigQuery as a Lakehouse? - -This is a subtle question about terminology versus architecture. Yes, and No. - -Databricks (using Delta Lake, its open format) pioneered the Lakehouse concept and is a native Lakehouse platform that sits atop open storage. The architecture perfectly aligns with the decoupled Lakehouse definition. - -Snowflake and BigQuery are primarily Data Warehouses. They excel through their proprietary, integrated storage and compute. However, they are adapting. Snowflake now supports features to read and manage data directly on an external S3/ADLS bucket (an external table), moving toward a Lakehouse-like capability. Similarly, BigQuery can query external data. The key distinction remains: when you fully leverage these platforms, you are using their proprietary storage, which forfeits the openness and portability that define the true Data Lakehouse philosophy. They are using Lakehouse features, but not fully adopting the open Lakehouse architecture. - -### 9.3 How do Data Lakehouses handle high concurrency reporting compared to Data Warehouses? - -The DW has a clear architectural advantage for high concurrency: its tightly coupled, proprietary storage is optimized to serve thousands of concurrent queries by design. The LH requires more explicit work. While modern query engines (Trino, Spark) can deliver competitive speed for analytical queries, handling high-concurrency BI serving demands a highly tuned environment. - -The pragmatic solution is to implement the Unified Hybrid Architecture (Section 5). The LH handles the large, complex transformations cheaply, but the final, highly aggregated Gold consumption layer is copied to a high-performance DW which is then used exclusively for high-concurrency dashboards. The LH is fast; the DW is still faster for specific, high-concurrency BI serving. - ## 10. Conclusion The debate between the Data Warehouse and the Data Lakehouse is not merely technical; it is a question of integrated simplicity versus strategic freedom. Having systematically deconstructed the core architectural differences and feature sets, we can now offer a definitive final perspective. @@ -322,7 +300,52 @@ For the vast majority of modern enterprises, the architectural decision should l The most robust and pragmatic solution for large organizations remains the unified hybrid architecture. Use the Data Lakehouse to manage the complex, high-volume, and raw data layers (Bronze/Silver), reaping the benefits of its low-cost storage and feature flexibility. Use the Data Warehouse only as a high-performance serving layer for the final, aggregated Gold data, leveraging its integrated speed precisely where sub-second latency matters most. -Ready to build your Data Lakehouse? [OLake](https://github.com/datazip-inc/olake) helps you replicate data from operational databases directly to Apache Iceberg tables with CDC capabilities, providing the foundation for a modern lakehouse architecture. Check out the [GitHub repository](https://github.com/datazip-inc/olake) and join the [Slack community](https://join.slack.com/t/getolake/shared_invite/zt-2usyz3i6r-8I8c9MtfcQUINQbR7vNtCQ) to get started. +## FAQs + + +

    The core difference is coupling. A data warehouse tightly integrates storage, compute, and governance into one proprietary platform, which delivers strong out-of-the-box performance but at premium cost and with vendor lock-in. A data lakehouse decouples them: data sits in cheap open-format files (Parquet/ORC) on cloud object storage, and an open table format like Apache Iceberg adds the ACID transactions, schema management, and governance that used to be exclusive to the warehouse.

    +

    In short, the warehouse trades flexibility and cost for integrated speed, while the lakehouse trades some out-of-the-box performance for openness, scale, and dramatically lower storage cost.

    + + }, + { + question: "Q2. Is a data lakehouse cheaper than a data warehouse?", + answer:
    +

    At scale, usually yes, but the saving comes from the cost model, not a single line item. A warehouse bundles compute and proprietary storage, so cost climbs steeply as data grows. A lakehouse stores data on commodity object storage (S3, ADLS, GCS) and decouples compute, so you pay commodity prices for storage and only for the compute you actually run.

    +

    The tradeoff is that lakehouse savings depend on active engineering. Without file compaction, partitioning, and egress-aware design, decoupled compute and data-transfer costs can leak. The warehouse hides these costs by managing them for you at a premium.

    +
    + }, + { + question: "Q3. How does schema evolution work in a lakehouse compared to a warehouse?", + answer:
    +

    In a warehouse, schema changes are enforced through DDL, and the operation required depends on the specific implementation and the type of change: some changes are lightweight metadata updates, others require table locks, and some force a full data rewrite. A stream arriving with a slightly changed schema can be rejected or trigger one of these heavier operations, depending on how the warehouse handles that change. In a lakehouse using a format like Iceberg, schema evolution is a metadata operation, not a data rewrite.

    +
    + }, + { + question: "Q4. What is time travel in a data lakehouse and why does it matter?", + answer:
    +

    Time travel is the ability to query a table as of a past state or roll it back to an earlier version, and it comes built in with open table formats. Because every write creates a new immutable snapshot in the metadata pointing to a set of data files, older snapshots are preserved and the underlying files are never overwritten, only logically retired.

    +

    The practical payoff is recovery. If a bad job deletes data, you roll back to a prior snapshot with a single metadata command instead of restoring from external backups. In a traditional warehouse this recovery is a slow, high-risk operation.

    +
    + }, + { + question: "Q5. Does a lakehouse replace a data warehouse for high-concurrency BI?", + answer:
    +

    Not entirely. A lakehouse now matches the warehouse on ACID transactions, schema management, and governance, which makes it the stronger foundation for unifying batch, streaming, and ML/AI workloads at scale. The one area where the warehouse still leads is high-concurrency, sub-second BI serving for thousands of simultaneous dashboard users.

    +

    The common answer is a hybrid architecture: use the lakehouse to manage raw and refined layers (Bronze/Silver) cheaply, then load the small, aggregated Gold layer into a warehouse or a real-time query engine used only for the high-concurrency serving tier.

    +
    + }, + { + question: "Q6. What are the biggest operational pitfalls when moving to a lakehouse?", + answer:
    +

    The main ones are responsibilities the warehouse used to handle silently. The small-files problem is the biggest: ingestion can create millions of tiny files that cripple query performance, so you need scheduled compaction to consolidate them into optimally sized files (roughly 128MB to 512MB). Tools like OLake Fusion automate this by monitoring Iceberg table health and running compaction for you, closing the gap with the warehouse's invisible maintenance.

    +

    Two others matter. Optimistic concurrency behaves differently from a warehouse's locking, so long-running updates can hit more conflicts and benefit from smaller atomic writes. And egress costs can surprise you, so keep compute co-located with storage in the same region to avoid paying to move data out.

    +
    + }, +]} /> + diff --git a/blog/2025-11-29-iceberg-variant-geospatial.mdx b/blog/2025-11-29-iceberg-variant-geospatial.mdx index 3286565db..c69ed9f64 100644 --- a/blog/2025-11-29-iceberg-variant-geospatial.mdx +++ b/blog/2025-11-29-iceberg-variant-geospatial.mdx @@ -257,7 +257,69 @@ Apache Iceberg v3's integration of Variant and Geospatial data types marks a piv These advancements not only enhance Iceberg's ability to manage evolving data modalities but also improve performance across query engines, thanks to standardized encoding formats and predicate pushdowns. With engines like Apache Spark, Trino, and Flink actively updating to support these new types, Iceberg's role as a universal data format is solidified, providing a consistent, open standard for complex data workflows. As Iceberg v3 gains traction, it ensures that organizations can build future-proof, extensible data architectures that unify structured, semi-structured, and geospatial data under a single, scalable framework. This sets the stage for seamless interoperability across tools, optimized data pipelines, and a unified data ecosystem that can handle the demands of next-generation analytics. -Ready to leverage Apache Iceberg for your data lakehouse? [OLake](https://github.com/datazip-inc/olake) provides seamless CDC replication from operational databases directly to Iceberg tables, helping you build a modern lakehouse architecture with support for structured, semi-structured, and spatial data. Check out the [GitHub repository](https://github.com/datazip-inc/olake) and join the [Slack community](https://join.slack.com/t/getolake/shared_invite/zt-2usyz3i6r-8I8c9MtfcQUINQbR7vNtCQ) to get started. +## FAQs + +

    The VARIANT type in Apache Iceberg v3 allows semi-structured data such as JSON payloads, IoT event streams, or API responses to be stored natively in a compact binary format within an Iceberg column.

    +

    It is ideal for data with evolving or flexible schemas where pre-flattening into rigid columns is impractical. VARIANT supports efficient filter pushdown and nested field extraction far faster than either:

    +
      +
    • Storing JSON as plain text strings (requires full parsing at query time)
      • +
    • Rigid wide schemas with thousands of nullable columns (causes frequent schema changes and metadata bloat)
      • +
    +

    VARIANT gives you the flexibility of schema-on-read with the performance of columnar storage.

    + + }, + { + question: "Q2. How does Apache Iceberg v3 support geospatial data?", + answer:
    +

    Iceberg v3 introduces two native spatial data types:

    +
      +
    • GEOMETRY: handles planar coordinate geometry for flat-surface spatial calculations
      • +
    • GEOGRAPHY: handles spherical (Earth-surface) coordinates, accounting for the curvature of the Earth
      • +
    +

    These types enable map-based analytics, sensor trajectory analysis, and location-based queries directly within Iceberg tables without requiring separate spatial databases.

    +

    Caveat: Geospatial-specific partition transforms (such as xz2) are not yet defined in the Iceberg v3 core specification. Spatial partitioning optimizations currently rely on engine-level implementations rather than a standardized spec-level transform.

    +
    + }, + { + question: "Q3. Why is native VARIANT type support important for semi-structured data in lakehouses?", + answer:
    +

    Before VARIANT, teams faced two imperfect approaches:

    +
      +
    1. Pre-flatten JSON into fixed columns: rigid schemas break whenever the source evolves, requiring expensive migrations
      2. +
    2. Store as raw text strings: flexible but forces full parsing at query time with no predicate pushdown
      3. +
    +

    VARIANT solves this by storing binary-encoded semi-structured data that engines can filter and extract fields from efficiently, combining schema flexibility with columnar performance.

    +
    + }, + { + question: "Q4. What performance advantages does the VARIANT type offer over storing JSON as strings?", + answer:
    +

    Using VARIANT instead of plain JSON strings provides three key advantages:

    +
      +
    • Reduced storage footprint: binary encoding is significantly more compact than text JSON
      • +
    • Predicate pushdown into nested structures: filters can be applied deep inside nested objects without full parsing
      • +
    • No query-time parsing overhead: fields can be accessed directly from the binary structure
      • +
    +
    + }, + { + question: "Q5. Which query engines support VARIANT and Geospatial types introduced in Iceberg v3?", + answer:
    +

    Support for Iceberg v3 VARIANT and Geospatial types is actively evolving:

    +
      +
    • Apache Parquet: most mature encoding support for VARIANT (binary encoding defined at Parquet level)
      • +
    • Snowflake: supports VARIANT in Iceberg v3 tables across batch, microbatch, and streaming pipelines
      • +
    • Apache Spark and Apache Parquet communities: actively developing support upstream
      • +
    • Trino, DuckDB: implementations in progress
      • +
    +

    Recommendation: Since v3 support is still rolling out, always verify your specific engine’s current support status before adopting VARIANT or Geospatial types in production.

    +
    + } +]} /> + diff --git a/blog/2025-12-10-build-data-lakehouse-iceberg-clickhouse-olake.mdx b/blog/2025-12-10-build-data-lakehouse-iceberg-clickhouse-olake.mdx index 099c21a11..0dd83e988 100644 --- a/blog/2025-12-10-build-data-lakehouse-iceberg-clickhouse-olake.mdx +++ b/blog/2025-12-10-build-data-lakehouse-iceberg-clickhouse-olake.mdx @@ -1,6 +1,6 @@ --- -title: "Building a Data Lakehouse with Apache Iceberg + ClickHouse + OLake" -description: "Learn how to build a complete data lakehouse using Apache Iceberg, ClickHouse, OLake and MinIO for real-time CDC, scalable storage, and fast analytics. Step-by-step guide with Docker setup." +title: "Building a Data Lakehouse with Apache Iceberg + ClickHouse + OLake Go" +description: "Learn how to build a complete data lakehouse using Apache Iceberg, ClickHouse, OLake Go and MinIO for real-time CDC, scalable storage, and fast analytics. Step-by-step guide with Docker setup." slug: build-data-lakehouse-iceberg-clickhouse-olake date: 2025-12-10 authors: [sandeep] @@ -10,9 +10,9 @@ image: /img/blog/cover/build-data-lakehouse-iceberg-clickhouse-olake-cover.webp ![Building a Data Lakehouse with Apache Iceberg + ClickHouse + OLake](/img/blog/cover/build-data-lakehouse-iceberg-clickhouse-olake-cover.webp) -# Building a Data Lakehouse with Apache Iceberg + ClickHouse + OLake +# Building a Data Lakehouse with Apache Iceberg + ClickHouse + OLake Go -If you're serious about building a modern data architecture, you'll love this one. We'll put together a fully open-source lakehouse platform using Apache Iceberg, ClickHouse, OLake and MinIO — and you can spin it up on your laptop using Docker in a few steps. +If you're serious about building a modern data architecture, you'll love this one. We'll put together a fully open-source lakehouse platform using Apache Iceberg, ClickHouse, OLake Go and MinIO — and you can spin it up on your laptop using Docker in a few steps. ## What is a data lakehouse? @@ -24,7 +24,7 @@ Here's the architecture we'll build: - **Source**: MySQL – the operational database -- **Ingestion**: OLake UI captures CDC (change-data-capture) from MySQL and writes into Iceberg tables stored in MinIO +- **Ingestion**: OLake Go UI captures CDC (change-data-capture) from MySQL and writes into Iceberg tables stored in MinIO - **Storage**: MinIO serves as the S3-compatible object storage for both raw and curated Iceberg tables @@ -68,7 +68,7 @@ This architecture lets you move data from MySQL → Iceberg (raw) → queryable ## Architecture at a Glance -The following diagram illustrates the complete data flow from MySQL through OLake CDC, into MinIO as Iceberg tables, and finally into ClickHouse for analytics: +The following diagram illustrates the complete data flow from MySQL through OLake Go CDC, into MinIO as Iceberg tables, and finally into ClickHouse for analytics: ![Data Lakehouse Architecture](/img/blog/2025/25/architecture.webp) @@ -82,13 +82,13 @@ The following diagram illustrates the complete data flow from MySQL through OLak **Key components** -* **MySQL 8.0** – Demo OLTP workload with GTID + binlog enabled for OLake CDC. +* **MySQL 8.0** – Demo OLTP workload with GTID + binlog enabled for OLake Go CDC. * **OLake UI (separate docker-compose)** – Configures the source, destination, and `iceberg_job` pipeline that writes Iceberg tables to MinIO through the REST catalog. * **MinIO** – Acts as the S3-compatible warehouse holding both the raw namespace (`iceberg_job_demo_db`) and the curated Silver namespace (`demo_lakehouse_silver`). -* **Iceberg REST Catalog + PostgreSQL** – Serves metadata to both OLake and ClickHouse, ensuring all engines see the same table definitions. +* **Iceberg REST Catalog + PostgreSQL** – Serves metadata to both OLake Go and ClickHouse, ensuring all engines see the same table definitions. * **ClickHouse** – Queries raw Iceberg via REST, writes the Silver Iceberg table back to MinIO, and stores Gold aggregates locally for sub-10ms dashboards. @@ -96,7 +96,7 @@ The following diagram illustrates the complete data flow from MySQL through OLak ## Setting Up OLake UI - CDC Engine -OLake has one of its unique offerings the OLake UI, which we will be using for our setup. This is a user-friendly control center for managing data pipelines without relying heavily on CLI commands. It allows you to configure sources, destinations, and jobs visually, making the setup more accessible and less error-prone. Many organizations actively use OLake UI to reduce manual CLI work, streamline CDC pipelines, and adopt a no-code-friendly approach. +OLake Go has one of its unique offerings the OLake UI, which we will be using for our setup. This is a user-friendly control center for managing data pipelines without relying heavily on CLI commands. It allows you to configure sources, destinations, and jobs visually, making the setup more accessible and less error-prone. Many organizations actively use OLake UI to reduce manual CLI work, streamline CDC pipelines, and adopt a no-code-friendly approach. For our setup, we will be working with the OLake UI. We'll start by cloning the repository from GitHub and bringing it up using Docker Compose. Once the UI is running, it will serve as our control hub for creating and monitoring all CDC pipelines. @@ -123,7 +123,7 @@ export PWD=$(pwd) cd olake-ui ``` -The OLake UI docker-compose file uses `${PWD}/olake-data` as the host persistence path. This means all your OLake configurations, job states, and metadata will be saved to an olake-data folder in your current directory. Well, that's exactly what we want - persistent storage that survives container restarts! +The OLake UI docker-compose file uses `${PWD}/olake-data` as the host persistence path. This means all your OLake Go configurations, job states, and metadata will be saved to an olake-data folder in your current directory. Well, that's exactly what we want - persistent storage that survives container restarts! Now let's fire up the OLake UI: @@ -244,7 +244,7 @@ The MinIO Console provides a web-based interface where you can: The `warehouse` and `olake-data` buckets are automatically created by the `mc` service in docker-compose.yml. -**Note:** The `mc` container exits after successfully creating the buckets (exit code 0) - this is expected behavior. You can verify the buckets exist by checking the MinIO Console (`http://localhost:9091`) or by checking the `mc` container logs. Once OLake starts writing data, you'll see directories for each table (e.g., `iceberg_job_demo_db/users/`, `iceberg_job_demo_db/products/`, etc.) containing Iceberg metadata and Parquet data files. The namespace format is `_`. +**Note:** The `mc` container exits after successfully creating the buckets (exit code 0) - this is expected behavior. You can verify the buckets exist by checking the MinIO Console (`http://localhost:9091`) or by checking the `mc` container logs. Once OLake Go starts writing data, you'll see directories for each table (e.g., `iceberg_job_demo_db/users/`, `iceberg_job_demo_db/products/`, etc.) containing Iceberg metadata and Parquet data files. The namespace format is `_`. --- @@ -309,7 +309,7 @@ Before configuring OLake UI, you may want to inspect what data is available in M ## Prepare ClickHouse for the Iceberg REST Catalog -ClickHouse ships with experimental Iceberg support disabled by default. The repo already enables the necessary flags inside `clickhouse-config/config.xml` and expects an Iceberg REST catalog provided by OLake. +ClickHouse ships with experimental Iceberg support disabled by default. The repo already enables the necessary flags inside `clickhouse-config/config.xml` and expects an Iceberg REST catalog provided by OLake Go. **Iceberg REST Catalog Details:** @@ -319,7 +319,7 @@ ClickHouse ships with experimental Iceberg support disabled by default. The repo - **Full API endpoint**: `http://localhost:8181/v1/config` (for health checks from host) -- **Namespace**: `iceberg_job_demo_db` (format: `_` - where OLake writes the raw Iceberg tables) +- **Namespace**: `iceberg_job_demo_db` (format: `_` - where OLake Go writes the raw Iceberg tables) - **No authentication required** (Iceberg REST catalog doesn't use auth by default) @@ -345,7 +345,7 @@ Now let's configure OLake UI to replicate data from MySQL to Iceberg tables in M - Password: `password` -Once logged in, you'll see the OLake dashboard. We need to configure two things: a **Source** (MySQL) and a **Destination** (Iceberg on MinIO). +Once logged in, you'll see the OLake Go dashboard. We need to configure two things: a **Source** (MySQL) and a **Destination** (Iceberg on MinIO). **Step 2: Register the MySQL Source** @@ -409,7 +409,7 @@ Great! Your MySQL source is now registered. ![Save destination](/img/blog/2025/25/save-destination.webp) -Perfect! Now OLake knows where to write the Iceberg tables. +Perfect! Now OLake Go knows where to write the Iceberg tables. The destination is configured to use: @@ -514,9 +514,9 @@ Now we'll create a pipeline that connects the MySQL source to the Iceberg destin ![Sync now](/img/blog/2025/25/sync-now.webp) -That's it! OLake will now start syncing your MySQL data to Iceberg tables. Here's what happens behind the scenes: +That's it! OLake Go will now start syncing your MySQL data to Iceberg tables. Here's what happens behind the scenes: -- OLake takes an initial snapshot of all data from MySQL (this may take 2-5 minutes with 10,000+ orders) +- OLake Go takes an initial snapshot of all data from MySQL (this may take 2-5 minutes with 10,000+ orders) - It creates Iceberg tables in the namespace `iceberg_job_demo_db` (format: `_`) @@ -611,7 +611,7 @@ Once you've verified the data is there, you're ready to query it with ClickHouse ## Query Iceberg Tables from ClickHouse -Now that OLake has written the Iceberg tables to MinIO, let's connect ClickHouse to query them. ClickHouse uses the **DataLakeCatalog** engine to connect to the Iceberg REST catalog. +Now that OLake Go has written the Iceberg tables to MinIO, let's connect ClickHouse to query them. ClickHouse uses the **DataLakeCatalog** engine to connect to the Iceberg REST catalog. **How it works:** @@ -623,7 +623,7 @@ We've broken down the setup into three separate scripts so you can verify each s **Step 1: Query Raw Iceberg Tables** -First, verify that ClickHouse can connect to and query the raw Iceberg tables written by OLake: +First, verify that ClickHouse can connect to and query the raw Iceberg tables written by OLake Go: ```bash docker exec -it clickhouse-client clickhouse-client --host clickhouse --time --queries-file /scripts/iceberg-query-raw.sql @@ -903,7 +903,7 @@ All three layers are now ready for querying! - **"All Layers Summary"**: This confirms all three layers are now accessible: - - **Raw**: Original Iceberg tables in MinIO (written by OLake) + - **Raw**: Original Iceberg tables in MinIO (written by OLake Go) - **Silver**: Optimized Iceberg table in MinIO (written by ClickHouse) @@ -917,7 +917,7 @@ All three layers are now ready for querying! The data architecture uses three layers for optimal performance: -1. **Raw Iceberg tables** (in MinIO) - Written by OLake from MySQL +1. **Raw Iceberg tables** (in MinIO) - Written by OLake Go from MySQL * Namespace: `iceberg_job_demo_db` (format: `_`) @@ -953,7 +953,7 @@ The setup scripts create: **Why this architecture matters:** - * **Raw Iceberg**: Proves ClickHouse can read OLake-managed data, but queries are slower due to unoptimized layout and network I/O from MinIO + * **Raw Iceberg**: Proves ClickHouse can read OLake Go-managed data, but queries are slower due to unoptimized layout and network I/O from MinIO * **Silver**: ClickHouse writes an optimized Iceberg table back to MinIO with curated columns and identity partitions. Queries are faster than raw because of the optimized schema and partitioning, but still require network I/O to MinIO. The table is accessible to any Iceberg-compatible engine. @@ -1102,9 +1102,9 @@ Need a completely fresh start (wipes data, buckets, Postgres catalog, etc.)? Use ## Where to Go Next -* **Scale Up:** Point additional OLTP sources (PostgreSQL, SQL Server, Mongo CDC) into OLake while reusing the same Iceberg destination and ClickHouse readers. +* **Scale Up:** Point additional OLTP sources (PostgreSQL, SQL Server, Mongo CDC) into OLake Go while reusing the same Iceberg destination and ClickHouse readers. -* **Optimize:** Automate silver/gold refreshes via cron or OLake webhooks, and add MergeTree materialized views for queries that still need sub-second response. +* **Optimize:** Automate silver/gold refreshes via cron or OLake Go webhooks, and add MergeTree materialized views for queries that still need sub-second response. * **Visualize:** Connect Superset, Grafana, or Hex directly to ClickHouse; use raw/silver for exploratory stories and gold for executive dashboards that must always be instant. @@ -1112,10 +1112,91 @@ Need a completely fresh start (wipes data, buckets, Postgres catalog, etc.)? Use * **Production-Ready:** Add MinIO lifecycle policies, bucket versioning, encryption, or replicate to real S3 to mimic production storage guarantees. -* **Monitor:** Track pipeline SLAs by scraping OLake job metrics, ClickHouse system tables, and MinIO health—set alerts when sync lag grows or catalog health checks fail. +* **Monitor:** Track pipeline SLAs by scraping OLake Go job metrics, ClickHouse system tables, and MinIO health—set alerts when sync lag grows or catalog health checks fail. + +**The beauty of OLake Go is its simplicity** - what used to require complex Debezium configurations now takes just a few clicks through the UI. You've built a complete data lakehouse that combines the best of data lakes and data warehouses! + +Enjoy building your data lakehouse with ClickHouse and OLake Go! + +## FAQs + +

    ClickHouse connects to Apache Iceberg through its DataLakeCatalog engine, which integrates with an Iceberg REST catalog that tracks table metadata and points to data files in MinIO.

    +

    You create a database (not individual tables) using the DataLakeCatalog engine, giving ClickHouse access to all tables in the specified namespace:

    + 
    {`CREATE DATABASE demo
+ENGINE = DataLakeCatalog('http://rest:8181/v1', 'admin', 'password')
+SETTINGS
+    catalog_type = 'rest',
+    storage_endpoint = 'http://minio:9000/lakehouse',
+    warehouse = 'demo';`}
    +

    Note: Backticks are required when querying tables with multi-level namespace paths, as ClickHouse does not natively support more than one namespace level in dot notation.

    +

    Once configured, ClickHouse queries Iceberg tables through this database connection as if they were native ClickHouse tables.

    + + }, + { + question: "Q2. What is OLake Go's role in the ClickHouse and Apache Iceberg lakehouse architecture?", + answer:
    +

    OLake Go acts as the CDC ingestion engine that captures changes from MySQL (or other databases) via binlog replication and writes them directly as Apache Iceberg tables in MinIO.

    +

    It orchestrates the full pipeline from source to Iceberg without requiring Kafka or Spark, making the data immediately available for ClickHouse to query through the Iceberg REST catalog.

    +
    + }, + { + question: "Q3. How does CDC from MySQL work in the OLake Go and ClickHouse Iceberg setup?", + answer:
    +

    OLake Go uses MySQL binlog-based CDC to capture every INSERT, UPDATE, and DELETE from the source database in real time.

    +

    The full flow:

    +
      +
    1. OLake Go reads the MySQL binlog stream and processes change events
      2. +
    2. Changes are written as Iceberg snapshots to a MinIO bucket
      3. +
    3. An Iceberg REST catalog, backed by PostgreSQL metadata storage, tracks table state, schemas, snapshots, and manifest locations
      4. +
    4. ClickHouse queries this catalog to discover schemas and file locations, always reading the latest committed snapshot
      5. +
    +
    + }, + { + question: "Q4. What is the three-layer architecture in a ClickHouse Iceberg lakehouse?", + answer:
    +

    The architecture follows the standard medallion model:

    + + + + + + + + + + + + + + + + + + + + + + + + + +
    LayerContentsPerformance
    RawRaw Iceberg tables written directly by OLake Go CDC (unfiltered, original partitioning)Slower; unoptimized file layout
    SilverCleaned and optimized Iceberg tables rewritten by ClickHouseFaster; optimized layout and partitioning
    GoldPre-aggregated business-ready tables stored in ClickHouse MergeTreeFastest; pre-computed metrics for dashboards
    +

    Queries on raw data are slower due to unoptimized layouts, while Silver and Gold layers provide significantly faster analytics for production workloads.

    +
    + }, + { + question: "Q5. Can I run this entire ClickHouse and Iceberg lakehouse stack locally with Docker?", + answer:
    +

    Yes. The complete stack (OLake UI, MySQL source database, MinIO object storage, Iceberg REST catalog, and ClickHouse) can be launched locally using a single command:

    + 
    {`docker compose up -d`}
    +

    This makes it easy to prototype the full lakehouse architecture before deploying to a cloud environment.

    +
    + } +]} /> -**The beauty of OLake is its simplicity** - what used to require complex Debezium configurations now takes just a few clicks through the UI. You've built a complete data lakehouse that combines the best of data lakes and data warehouses! - -Enjoy building your data lakehouse with ClickHouse and OLake! diff --git a/blog/2025-12-18-olake-now-an-arrow-based-iceberg-ingestion-tool.mdx b/blog/2025-12-18-olake-now-an-arrow-based-iceberg-ingestion-tool.mdx index 0c1f9a915..3888f6bcb 100644 --- a/blog/2025-12-18-olake-now-an-arrow-based-iceberg-ingestion-tool.mdx +++ b/blog/2025-12-18-olake-now-an-arrow-based-iceberg-ingestion-tool.mdx @@ -10,11 +10,11 @@ image: /img/blog/cover/arrow_olake_cover_image.webp --- -# OLake — now an Arrow-based Iceberg Ingestion Tool +# OLake Go — now an Arrow-based Iceberg Ingestion Tool ![Apache Arrow OLake cover image](/img/blog/cover/arrow_olake_cover_image.webp) -At OLake, our target has always been pretty straightforward — to make the best ingestion tool in the market to replicate data from databases to Iceberg faster and reliably. +At OLake Go, our target has always been pretty straightforward — to make the best ingestion tool in the market to replicate data from databases to Iceberg faster and reliably. As we continued to optimize our writer engine — the part responsible for moving data into Apache Iceberg tables — we realized that our traditional serialization approach was hitting performance limits, especially when handling terabytes of data. @@ -39,7 +39,7 @@ It gives you, - *Language Agnostic* : Working across different programming languages (Go, Java, Python, etc.) - *Native Parquet Integration* : Built-in support for writing Parquet files efficiently -## OLake's New Arrow-Writer Architecture +## OLake Go's New Arrow-Writer Architecture ![OLake Arrow-based Iceberg ingestion architecture](/img/blog/2025/26/olake-arrow-writer-architecture.webp) @@ -78,7 +78,7 @@ This eliminates the expensive Go → Java bridge for data writes while using Jav ### High-Level Architecture -On a very high level in our Arrow writer architecture, OLake, as an ingestion tool, runs on multiple threads in a highly parallel and concurrent environment, and continuously dumps your data in the form of Parquets into your object store and then finally generates the Iceberg table format on top of it. +On a very high level in our Arrow writer architecture, OLake Go, as an ingestion tool, runs on multiple threads in a highly parallel and concurrent environment, and continuously dumps your data in the form of Parquets into your object store and then finally generates the Iceberg table format on top of it. As each thread finishes its chunk of data (writing them in the form of Parquets in your object store), we hit the Java API of Iceberg to take these Parquet files into consideration under its table format. This is similar to the *AddFiles( )* operation in iceberg-go — something we refer to as **REGISTER** in our OLake terminology. @@ -107,7 +107,7 @@ Arrow exposes raw buffer pointers directly. No memory allocation or copying is r ![OLake Arrow-based Iceberg ingestion architecture](/img/blog/2025/26/arrow-record-batch.webp) -Arrow operates entirely on batches. OLake writes an entire [**RecordBatch**](https://arrow.apache.org/docs/python/data.html#record-batches) in one call — 10,000 rows processed in microseconds. +Arrow operates entirely on batches. OLake Go writes an entire [**RecordBatch**](https://arrow.apache.org/docs/python/data.html#record-batches) in one call — 10,000 rows processed in microseconds. Every chunk of data coming from the source side is broken down into 10,000-size mini batches as an **arrow.RecordBatch**. A Record Batch is a single, in-memory, columnar block of data — basically, a set of columns (arrays) that all share the same schema and the same number of rows. @@ -135,7 +135,7 @@ We, being a golang project, we use the **arrow-go** library, the latest **v18** - **Dictionary encoding**: Enabled :::note -These properties are currently hard-coded but will be made configurable in the coming versions of OLake. +These properties are currently hard-coded but will be made configurable in the coming versions of OLake Go. ::: ### For a Non-Partitioned Table @@ -144,17 +144,17 @@ Each thread is associated with a chunk of data and is dedicated a rolling data w ### For a Partitioned Table -We implement a **fan-out strategy**. Every chunk of data is distributed across multiple partition keys generated over the provided partition columns and transform information. Currently, OLake supports all the partition transforms provided by Iceberg. +We implement a **fan-out strategy**. Every chunk of data is distributed across multiple partition keys generated over the provided partition columns and transform information. Currently, OLake Go supports all the partition transforms provided by Iceberg. Every partition key is thus dedicated a rolling data writer of its own. ### The Equality Delete Writer -OLake, being an ingestion tool, our aim was to ingest faster. Thus, right now, we support writing CDC in the form of equality delete files. +OLake Go, being an ingestion tool, our aim was to ingest faster. Thus, right now, we support writing CDC in the form of equality delete files. An [**Equality Delete File**](https://iceberg.apache.org/spec/?h=equality#equality-delete-files) is simply a Parquet file that tells a query engine to mark a row deleted by one or more column values. It is different from a [**Positional Delete File**](https://iceberg.apache.org/spec/?h=equality#position-delete-files), which would also mention the Parquet file location along with the position of rows to skip. -Nevertheless, in case of CDC, OLake writes the equality delete files into Iceberg. The delete files are nothing but in the form of Parquets itself, thus, writing them directly into object storage was never a big deal. We use the same concept of rolling writer strategy, but this time to write delete files into Iceberg. +Nevertheless, in case of CDC, OLake Go writes the equality delete files into Iceberg. The delete files are nothing but in the form of Parquets itself, thus, writing them directly into object storage was never a big deal. We use the same concept of rolling writer strategy, but this time to write delete files into Iceberg. Thus, for deletes/updates we track them separately using **_olake_id** , as equality field, with the maximum file size a delete file can go up to is 64 MB. @@ -175,7 +175,7 @@ An equality delete with an unpartitioned spec acts as a **global equality delete [**Sequence Number**](https://iceberg.apache.org/spec/#sequence-numbers) in Iceberg is a monotonically increasing long value that tracks the order of commits in an Iceberg table. You can think of it as a logical timestamp that establishes a total ordering of all changes made to the table. -Since we are creating equality delete files from the OLake side and "registering" them into Iceberg using Apache Iceberg Java API, we handle this with care. +Since we are creating equality delete files from the OLake Go side and "registering" them into Iceberg using Apache Iceberg Java API, we handle this with care. As we commit in the Iceberg table, it creates a new snapshot for the table with a new sequence number. For any data file or delete file, initially, their sequence numbers are "null" in their Parquet file manifests, but eventually, they acquire the snapshot sequence number as their sequence number. @@ -193,7 +193,7 @@ You can read more about this in [**Scan Planning**](https://iceberg.apache.org/s ## Wrapping Up -With so many improvements and optimizations in OLake in the arrow-writer side, we have many advantages over our traditional java-writer approach: +With so many improvements and optimizations in OLake Go in the arrow-writer side, we have many advantages over our traditional java-writer approach: - Low CPU overhead from Protobuf serialization - Almost null java heap pressure from deserializing records @@ -204,7 +204,56 @@ Along with that, we also come up with many other performance benefits of arrow l Yet, the only issue we see with the current architecture is the use of **recordBuilders** in arrow. Though it doesn’t prove to be that problematic, we have plans to completely get rid of it and optimize more on the arrow-writer side in the upcoming releases. -

    -

    - -Cheers! +## FAQs + + +

    Apache Arrow is a language-agnostic, columnar in-memory data format that enables zero-copy reads and efficient data exchange between systems without serialization overhead.

    +

    OLake Go adopted Arrow because it provides a shared memory format that allows Go and Java components to understand the same data structure natively. This eliminates slow serialization/deserialization steps and enables direct Parquet file writes without routing data through a Java Iceberg server.

    + + }, + { + question: "Q2. How does OLake Go's Arrow-based writer improve ingestion performance?", + answer:
    +

    In the traditional architecture, OLake Go serialized Go records into Protobuf, sent them over gRPC to a Java service, and relied on Java to write Iceberg/Parquet files, introducing multiple serialization and network hops.

    +

    The Arrow-based writer eliminates this data bridge:

    +
      +
    • OLake Go writes Arrow records directly to Parquet files in Go
      • +
    • Java is used only for Iceberg metadata operations
      • +
    +

    This reduces serialization overhead and improves throughput, delivering approximately 1.75× faster ingestion performance in observed workloads.

    +
    + }, + { + question: "Q3. What is zero-copy data transfer in Apache Arrow?", + answer:
    +

    Zero-copy transfer means multiple systems can read the same data in memory without creating additional copies.

    +

    Apache Arrow achieves this through a standardized columnar memory layout. Any Arrow-compatible runtime (Go, Java, Python, C++) can directly access Arrow buffers without deserialization.

    +

    In OLake Go’s case, this eliminates the overhead of translating data between Go and Java representations during Iceberg writes.

    +
    + }, + { + question: "Q4. How does the OLake Go Arrow writer handle Iceberg metadata management?", + answer:
    +

    In the Arrow-based architecture:

    +
      +
    • The Go layer handles all data processing and writes Parquet files directly to object storage
      • +
    • The Java Iceberg library is invoked only for metadata operations: registering data files, updating manifests, and committing snapshots
      • +
    +

    This separation ensures that Java’s mature Iceberg implementation guarantees correctness, while Go handles performance-critical data throughput.

    +
    + }, + { + question: "Q5. Is OLake Go's Apache Arrow writer production-ready?", + answer:
    +

    The Arrow writer was initially introduced as a beta release, offering approximately 1.75× performance improvement for Iceberg ingestion workloads.

    +

    It is particularly beneficial for high-volume pipelines where serialization overhead becomes a bottleneck.

    +

    Recommendation: Check the latest OLake Go release notes and documentation for current stability status and production readiness before adopting it in critical pipelines.

    +
    + } +]} /> + + + diff --git a/blog/2025-12-24-snowflake-mor-to-cow.mdx b/blog/2025-12-24-snowflake-mor-to-cow.mdx index 853875100..276979f2c 100644 --- a/blog/2025-12-24-snowflake-mor-to-cow.mdx +++ b/blog/2025-12-24-snowflake-mor-to-cow.mdx @@ -12,23 +12,23 @@ import TabItem from '@theme/TabItem'; ![Snowflake COW cover image](/img/blog/cover/snowflake_cow.webp) -If you're using OLake to replicate database changes to Apache Iceberg and Databricks for analytics, you've probably hit a frustrating roadblock: Databricks doesn't support equality delete files. OLake writes data efficiently using Merge-on-Read (MOR) with equality deletes for CDC operations, but when you try to query those tables in Databricks, the deletions, updates and inserts simply aren't honored. Your query results become incorrect, missing critical data changes. +If you're using OLake Go to replicate database changes to Apache Iceberg and Databricks for analytics, you've probably hit a frustrating roadblock: Databricks doesn't support equality delete files. OLake Go writes data efficiently using Merge-on-Read (MOR) with equality deletes for CDC operations, but when you try to query those tables in Databricks, the deletions, updates and inserts simply aren't honored. Your query results become incorrect, missing critical data changes. This isn't just a Databricks limitation—several major query engines including Snowflake face the same challenge. While these platforms are incredibly powerful for analytics, their Iceberg implementations only support Copy-on-Write (COW) tables or position deletes at best. -In this blog, I'll walk you through the problem and show you how we've solved it with a simple yet powerful MOR to COW write script that transforms OLake's MOR tables into COW-compatible tables that Databricks and other query engines can read correctly. +In this blog, I'll walk you through the problem and show you how we've solved it with a simple yet powerful MOR to COW write script that transforms OLake Go's MOR tables into COW-compatible tables that Databricks and other query engines can read correctly. ## The Problem: MOR vs COW in the Real World -Let's understand what's happening under the hood. When you use OLake for Change Data Capture (CDC), it writes data to Iceberg using a strategy called Merge-on-Read (MOR) with equality delete files. This approach is optimized for high-throughput writes: +Let's understand what's happening under the hood. When you use OLake Go for Change Data Capture (CDC), it writes data to Iceberg using a strategy called Merge-on-Read (MOR) with equality delete files. This approach is optimized for high-throughput writes: > **Note:** For a deeper understanding of MOR vs COW strategies in Apache Iceberg, refer to our detailed guide on [Merge-on-Read vs Copy-on-Write in Apache Iceberg](/iceberg/mor-vs-cow). -### How OLake Writes Data (MOR with Equality Deletes): +### How OLake Go Writes Data (MOR with Equality Deletes): -**1. Initial Full Refresh:** OLake performs a complete historical load of your table to Iceberg. This creates append only data files (No MOR). +**1. Initial Full Refresh:** OLake Go performs a complete historical load of your table to Iceberg. This creates append only data files (No MOR). -**2. CDC Updates:** As changes happen in your source database, OLake captures them and creates equality delete files and data files. +**2. CDC Updates:** As changes happen in your source database, OLake Go captures them and creates equality delete files and data files. **3. The Result:** After multiple CDC sync cycles, you have: - Multiple data files with your records @@ -50,8 +50,8 @@ We've built a PySpark script that automates this entire process. Here's how it w The workflow consists of the following steps: -- **Data Ingestion**: Multiple source databases (PostgreSQL, MySQL, Oracle, MongoDB, Kafka) are ingested through OLake -- **MOR Table Creation**: OLake creates MOR-Equality-delete-tables +- **Data Ingestion**: Multiple source databases (PostgreSQL, MySQL, Oracle, MongoDB, Kafka) are ingested through OLake Go +- **MOR Table Creation**: OLake Go creates MOR-Equality-delete-tables - **COW Write**: Spark script to transforms MOR tables into Copy-on-Write (COW) format by rewriting data files with equality deletes applied - **Storage**: COW tables are stored in object storage (S3, Azure Blob Storage, GCS, etc.) - **Querying**: Databricks queries COW tables as external Iceberg tables with all deletes and updates properly applied @@ -1125,7 +1125,7 @@ These configurations ensure proper resource management and prevent job failures The MOR to COW write process is designed to be safe, repeatable, and compatible with continuous CDC ingestion. Key features: -- **Non-intrusive**: Works alongside OLake's ongoing syncs using Iceberg's snapshot isolation +- **Non-intrusive**: Works alongside OLake Go's ongoing syncs using Iceberg's snapshot isolation - **Unified incremental mode**: Uses a single function that handles both first-time and subsequent runs. On the first run, it checks if the COW table exists—if not, it creates the COW table with a full resolved dataset from the MOR table. On subsequent runs, it updates the existing COW table with only the latest changes. **1. Read the last successful truncate ID from the COW table:** The process starts by checking the COW table metadata to determine whether a previous MOR → COW run has completed successfully. @@ -1201,7 +1201,7 @@ On reruns, the script uses `last_successful_truncate_snapshot_id` as the effecti #### How recovery works -Let us assume that `trunc0` was the most recent successful truncate operation and while running `trunc1` the script failed. By the time we re run the script, OLake might have ingested some more CDC changes. This is how the workflow will behave: +Let us assume that `trunc0` was the most recent successful truncate operation and while running `trunc1` the script failed. By the time we re run the script, OLake Go might have ingested some more CDC changes. This is how the workflow will behave: - The script checks the COW table's snapshot history and finds the latest WAP ID containing `trunc0_snapshot_id`. - It re-publishes this WAP ID to ensure the data written to COW will be visible to the query engine. - Then it fetches the `trunc0` stored in WAP ID and uses it as the starting point for the current run. The script truncates the MOR table again, creating `trunc2` as the boundary for the run. @@ -1372,5 +1372,68 @@ To understand how the MOR to COW write script works and see it in action, you ca ## Conclusion -By implementing this automated MOR to COW write solution, you can now enjoy the best of both worlds: OLake's high-performance Merge-on-Read (MOR) writes for efficient CDC ingestion, combined with Databricks-compatible Copy-on-Write (COW) tables for accurate analytics queries. - +By implementing this automated MOR to COW write solution, you can now enjoy the best of both worlds: OLake Go's high-performance Merge-on-Read (MOR) writes for efficient CDC ingestion, combined with Databricks-compatible Copy-on-Write (COW) tables for accurate analytics queries. + +## FAQs + + +

    Databricks' Iceberg support has historically had limited or evolving support for equality delete files, which are commonly used in Merge-on-Read (MOR) tables for CDC workloads.

    +

    When an engine does not correctly apply equality deletes, query results may include rows that should have been deleted or updated, leading to inconsistent or incorrect results.

    +

    Because OLake Go uses equality deletes for efficient CDC ingestion, a conversion step is often required to ensure compatibility with engines that expect fully materialized data.

    + + }, + { + question: "Q2. What is the difference between MOR and COW in Apache Iceberg?", + answer:
    +

    Merge-on-Read (MOR): Data and delete files are written separately. Deletes are recorded as equality or positional delete files and merged with data at query time.

    +
      +
    • Fast and efficient writes
      • +
    • Additional read overhead due to merge operations
      • +
    +

    Copy-on-Write (COW): Deletes and updates are applied immediately by rewriting affected Parquet files.

    +
      +
    • Fast reads (no delete files to reconcile)
      • +
    • More expensive writes due to full file rewrites
      • +
    +
    + }, + { + question: "Q3. How does the MOR-to-COW conversion script work for Databricks compatibility?", + answer:
    +

    The conversion process materializes the final state of the table:

    +
      +
    1. The PySpark script reads the MOR Iceberg table with all delete files applied
      2. +
    2. It writes the resolved dataset into a new Iceberg table in COW format
      3. +
    3. The resulting table stores fully materialized Parquet files with all updates and deletes applied
      4. +
    +

    Databricks can then query this COW table as an external Iceberg table and return correct results without needing delete file support.

    +
    + }, + { + question: "Q4. Which other query engines besides Databricks have issues with MOR equality delete files?", + answer:
    +

    Support for equality delete files varies across engines:

    +
      +
    • Snowflake: support has historically been limited or evolving depending on configuration and version
      • +
    • Other engines: some may have partial or version-dependent support for equality deletes
      • +
    +

    Because of this variability, MOR-to-COW conversion is a practical strategy for ensuring compatibility with engines that expect fully materialized (COW-style) tables.

    +
    + }, + { + question: "Q5. How should I manage storage costs when running MOR-to-COW conversions regularly?", + answer:
    +

    To avoid duplicate storage costs from maintaining both MOR and COW versions:

    +
      +
    • Verify the correctness of the COW table after conversion
      • +
    • Run Iceberg snapshot expiry to remove old snapshots from the MOR table
      • +
    +

    Expiring snapshots older than 5–7 days (depending on your compaction cadence) removes orphaned files and eliminates unnecessary storage overhead.

    +
    + } +]} /> + + diff --git a/blog/2026-01-25-ingesting-files-from-s3-with-olake-turn-buckets-into-reliable-streams.mdx b/blog/2026-01-25-ingesting-files-from-s3-with-olake-turn-buckets-into-reliable-streams.mdx index 4df7aa36c..295a1ded6 100644 --- a/blog/2026-01-25-ingesting-files-from-s3-with-olake-turn-buckets-into-reliable-streams.mdx +++ b/blog/2026-01-25-ingesting-files-from-s3-with-olake-turn-buckets-into-reliable-streams.mdx @@ -17,7 +17,7 @@ Exports land in S3. Logs get dumped into folders. Partners upload daily drops. B The problem is: S3 is storage, not a dataset manager. It won't tell you what changed since the last run. It won't infer schema. It won't group files into logical datasets. And it definitely won't help you scale ingestion when the bucket gets big. -That's exactly what the OLake S3 Source connector is meant to solve. +That's exactly what the OLake Go S3 Source connector is meant to solve. It lets you ingest data from Amazon S3 and S3-compatible storage like MinIO and LocalStack, and it does it in a way that matches how buckets are usually structured in real life: folders represent datasets, files arrive over time, and you want ingestion to be incremental and fast. @@ -25,7 +25,7 @@ You can configure it from the OLake UI or run it locally (Docker) if you're keep ## What this connector is really doing -Instead of treating S3 as "one giant bucket of files", OLake treats it like a place where multiple datasets naturally exist side-by-side. +Instead of treating S3 as "one giant bucket of files", OLake Go treats it like a place where multiple datasets naturally exist side-by-side. If you've got data organized like: @@ -33,7 +33,7 @@ If you've got data organized like: - `orders/…` - `products/…` -…then you already implicitly have multiple streams. OLake just makes that explicit. +…then you already implicitly have multiple streams. OLake Go just makes that explicit. Once it identifies those streams, it focuses on three things you always want in S3 ingestion: @@ -43,7 +43,7 @@ Once it identifies those streams, it focuses on three things you always want in That's why the connector includes format support, schema inference, stream discovery through folder grouping, and incremental sync using S3 metadata—without you building that machinery yourself. -![S3 to Iceberg flowchart: OLake orchestration, S3 driver, incremental vs backfill, range reader, parsers, and Iceberg tables](/img/blog/2026/3/s3-flowchart.webp) +![S3 to Iceberg flowchart: OLake Go orchestration, S3 driver, incremental vs backfill, range reader, parsers, and Iceberg tables](/img/blog/2026/3/s3-flowchart.webp)

    (click to zoom in)

    @@ -75,13 +75,13 @@ The connector reads files (supports Parquet range reads for efficiency) and deco ### Schema & Type Mapping -**CSV files:** OLake samples rows and picks the safest data type that works across all values (for example, treating mixed values as strings if needed). +**CSV files:** OLake Go samples rows and picks the safest data type that works across all values (for example, treating mixed values as strings if needed). **JSON files:** Primitive types like strings, numbers, and booleans are detected automatically. Nested objects or arrays are stored as JSON strings. -**Parquet files:** OLake reads the schema directly from the file metadata, so no inference is needed. +**Parquet files:** OLake Go reads the schema directly from the file metadata, so no inference is needed. -OLake automatically figures out column types while reading files, so you don't need to define schemas manually: +OLake Go automatically figures out column types while reading files, so you don't need to define schemas manually: ### Emit & Record Metadata @@ -97,11 +97,11 @@ Transient failures are retried according to your `retry_count`. Hard parsing err This flow gives you visibility and speed—you only process what changed and you keep track of per-stream progress. -## Stream grouping: how OLake turns folder structure into datasets +## Stream grouping: how OLake Go turns folder structure into datasets This is the feature that makes the connector feel like it was built by people who've actually dealt with messy buckets. -OLake automatically groups files into streams based on folder structure. Here's the mental model: +OLake Go automatically groups files into streams based on folder structure. Here's the mental model: **The first folder after your configured `path_prefix` becomes the stream name.** @@ -119,7 +119,7 @@ s3://my-bucket/data/ └── products.csv.gz ``` -…and your `path_prefix` is `data/`, OLake creates: +…and your `path_prefix` is `data/`, OLake Go creates: - `users` stream - `orders` stream @@ -131,7 +131,7 @@ To keep your expectations up and not worried on a friday production the grouping This is usually what you want because it matches the "dataset folder" style most teams use. -## Formats supported: OLake reads what people actually store in buckets +## Formats supported: OLake Go reads what people actually store in buckets S3 buckets almost always end up storing a mix of: @@ -139,11 +139,11 @@ S3 buckets almost always end up storing a mix of: - JSON events/logs - Parquet outputs from batch/streaming jobs -OLake supports all three and handles them in the "obvious, non-annoying" way. +OLake Go supports all three and handles them in the "obvious, non-annoying" way. ### CSV (plain or gzipped) -CSV is messy, but it's common, so the connector gives you enough control to make it work reliably: delimiter, header detection, quote character, and skipping initial rows when needed. Schema is inferred from header + sampling, but OLake stays conservative because CSV is inherently ambiguous. +CSV is messy, but it's common, so the connector gives you enough control to make it work reliably: delimiter, header detection, quote character, and skipping initial rows when needed. Schema is inferred from header + sampling, but OLake Go stays conservative because CSV is inherently ambiguous. **Supported:** - `.csv` @@ -151,7 +151,7 @@ CSV is messy, but it's common, so the connector gives you enough control to make ### JSON (multiple shapes, plain or gzipped) -JSON is even more inconsistent across teams, so OLake handles the common real-world patterns: +JSON is even more inconsistent across teams, so OLake Go handles the common real-world patterns: - JSONL (line-delimited) - JSON arrays @@ -165,14 +165,14 @@ It auto-detects which one you've got and infers schema from primitives. Nested o ### Parquet (native schema, efficient reads) -Parquet is the easiest case. OLake reads schema directly from Parquet metadata, so there's no guessing and it scales well. It also supports efficient streaming reads with S3 range requests, which matters for large files. +Parquet is the easiest case. OLake Go reads schema directly from Parquet metadata, so there's no guessing and it scales well. It also supports efficient streaming reads with S3 range requests, which matters for large files. **Supported:** - `.parquet` ### Compression: you don't need to configure -A small but important quality-of-life thing: if your files end with `.gz`, OLake automatically decompresses them. That's it. No extra "compression" field, no special mode, no separate connector. +A small but important quality-of-life thing: if your files end with `.gz`, OLake Go automatically decompresses them. That's it. No extra "compression" field, no special mode, no separate connector. This is especially useful for S3 ingestion because `.gz` is often the default for CSV and JSON exports. @@ -194,13 +194,13 @@ But in production, re-reading every object every time gets expensive and slow. Incremental is where this connector becomes operationally clean. -OLake uses the S3 object `LastModified` timestamp as the cursor for incremental syncs. +OLake Go uses the S3 object `LastModified` timestamp as the cursor for incremental syncs. This means: - If a file is new or updated, it will be picked up in the next sync - If a file is unchanged, it will be skipped -- If a file is deleted from S3, OLake does not track or emit delete events +- If a file is deleted from S3, OLake Go does not track or emit delete events :::warning Important Incremental sync only detects additions and updates. Deletions in S3 are not propagated to downstream systems. @@ -214,7 +214,7 @@ If deletions must be reflected, run a full refresh to reconcile the destination To avoid the access denied issue make sure that the necessary policies exist. -The OLake S3 Source connector works with Amazon S3 without any version restrictions and is fully compatible with standard AWS-managed buckets. For local development and testing using S3-compatible services, MinIO version 2020 or newer is required to ensure compatibility with the S3 API features used by the connector. +The OLake Go S3 Source connector works with Amazon S3 without any version restrictions and is fully compatible with standard AWS-managed buckets. For local development and testing using S3-compatible services, MinIO version 2020 or newer is required to ensure compatibility with the S3 API features used by the connector. When using LocalStack, a minimum version of 0.12+ is recommended for stable S3 behavior and IAM simulation. @@ -222,7 +222,7 @@ In terms of data formats, the connector supports CSV, JSON, and Parquet files. T ### Required permissions -OLake needs to list objects and read them: +OLake Go needs to list objects and read them: - `s3:ListBucket` - `s3:GetObject` @@ -261,11 +261,11 @@ Replace `` with your bucket name. #### AWS S3 -OLake always needs credentials to access S3, but you don't always have to enter them explicitly. +OLake Go always needs credentials to access S3, but you don't always have to enter them explicitly. -If credentials are not provided in the OLake configuration, the connector automatically uses the AWS default credential chain. This is the recommended approach for production deployments. +If credentials are not provided in the OLake Go configuration, the connector automatically uses the AWS default credential chain. This is the recommended approach for production deployments. -OLake checks for credentials in the following order: +OLake Go checks for credentials in the following order: 1. Static credentials in configuration (`access_key_id`, `secret_access_key`) 2. Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`) @@ -334,7 +334,7 @@ Parquet files generally require no tuning at all, since the schema is embedded d ## Data type mapping -OLake tries to keep ingestion stable and predictable while still giving useful typing. +OLake Go tries to keep ingestion stable and predictable while still giving useful typing. | File Format | Source Type | Destination Type | Notes | |-------------|-------------|------------------|-------| @@ -353,7 +353,7 @@ OLake tries to keep ingestion stable and predictable while still giving useful t | Parquet | DECIMAL | float | Converted to float64. May result in precision loss for high-precision decimal values. | | All formats | `_last_modified_time` | timestamptz | S3 LastModified metadata (added by connector) | -**Timezone:** OLake ingests timestamps in UTC (timestamptz) regardless of source timezone. +**Timezone:** OLake Go ingests timestamps in UTC (timestamptz) regardless of source timezone. Parquet DECIMAL types are converted to float64 during ingestion. @@ -361,7 +361,7 @@ While this works well for analytical use cases, very high-precision or fixed-sca ## Date and time handling (edge cases) -To keep downstream destinations happy, OLake normalizes problematic dates: +To keep downstream destinations happy, OLake Go normalizes problematic dates: - **Year = 0000:** replaced with epoch start `1970-01-01`. Example: `0000-05-10` → `1970-01-01`. - **Year > 9999:** capped at 9999 (month/day preserved). Example: `10000-03-12` → `9999-03-12`. @@ -371,7 +371,7 @@ These rules apply to date, time, and timestamp columns during transfer. ## Incremental sync details -Here's how olake tracks the change and how cursor moves: +Here's how olake Go tracks the change and how cursor moves: ![Incremental Sync Diagram](/img/blog/2026/3/incremental-sync-image.webp) @@ -512,4 +512,78 @@ If the issue isn't listed, post to the OLake Slack with connector config (omit s And once you're happy with your S3 setup and you're ready to expand your pipeline to other sources, check out our [other connector guides here](https://olake.io/docs/connectors/). +## FAQs + +

    OLake Go maps S3 data into logical streams based on folder structure. Each logical prefix (typically top-level folders under the configured path) is treated as a separate stream.

    +

    For example:

    +
      +
    • bucket/prefix/users/ → users stream
      • +
    • bucket/prefix/orders/ → orders stream
      • +
    +

    This allows your existing bucket organization to define datasets without additional configuration. Deeper nested structures may still be grouped depending on prefix configuration.

    + + }, + { + question: "Q2. What file formats does OLake Go's S3 connector support for ingestion?", + answer:
    +

    OLake Go supports the following formats:

    +
      +
    • CSV: Samples rows to infer the safest data types
      • +
    • JSON: Extracts primitive types into columns and stores nested objects/arrays as JSON strings
      • +
    • Parquet: Reads schema directly from file metadata (no inference required)
      • +
    +

    Gzip-compressed files (typically .gz for CSV/JSON) are decompressed automatically. Parquet files already use internal compression.

    +
    + }, + { + question: "Q3. How does OLake Go implement incremental sync from S3 to avoid re-reading all files?", + answer:
    +

    OLake Go uses the S3 object's LastModified timestamp as a cursor for each stream. On incremental runs, only files newer than the stored cursor are processed, which keeps subsequent syncs efficient since only new or changed files are scanned.

    +

    The behavior follows the file's LastModified state:

    +
      +
    • If a file is new or updated, it is picked up in the next sync, since an update gives the file a newer LastModified timestamp.
      • +
    • If a file is unchanged, it is skipped.
      • +
    • If a file is deleted from S3, OLake Go does not track or emit delete events.
      • +
    +
    + }, + { + question: "Q4. Can I use OLake Go's S3 connector with MinIO or LocalStack for local development?", + answer:
    +

    Yes. OLake Go's S3 connector works with any S3-compatible storage, including MinIO and LocalStack.

    +

    You simply configure:

    +
      +
    • Custom endpoint URL
      • +
    • Access key
      • +
    • Secret key
      • +
    +

    This allows you to build and test ingestion pipelines locally before deploying to AWS S3.

    +
    + }, + { + question: "Q5. What problem does the OLake Go S3 connector solve that S3 alone cannot?", + answer:
    +

    S3 is raw object storage it does not provide:

    +
      +
    • Change tracking between runs
      • +
    • Schema inference
      • +
    • Logical dataset grouping
      • +
    • Incremental processing capabilities
      • +
    +

    OLake Go adds these capabilities on top of S3:

    +
      +
    • Stream discovery: folder-to-stream mapping
      • +
    • Schema inference: per file format
      • +
    • Incremental sync: using LastModified cursors
      • +
    • Efficient reads: leveraging Parquet's columnar structure
      • +
    +

    This transforms a static object store into a reliable, repeatable data ingestion source.

    +
    + } +]} /> + + diff --git a/blog/2026-01-27-compaction-blog.mdx b/blog/2026-01-27-compaction-blog.mdx index 90e6d641f..9ca12d538 100644 --- a/blog/2026-01-27-compaction-blog.mdx +++ b/blog/2026-01-27-compaction-blog.mdx @@ -1,8 +1,8 @@ --- slug: olake-amoro-iceberg-lakehouse -title: "How to Compact Apache Iceberg Tables: Small Files + Automation with Apache Amoro" -description: A practical guide to fixing small-file bloat in Apache Iceberg, showing when and how to run compaction, the performance gains you can expect, and how Amoro automates it to turn Iceberg tables into self-optimizing lakehouses. -tags: [iceberg, olake, amoro, s3] +title: "How to Compact Apache Iceberg Tables: Small Files + Automation with Apache Amoro™" +description: A practical guide to fixing small-file bloat in Apache Iceberg, showing when and how to run compaction, the performance gains you can expect, and how Apache Amoro™ automates it to turn Iceberg tables into self-optimizing lakehouses. +tags: [iceberg, olake, "amoro", s3] authors: [anshika] image: /img/blog/cover/compaction_blog_cover_image.webp --- @@ -56,7 +56,7 @@ Compaction is the most visible part of **Iceberg table maintenance**—along wit **3. Scale makes the problem unavoidable -** What works fine at 100 GB breaks at 10 TB, and completely collapses at 1 PB. -### 1.4. Iceberg & Amoro as Solutions for Modern Lakehouses +### 1.4. Iceberg & Apache Amoro™ as Solutions for Modern Lakehouses Iceberg tracks every file, maintains detailed statistics, and supports atomic rewrites that don't disrupt concurrent readers or writers. The challenge? Iceberg gives you the tools, but you must orchestrate them. You need to: - Monitor table health continuously @@ -65,7 +65,7 @@ Iceberg tracks every file, maintains detailed statistics, and supports atomic re This operational complexity leads many organizations to build custom automation—or worse, neglect maintenance altogether. -**Enter Apache Amoro (incubating):** a lakehouse management system built specifically to solve this problem. Amoro provides self-optimizing capabilities that continuously monitor your Iceberg tables, automatically trigger compaction when needed, and maintain optimal table health without manual intervention. +**Enter Apache Amoro™ (incubating):** a lakehouse management system built specifically to solve this problem. Apache Amoro™ provides self-optimizing capabilities that continuously monitor your Iceberg tables, automatically trigger compaction when needed, and maintain optimal table health without manual intervention. ## 2. What Causes the Small Files Problem? @@ -397,19 +397,19 @@ If delete workload is high, treat **“data compaction”** and **“delete comp Compaction rewrites data files, but it doesn’t delete old files immediately. Those old files remain referenced by older snapshots until you expire them. If you only compact data files but never expire snapshots, you’ll keep paying for storage, and planning may still degrade because metadata history keeps growing. ::: -## 5. Enter Amoro: Automated Optimization for Iceberg +## 5. Enter Apache Amoro™: Automated Optimization for Iceberg -This section introduces Apache Amoro as the solution to operational complexity. While Iceberg provides the building blocks, Amoro provides the automation and intelligence to maintain table health continuously. +This section introduces Apache Amoro™ as the solution to operational complexity. While Iceberg provides the building blocks, Apache Amoro™ provides the automation and intelligence to maintain table health continuously. -### 5.1. Architecture of Amoro +### 5.1. Architecture of Apache Amoro™ -Amoro transforms Iceberg maintenance from a manual, engineer-driven process into a self-managing system. +Apache Amoro™ transforms Iceberg maintenance from a manual, engineer-driven process into a self-managing system. -![Amoro Architecture](/img/blog/2026/1/amoro_arch.webp) +![Apache Amoro™ Architecture](/img/blog/2026/1/amoro_arch.webp) -The main components of Amoro are: +The main components of Apache Amoro™ are: -**Amoro Management Service (AMS)** +**Apache Amoro™ Management Service (AMS)** AMS is the brain of the system. It constantly watches over all registered Iceberg tables and evaluates their health—looking for things like too many small files, growing delete files, or bloated metadata. Based on what it finds, AMS automatically decides what needs to be optimized and when. It also manages the pool of optimizers, tracks their capacity, and exposes everything through a clean UI and a set of APIs so teams can monitor and control optimization activities without manual intervention. @@ -417,9 +417,9 @@ AMS is the brain of the system. It constantly watches over all registered Iceber Optimizers are the workers that actually perform the heavy lifting. They run the compaction jobs, merge delete files, rewrite manifests, and clean up snapshots. These workers are organized into resource groups so you can isolate workloads (e.g., separate streaming table optimization from large batch compaction). They scale independently from query engines, which means optimization does not affect query performance. AMS simply assigns tasks, and the optimizers execute them in a distributed, fault-tolerant manner. -### 5.2. How Amoro Performs Continuous Small-File Optimization +### 5.2. How Apache Amoro™ Performs Continuous Small-File Optimization -Amoro keeps Iceberg tables healthy by running a continuous feedback loop. Every few seconds, AMS checks the state of your tables, decides what needs attention, and dispatches optimizers to fix problems without interrupting incoming writes. +Apache Amoro™ keeps Iceberg tables healthy by running a continuous feedback loop. Every few seconds, AMS checks the state of your tables, decides what needs attention, and dispatches optimizers to fix problems without interrupting incoming writes. **1. Monitoring**
    Every 30–60 seconds, AMS scans the metadata of each registered table. It looks at file counts, average file size, delete-file buildup, and other health indicators. If a table starts accumulating too many small files, AMS immediately flags it. @@ -438,7 +438,7 @@ AMS validates the commit, updates the table’s health score, and decides whethe ### 5.3. Minor vs Major vs Full Optimization Jobs -Amoro uses a two-tier optimization strategy that works similarly to how the JVM performs garbage collection. The idea is to keep tables healthy with frequent light operations, while occasionally running deeper optimizations when necessary. +Apache Amoro™ uses a two-tier optimization strategy that works similarly to how the JVM performs garbage collection. The idea is to keep tables healthy with frequent light operations, while occasionally running deeper optimizations when necessary. **Minor Optimization**
    Minor optimization runs very frequently typically every 5 to 15 minutes and focuses only on small “fragment” files that are under 16MB. It uses a fast bin-packing strategy to merge these tiny files, making it a lightweight process that finishes quickly and keeps write amplification low. The goal is simply to prevent heavy fragmentation before it grows. While it’s efficient and uses very few resources, minor optimization doesn’t completely reorganize the table, so the resulting layout is not perfectly optimal, and larger files remain untouched. @@ -451,7 +451,7 @@ Full optimization is the most intensive operation and runs only occasionally (us ### 5.4. Automatic Delete File Merging -Amoro handles delete files intelligently: +Apache Amoro™ handles delete files intelligently: **Detection** @@ -462,17 +462,17 @@ Amoro handles delete files intelligently: **Strategy Selection** - **If delete_ratio < 10%:**
    - Amoro performs simple consolidation, merging small delete files into fewer, larger ones so engines don’t waste time opening thousands of tiny delete files. + Apache Amoro™ performs simple consolidation, merging small delete files into fewer, larger ones so engines don’t waste time opening thousands of tiny delete files. - **If delete_ratio < 30%:**
    - Amoro performs partial application, rewriting only the data files that have accumulated the most deletes and hence reducing read-time overhead without rewriting everything. + Apache Amoro™ performs partial application, rewriting only the data files that have accumulated the most deletes and hence reducing read-time overhead without rewriting everything. - **Else (> 30%):**
    - Amoro performs full delete application, rewriting all affected data files so that all delete files are applied and removed completely. + Apache Amoro™ performs full delete application, rewriting all affected data files so that all delete files are applied and removed completely. **Result:** Delete files never accumulate to problematic levels. Read performance stays optimal. ### 5.5. Automatic Metadata Organization -Beyond data files, Amoro also continuously maintains Iceberg’s metadata to keep planning fast and storage clean. +Beyond data files, Apache Amoro™ also continuously maintains Iceberg’s metadata to keep planning fast and storage clean. **Manifest Optimization** @@ -495,4 +495,73 @@ Under the hood, this maps to Iceberg procedures like `rewrite_manifests`, `expir Compaction in Apache Iceberg is a core maintenance operation, but the right strategy depends on several factors including ingestion patterns, table size and growth rate, query latency requirements, delete behavior, orchestration design, and cloud storage cost constraints. In practice, the most robust production setups blend multiple techniques: continuous incremental compaction to prevent small-file buildup, periodic full table rewrites for deep optimization, metadata-driven triggers for intelligent scheduling, sorting during compaction to improve query performance, and regular snapshot expiration to keep storage lean. When these strategies are combined effectively, Iceberg evolves from a simple table format into a high-performance analytic engine capable of handling real-world streaming workloads and multi-terabyte–scale data pipelines with consistency and efficiency. +## FAQs + +

    Every write to an Iceberg table, whether from CDC, Kafka streaming, or batch jobs, creates new Parquet data files, and those files are not necessarily optimally sized. A small change writes a small file, and since writes keep arriving, small files keep accumulating faster than anything cleans them up.

    +

    Each file introduces overhead during query execution:

    +
      +
    • Separate object storage API calls (e.g., GET, LIST).
      • +
    • Metadata reads for each file.
      • +
    • Task scheduling and coordination across workers.
      • +
    +

    As file counts grow, query planning slows down significantly, often taking tens of seconds, dashboards may time out, and storage API costs increase due to excessive requests.

    + + }, + { + question: "Q2. What is table compaction in Apache Iceberg and when should you run it?", + answer:
    +

    Compaction rewrites many small Parquet files into fewer, larger files typically in the range of 128–512 MB.

    +

    This improves performance by:

    +
      +
    • Reducing the number of files scanned during queries
      • +
    • Improving metadata efficiency
      • +
    • Lowering object storage API costs
      • +
    +

    You should run compaction:

    +
      +
    • When file counts per partition exceed a few hundred
      • +
    • When query planning latency becomes noticeable
      • +
    • On a scheduled basis for streaming or CDC-heavy tables
      • +
    +
    + }, + { + question: "Q3. How does Apache Amoro automate Iceberg table compaction?", + answer:
    +

    Apache Amoro (incubating) is a lakehouse management system that continuously monitors Iceberg table health metrics such as file counts, file sizes, and snapshot age.

    +

    When thresholds are exceeded, it automatically triggers compaction jobs using appropriate rewrite strategies without manual intervention.

    +

    This turns compaction into a background, self-optimizing process similar in concept to automated maintenance tasks like VACUUM in traditional databases.

    +
    + }, + { + question: "Q4. What performance improvements can I expect after compacting Iceberg tables?", + answer:
    +

    Compaction can lead to significant performance gains:

    +
      +
    • Faster query planning: reduced from tens of seconds to near-instant in many cases
      • +
    • Improved scan performance: fewer, larger files enable better parallelism
      • +
    • Stable metadata operations: reduced risk of timeouts
      • +
    • Lower storage API costs: fewer GET and LIST requests
      • +
    +

    Actual improvements depend on workload patterns and how fragmented the table was before compaction.

    +
    + }, + { + question: "Q5. How do I set up Apache Amoro to work with OLake for automated Iceberg maintenance?", + answer:
    +

    To enable automated compaction:

    +
      +
    • Deploy Apache Amoro alongside your OLake and Iceberg stack
      • +
    • Connect Amoro to your Iceberg REST catalog (e.g., Lakekeeper, Polaris, or similar)
      • +
    • Configure compaction policies such as target file size, thresholds, and schedules via the UI
      • +
    +

    Once configured, Amoro continuously monitors tables and automatically compacts data as OLake writes new files, ensuring consistent performance without manual intervention.

    +
    + } +]} /> + + diff --git a/blog/2026-01-27-sync-mssql-to-your-lakehouse-with-olake.mdx b/blog/2026-01-27-sync-mssql-to-your-lakehouse-with-olake.mdx index f6dd2c892..d79817ab7 100644 --- a/blog/2026-01-27-sync-mssql-to-your-lakehouse-with-olake.mdx +++ b/blog/2026-01-27-sync-mssql-to-your-lakehouse-with-olake.mdx @@ -1,7 +1,7 @@ --- slug: sync-mssql-to-your-lakehouse-with-olake -title: "Sync MSSQL to Your Lakehouse with OLake" -description: A practical guide to syncing Microsoft SQL Server (MSSQL) into Apache Iceberg using OLake, covering sync modes, CDC setup, schema changes, data type mapping, and troubleshooting. +title: "Sync MSSQL to Your Lakehouse with OLake Go" +description: A practical guide to syncing Microsoft SQL Server (MSSQL) into Apache Iceberg using OLake Go, covering sync modes, CDC setup, schema changes, data type mapping, and troubleshooting. tags: [mssql, olake, iceberg, lakehouse, cdc, sync] authors: [akshay] image: /img/blog/cover/mssql-connector-cover-image.webp @@ -11,45 +11,45 @@ import BlogCTA from '@site/src/components/BlogCTA'; ![MSSQL Connector Cover Image](/img/blog/cover/mssql-connector-cover-image.webp) -If you're trying to sync Microsoft SQL Server (MSSQL) into Apache Iceberg using OLake, this guide is meant to feel like we're setting it up together—no heavy docs energy, just the things you actually need to know to get a clean, reliable pipeline running. +If you're trying to sync Microsoft SQL Server (MSSQL) into Apache Iceberg using OLake Go, this guide is meant to feel like we're setting it up together—no heavy docs energy, just the things you actually need to know to get a clean, reliable pipeline running. SQL Server shows up everywhere: product databases, internal tools, ERP-ish systems, customer dashboards, finance ops… and a lot of teams today want one thing: "Keep my operational SQL Server data flowing into my lakehouse, without babysitting it." -That's exactly what the OLake MSSQL connector is for. +That's exactly what the OLake Go MSSQL connector is for. We'll cover what the connector does, which sync mode to pick, how to enable CDC properly (super interesting part), how schema changes work, the limitations you should know upfront, and the practical setup steps in the OLake UI. I'll also call out CLI/Docker flows along the way so you can match this to your workflow. -## Overview: what the OLake MSSQL connector does +## Overview: what the OLake Go MSSQL connector does ![MSSQL Connector Overview](/img/blog/2026/4/mssql-overview-image.webp) -At a high level, the OLake MSSQL Source connector supports multiple synchronization modes and is built for "real tables" (large row counts, frequent updates, evolving schemas). +At a high level, the OLake Go MSSQL Source connector supports multiple synchronization modes and is built for "real tables" (large row counts, frequent updates, evolving schemas). A few features you'll feel immediately when you run it are: -- **Parallel chunking** helps OLake move large tables faster by reading in pieces instead of one slow scan. -- **Checkpointing** means OLake remembers progress so if something fails mid-way, it doesn't behave like "oops, start again from the beginning." -- **Automatic resume** for failed full loads is exactly what it sounds like: if a full refresh fails, OLake can resume instead of re-copying everything. +- **Parallel chunking** helps OLake Go move large tables faster by reading in pieces instead of one slow scan. +- **Checkpointing** means OLake Go remembers progress so if something fails mid-way, it doesn't behave like "oops, start again from the beginning." +- **Automatic resume** for failed full loads is exactly what it sounds like: if a full refresh fails, OLake Go can resume instead of re-copying everything. And you can run this connector in two ways: -- **Inside the OLake UI** (most common for teams getting started) +- **Inside the OLake Go UI** (most common for teams getting started) - **Locally via Docker / CLI flows** (handy for OSS workflows or if you want everything as code) **Quick note:** in this blog, I'm going to explain the setup from the UI point of view, because it's the easiest way to get to a working pipeline. If you prefer CLI, the same configuration fields apply and you can follow the [matching CLI guide in the docs](https://olake.io/docs/connectors/mssql/). ## Sync modes supported (and how to choose) -OLake supports multiple sync modes for MSSQL. The names are technical, but the decision is usually simple if you map them to what you're trying to achieve. +OLake Go supports multiple sync modes for MSSQL. The names are technical, but the decision is usually simple if you map them to what you're trying to achieve. ### 1) Full Refresh This copies the current state of your table(s). It's your "day 0 snapshot." **Use this when:** -- you're onboarding a new SQL Server database into OLake, +- you're onboarding a new SQL Server database into OLake Go, - you want a clean baseline, - or you're okay with "copy everything again" as your model. @@ -82,7 +82,7 @@ This assumes you already have a baseline (maybe created earlier, or managed sepa ## Prerequisites (don't skip these) -Before you configure OLake, make sure your SQL Server environment meets a few basics. +Before you configure OLake Go, make sure your SQL Server environment meets a few basics. ### Version prerequisite @@ -105,7 +105,7 @@ Let's walk through that properly because it's the number one source of confusion ### What CDC actually is -SQL Server CDC (Change Data Capture) records row-level changes (inserts/updates/deletes) into special "change tables." OLake reads those changes and applies them downstream so your destination stays aligned with what happened in the source. +SQL Server CDC (Change Data Capture) records row-level changes (inserts/updates/deletes) into special "change tables." OLake Go reads those changes and applies them downstream so your destination stays aligned with what happened in the source. CDC is powerful but only if it's enabled correctly. @@ -180,13 +180,13 @@ EXEC sys.sp_cdc_enable_table The important thing is: give it a new capture instance name (different from the old one). -### What OLake does during CDC capture-instance transitions +### What OLake Go does during CDC capture-instance transitions -When a new CDC capture instance is created for a table (usually after a schema change), OLake automatically detects that a newer capture instance exists. +When a new CDC capture instance is created for a table (usually after a schema change), OLake Go automatically detects that a newer capture instance exists. -OLake will continue reading from the older capture instance and will switch over to the newest one only when the event stream reaches a point where both capture instances are valid. This ensures continuity and avoids duplicate or out-of-order events. +OLake Go will continue reading from the older capture instance and will switch over to the newest one only when the event stream reaches a point where both capture instances are valid. This ensures continuity and avoids duplicate or out-of-order events. -In practice, this means you don't need to manually "cut over" pipelines at the exact right moment. OLake handles the transition safely and automatically once the timeline makes it safe to do so. +In practice, this means you don't need to manually "cut over" pipelines at the exact right moment. OLake Go handles the transition safely and automatically once the timeline makes it safe to do so. ### Important CDC caveat during schema changes @@ -194,14 +194,14 @@ There is one important limitation to be aware of when working with SQL Server CD **If inserts, updates, or deletes occur between the time a DDL change is applied and the time the new CDC capture instance is created, those CDC events related to the newly added or modified columns will not be captured.** -For example, if a user adds a new column X to a table, and rows are inserted or updated before a new capture instance is created, changes to column X during that window will not appear in CDC events. This behavior is inherent to how SQL Server CDC works and is not specific to OLake. +For example, if a user adds a new column X to a table, and rows are inserted or updated before a new capture instance is created, changes to column X during that window will not appear in CDC events. This behavior is inherent to how SQL Server CDC works and is not specific to OLake Go. To minimize data gaps, it's best practice to: - apply schema changes during low-write windows, and - create the new capture instance immediately after the DDL change. -OLake will then pick up from the correct point and transition cleanly once the stream is aligned. +OLake Go will then pick up from the correct point and transition cleanly once the stream is aligned. ### Columnstore indexes @@ -215,16 +215,16 @@ CDC does not support values for computed columns (even if persisted). If computed columns are included in a capture instance, they will show as NULL in CDC output. -That's not OLake—it's how SQL Server CDC behaves. +That's not OLake Go—it's how SQL Server CDC behaves. ::: ## Configuration (UI-first, but the same fields apply to CLI) -Once prerequisites are met (and CDC enabled if you need it), setting up the source in OLake is straightforward. +Once prerequisites are met (and CDC enabled if you need it), setting up the source in OLake Go is straightforward. ### Step 1: Navigate to the source setup screen -1. Log in to OLake after you have done the [setup using docs](https://olake.io/docs/install/olake-ui/) +1. Log in to OLake Go after you have done the [setup using docs](https://olake.io/docs/install/olake-ui/) 2. Go to **Sources** (left sidebar) 3. Click **Create Source** (top right) 4. Select **MSSQL** from the connector list @@ -261,7 +261,7 @@ Once the source is created, you can configure jobs on top of it (choose tables, ## Data type mapping -This is how your columns are treated downstream and OLake maps MSSQL types into predictable destination types so downstream systems don't get messy surprises. +This is how your columns are treated downstream and OLake Go maps MSSQL types into predictable destination types so downstream systems don't get messy surprises. | MSSQL Data Types | Destination Type | |------------------|------------------| @@ -278,7 +278,7 @@ If you're syncing into a lakehouse and later querying through engines like Trino Dates are one of those things that feel normal until one row breaks your job. -During transfer, OLake normalizes values in date, time, and timestamp columns to ensure valid calendar ranges and destination compatibility. +During transfer, OLake Go normalizes values in date, time, and timestamp columns to ensure valid calendar ranges and destination compatibility. ### Case I: Year = 0000 @@ -328,7 +328,7 @@ If you paste the exact error and mention whether it happened during Test Connect ## Wrap-up -If you're wiring up SQL Server → OLake, you're already doing the most important thing right: keeping the first version simple and stable. +If you're wiring up SQL Server → OLake Go, you're already doing the most important thing right: keeping the first version simple and stable. A good flow is to start with a full refresh so you know the connection, permissions, and table selection are all solid. Once that baseline is in place, you can move to incremental or CDC depending on how often your tables change (and how important updates/deletes are for your downstream use cases). @@ -339,4 +339,79 @@ And if you do go the CDC route, just keep these two practical rules in mind beca When you're ready to bring in more systems, you can follow our [other connector walkthroughs as well here](https://olake.io/docs/connectors/). +## FAQs + + +

    OLake Go's MSSQL connector connects to SQL Server, captures data using configurable sync modes, and writes it directly as Apache Iceberg tables on your chosen object storage.

    +

    For continuous pipelines, it supports Change Data Capture (CDC) using SQL Server’s native CDC feature to capture inserts, updates, and deletes in near real time.

    +

    For large tables, OLake Go performs parallelized initial loads using checkpointed chunking, enabling:

    +
      +
    • Fast full-table ingestion
      • +
    • Failure recovery with automatic resume
      • +
    • No long-running table locks
      • +
    + + }, + { + question: "Q2. What sync modes does OLake Go support for Microsoft SQL Server?", + answer:
    +

    OLake Go supports four sync modes for MSSQL:

    +
      +
    • Full Refresh: Reloads the entire table on each run
      • +
    • Full Refresh + Incremental: Initial snapshot followed by cursor-based updates
      • +
    • Full Refresh + CDC: Snapshot followed by real-time CDC (captures inserts, updates, deletes)
      • +
    • CDC Only: Streams only changes from the current CDC position (no initial snapshot)
      • +
    +

    Recommendation: Full Refresh + CDC is best for production workloads with frequent updates and deletes.

    +
    + }, + { + question: "Q3. How do I enable CDC (Change Data Capture) on Microsoft SQL Server for OLake Go?", + answer:
    +

    CDC must be enabled at both the database and table level before OLake Go can consume changes.

    +

    This is done using SQL Server system procedures:

    +
      +
    • sp_cdc_enable_db → enables CDC for the database
      • +
    • sp_cdc_enable_table → enables CDC for specific tables
      • +
    +

    Requirements:

    +
      +
    • SQL Server Agent must be running
      • +
    • Supported editions include Enterprise, Developer, and Standard
      • +
    +
    + }, + { + question: "Q4. How does OLake Go handle schema changes during MSSQL replication to Iceberg?", + answer:
    +

    The important part is on the SQL Server side. When you change a source table's schema (add or drop columns, change types), SQL Server does not automatically update its CDC change table to match. So the fix is to create a new CDC capture instance with a new name that reflects the updated schema:

    +
      +
    • After the DDL change, run sp_cdc_enable_table again with a new @capture_instance name (for example, dbo_my_table_v2).
      • +
    • OLake Go automatically detects that a newer capture instance exists. It keeps reading from the older instance and switches to the newer one only once the event stream reaches a point where both are valid, so there is no manual cutover and no duplicate or out-of-order events.
      • +
    +

    Automating this: OLake Go can also manage capture instances for you. Instead of performing these steps manually, enable the Manage Capture Instance toggle in the UI and OLake Go handles capture instance management automatically. For this to work, the capture user must have db_owner role membership on the source database. See the MSSQL connector documentation for details.

    +

    Caveat: Any inserts, updates, or deletes that happen between the DDL change and the creation of the new capture instance may not be captured for the newly added or modified columns. This is how SQL Server CDC works, not specific to OLake Go. To minimize gaps, apply schema changes during low-write windows and create the new capture instance immediately after.

    +
    +}, + { + question: "Q5. Can I configure OLake Go's MSSQL connector without writing code using the UI?", + answer:
    +

    Yes. OLake Go provides a web-based UI for configuring MSSQL ingestion without writing code.

    +

    You can:

    +
      +
    • Enter connection details (host, port, database, credentials)
      • +
    • Select sync mode
      • +
    • Choose tables to replicate
      • +
    • Configure the Iceberg destination
      • +
    +

    The same configuration can also be managed via CLI or Docker for teams that prefer infrastructure-as-code workflows.

    +
    + } +]} /> + + + diff --git a/blog/2026-01-28-ibm-db2-luw-to-lakehouse-sync-apache-iceberg-olake.mdx b/blog/2026-01-28-ibm-db2-luw-to-lakehouse-sync-apache-iceberg-olake.mdx index 8ace09602..f42c0a7cb 100644 --- a/blog/2026-01-28-ibm-db2-luw-to-lakehouse-sync-apache-iceberg-olake.mdx +++ b/blog/2026-01-28-ibm-db2-luw-to-lakehouse-sync-apache-iceberg-olake.mdx @@ -1,7 +1,7 @@ --- slug: ibm-db2-luw-to-lakehouse-sync-apache-iceberg-olake -title: "IBM Db2 LUW to Lakehouse: Sync to Apache Iceberg Using OLake" -description: A practical guide to syncing IBM Db2 for LUW databases to Apache Iceberg using OLake, covering setup, configuration, sync modes, troubleshooting, and DB2-specific considerations like RUNSTATS and REORG. +title: "IBM Db2 LUW to Lakehouse: Sync to Apache Iceberg Using OLake Go" +description: A practical guide to syncing IBM Db2 for LUW databases to Apache Iceberg using OLake Go, covering setup, configuration, sync modes, troubleshooting, and DB2-specific considerations like RUNSTATS and REORG. tags: [db2, iceberg, olake, lakehouse, sync] authors: [akshay] image: /img/blog/cover/db2-luw-to-lakehouse-cover.webp @@ -11,11 +11,11 @@ import BlogCTA from '@site/src/components/BlogCTA'; ![IBM Db2 LUW to Lakehouse cover image](/img/blog/cover/db2-luw-to-lakehouse-cover.webp) -If you're trying to sync an IBM Db2 for LUW (Linux/Unix/Windows) database to Iceberg using OLake, this guide is for you. +If you're trying to sync an IBM Db2 for LUW (Linux/Unix/Windows) database to Iceberg using OLake Go, this guide is for you. Db2 doesn't always show up in "modern stack" discussions, but in the real world, it's still powering a lot of serious, business-critical systems. Teams keep Db2 around because it's stable, fast, and quite tested for high-volume transactional workloads. -And that's exactly where OLake fits in—it helps you take data that lives in Db2 and move it into your lakehouse, which can be Iceberg tables, downstream analytics, AI/ML, or even reporting, without turning it into a multi-month migration project. +And that's exactly where OLake Go fits in—it helps you take data that lives in Db2 and move it into your lakehouse, which can be Iceberg tables, downstream analytics, AI/ML, or even reporting, without turning it into a multi-month migration project. This blog will walk you through: - what the connector does @@ -49,11 +49,11 @@ So the question becomes: how do you unlock that Db2 data for analytics and lakeh That's the job of this connector. -## What the OLake Db2 connector does +## What the OLake Go Db2 connector does ![Db2 connector working diagram](/img/blog/2026/2/db2-working-image.webp) -This connector is the bridge between Db2 and OLake. +This connector is the bridge between Db2 and OLake Go. It can do two big things: @@ -118,18 +118,18 @@ More threads can speed up big tables, but don't crank it blindly. Db2 is fast, b ## Test Connection - what to expect -When you click **Test Connection**, OLake does a quick "sanity check" before you spend time setting up syncs. +When you click **Test Connection**, OLake Go does a quick "sanity check" before you spend time setting up syncs. -Under the hood, OLake is basically trying to answer: +Under the hood, OLake Go is basically trying to answer: "Can I reach this Db2 server over the network, and can I log in successfully using the credentials you gave me?" -### What OLake actually does during the test +### What OLake Go actually does during the test Even though it feels like a single button click, a few things happen in sequence: 1. **Network reachability check (implicit)** - - OLake attempts to connect to the host and port you provided. If the port isn't reachable, the connection will fail before it even gets to authentication. + - OLake Go attempts to connect to the host and port you provided. If the port isn't reachable, the connection will fail before it even gets to authentication. 2. **JDBC handshake + session creation** - If the port is reachable, the DB2 driver tries to establish a session with the database. This is where driver-level settings and SSL mode start to matter. @@ -158,11 +158,11 @@ If you think humans make errors, then you are right and you can check these: **What you can check:** - Confirm the Db2 listener port with your DB team -- Try the host from the same network where OLake is running (not from your laptop) +- Try the host from the same network where OLake Go is running (not from your laptop) #### 2) Firewall / security group / private networking -Even if the host and port are correct, OLake must be allowed to reach Db2 over the network path: +Even if the host and port are correct, OLake Go must be allowed to reach Db2 over the network path: - Security group rules (cloud) - VPC routing / peering - Firewall rules on the VM @@ -170,7 +170,7 @@ Even if the host and port are correct, OLake must be allowed to reach Db2 over t **What you can try:** - If Db2 is in a private subnet and not directly reachable, use SSH tunneling (often the quickest fix) -- Or ask for the OLake runtime IP/CIDR to be allowed +- Or ask for the OLake Go runtime IP/CIDR to be allowed #### 3) Missing privileges (connection works, but access fails later) @@ -192,7 +192,7 @@ This one trips people up because it can feel inconsistent. SELECT 1 FROM schema.table FETCH FIRST 1 ROW ONLY; ``` -## Data type mapping (how Db2 types land in OLake) +## Data type mapping (how Db2 types land in OLake Go) When you replicate into a lakehouse, type stability matters. So we map Db2 types into predictable destination types. @@ -213,7 +213,7 @@ That helps avoid "time drift" when downstream tools assume a single timeline. ## RUNSTATS: Highlight of DB2 -OLake requires updated Db2 statistics for sync. If table/index stats are stale, planning gets worse. And for ingestion tools, stale stats can lead to inefficient chunking decisions. +OLake Go requires updated Db2 statistics for sync. If table/index stats are stale, planning gets worse. And for ingestion tools, stale stats can lead to inefficient chunking decisions. So before you run syncs, run: @@ -234,7 +234,7 @@ Dates are where pipelines die silently or painfully. Some systems allow weird values (year 0000, invalid dates, etc.). Many downstream engines don't. -So OLake normalizes those "bad" values during transfer using simple rules: +So OLake Go normalizes those "bad" values during transfer using simple rules: - **Year = 0000** → replaced with epoch start - `0000-05-10` → `1970-01-01` @@ -274,12 +274,47 @@ Most teams resolve it by step 2 or 3. ## Wrap-up -If you're setting up Db2 → OLake, you're on the right track. The best way to do this is exactly what you're doing: start simple, get a clean first sync working, and then build from there. +If you're setting up Db2 → OLake Go, you're on the right track. The best way to do this is exactly what you're doing: start simple, get a clean first sync working, and then build from there. -Db2 is still a big part of how a lot of enterprises run their core systems—and OLake makes it much easier to bring that Db2 data into open lakehouse formats, so you can actually use it for analytics, reporting, and downstream workloads without touching (or rewriting) the source system. +Db2 is still a big part of how a lot of enterprises run their core systems—and OLake Go makes it much easier to bring that Db2 data into open lakehouse formats, so you can actually use it for analytics, reporting, and downstream workloads without touching (or rewriting) the source system. If anything breaks along the way, don't stress around and drop at the OLake community and devs would be there to help you in no time. Most of the time it's a small network/permission/SSL thing and we can point you to the fix quickly. And once you're happy with your Db2 setup and you're ready to expand your pipeline to other sources, check out our [other connector guides here](https://olake.io/docs/connectors/). +## FAQs + +

    IBM Db2 for LUW (Linux/Unix/Windows) is IBM's relational database for on-premise and cloud enterprise environments. It is commonly used in financial services, manufacturing, retail, telecom, and government because it is battle-tested for high-volume transactional workloads, extremely stable, and has powered business-critical systems for decades. Organizations keep Db2 because it handles real revenue and operations reliably, not because they want to migrate.

    + + }, + { + question: "Q2. How does OLake Go sync IBM Db2 LUW data to Apache Iceberg?", + answer:
    +

    OLake Go's Db2 connector connects to the database using standard JDBC, performs a full snapshot of selected tables in the initial load using parallel chunking for large tables, and then uses incremental sync to keep Iceberg tables updated with only new or changed rows since the last sync. The data lands in Apache Iceberg format on your object storage (S3, GCS, or Azure Blob), ready for analytics without touching the source Db2 system.

    +
    + }, + { + question: "Q3. What are Db2-specific setup considerations I should know before using OLake Go?", + answer:
    +

    The main Db2-specific requirement is RUNSTATS: OLake Go requires up-to-date table statistics for sync, so run RUNSTATS on your tables (and their indexes) before syncing, especially after bulk loads or layout changes. Stale statistics can lead to inefficient chunking decisions and slower syncs. Beyond that, standard prerequisites apply: Db2 version 11.5.3 or higher and a user with read access to the tables you want to sync.

    +
    + }, + { + question: "Q4. What sync modes does OLake Go support for IBM Db2 replication?", + answer:
    +

    OLake Go supports Full Refresh (complete table copy, ideal for the initial baseline) and Incremental sync (pulls only new and changed rows since the last run using a cursor column). After establishing a full refresh baseline, switching to incremental mode keeps your Iceberg tables fresh efficiently. OLake Go also includes parallel chunking to speed up large initial loads and checkpointing to resume from where a failed sync left off.

    +
    + }, + { + question: "Q5. Can OLake Go sync Db2 data without disrupting the source production database?", + answer:
    +

    Yes. OLake Go reads Db2 data using standard SELECT queries with cursor-based chunking, which does not place exclusive locks or block writes to the source tables. For large tables, the parallel chunking distributes the load into manageable segments. The Db2 instance continues serving its application workloads normally while OLake Go reads data in the background for replication to Iceberg.

    +
    + } +]} /> + + diff --git a/blog/2026-02-25-apache-iceberg-lakehouse-observability-metadata-monitoring.mdx b/blog/2026-02-25-apache-iceberg-lakehouse-observability-metadata-monitoring.mdx index dfee48456..34341c578 100644 --- a/blog/2026-02-25-apache-iceberg-lakehouse-observability-metadata-monitoring.mdx +++ b/blog/2026-02-25-apache-iceberg-lakehouse-observability-metadata-monitoring.mdx @@ -438,4 +438,44 @@ For teams adopting Iceberg, the practical next step is to integrate these capabi In the end, Apache Iceberg exemplifies the evolution of data lakes towards being more **self-describing and self-managing**. Observability is not an afterthought but a core feature of the table format. For data engineers, this means easier troubleshooting, proactive maintenance, and confidence in the integrity and performance of their data platform. As the data ecosystem continues to grow, leveraging Iceberg’s monitoring and metrics features can be a game-changer in operating a modern, **transparent** data lake that you can trust. +## FAQs + +

    Iceberg metadata tables are built-in, SQL-queryable system tables that expose a table's current state and full history, so you can monitor table health with the same engine you query data with. The most useful ones for observability are snapshots and history (a versioned log of every commit and what it changed), files (every data and delete file in the current state, with size, partition, and record count), partitions (per-partition file counts, sizes, and deletes), and metadata_log_entries (the log of metadata versions).

    +

    Because this information lives in the table itself, you can answer questions like how many files a table has or whether its schema changed recently without scanning object storage or parsing external logs.

    + + }, + { + question: "Q2. How do you find the small files problem in an Iceberg table?", + answer:
    +

    Query the files metadata table, filtering to data files with content = 0, then group by partition and flag any partition whose average file size falls below a threshold such as 64 MB. That pinpoints exactly which partitions are fragmented and need compaction.

    +

    For continuous detection, enable Iceberg metrics reporting (available since Iceberg 1.1.0), which emits CommitReport and ScanReport events into your monitoring stack. The pattern is to let metrics raise the alert, then use the metadata tables to diagnose the cause.

    +

    Alternatively, OLake Fusion monitors table health continuously and triggers compaction automatically, so you don't have to run this diagnosis by hand.

    +
    + }, + { + question: "Q3. How do you detect schema changes in Apache Iceberg?", + answer:
    +

    Query the metadata_log_entries table and compare each entry's latest_schema_id against the previous one. This is more reliable than the snapshots table, because a schema or partition-spec change writes a new metadata version but does not always create a new snapshot, so snapshot-based checks can miss it.

    +

    A production pattern is to build a schema-evolution timeline from this log, alert when the schema ID changes outside an approved window, and attach the metadata file to the alert so an engineer can inspect the exact version.

    +
    + }, + { + question: "Q4. How do you measure storage bloat from Iceberg snapshots?", + answer:
    +

    Compare the files table filtered to data files with content = 0 (active data) against all_data_files (data files referenced across all tracked snapshots) and take the ratio of their total sizes. Filtering on content = 0 matters because the files table lists delete files as well as data files. If total storage is much larger than active storage, for example 5x or more, retained snapshots and time-travel history are consuming significant extra storage.

    +

    The fix is to run expire_snapshots on a schedule, followed by orphan-file cleanup. Snapshot expiration is not automatic in core Iceberg, so it has to be operationalized rather than assumed.

    +
    + }, + { + question: "Q5. What is the difference between pull-based and push-based Iceberg monitoring?", + answer:
    +

    Pull-based monitoring runs scheduled queries against the metadata tables and materializes the results into a metrics store. It works with any catalog, needs no pipeline changes, and suits dashboards, trend analysis, and capacity planning.

    +

    Push-based monitoring uses Iceberg's metrics reporter framework to emit CommitReport and ScanReport events at commit and scan time, giving low-latency signals that catch bad writes as they happen. Most teams run both: push for early warning during active workloads, pull for durable history and governance.

    +
    + } +]} /> + \ No newline at end of file diff --git a/blog/2026-02-27-compaction-experiment.mdx b/blog/2026-02-27-compaction-experiment.mdx index ab3590130..a6f8ac4d5 100644 --- a/blog/2026-02-27-compaction-experiment.mdx +++ b/blog/2026-02-27-compaction-experiment.mdx @@ -24,7 +24,7 @@ This blog walks through the entire journey: the ingestion, the deliberate manipu The TPC-H benchmark is a standard set of eight interrelated tables (region, nation, supplier, part, partsupp, orders, lineitem, and customer) designed to simulate a realistic business analytics workload. At scale factor 1000, it produces roughly 1 TB of data. -We used OLake to ingest this data from PostgreSQL directly into Apache Iceberg tables stored in S3, with AWS Glue as the catalog. After the full load completed, we had eight clean Iceberg tables sitting in S3, backed by a Glue catalog, with zero delete files and well-sized Parquet data files. Query performance at this point was healthy. +We used OLake Go to ingest this data from PostgreSQL directly into Apache Iceberg tables stored in S3, with AWS Glue as the catalog. After the full load completed, we had eight clean Iceberg tables sitting in S3, backed by a Glue catalog, with zero delete files and well-sized Parquet data files. Query performance at this point was healthy. :::info For those benchmark results—including query times, memory utilization, and a comparison with Databricks—see [Running TPC-H queries](https://olake.io/iceberg/databricks-vs-iceberg/#c-running-tpch-queries) in our **Databricks vs Apache Iceberg** blog. @@ -1826,4 +1826,97 @@ But beyond the raw numbers, what this experiment demonstrated is that compaction If you run CDC into Iceberg, schedule compaction as part of your maintenance routine. Your queries and your cloud bill will thank you. +## FAQs + +

    This benchmark ran all 22 TPC-H queries on 1 TB of data (scale factor 1000) across eight Iceberg tables, first on tables fragmented with 1,000 equality delete files each to simulate CDC-heavy ingestion, then again after compaction on the same cluster and Spark configuration.

    + + + + + + + + + + + + + + + + + + + + + + + + +
    MetricBefore compactionAfter compaction
    Total query time34,635 s (~9.7 hours)7,377 s (~2 hours)
    Overall speedup~4.7x faster
    Queries completed21 of 22 (Query 13 failed)22 of 22
    +

    The pattern: the more joins, shuffles, and aggregations a query performs, the more it suffers from fragmented files and the more it gains from compaction. Complex multi-table joins improved by 5x to nearly 20x, while simple single-table queries like Q1 and Q6 stayed roughly the same.

    + + }, + { + question: "Q2. What are equality delete files in Apache Iceberg and why do they degrade query performance at scale?", + answer:
    +

    Equality delete files are part of Iceberg's Merge-on-Read (MOR) strategy. Instead of rewriting a full data file when rows are updated or deleted, Iceberg appends a small Parquet file recording which rows should be treated as deleted, matched by column values (usually the primary key). This makes writes fast, since only a tiny file is added rather than a large one rewritten.

    +

    The cost shows up at read time. Every query has to load all active delete files, check them against the data files to find logically deleted rows, and filter those out before returning results. CDC pipelines make this worse over time: each update writes a new delete file, so the count grows continuously. Without compaction, queries end up opening thousands of small files per scan, and the ones with heavy joins and aggregations degrade the most.

    +
    + }, + { + question: "Q3. What is the real cost impact of skipping Apache Iceberg compaction?", + answer:
    +

    On this benchmark's EMR cluster, the same 22-query TPC-H run cost about 6x more on the fragmented table than on the compacted one:

    + + + + + + + + + + + + + + + + + + + + + + + + + +
    ScenarioCostRuntime
    Before compaction (fragmented)~$27.31~9 h 39 min
    After compaction~$4.61~2 h 7 min
    Compaction job (one-time)~$8.37~3 h
    +

    Even including the one-time compaction cost, the compacted path totals about $13 versus $27.31 for a single fragmented run, and the gap widens with every additional run since fragmentation keeps growing without compaction. Compacted tables also run reliably on smaller clusters, avoiding the extra RAM and storage the fragmented table needed to stay stable.

    +
    + }, + { + question: "Q4. How does Apache Iceberg bin-pack compaction work and what file count changes should you expect?", + answer:
    +

    Bin-pack compaction is Iceberg's default compaction strategy. It groups input files, both data files and delete files (equality and positional), into "bins" that each target a specific output size, adding files to a bin until it reaches max-file-group-size-bytes, then starting a new bin. Each bin is rewritten as a single output file close to target-file-size-bytes.

    +

    Importantly, bin-pack compaction does more than merge small files. When delete files, whether equality or positional, are part of a rewrite group, it physically applies the deletes to the base data, removing the deleted rows and eliminating the delete files entirely. So after compaction you should expect the many small data files to consolidate into fewer, evenly sized files, and the delete file count to drop to zero.

    +

    For query engines, the result is fewer S3 GET requests, simpler scan planning, and no more Merge-on-Read overhead, which is what made complex analytical queries slow before compaction.

    +
    + }, + { + question: "Q5. What is S3 port exhaustion in Apache Iceberg queries and how does compaction prevent it?", + answer:
    +

    S3 port exhaustion happens when a query opens so many simultaneous TCP connections to S3 that the operating system runs out of available ephemeral ports (typically 32,768 to 60,999 on Linux), and new connections fail with "Cannot assign requested address."

    +

    In Iceberg's Merge-on-Read model, each equality delete file needs its own S3 GET request to be fetched and applied during a scan. When a query touches many partitions across a table with thousands of delete files, the engine tries to open a large number of concurrent S3 connections at once, which can exhaust the port range. This isn't something you can tune away, it's a direct consequence of how many files the engine has to access per scan.

    +

    Compaction fixes it by cutting the file count. When bin-pack compaction applies the deletes and consolidates data files, the delete-file count drops to zero and the query needs far fewer concurrent S3 connections per scan, so it stays well within the available port range.

    +
    + }, +]} /> + + \ No newline at end of file diff --git a/blog/2026-03-05-architect-guide-cdc-apache-iceberg.mdx b/blog/2026-03-05-architect-guide-cdc-apache-iceberg.mdx index c2086a19b..8fd724175 100644 --- a/blog/2026-03-05-architect-guide-cdc-apache-iceberg.mdx +++ b/blog/2026-03-05-architect-guide-cdc-apache-iceberg.mdx @@ -52,7 +52,7 @@ The recommended Medallion approach combines the strengths of the previous two pa ### Pattern 4: Continuous Compaction -This advanced pattern, inspired by projects like **Apache Amoro**, reimagines how we handle deletions. Instead of waiting for a massive rewrite job, the system ingests all data in an Equality Delete format. This allows ingestion to continue at high speed without interruption. We then run a custom, tiered compaction process in the background that does not require stopping the world. +This advanced pattern, inspired by projects like **Apache Amoro™**, reimagines how we handle deletions. Instead of waiting for a massive rewrite job, the system ingests all data in an Equality Delete format. This allows ingestion to continue at high speed without interruption. We then run a custom, tiered compaction process in the background that does not require stopping the world. Think of this like a multi-stage sorting facility. In the **Minor** stage, we quickly convert expensive Equality Deletes into more efficient Positional Deletes. In the **Major** stage, we bundle small, fragmented files into medium-sized ones to reduce metadata overhead. Finally, in the **Full** stage, we optimize everything into the target file size (e.g., 512MB). This approach is **future-proof** because it allows for parallel ingestion and compaction, ensuring that small file overhead never accumulates to the point of system failure. @@ -78,9 +78,9 @@ To prevent the overwrite of increased delete files from making queries too slow, ## Implementation -You can implement CDC with Apache Iceberg from different databases like MySQL, Postgres, Oracle, MongoDB, etc using OLake. The detailed steps for implementing the CDC pipeline are mentioned [here](https://olake.io/docs/community/setting-up-a-dev-env/). +You can implement CDC with Apache Iceberg from different databases like MySQL, Postgres, Oracle, MongoDB, etc using OLake Go. The detailed steps for implementing the CDC pipeline are mentioned [here](https://olake.io/docs/community/setting-up-a-dev-env/). -While the present CDC implementation into Apache Iceberg with OLake is quite efficient, we are introducing the efficient continuous compaction pattern very soon. Look out for this launch, and definitely try it out! You will be pretty amazed at the compact efficiency that you can achieve with this pattern. +While the present CDC implementation into Apache Iceberg with OLake Go is quite efficient, we are introducing the efficient continuous compaction pattern very soon. Look out for this launch, and definitely try it out! You will be pretty amazed at the compact efficiency that you can achieve with this pattern. ## Technical Deep Dive @@ -129,6 +129,162 @@ Building a reliable CDC pipeline with Apache Iceberg is about selecting the righ The foundation of this path lies in the Medallion Architecture, using a raw change log (Bronze) to ensure data durability and an asynchronous merge process (Silver/Gold) to handle the heavy lifting of materialization. By adopting Merge-on-Read (MoR) for high-velocity streams and augmenting it with a tiered, continuous compaction strategy, you eliminate the bottlenecks that typically are the major pain points for large-scale data lakes. This approach ensures that your system stays flexible, allowing you to ingest thousands of changes per second without forcing users to wait minutes for their queries to finish. -As the data lakehouse ecosystem continues to mature, the tools for managing these tables are becoming increasingly autonomous. Systems that self-optimize, such as Apache Amoro, represent the next step in this evolution. By following these architectural principles, you aren't just building a pipeline for today; you are constructing a performant and reliable foundation that will scale alongside your organization’s data needs for years to come. +As the data lakehouse ecosystem continues to mature, the tools for managing these tables are becoming increasingly autonomous. Systems that self-optimize, such as Apache Amoro™, represent the next step in this evolution. By following these architectural principles, you aren't just building a pipeline for today; you are constructing a performant and reliable foundation that will scale alongside your organization’s data needs for years to come. + +## FAQs + + +

    Change Data Capture is a data integration technique that monitors a source database's transaction logs to capture every individual INSERT, UPDATE, and DELETE as it happens in real time, rather than periodically copying the entire table.

    + +

    Traditional snapshot ETL (full table export once every 24 hours) creates three major problems that CDC eliminates:

    + +
      +
    1. Data staleness: Any analysis is based on data up to 24 hours old, making time-sensitive decisions unreliable.
      2. +
    2. Source database strain: Full table scans during off-hours degrade production performance and create fragile scheduling dependencies (a single failure leads to outdated data for the entire day).
      3. +
    3. Operational brittleness: Batch jobs either fully succeed or fully fail, with no graceful degradation.
      4. +
    + +

    CDC solves these issues by streaming row-level changes continuously from transaction logs, minimizing load on the source system and processing data incrementally so failures affect only small time windows instead of entire batch runs.

    + + }, + + { + question: "Q2. What are the four main architectural patterns for CDC ingestion into Apache Iceberg and when should you use each?", + answer:
    +

    CDC ingestion into Apache Iceberg typically follows four architectural patterns, each with different trade-offs:

    + +
      +
    1. + Direct Materialization +
        +
      • Streams CDC events from Kafka via Flink or Spark
        • +
      • Performs immediate UPSERTs into Iceberg tables
        • +
      • Pros: Lowest latency ingestion
        • +
      • Cons: High number of small delete files and frequent snapshots
        • +
      • Use when: Sub-minute freshness is critical and compaction is in place
        • +
      +
      2. + +
    2. + Raw Change Log +
        +
      • Appends every CDC event as a new row
        • +
      • No reconciliation or rewriting of data
        • +
      • Pros: Perfect audit trail, easy replay
        • +
      • Cons: Expensive reads due to merge-on-read processing
        • +
      • Use when: Compliance and audit requirements dominate query performance needs
        • +
      +
      3. + +
    3. + Hybrid Medallion Approach +
        +
      • Bronze layer stores raw CDC events
        • +
      • Silver/Gold layers updated via asynchronous MERGE INTO jobs
        • +
      • Pros: Decouples ingestion speed from query performance
        • +
      • Cons: More pipeline complexity
        • +
      • Use when: Most production analytics systems
        • +
      +
      4. + +
    4. + Continuous Compaction +
        +
      • Ingests data using equality deletes
        • +
      • Runs tiered compaction (Minor → Major → Full)
        • +
      • Gradually converts delete files into clean data files
        • +
      • Pros: Prevents accumulation of delete files
        • +
      • Cons: Requires sophisticated orchestration
        • +
      • Use when: High-scale CDC systems requiring stable long-term performance
        • +
      +
      5. +
    +
    + }, + + { + question: "Q3. What is the difference between Copy-on-Write and Merge-on-Read in Apache Iceberg for CDC workloads and which should you choose?", + answer:
    +

    Copy-on-Write (CoW) and Merge-on-Read (MoR) are two different strategies for handling updates and deletes in Iceberg.

    + +
      +
    1. + Copy-on-Write (CoW) +
        +
      • Rewrites entire data files when rows change
        • +
      • Pros: Fast reads, no runtime merge needed
        • +
      • Cons: Expensive writes for high-churn data, slow ingestion under heavy updates
        • +
      +
      2. + +
    2. + Merge-on-Read (MoR) +
        +
      • Writes delete files instead of rewriting data files
        • +
      • Uses: +
          +
        • Position delete files (row-level location-based deletion)
          • +
        • Equality delete files (primary key-based deletion)
          • +
        +
        • +
      • Pros: Fast ingestion, works well with high update frequency
        • +
      • Cons: Read-time overhead due to merge processing ("Read Tax")
        • +
      +
      3. +
    + +

    Recommendation: Use Merge-on-Read for CDC ingestion, and control read overhead through scheduled compaction. A hybrid approach often converts MoR outputs into CoW-style clean files during background compaction.

    +
    + }, + + { + question: "Q4. How does Apache Iceberg handle schema evolution in CDC pipelines without breaking downstream consumers?", + answer:
    +

    Iceberg handles schema evolution using immutable column IDs instead of names or positions, making it safe for CDC pipelines.

    + +
      +
    1. Column identity: Every column is assigned a unique, permanent ID at creation time.
      2. +
    2. Renaming columns: Only metadata changes; data files remain valid because they reference column IDs.
      3. +
    3. Adding columns: New column is added with a new ID; existing files return null for that column.
      4. +
    4. No rewrites required: Existing Parquet files and downstream queries continue working without modification.
      5. +
    + +

    This design prevents pipeline breakage and avoids costly data rewrites when schemas evolve in upstream systems.

    +
    + }, + + { + question: "Q5. Why is time-based partitioning a poor choice for CDC workloads in Apache Iceberg and what should you use instead?", + answer:
    +

    Time-based partitioning (event_day, created_at_month, etc.) works well for append-only systems but performs poorly in CDC workloads where historical data is frequently updated.

    + +
      +
    1. + Scattered writes: Updates to old records force writes into old partitions, spreading I/O across the entire dataset. +
      2. +
    2. + High fragmentation: Frequent updates create many small files across many partitions, degrading performance. +
      3. +
    3. + Inefficient compaction: Cleanup operations must scan across multiple historical partitions. +
      4. +
    + +

    Better alternative: Use bucketing or hidden partitioning based on primary key (e.g., bucket(user_id, 128)). This ensures:

    + +
      +
    • Updates for the same entity land in the same bucket
      • +
    • Localized writes instead of scattered historical writes
      • +
    • Reduced fragmentation and faster compaction
      • +
    + +

    Iceberg’s hidden partitioning makes this transparent to ingestion pipelines while significantly improving CDC performance.

    +
    + } +]} /> + \ No newline at end of file diff --git a/blog/2026-04-22-spark-vs-fusion-compaction.mdx b/blog/2026-04-22-spark-vs-fusion-compaction.mdx new file mode 100644 index 000000000..50bd395e8 --- /dev/null +++ b/blog/2026-04-22-spark-vs-fusion-compaction.mdx @@ -0,0 +1,1139 @@ +--- +slug: iceberg-compaction-spark-vs-fusion-benchmark +title: "50% Cheaper (2x Faster) Iceberg Compaction: OLake Fusion (Open Source) Beats Spark" +description: "We benchmark Spark rewrite_data_files against OLake Fusion compaction on Apache Iceberg by running a full TPCH lineitem load from Postgres to GCP, applying 200k-record CDC batches every 2 minutes, and tracking TPC-H Query 6 performance, runtime, resource usage, and infrastructure cost." +tags: [iceberg, tpch, compaction, benchmark, spark, olake, fusion] +authors: [nayan] +image: /img/blog/2026/4/fusion_vs_spark.webp +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +
    + compaction diagram +
    + +Over the last few years, most data teams have either moved to a lakehouse architecture or are actively moving in that direction. That shift solves several legacy warehouse and data lake limitations, but migration alone is not the finish line. Once you're on a lakehouse, you still need to manage table health carefully to keep performance stable and costs under control. + +Most data lakehouse problems are loud. Pipelines fail and errors show up in logs. But there is one problem that stays completely silent. Your queries just get a little slower each day until one morning something that used to finish in seconds is now taking minutes. The reason is almost always small files. In CDC-heavy pipelines, every incremental sync writes a new batch of small data files and delete files into your Iceberg table, and over time these accumulate into thousands of tiny files that make every scan slower and more expensive. The solution is compaction. + +But here's the thing not all compaction is the same. The way you compact, how often you compact, and what engine you use to compact can make a dramatic difference in both query performance and resource plus operational cost. That's exactly what we set out to measure in this experiment. + +To answer this properly, we ran a controlled benchmark on Apache Iceberg tables under continuous CDC ingestion. We compared **Apache Spark's** `rewrite_data_files` with **OLake Fusion** compaction while keeping the data, infrastructure, and ingestion engine (i.e. **OLake Go**) exactly the same in both runs. The results were eye-opening. + +:::info Recommended background reading +If you want a conceptual foundation on compaction strategies and tiered maintenance for Iceberg, check out [OLake, Apache Amoro™, and the Iceberg Lakehouse Maintenance Playbook](/blog/olake-amoro-iceberg-lakehouse). +::: + +In this blog, we skip the theory and go straight to the battlefield. + +## Benchmarking Design: + +### 1. Data + +We used the TPC-H dataset (300 GB total across all tables) and focused exclusively on the **lineitem** table — the largest table in the TPC-H schema. When loaded into Apache Iceberg on Google Cloud Storage, the lineitem table came to approximately **85 GB**. The table was **unpartitioned** in both runs, meaning all files were stored together without any partitioning, so any file count growth from CDC directly impacts how long queries take to scan the table. + +### 2. Source + +PostgreSQL served as the OLTP source, where we continuously issued row-level updates every 2 mins to emulate production-like CDC traffic. + +### 3. Destination + +Apache Iceberg on Google Cloud Storage was the lakehouse destination, where frequent CDC commits gradually create small files and metadata overhead. + +### 4. Catalog + +Lakekeeper served as the Iceberg catalog, managing table metadata, snapshots, and commit coordination consistently across both environments. + +### 5. Ingestion Pipeline + +For both environments, we used **OLake Go** as the ingestion engine — one of the fastest open-source tools for replicating data into Apache Iceberg table format. + +The ingestion pipeline ran identically in both environments: + +- **OLake Go** performed a full load of the lineitem table into Iceberg. +- A Python script then began continuously updating a string column in the source PostgreSQL lineitem table — 200,000 rows per iteration, cycling endlessly. +
    + Python CDC updater script used in this benchmark + + ```python + import psycopg2 + import time + + HOST = " 0: + sleep_seconds = next_run_monotonic - time.monotonic() + if sleep_seconds > 0: + print(f" Sleeping {round(sleep_seconds, 1)}s until next run ...") + time.sleep(sleep_seconds) + next_run_monotonic += INTERVAL_SECONDS + + print(f"\n[{run_idx + 1}/{RUNS}] Running UPDATE on {SCHEMA}.{TABLE} - ~200k random rows ...") + start = time.time() + cursor.execute(update_sql) + rows_affected = cursor.rowcount + conn.commit() + elapsed = round(time.time() - start, 2) + print(f"[{run_idx + 1}/{RUNS}] Done. Rows updated: {rows_affected} | Time taken: {elapsed}s") + + except Exception as e: + print(f"ERROR: {e}") + if conn: + conn.rollback() + print("Transaction rolled back.") + raise + + finally: + if cursor: + cursor.close() + if conn: + conn.close() + print("\nConnection closed.") + ``` + +
    +- **OLake Go** was configured to sync every 2 minutes, picking up each batch of updates and writing them to the Iceberg destination as small incremental files. Since these are CDC upserts, **each ingestion cycle produces new equality delete files** alongside data files — meaning with every sync, both file count and delete-file overhead grow steadily. + +This is as close to a real-world production CDC scenario as you can get in a benchmark. + +### 6. Timeline + +The total experiment ran for **2 hours and 10 minutes**, structured deliberately: + +- **First 10 minutes** CDC starts and the first 5 ingestion cycles run (one every 2 minutes) +- **After first 2 CDC ingestions** TPC-H Query 6 starts executing continuously so we can track query time from the beginning. Here is the query we used: + + ```sql + SELECT + SUM(l_extendedprice * l_discount) AS revenue + FROM + lineitem + WHERE + l_shipdate >= DATE '1994-01-01' + AND l_shipdate < DATE '1995-01-01' + AND l_discount BETWEEN 0.05 AND 0.07 + AND l_quantity < 24 + ``` + + :::note + TPC-H Query 6 was kept consistent across both runs to ensure a fair Spark vs Fusion compaction comparison. + ::: + +- **After first 5 CDC ingestions** compaction kicks in to take on the growing small-file load. +- The **2-hour clock** starts the moment compaction kicks in — this is our primary observation window. + +This structure gives us three distinct phases to observe: pre-compaction degradation, the immediate post-compaction response, and long-term steady-state performance. + +## What Happens Without Compaction? + +Without compaction, query execution time climbs steadily as each CDC cycle adds more small files. In our earlier benchmark comparing Iceberg query performance with and without compaction, the difference was significant. The full breakdown is available here: [Iceberg Compaction: How Much Faster Are TPC-H Queries?](/blog/iceberg-compaction-tpch-benchmark) + +## Infrastructure and Resources + +To keep this benchmark clean and unbiased, we provisioned **Fusion** and **Spark** with equivalent infrastructure and reused the **same TPCH query resources** in both runs so query conditions stayed identical. + +Fusion shows one additional resource because that node runs the Fusion service itself, not extra compaction compute, so it does not create an unfair performance advantage. + + + + + | Node role | Instance type | vCPUs | RAM | + |---|---|---|---| + | Master | c4a-standard-8 | 8 | 32 GB | + | Worker | c4a-highmem-16 | 16 | 128 GB | + | Worker | c4a-highmem-16 | 16 | 128 GB | + + + + + | Node role | Instance type | vCPUs | RAM | + |---|---|---|---| + | Master | c4a-standard-8 | 8 | 32 GB | + | Worker | c4a-highmem-16 | 16 | 128 GB | + | Worker | c4a-highmem-16 | 16 | 128 GB | + + + + + | Node role | Instance type | vCPUs | RAM | + |---|---|---|---| + | Master | c4a-standard-8 | 8 | 32 GB | + | Worker | c4a-standard-32 | 32 | 128 GB | + | Worker | c4a-standard-32 | 32 | 128 GB | + + + + +## Compaction Setup and Configuration + +### 1. OLake Fusion Configuration + +OLake Fusion takes a tiered compaction approach, giving you three levels of optimization that can be scheduled independently based on how aggressively you want to maintain your tables. + +All three compaction tiers are explained there in detail, so definitely check out the [OLake Iceberg Maintenance docs](/docs/iceberg-maintenance/compaction/overview/). + +For this benchmark, we used a **512 MB target file size**, so each tier behaves as follows: + +- **Lite:** Files smaller than **64 MB** (1/8 of 512 MB) are merged into approximately **64 MB** output files. Fast, low-cost, and suitable for frequent runs during active ingestion. +- **Medium:** Deletes are applied and files are merged into outputs that typically range from **64 MB to 512 MB**, depending on available data in each optimization cycle. +- **Full:** Performs a complete table rewrite to align output files close to the **512 MB** target, used when deeper cleanup and maximum query performance are required. + +We used the following Fusion cluster configuration for running the compaction: + +| Setting | Value | +|---|---| +| `spark-conf.spark.executor.instances` | 4 | +| `spark-conf.spark.executor.cores` | 7 | +| `spark-conf.spark.executor.memory` | 45g | +| `spark-conf.spark.executor.memoryOverhead` | 12g | +| `spark-conf.spark.driver.memory` | 18g | +| `spark-conf.spark.driver.memoryOverhead` | 9g | +| `spark.dynamicAllocation.enabled` | false | + +Fusion runs table optimization on Spark, and these `spark-conf` values size that workload: enough **executors and cores** to keep compaction moving in parallel, and enough **memory** overall so rewrite and merge steps are not constantly fighting for space. + +### How We Scheduled It + +For this experiment, Fusion ran on the following schedule: + +- Lite compaction: every 20 minutes +- Medium compaction: every 40 minutes + +:::info Why Full compaction was skipped +- We intentionally skipped Full compaction in this benchmark because the observation window and total data size were quite small. It would have had limited time to show its optimal long-term impact, so including it would likely not have changed the results meaningfully. +- Even without Full compaction, Fusion's query performance remained competitive with Spark throughout the experiment, which shows that Lite and Medium alone are sufficient for short to medium-term CDC workloads. +- Full compaction becomes most valuable after several days or even a week of ingestion, when Medium compaction has accumulated a large number of files in the size range between the target file size and one-eighth of it. At that point a Full rewrite consolidates everything cleanly. +::: + +### Compaction Parameters + +We used the following parameters for the compaction: + +- `target-file-size-bytes`: 512 MB (the desired output file size after compaction) + +:::note +By default, `max-task-size-bytes` is set to the same value as `target-file-size-bytes`. +::: + +### 2. Spark Configuration + +Spark's `rewrite_data_files` is the standard compaction tool in the Iceberg ecosystem for Spark-based environments. It rewrites data files based on configurable size bounds, consolidating fragmented files into right-sized outputs. + +We used the following Spark cluster configuration for running the compaction: + +| Setting | Value | +|---|---| +| `spark.executor.instances` | 4 | +| `spark.executor.cores` | 7 | +| `spark.executor.memory` | 45g | +| `spark.executor.memoryOverhead` | 12g | +| `spark.driver.memory` | 18g | +| `spark.driver.memoryOverhead` | 10g | +| `spark.dynamicAllocation.enabled` | false | + +This configuration was chosen to balance parallelism and memory for Iceberg compaction. We used **4 executors** with **7 cores each** so `rewrite_data_files` could drive enough concurrent work, while keeping **generous executor memory** because compaction is **memory-intensive** (reads, shuffles, merges, and writes) and too little memory tends to slow the job down or even lead to OOM crashes. + +### Compaction Parameters + +We used the following parameters for the compaction: + +- `target-file-size-bytes`: **512 MB** + Spark tries to produce output files close to 512 MB after compaction. +- `max-file-size-bytes`: **614 MB** + Any file larger than 614 MB is treated as oversized and selected for rewrite. +- `min-file-size-bytes`: **384 MB** + Any file smaller than 384 MB is treated as undersized and selected for merge. +- `max-concurrent-file-group-rewrites`: **28** + Spark can rewrite up to 28 file groups at the same time, which controls compaction parallelism. +- `partial-progress.enabled`: **false** + Spark commits only if the entire compaction job succeeds; partial results are not committed. +- `delete-file-threshold`: **1** + If a data file has even 1 delete file attached, Spark includes it in compaction. + +### How We Scheduled It + +In this experiment, Spark compaction was scheduled to run every **20 minutes**. We kept this interval fixed to reflect a practical CDC maintenance pattern, where small files keep accumulating and need regular cleanup. + +
    + Python script used to run Spark compaction + + ```python + #!/usr/bin/env python3 + import logging + import time + import datetime + from pyspark.sql import SparkSession + + + # Logging Setup + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s [%(levelname)s] %(message)s", + datefmt="%Y-%m-%d %H:%M:%S", + ) + log = logging.getLogger("iceberg-compaction") + + + CATALOG = "benchmark" + DATABASE = "spark_compact" + TABLES = [ + "lineitem", + ] + + + # Scheduler Config + TOTAL_RUNS = 20 + INTERVAL_MINUTES = 20 + INTERVAL_SECONDS = INTERVAL_MINUTES * 60 + + + # TPC-H Q6 - Forecasting Revenue Change + def build_tpch_q6(catalog, db): + return { + "Q6 - Forecasting Revenue Change": f""" + SELECT + SUM(l_extendedprice * l_discount) AS revenue + FROM + {catalog}.{db}.lineitem + WHERE + l_shipdate >= date '1994-01-01' + AND l_shipdate < date '1995-01-01' + AND l_discount BETWEEN 0.05 AND 0.07 + AND l_quantity < 24 + """ + } + + + # Helper: Per-table file stats (optionally pinned to a snapshot) + def get_table_file_stats(spark, catalog, table_full, snapshot_id=None): + """ + content = 0 -> data files + content = 2 -> equality delete files + snapshot_id -> if provided, reads file state at that exact snapshot (time travel) + if None, reads current state + """ + version_clause = f"VERSION AS OF {snapshot_id}" if snapshot_id else "" + + try: + row = spark.sql( + f"SELECT COUNT(*) AS file_count, SUM(file_size_in_bytes) AS total_bytes " + f"FROM {catalog}.{table_full}.files {version_clause} WHERE content = 0" + ).collect()[0] + data_count = row["file_count"] + data_mb = round(row["total_bytes"] / (1024 * 1024), 2) if row["total_bytes"] else 0 + except Exception as e: + log.warning(f" Could not fetch data file stats: {e}") + data_count, data_mb = "N/A", "N/A" + + try: + eq_count = spark.sql( + f"SELECT COUNT(*) AS file_count " + f"FROM {catalog}.{table_full}.files {version_clause} WHERE content = 2" + ).collect()[0]["file_count"] + except Exception as e: + log.warning(f" Could not fetch equality delete file stats: {e}") + eq_count = "N/A" + + return data_count, data_mb, eq_count + + + # Helper: Find the snapshot ID that compaction created + def get_compaction_snapshot(spark, catalog, table_full, compaction_started_at): + """ + After rewrite_data_files completes, find the snapshot with operation='replace' + that was committed after compaction started. This is the exact compaction snapshot. + Returns (snapshot_id, committed_at) or (None, None) if nothing was rewritten. + """ + try: + rows = spark.sql( + f"SELECT snapshot_id, committed_at " + f"FROM {catalog}.{table_full}.snapshots " + f"WHERE operation = 'replace' " + f"AND committed_at >= TIMESTAMP '{compaction_started_at}' " + f"ORDER BY committed_at ASC " + f"LIMIT 1" + ).collect() + if rows: + snap_id = rows[0]["snapshot_id"] + snap_ts = str(rows[0]["committed_at"]) + log.info(f" Compaction snapshot found : id={snap_id} | committed_at={snap_ts}") + return snap_id, snap_ts + else: + log.warning(f" No replace snapshot found after {compaction_started_at} - " + f"compaction may have found nothing to rewrite.") + return None, None + except Exception as e: + log.warning(f" Could not find compaction snapshot: {e}") + return None, None + + + # Helper: Count OLake ingestion appends during compaction window + def count_ingestions_during_compaction(spark, catalog, table_full, + pre_committed_at, compaction_committed_at): + """ + Counts append snapshots committed strictly AFTER pre_committed_at and + up to AND INCLUDING compaction_committed_at. + + Each OLake ingestion commit adds exactly: 1 data file + 1 equality-delete file. + We subtract this count from the raw POST snapshot counts to isolate the + pure compaction effect (no ingestion noise). + + Returns 0 safely if either timestamp is None. + """ + if not pre_committed_at or not compaction_committed_at: + log.warning(" Skipping ingestion count - missing pre or compaction timestamp.") + return 0 + try: + row = spark.sql( + f"SELECT COUNT(*) AS cnt " + f"FROM {catalog}.{table_full}.snapshots " + f"WHERE operation IN ('append', 'overwrite') " + f"AND committed_at > TIMESTAMP '{pre_committed_at}' " + f"AND committed_at <= TIMESTAMP '{compaction_committed_at}'" + ).collect()[0] + cnt = row["cnt"] + log.info(f" OLake ingestion commits during compaction window : {cnt} " + f"(each = +1 data file, +1 eq-delete file)") + return cnt + except Exception as e: + log.warning(f" Could not count ingestion snapshots during compaction: {e}") + return 0 + + + # Helper: Run TPC-H Q6 and return elapsed time + def run_tpch_q6(spark, label, tpch_queries): + log.info("=" * 70) + log.info(f" TPC-H Query Benchmark [{label}]") + log.info("=" * 70) + results = {} + for qname, qsql in tpch_queries.items(): + try: + q_start = time.time() + spark.sql(qsql).collect() + q_elapsed = round(time.time() - q_start, 2) + log.info(f" {qname:<40} : {q_elapsed}s ({round(q_elapsed / 60, 2)} min)") + results[qname] = q_elapsed + except Exception as e: + log.warning(f" {qname:<40} : FAILED - {e}") + results[qname] = "FAILED" + log.info("=" * 70) + return results + + + # Core compaction logic for a single run + def run_compaction(spark, run_number, tpch_queries): + overall_start_time = time.time() + overall_start_ts = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") + + log.info("=" * 70) + log.info(f" OVERALL COMPACTION RUN STARTED AT : {overall_start_ts}") + log.info(f" Tables to compact : {TABLES}") + log.info("=" * 70) + + for table_name in TABLES: + TABLE = f"{DATABASE}.{table_name}" + FULL_TABLE = f"{CATALOG}.{TABLE}" + + log.info("") + log.info("=" * 70) + log.info(f" TABLE: {FULL_TABLE}") + log.info("=" * 70) + + # Last 5 snapshots - pre + try: + snaps = spark.sql( + f"SELECT snapshot_id, committed_at, operation " + f"FROM {CATALOG}.{TABLE}.snapshots " + f"ORDER BY committed_at DESC LIMIT 5" + ).collect() + log.info(" [PRE] Last 5 snapshots:") + for row in snaps: + log.info(f" snapshot_id={row['snapshot_id']} | " + f"committed_at={row['committed_at']} | " + f"operation={row['operation']}") + except Exception as e: + log.warning(f" Could not fetch pre-compaction snapshots: {e}") + + # File stats - pre + data_cnt, data_mb, eq_del_cnt = get_table_file_stats(spark, CATALOG, TABLE) + log.info(f" [PRE] Data files (content=0) : {data_cnt} files | {data_mb} MB") + log.info(f" [PRE] Eq-delete files (content=2) : {eq_del_cnt} files") + + # Capture PRE boundary timestamp inline - same snapshot the PRE stats just read from. + # Used as the left edge of the ingestion-counting window. + pre_committed_at = None + try: + pre_row = spark.sql( + f"SELECT committed_at FROM {CATALOG}.{TABLE}.snapshots " + f"ORDER BY committed_at DESC LIMIT 1" + ).collect() + pre_committed_at = str(pre_row[0]["committed_at"]) if pre_row else None + log.info(f" [PRE] Snapshot boundary : {pre_committed_at}") + except Exception as e: + log.warning(f" Could not fetch PRE snapshot boundary: {e}") + + # Disable vectorization at table level + try: + spark.sql( + f"ALTER TABLE {FULL_TABLE} " + f"SET TBLPROPERTIES ('read.parquet.vectorization.enabled' = 'false')" + ) + log.info(f" Disabled vectorization for {table_name}.") + except Exception as e: + log.warning(f" Could not set vectorization property: {e}") + + # Compaction config banner + log.info(" Compaction config:") + log.info(" strategy : binpack") + log.info(" target-file-size-bytes : 536870912 (512 MB)") + log.info(" max-file-size-bytes : 644245094 (614.4 MB)") + log.info(" min-file-size-bytes : 402653184 (384 MB)") + log.info(" max-concurrent-file-group-rewrites: 28") + log.info(" partial-progress.enabled : false") + log.info(" delete-file-threshold : 1") + + table_start_time = time.time() + table_start_ts = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") + log.info(f" Compaction started at : {table_start_ts}") + + try: + result = spark.sql( + f""" + CALL {CATALOG}.system.rewrite_data_files( + table => '{TABLE}', + strategy => 'binpack', + options => map( + 'target-file-size-bytes', '536870912', + 'max-file-size-bytes', '644245094', + 'min-file-size-bytes', '402653184', + 'max-concurrent-file-group-rewrites', '28', + 'partial-progress.enabled', 'false', + 'delete-file-threshold', '1' + ) + ) + """ + ) + rows = result.collect() + table_elapsed = round(time.time() - table_start_time, 2) + table_end_ts = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") + + log.info(f" Compaction finished at : {table_end_ts}") + log.info(f" Compaction time for this table: {table_elapsed}s " + f"({round(table_elapsed / 60, 2)} min)") + + for row in rows: + rewritten_files = row["rewritten_data_files_count"] + added_files = row["added_data_files_count"] + rewritten_bytes = ( + row["rewritten_bytes_count"] + if "rewritten_bytes_count" in row.__fields__ + else "N/A" + ) + rewritten_mb = ( + round(rewritten_bytes / (1024 * 1024), 2) + if isinstance(rewritten_bytes, int) + else "N/A" + ) + log.info(" Compaction Result:") + log.info(f" Files rewritten (input) : {rewritten_files}") + log.info(f" Files added (output) : {added_files}") + log.info(f" Total bytes rewritten : {rewritten_bytes} ({rewritten_mb} MB)") + + except Exception as e: + elapsed = round(time.time() - table_start_time, 2) + log.error(f" Compaction FAILED for {table_name} after {elapsed}s") + log.error(f" Error: {e}") + continue + + # Step 1: Find the exact compaction snapshot (operation=replace) + compaction_snapshot_id, compaction_committed_at = get_compaction_snapshot( + spark, CATALOG, TABLE, table_start_ts + ) + + # Step 2: Count OLake ingestions that committed DURING compaction + # Window: strictly after PRE snapshot -> up to and including compaction snapshot + # Each ingestion = +1 data file, +1 eq-delete file (OLake guarantee) + ingestions_during_compaction = count_ingestions_during_compaction( + spark, CATALOG, TABLE, pre_committed_at, compaction_committed_at + ) + + # Step 3: Read POST file counts pinned to the compaction snapshot + data_cnt_p, data_mb_p, eq_del_cnt_p = get_table_file_stats( + spark, CATALOG, TABLE, snapshot_id=compaction_snapshot_id + ) + + # Step 4: Subtract ingestion noise -> pure compaction POST counts + # The compaction snapshot inherits ingested files via Iceberg's linear chain. + # Subtracting ingestions_during_compaction isolates the compaction-only effect. + true_data_cnt_p = ( + data_cnt_p - ingestions_during_compaction + if isinstance(data_cnt_p, int) + else data_cnt_p + ) + true_eq_del_cnt_p = ( + eq_del_cnt_p - ingestions_during_compaction + if isinstance(eq_del_cnt_p, int) + else eq_del_cnt_p + ) + + if compaction_snapshot_id: + log.info(f" [POST] File stats pinned to compaction snapshot {compaction_snapshot_id}:") + else: + log.info(f" [POST] File stats (no replace snapshot found - showing current state):") + + log.info(f" [POST] Data files (content=0) at snapshot : {data_cnt_p} files") + log.info(f" [POST] Eq-delete (content=2) at snapshot : {eq_del_cnt_p} files") + log.info(f" [POST] Minus ingestion files during window : -{ingestions_during_compaction} (data), -{ingestions_during_compaction} (eq-delete)") + log.info(f" [POST] True post-compaction data files : {true_data_cnt_p} files | {data_mb_p} MB") + log.info(f" [POST] True post-compaction eq-delete files : {true_eq_del_cnt_p} files") + + # Delta - pure compaction effect only + if isinstance(data_cnt, int) and isinstance(true_data_cnt_p, int): + log.info(f" [DELTA] Data files : {data_cnt} -> {true_data_cnt_p} " + f"(change: {true_data_cnt_p - data_cnt:+d})") + if isinstance(eq_del_cnt, int) and isinstance(true_eq_del_cnt_p, int): + log.info(f" [DELTA] Eq-del files : {eq_del_cnt} -> {true_eq_del_cnt_p} " + f"(change: {true_eq_del_cnt_p - eq_del_cnt:+d})") + + # Last 3 snapshots - post + try: + snaps_after = spark.sql( + f"SELECT snapshot_id, committed_at, operation " + f"FROM {CATALOG}.{TABLE}.snapshots " + f"ORDER BY committed_at DESC LIMIT 3" + ).collect() + log.info(" [POST] Latest snapshots:") + for row in snaps_after: + log.info(f" snapshot_id={row['snapshot_id']} | " + f"committed_at={row['committed_at']} | " + f"operation={row['operation']}") + except Exception as e: + log.warning(f" Could not fetch post-compaction snapshots: {e}") + + # Per-run Summary + overall_elapsed = round(time.time() - overall_start_time, 2) + overall_end_ts = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") + + log.info("") + log.info("=" * 70) + log.info(f" RUN {run_number} - COMPACTION SUMMARY") + log.info("=" * 70) + log.info(f" Started at : {overall_start_ts}") + log.info(f" Ended at : {overall_end_ts}") + log.info(f" Total time : {overall_elapsed}s ({round(overall_elapsed / 60, 2)} min)") + log.info("=" * 70) + + + # SPARK SESSION + log.info("Initializing SparkSession...") + spark = ( + SparkSession.builder + .appName("iceberg-compaction-tpch-spark_compact") + .config("spark.sql.extensions", + "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") + .config("spark.sql.catalog.benchmark", + "org.apache.iceberg.spark.SparkCatalog") + .config("spark.sql.catalog.benchmark.catalog-impl", + "org.apache.iceberg.rest.RESTCatalog") + .config("spark.sql.catalog.benchmark.uri", + "http://10.20.0.64:30081/catalog") + .config("spark.sql.catalog.benchmark.warehouse", + "benchmarking") + .config("spark.sql.catalog.benchmark.io-impl", + "org.apache.iceberg.aws.s3.S3FileIO") + .config("spark.sql.catalog.benchmark.s3.endpoint", + "https://storage.googleapis.com/") + .config("spark.sql.catalog.benchmark.s3.path-style-access", "true") + .config("spark.sql.catalog.benchmark.client.region", "ap-south-1") + .config("spark.hadoop.fs.s3a.endpoint", "https://storage.googleapis.com") + .config("spark.hadoop.fs.s3a.path.style.access", "true") + .config("spark.hadoop.fs.s3a.endpoint.region", "ap-south-1") + .config("spark.sql.shuffle.partitions", "128") + .config("spark.sql.defaultCatalog", "benchmark") + .getOrCreate() + ) + + spark.conf.set("spark.sql.iceberg.vectorization.enabled", "false") + spark.sparkContext.setLogLevel("INFO") + + log.info("SparkSession initialized successfully.") + log.info(f" Spark version : {spark.version}") + log.info(f" App name : {spark.sparkContext.appName}") + log.info(f" Master : {spark.sparkContext.master}") + log.info(f" Default parallelism : {spark.sparkContext.defaultParallelism}") + + TPCH_QUERIES = build_tpch_q6(CATALOG, DATABASE) + + + # SCHEDULED LOOP + schedule_anchor = time.time() + + for run_number in range(1, TOTAL_RUNS + 1): + + scheduled_start = schedule_anchor + (run_number - 1) * INTERVAL_SECONDS + now = time.time() + wait_seconds = scheduled_start - now + + if wait_seconds > 0: + next_run_ts = datetime.datetime.fromtimestamp(scheduled_start).strftime("%Y-%m-%d %H:%M:%S") + log.info("") + log.info("~" * 70) + log.info(f" Waiting {round(wait_seconds, 1)}s until next scheduled run at {next_run_ts} ...") + log.info("~" * 70) + time.sleep(wait_seconds) + + log.info("") + log.info("#" * 70) + log.info(f"#{'':^68}#") + log.info(f"#{'RUN ' + str(run_number) + ' OF ' + str(TOTAL_RUNS):^68}#") + log.info(f"#{'':^68}#") + log.info("#" * 70) + + run_compaction(spark, run_number, TPCH_QUERIES) + + log.info("") + log.info("-" * 70) + log.info(f" END OF RUN {run_number} OF {TOTAL_RUNS}") + log.info("-" * 70) + + remaining = TOTAL_RUNS - run_number + if remaining > 0: + log.info(f" Remaining runs : {remaining}") + for future_run in range(run_number + 1, TOTAL_RUNS + 1): + future_ts = datetime.datetime.fromtimestamp( + schedule_anchor + (future_run - 1) * INTERVAL_SECONDS + ).strftime("%Y-%m-%d %H:%M:%S") + log.info(f" Run {future_run} scheduled at : {future_ts}") + else: + log.info(" All scheduled runs completed. No further runs.") + log.info("-" * 70) + + + # ALL RUNS DONE + log.info("") + log.info("=" * 70) + log.info(" ALL COMPACTION RUNS FINISHED") + log.info(f" Total runs executed : {TOTAL_RUNS}") + log.info(f" Interval : every {INTERVAL_MINUTES} minute(s)") + log.info("=" * 70) + log.info("") + log.info("All done. Stopping SparkSession.") + spark.stop() + ``` + +
    + +
    + Spark-submit command used to run this script + + ```bash + gcloud dataproc jobs submit pyspark gs://dz-benchmark/dataproc_compaction.py \ + --cluster=dz-olake-compaction-21042026 \ + --region=asia-south1 \ + --properties="^#^spark.jars.packages=org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.7.2,org.apache.iceberg:iceberg-aws-bundle:1.7.2#spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions#spark.sql.catalog.benchmark=org.apache.iceberg.spark.SparkCatalog#spark.sql.catalog.benchmark.catalog-impl=org.apache.iceberg.rest.RESTCatalog#spark.sql.catalog.benchmark.uri=http://10.20.0.64:30081/catalog#spark.sql.catalog.benchmark.warehouse=benchmarking#spark.sql.catalog.benchmark.io-impl=org.apache.iceberg.aws.s3.S3FileIO#spark.sql.catalog.benchmark.s3.endpoint=https://storage.googleapis.com/#spark.sql.catalog.benchmark.s3.path-style-access=true#spark.sql.catalog.benchmark.client.region=ap-south-1#spark.sql.defaultCatalog=benchmark#spark.hadoop.fs.s3a.endpoint=https://storage.googleapis.com#spark.hadoop.fs.s3a.path.style.access=true#spark.hadoop.fs.s3a.endpoint.region=ap-south-1#spark.dynamicAllocation.enabled=false#spark.executor.instances=4#spark.executor.cores=7#spark.executor.memory=45g#spark.executor.memoryOverhead=12g#spark.driver.memory=18g#spark.driver.memoryOverhead=10g#spark.sql.parquet.enableVectorizedReader=false#spark.sql.iceberg.vectorization.enabled=false#spark.hadoop.io.native.lib.available=false#spark.executor.extraJavaOptions=-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35 -XX:G1ReservePercent=20 -XX:UseAVX=0 -Darrow.enable_unsafe_memory_access=false -Darrow.enable_null_check_for_get=true -Dhadoop.io.native.lib.available=false#spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35 -Xss8m -XX:UseAVX=0 -Darrow.enable_unsafe_memory_access=false -Darrow.enable_null_check_for_get=true" + ``` + +
    + +## TPC-H Query Execution + +TPC-H Query 6 is a scan-heavy analytical query with filter predicates, making it extremely sensitive to file count and layout. It's the perfect canary for detecting compaction effectiveness. + +Alongside compaction, we ran **TPC-H Query 6 continuously** throughout the experiment with no wait time between runs - each new execution started immediately after the previous one finished. This allowed us to capture performance across all table states (before compaction, immediately after compaction, and later as new CDC files accumulated) and compute an unbiased average without overrepresenting any single phase. + +We used the following Spark cluster configuration for TPC-H Query 6 runs: + +| Setting | Value | +|---|---| +| `spark.executor.instances` | 12 | +| `spark.executor.cores` | 5 | +| `spark.executor.memory` | 15g | +| `spark.executor.memoryOverhead` | 4g | +| `spark.driver.memory` | 18g | +| `spark.driver.memoryOverhead` | 10g | +| `spark.dynamicAllocation.enabled` | false | + +This configuration was chosen to keep Query 6 execution consistently fast by prioritizing parallelism (more executors and cores) while using practical memory limits. Since Query 6 is mostly scan-heavy rather than memory-heavy, this setup delivers better throughput without unnecessary memory over-provisioning. + +
    + Python script used to execute TPC-H Query 6 + + ```python + #!/usr/bin/env python3 + import logging + import time + import datetime + from pyspark.sql import SparkSession + + # Logging Setup + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s [%(levelname)s] %(message)s", + datefmt="%Y-%m-%d %H:%M:%S", + ) + log = logging.getLogger("tpch-benchmark") + + CATALOG = "benchmark" + # Iceberg namespace in Lakekeeper (REST catalog); must hold `lineitem`. + DATABASE = "spark_compact" + # Lakekeeper REST catalog warehouse id (matches spark.sql.catalog.benchmark.warehouse). + LAKEKEEPER_WAREHOUSE = "benchmarking" + LAKEKEEPER_URI = "http://10.20.0.64:30081/catalog" + + # Run Config + TOTAL_RUNS = 200 + BREAK_SECONDS = 0 # no pause between runs + + # TPC-H Q6 - Forecasting Revenue Change + # Single table scan on lineitem only. No joins, no GROUP BY. + # Pure sequential scan - best query to measure compaction impact. + TPCH_Q6 = f""" + SELECT + SUM(l_extendedprice * l_discount) AS revenue + FROM + {CATALOG}.{DATABASE}.lineitem + WHERE + l_shipdate >= DATE '1994-01-01' + AND l_shipdate < DATE '1995-01-01' + AND l_discount BETWEEN 0.05 AND 0.07 + AND l_quantity < 24 + """ + + # SparkSession + log.info("Initializing SparkSession...") + spark = ( + SparkSession.builder + .appName("tpch-q6-benchmark") + .config("spark.sql.extensions", + "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") + .config("spark.sql.catalog.benchmark", + "org.apache.iceberg.spark.SparkCatalog") + .config("spark.sql.catalog.benchmark.catalog-impl", + "org.apache.iceberg.rest.RESTCatalog") + .config("spark.sql.catalog.benchmark.uri", + LAKEKEEPER_URI) + .config("spark.sql.catalog.benchmark.warehouse", + LAKEKEEPER_WAREHOUSE) + .config("spark.sql.catalog.benchmark.io-impl", + "org.apache.iceberg.aws.s3.S3FileIO") + .config("spark.sql.catalog.benchmark.s3.endpoint", + "https://storage.googleapis.com/") + .config("spark.sql.catalog.benchmark.s3.path-style-access", "true") + .config("spark.sql.catalog.benchmark.client.region", "ap-south-1") + .config("spark.hadoop.fs.s3a.endpoint", "https://storage.googleapis.com") + .config("spark.hadoop.fs.s3a.path.style.access", "true") + .config("spark.hadoop.fs.s3a.endpoint.region", "ap-south-1") + .config("spark.sql.shuffle.partitions", "128") + .config("spark.sql.defaultCatalog", "benchmark") + .getOrCreate() + ) + + spark.conf.set("spark.sql.iceberg.vectorization.enabled", "false") + spark.sparkContext.setLogLevel("INFO") + + log.info("SparkSession initialized successfully.") + log.info(f" Spark version : {spark.version}") + log.info(f" App name : {spark.sparkContext.appName}") + log.info(f" Master : {spark.sparkContext.master}") + log.info(f" Default parallelism : {spark.sparkContext.defaultParallelism}") + log.info(f" Lakekeeper URI : {LAKEKEEPER_URI}") + log.info(f" Catalog warehouse : {LAKEKEEPER_WAREHOUSE}") + log.info(f" Namespace (db) : {DATABASE}") + log.info(f" Total Q6 runs : {TOTAL_RUNS}") + log.info(f" Break between runs : {BREAK_SECONDS}s") + + run_results = [] # {run, started_at, finished_at, elapsed_s, revenue, status} + + # BENCHMARK LOOP - Q6 x TOTAL_RUNS, no pause between runs + for run_number in range(1, TOTAL_RUNS + 1): + log.info("") + log.info("#" * 70) + log.info(f"#{'':^68}#") + log.info(f"#{'RUN ' + str(run_number) + ' OF ' + str(TOTAL_RUNS):^68}#") + log.info(f"#{'':^68}#") + log.info("#" * 70) + + log.info("") + log.info("=" * 70) + log.info(" TPC-H Q6 - Forecasting Revenue Change") + log.info(f" Table : {CATALOG}.{DATABASE}.lineitem") + log.info(" Type : Single table scan | No joins | No GROUP BY") + log.info("=" * 70) + + run_start_ts = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") + log.info(f" Query started at : {run_start_ts}") + + q_start = time.time() + try: + result = spark.sql(TPCH_Q6) + rows = result.collect() + q_elapsed = round(time.time() - q_start, 2) + run_end_ts = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") + + revenue_val = rows[0]["revenue"] if rows else None + + log.info(f" Query finished at : {run_end_ts}") + log.info(f" Query time : {q_elapsed}s ({round(q_elapsed / 60, 2)} min)") + log.info("") + log.info(" Result:") + log.info(f" revenue = {revenue_val}") + + run_results.append({ + "run": run_number, + "started_at": run_start_ts, + "finished_at": run_end_ts, + "elapsed_s": q_elapsed, + "revenue": revenue_val, + "status": "OK", + }) + + except Exception as e: + q_elapsed = round(time.time() - q_start, 2) + run_end_ts = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") + log.error("=" * 70) + log.error(f" Q6 FAILED after {q_elapsed}s") + log.error(f" Error: {e}") + log.error("=" * 70) + run_results.append({ + "run": run_number, + "started_at": run_start_ts, + "finished_at": run_end_ts, + "elapsed_s": q_elapsed, + "revenue": None, + "status": f"FAILED: {e}", + }) + + log.info("") + log.info("-" * 70) + log.info(f" END OF RUN {run_number} OF {TOTAL_RUNS}") + log.info(f" Elapsed : {q_elapsed}s ({round(q_elapsed / 60, 2)} min)") + remaining = TOTAL_RUNS - run_number + if remaining > 0: + log.info(f" Remaining runs : {remaining}") + else: + log.info(" All runs completed.") + log.info("-" * 70) + + if remaining > 0 and BREAK_SECONDS > 0: + log.info(f" Cooling down for {BREAK_SECONDS}s before next run...") + time.sleep(BREAK_SECONDS) + + # FINAL SUMMARY + log.info("") + log.info("=" * 70) + log.info(" BENCHMARK COMPLETE - ALL RUNS SUMMARY") + log.info("=" * 70) + log.info(f" {'RUN':<6} {'STARTED AT':<22} {'ELAPSED (s)':<14} {'STATUS'}") + log.info(f" {'-'*4:<6} {'-'*19:<22} {'-'*11:<14} {'-'*10}") + + ok_times = [] + for r in run_results: + elapsed_str = str(r["elapsed_s"]) if r["status"] == "OK" else "FAILED" + log.info(f" {r['run']:<6} {r['started_at']:<22} {elapsed_str:<14} {r['status']}") + if r["status"] == "OK": + ok_times.append(r["elapsed_s"]) + + log.info("") + if ok_times: + log.info(f" Successful runs : {len(ok_times)} / {TOTAL_RUNS}") + log.info(f" Min query time : {min(ok_times)}s") + log.info(f" Max query time : {max(ok_times)}s") + log.info(f" Avg query time : {round(sum(ok_times) / len(ok_times), 2)}s") + log.info(f" Total query time : {round(sum(ok_times), 2)}s " + f"({round(sum(ok_times) / 60, 2)} min)") + else: + log.error(" No successful runs to summarize.") + + log.info("=" * 70) + log.info("") + log.info("All done. Stopping SparkSession.") + spark.stop() + ``` + +
    + +
    + Spark-submit command used for the TPC-H query script + + ```bash + gcloud dataproc jobs submit pyspark gs://dz-benchmark/tpch_dataproc.py \ + --cluster=dz-olake-tpch-21042026 \ + --region=asia-south1 \ + --properties="^#^spark.jars.packages=org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.7.2,org.apache.iceberg:iceberg-aws-bundle:1.7.2#spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions#spark.sql.catalog.benchmark=org.apache.iceberg.spark.SparkCatalog#spark.sql.catalog.benchmark.catalog-impl=org.apache.iceberg.rest.RESTCatalog#spark.sql.catalog.benchmark.uri=http://10.20.0.64:30081/catalog#spark.sql.catalog.benchmark.warehouse=benchmarking#spark.sql.catalog.benchmark.io-impl=org.apache.iceberg.aws.s3.S3FileIO#spark.sql.catalog.benchmark.s3.endpoint=https://storage.googleapis.com/#spark.sql.catalog.benchmark.s3.path-style-access=true#spark.sql.catalog.benchmark.client.region=ap-south-1#spark.sql.defaultCatalog=benchmark#spark.hadoop.fs.s3a.endpoint=https://storage.googleapis.com#spark.hadoop.fs.s3a.path.style.access=true#spark.hadoop.fs.s3a.endpoint.region=ap-south-1#spark.dynamicAllocation.enabled=false#spark.executor.instances=12#spark.executor.cores=5#spark.executor.memory=15g#spark.executor.memoryOverhead=4g#spark.driver.memory=18g#spark.driver.memoryOverhead=10g#spark.sql.parquet.enableVectorizedReader=false#spark.sql.iceberg.vectorization.enabled=false#spark.hadoop.io.native.lib.available=false#spark.executor.extraJavaOptions=-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35 -XX:G1ReservePercent=20 -XX:UseAVX=0 -Darrow.enable_unsafe_memory_access=false -Darrow.enable_null_check_for_get=true -Dhadoop.io.native.lib.available=false#spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35 -Xss8m -XX:UseAVX=0 -Darrow.enable_unsafe_memory_access=false -Darrow.enable_null_check_for_get=true" + ``` + +
    + +## Benchmark Results: Who Actually Wins? + +To make this comparison fair, we first made sure **query execution behavior stayed as aligned as possible** across both setups. Our goal was to avoid drawing conclusions from query-pattern differences and instead focus on what we actually wanted to compare: **compaction time, resource usage, and cost** for Spark vs Fusion. + +Once query-time behavior was in a similar range, we had a stable baseline for evaluating the other constraints honestly. That let us compare both systems on operational efficiency, not just raw query outcomes. + +### 1. TPC-H Query Time + +In this section, we report the **minimum**, **maximum**, and **average** execution time of TPC-H Query 6 for both scenarios (Spark compaction and Fusion compaction). This gives a clear view of best-case, worst-case, and steady-state query behavior under continuous CDC and active maintenance. + + + + + ![TPC-H Query 6 execution time — Fusion compaction](/img/blog/2026/4/fusion_tpch.webp) + + {/* Over **134** consecutive runs with Fusion compaction enabled, TPC-H Query 6 was about **43s at best**, up to roughly **77s at worst**, with an **average near 56s**. Compaction keeps pulling the table back from small-file bloat, so query time does not stay stuck at the high end—it spikes and then settles again instead of drifting upward without bound. + */} + + + + ![TPC-H Query 6 execution time — Spark compaction](/img/blog/2026/4/spark_tpch.webp) + + {/* Over about **123** consecutive runs with Spark `rewrite_data_files` compaction on the same schedule, TPC-H Query 6 was about **45s at best**, up to roughly **83s at worst**, with an **average near 59s**. Periodic Spark compaction keeps the layout from degrading unchecked, so runtimes climb during small-file buildup then **drop back** instead of staying pinned near the worst case. + */} + + + +Comparison between Fusion and Spark TPC-H Query 6 execution time: + +| Metric | Fusion | Spark | +| --- | --- | --- | +| Average query time | 56.00 s | 58.72 s | +| Min query time | 42.90 s | 45.62 s | +| Max query time | 76.71 s | 83.46 s | + +### 2. Compaction Time + +Here we compare how long each compaction approach took to run under the same ingestion and observation window. Use the tabs below for Fusion vs Spark. + + + + + ![Fusion compaction time](/img/blog/2026/4/fusion_compaction.webp) + + **Summary (Fusion):** + - **Total compaction time:** 27 mins 02 secs + - **Average per run:** 4 mins 30 secs + - **Min:** 4 mins 11 secs + - **Max:** 4 mins 46 secs + + + + + ![Spark compaction time](/img/blog/2026/4/spark_compaction.webp) + + **Summary (Spark):** + - **Total compaction time:** 55 mins 47 secs + - **Average per run:** 9 mins 18 secs + - **Min:** 9 mins 03 secs + - **Max:** 9 mins 29 secs + + + + +Both Fusion and Spark compaction ran **6 times** in this window: + +| Run | Fusion | Spark | Conclusion | +| --- | --- | --- | --- | +| 1 | 4 mins 12 secs | 9 mins 21 secs | **Fusion ~2.2x faster** | +| 2 | 4 mins 11 secs | 9 mins 03 secs | **Fusion ~2.2x faster** | +| 3 | 4 mins 33 secs | 9 mins 13 secs | **Fusion ~2.0x faster** | +| 4 | 4 mins 46 secs | 9 mins 16 secs | **Fusion ~1.9x faster** | +| 5 | 4 mins 36 secs | 9 mins 25 secs | **Fusion ~2.0x faster** | +| 6 | 4 mins 44 secs | 9 mins 29 secs | **Fusion ~2.0x faster** | + +**Final compaction-time verdict:** In this benchmark, **Fusion clearly outperforms Spark**. It finishes each compaction cycle in **4 mins 30 secs** on average, compared with **9 mins 18 secs** for Spark (**~2.06x faster**). Across the full 6-run window, Fusion takes **27 mins 02 secs**, while Spark takes **55 mins 47 secs**. + +### 3. Cost + +In this section, we compare the compaction cost for Spark and Fusion based on the resources used and the time each compaction workflow took to run. The objective is to show the operational trade-off clearly: not just which approach works, but which one delivers better efficiency per unit of compute spend. + +Before calculating total compaction cost, here are the hourly rates for each instance type used in this benchmark: + +| Instance type | Disk size | Cost per hour (USD) | +| --- | --- | --- | +| `c4a-standard-8` | 100 GB | $0.38 | +| `c4a-highmem-16` | 100 GB | $0.99 | +| `c4a-standard-32` | 1000 GB | $1.60 | + +Using the compaction infrastructure and total compaction durations from this benchmark: + +| Engine | Resources included for compaction cost | Hourly infrastructure cost | Total compaction time | Total compaction cost | +| --- | --- | --- | --- | --- | +| Fusion | `c4a-standard-8`
    2 x `c4a-highmem-16` | $2.36/hour | 27 mins 02 secs | **$1.06** | +| Spark | `c4a-standard-8`
    2 x `c4a-highmem-16` | $2.36/hour | 55 mins 47 secs | **$2.19** | + +#### Cost Summary + +- **Fusion total compaction cost:** **$1.06** +- **Spark total compaction cost:** **$2.19** +- **Difference:** Spark costs about **$1.13 more** in this window +- **Relative:** Fusion is about **51.6% lower cost** (or Spark is about **2.06x more expensive**) for compaction in this benchmark + +If we project this same 2-hour pattern linearly, the cost gap compounds quickly: + +| Period | Fusion cost | Spark cost | Spark extra cost | +| --- | --- | --- | --- | +| Daily | $12.72 | $26.28 | **$13.56** | +| Weekly | $89.04 | $183.96 | **$94.92** | +| Monthly (30 days) | $381.60 | $788.40 | **$406.80** | + +_These are directional projections based on this benchmark window; real production cost will vary with ingestion rate, file growth, and compaction frequency._ + +## Conclusion + +Compaction strategy is not a background maintenance detail; it directly impacts **query latency**, **compaction runtime**, and **infrastructure cost**. + +Under the same ingestion pressure and comparable infrastructure, **Fusion outperformed Spark** across core metrics in this benchmark window: + +- **Query time:** ~**56s** average on Fusion vs ~**59s** on Spark +- **Compaction time:** **27m 02s** for Fusion vs **55m 47s** for Spark (**~2.06x faster**) +- **Cost:** **$1.06** on Fusion vs **$2.19** on Spark (**~51.6% lower**) + +**Most important takeaway:** It is **not** necessary to run **Full compaction** in every cycle. For continuous CDC-heavy workloads, the day-to-day strategy should be **Lite** and **Medium** compaction to control file growth with much lower runtime and cost. **Full compaction** should be treated as a periodic deep-clean operation, triggered less frequently when long-term small-file accumulation demands it. diff --git a/blog/2026-04-28-olake-fusion-introduction-blog.mdx b/blog/2026-04-28-olake-fusion-introduction-blog.mdx new file mode 100644 index 000000000..4cd79c0dc --- /dev/null +++ b/blog/2026-04-28-olake-fusion-introduction-blog.mdx @@ -0,0 +1,213 @@ +--- +slug: apache-iceberg-table-maintenance-olake-fusion +title: "Apache Iceberg Table Maintenance Made Easier with OLake Fusion" +description: OLake Fusion is an Apache Iceberg table maintenance solution for CDC tables, helping manage small files and delete files with tiered scheduling, metrics, and lower Spark costs. +date: 2026-04-28 +tags: [iceberg, olake, fusion, compaction, optimization, apache-iceberg, iceberg-tables, iceberg-maintenance, small-files, binpack-compaction, sort-compaction, manifest-rewrite, lakehouse, metadata-optimization] +authors: [siddharth] +image: /img/blog/cover/OLakeFusionBlogCoverImage.webp +--- +![OLake fusion cover image](/img/blog/cover/OLakeFusionBlogCoverImage.webp) + +# Apache Iceberg Maintenance Made Easier with OLake Fusion + +Apache Iceberg is the right choice for most modern lakehouses ([read more](https://olake.io/blog/apache-iceberg-features-benefits/)). It gives you ACID guarantees, schema evolution, time travel, and genuinely fast analytical queries — without locking you into any single vendor or engine. The adoption numbers back it up: Iceberg has quietly become the default open table format for teams building serious data infrastructure. + +But here's what nobody tells you when you're getting started: picking the right table format is only half the job. The other half is *keeping those tables healthy*. And that part? It's a lot harder than it looks. + +This blog is about that second half — specifically, why Iceberg table maintenance tends to become a full-time headache, what teams are doing today to cope with it, and what we built at OLake to actually solve it. +## The Apache Iceberg Small Files Problem + +The Apache Iceberg small files problem shows up when streaming ingestion, CDC syncs, or frequent micro-batches create thousands of small Parquet files instead of fewer well-sized data files. It builds quietly until query performance is already degraded. + +Every time a CDC pipeline writes data to Iceberg, it creates new files in object storage. That's how the format works. The trouble is that modern CDC pipelines write constantly. Row-level changes streaming in every few seconds, each batch producing a tiny new file. What should've been 50 well-sized Parquet files has turned into 50,000 tiny ones spread across your table. + +![Small files problem diagram](/img/blog/2026/5/small-file-problem.webp) + +This is the small files problem, and it triggers a cascade of issues. + +**Query engines have to work much harder.** Engines like Spark, Trino, or Athena don't read a table as a single unit. They read individual files. With 50,000 small files, every query involves thousands of extra file listings, metadata reads, and I/O round trips. The total data size hasn't changed, but the work has grown by orders of magnitude. + +![Query slowdown diagram](/img/blog/2026/5/query-slowdown.webp) + +**Metadata becomes a bottleneck on its own.** Iceberg lists every data file in manifests. The more files you have, the heavier those lists get. Planning a query or committing a write then takes longer, because the engine has to scan a much larger inventory before it can do real work. + +**Delete file accumulation makes this even worse.** In CDC-heavy pipelines, every sync doesn't just create new data files. It also creates delete files that track which rows were updated or deleted. These delete files are how Iceberg handles upserts without rewriting entire data files on every change. But delete files have a cost: every query has to apply them at read time to get the correct view of the data. As they pile up, the overhead of applying deletes during reads becomes significant. A table with thousands of delete files will be noticeably slower than the same table after they've been resolved. + +![Delete file problem diagram](/img/blog/2026/5/delete-file-problem.webp) + +**Object storage costs creep up silently.** Cloud storage doesn't just charge for how much data you store. It also charges per API request. More files means more reads, more listings, more API calls on every operation. You won't notice it until the bill shows up, and by then you've been overpaying for weeks. + +![Storage Costs problem diagram](/img/blog/2026/5/storage-cost.webp) + +None of this happens suddenly. It builds up quietly, which is exactly why it catches teams off guard. By the time performance is obviously degraded, the tables are already in rough shape. + +## Why Vanilla Spark compaction is hard to operate + +The standard fix for small files and delete accumulation is **compaction**: periodically rewriting fragmented small files into larger, well-organized ones and resolving accumulated deletes into the data. Iceberg's ecosystem supports this, and Apache Spark has become the de facto tool for it via `rewrite_data_files`. + +So most teams end up doing something like this: + +They write a Python script that calls `rewrite_data_files(...)` with the right parameters. They figure out executor counts, memory settings, file size bounds, and parallelism through trial and error. They wire it up to a job scheduler or a cron job to run every 20 or 30 minutes. A few weeks later, their ingestion rate changes, and the parameters they tuned are no longer appropriate for the table's current state. + +This works. Teams do make it work. But look at what they're actually doing: + +**Writing custom spark scripts to maintain Iceberg tables.** The compaction script itself becomes a thing that needs documentation, version control, incident response, and occasional debugging at 2am when a job fails and nobody knows why. That's before you account for the fact that most teams have more than one Iceberg table. + +**Running one compaction setup for situations that need different treatment.** Some tables need frequent, aggressive compaction, while others only need lighter compaction on a slower schedule. Spark's `rewrite_data_files` doesn't differentiate. It processes whatever files fall within your size bounds, regardless of whether that's the right level of intervention for the current table state. + +**Figuring out what happened by digging through scattered logs.** Most setups save something like: scheduler run history, Spark driver logs, or files on disk. The hard part is connecting that output back to the table itself: whether file layout improved, deletes were absorbed, and whether the job really helped. When a run fails, or shows success while queries stay slow, **why** is often still unclear. Errors and exit codes alone rarely say what went wrong; you hunt through executor logs and put the picture together by hand. + +## Introducing OLake Fusion: Simplified Apache Iceberg Table Maintenance + +![Introducing OLake Fusion](/img/blog/cover/OLakeFusionBlogCoverImage.webp) + +OLake Fusion is a dedicated Iceberg maintenance service built for CDC-heavy Apache Iceberg tables. It handles compaction on a per-table cron schedule you configure, with tiered compaction levels, built-in metrics, and enough observability to actually understand what's happening to your tables. + +No custom Spark scripts. No wondering if last night's compaction job did anything useful. + +### Tiered Iceberg Compaction: Lite, Medium, and Full + +The most important thing about OLake Fusion's approach is that it doesn't treat all compaction as the same operation. It offers three compaction tiers that you can schedule independently, each designed for a different kind of table maintenance need. + +**Lite:** Designed for small, frequent cleanup tasks. It keeps tables from slowly sliding into bad shape without using much compute, so you can run it often. + +**Medium:** Designed for regular cleanup when small files and deletes are starting to slow reads. It does more work than Lite, but avoids the cost of rewriting the whole table. + +**Full:** Designed for deep cleanup tasks where the whole table needs to be laid out fresh. It uses the most compute, so it makes sense for occasional resets, not frequent runs. + +One important detail: if multiple tiers are scheduled to run at the same time, Fusion automatically runs only the highest one. Medium overrides Lite. Full overrides both. You don't end up doing redundant work when schedules overlap. + +For the exact details of what each tier does, see the [Types of Compaction](https://olake.io/docs/iceberg-maintenance/compaction/overview/). + + + +This tiered approach matters because it lets you optimize for cost and efficiency at the same time. Running Full compaction every few minutes on a CDC table is wasteful. You're rewriting data that doesn't need rewriting. Running only Lite is insufficient if delete files are building up and impacting read performance. The right answer is run Lite frequently, Medium regularly, Full occasionally. Fusion makes it easy to express exactly that. + +### OLake Fusion vs Vanilla Spark Compaction: Faster, Cheaper Iceberg Maintenance + +On comparable infrastructure, Fusion costs about **50% less** than Apache Spark’s `rewrite_data_files` for the same compaction workload without giving up table layout quality. Run-by-run timings, query checks, methodology, and cost breakdown are covered in [OLake Fusion vs Spark compaction benchmark](https://olake.io/blog/iceberg-compaction-spark-vs-fusion-benchmark/) + +| | Apache Spark `rewrite_data_files` | OLake Fusion | +|---|---|---| +| Setup | Custom script per table | Per-table config in UI | +| Scheduling | Manual | Built-in cron per tier | +| Compaction tiers | Single operation | Lite / Medium / Full | +| Observability | Manual log digging | Built-in metrics per run | +| Cost vs Spark | Baseline | ~50% lower | +| Speed vs Spark | Baseline | ~2x faster | + + +### Iceberg Maintenance Observability and Table Health Metrics + +Here's a problem that doesn't get talked about enough: with custom Spark compaction scripts, visibility is usually something you have to build and maintain yourself. + +You can query Iceberg metadata tables before and after a Spark job to calculate file counts and delete counts. But in practice, teams still have to wire that into the job, store the results, connect them to run history, and make them easy to inspect when something feels slow. Fusion makes that visibility part of the product instead of another extra script. + +Fusion comes with observability built in, at two levels. + +**Per-run logs and metrics.** Fusion keeps logs and metrics for each compaction run, so you can see what happened and dig in without starting from unrelated job noise. More in [Runs and logs](https://olake.io/docs/iceberg-maintenance/runs-and-logs). + +![Runs page](/img/docs/iceberg-maintenance/runs-and-logs/runs-page.webp) + +**Input vs output for each run.** After each compaction run, Fusion shows metrics for inputs and outputs: counts and sizes for data files and deletes, recorded before versus after each run. You read them straight from the UI instead of reconstructing totals only from unstructured logs. + +![Runs Metrics](/img/blog/2026/5/run-metrics.webp) + +**Table-level Metrics:** Fusion shows metrics for each table's current state so you can understand and decide whether it needs compaction. + +![Table Metrics](/img/docs/iceberg-maintenance/metrics/table-metrics.webp) + +The **Tables** page shows an overall **health score** for each table, so you get a first-pass view of whether compaction looks necessary before you dive into detailed metrics. + +![Table Health Score](/img/docs/iceberg-maintenance/metrics/health-score.webp) + +This is the kind of visibility that makes the difference between proactively maintaining your tables and reactively debugging performance issues after users are already complaining. + +To know more about metrics, refer [here](https://olake.io/docs/iceberg-maintenance/metrics). + +## Configure Once, Maintain Continuously + +Fusion connects to your Iceberg catalog. For each table, you configure the compaction schedule — which tiers to enable, and how often each one should run. You can think of it like cron: you define the cadence, Fusion executes it. + +A typical setup depends on your CDC ingestion frequency. For example, if ingestion runs every 2 minutes, you might schedule Lite every 30 minutes, Medium every 6 hours, and Full every 2 days. Fusion handles the execution, the logging, and the metrics. If a run fails, you see it immediately in the runs view without having to dig through your job scheduler's logs or SSH into a Spark driver node. + +If you're already using OLake for CDC ingestion, Fusion integrates naturally — same catalog, same UI. But it also works as a standalone service if you're using a different ingestion tool. + +Refer here for a walkthrough guide: [Getting Started with Fusion](https://olake.io/docs/getting-started/configure-first-compaction) + + +## Summary: Apache Iceberg Table Maintenance with OLake Fusion + +If you're running Iceberg with CDC pipelines, table maintenance isn't optional, it's the difference between a lakehouse that stays fast and one that gradually becomes unusable. The small files problem and delete file accumulation are real, they compound over time, and they're hard to notice until performance is already degraded. + +Spark-based compaction works, but only if you build and run those jobs yourself. They are often slow and expensive, and it can be hard to tell if each run really helped. + +OLake Fusion is built specifically for this. Tiered compaction that matches the level of work to what the table actually needs. 2x faster than Spark. About half the cost. And enough observability to actually understand what's happening to your tables, before your users start asking why queries are slow. + +## FAQs + +### Q1. What is table maintenance in Apache Iceberg? + +Table maintenance in Apache Iceberg is the set of routine operations that keep analytical queries fast and tables storage-efficient as data is written, updated, and deleted. Because every write creates new files and a new snapshot, tables accumulate small files, stale snapshots, orphan files, and bloated metadata over time. + +A typical maintenance workflow includes compaction (`rewrite_data_files`) to merge small files and resolve deletes, snapshot expiration (`expire_snapshots`) to drop old snapshots, orphan file removal (`remove_orphan_files`) to clean up unreferenced files, and manifest rewriting (`rewrite_manifests`) to speed up query planning. + +### Q2. What is Apache Iceberg compaction? +Apache Iceberg compaction rewrites small, fragmented data files into larger optimized Parquet files and resolves accumulated delete files, reducing query planning overhead, merge-on-read costs, and object storage API charges on Iceberg tables. + +### Q3. Why do CDC pipelines create small files in Iceberg? + +CDC pipelines write data continuously, often every few seconds or minutes. Each write +typically creates a new set of data files and delete files. Because writes are frequent +and small, the result over time is thousands of tiny files instead of a smaller number +of optimally sized ones. Iceberg's merge-on-read model means query engines must scan +all of these files at read time, which compounds the performance penalty. + +### Q4. What is Spark rewrite_data_files and what does it do? + +`rewrite_data_files` is an Apache Iceberg table maintenance procedure, typically run +via Apache Spark, that reads existing data files and rewrites them into larger, +better-organized Parquet files. It can also resolve equality delete files into the +data, removing them from the read path. Teams use it to undo the effects of small +file accumulation, but it requires a running Spark cluster, manual configuration, +and custom scheduling. + +### Q5. What is the difference between equality delete files and position delete files? + +Equality delete files record deleted rows by column value (e.g., "delete all rows +where `id = 42`"). They require the query engine to scan most of the data files to apply +the delete, which is expensive at scale. Position delete files are more efficient: they +record the exact file path and row offset of deleted rows. Fusion converts equality deletes into position deletes as an intermediate step before fully resolving them into the data files. + +### Q6. What is binpack compaction in Apache Iceberg? + +Binpack compaction is a strategy that packs existing small files into target-size +bins (e.g., 512MB) without reordering data. It minimizes the number of files while +keeping write amplification low. It's the right choice for frequent, lightweight +compaction runs where you want to reduce file count without the cost of a full sort. + +### Q7. What is sort compaction in Apache Iceberg? + +Sort compaction rewrites data files while also sorting rows by one or more columns +(e.g., a partition key or frequently filtered column). Sorted files allow query +engines to skip entire files using min/max statistics, dramatically reducing I/O +for selective queries. Sort compaction is more expensive than binpack and is best +reserved for periodic deep-maintenance runs. + +### Q8. How often should Iceberg tables be compacted? + +Compaction frequency depends on your ingestion rate. A typical setup for a CDC-heavy table absolutely depends on how frequently your CDC ingetions are scheduled. Given a case, where your CDC ingestion are scheduled every 2 mins, you may schedules your Lite compaction every 30 minutes, Medium every 6 hours and Full every 2 days. + +### Q9. Does Iceberg compaction remove delete files? + +Yes. When compaction rewrites data files, it applies any pending delete files and +incorporates their changes into the rewritten data. After compaction, those delete +files are no longer part of the table's live snapshot and are removed from the read +path. This is one of the primary performance benefits of compaction for CDC workloads. + +### Q10. How does OLake Fusion differ from running Vanilla Spark's `rewrite_data_files`? + +OLake Fusion offers three compaction tiers (Lite, Medium, Full) that you schedule independently per table via cron expressions. It runs approximately 2x faster than Spark's rewrite_data_files on comparable infrastructure and at about 50% of the cost. It also includes built-in metrics, file counts and sizes before and after each run. So you can verify that compaction actually improved your table's state. + + diff --git a/blog/2026-05-13-conflict-free-cdc-into-apache-iceberg.mdx b/blog/2026-05-13-conflict-free-cdc-into-apache-iceberg.mdx new file mode 100644 index 000000000..ddcd5599d --- /dev/null +++ b/blog/2026-05-13-conflict-free-cdc-into-apache-iceberg.mdx @@ -0,0 +1,190 @@ +--- +slug: conflict-free-cdc-into-apache-iceberg +title: "Conflict-Free CDC into Apache Iceberg: Architecting Temporal Memory for Autonomous Agents" +description: "Learn how to build 'Temporal Memory' for Agentic AI. Bypassing the Iceberg Read Amplification Wall using OLake CDC and ClickHouse for sub-second network analytics." +authors: [shuva] +tags: [iceberg, cdc, lakehouse, agentic-ai, google-cloud-lakehouse, mcp] +date: 2026-05-13 +image: /img/blog/cover/conflict-free-cdc.webp +--- + +import BlogCTA from '@site/src/components/BlogCTA'; + + +# Conflict-Free CDC into Apache Iceberg: Architecting Temporal Memory for Autonomous Agents + +The transition from passive dashboards to autonomous Agentic AI introduces a unique architectural challenge to the modern data stack: Data agents are inherently stateless, but threat detection is inherently temporal. +When building an autonomous security framework, agents rely on the Model Context Protocol (MCP) to query the network state. However, when an anomaly—such as a subtle DNS rebinding attack—is detected, querying the current state of the network graph is often insufficient. By the time an agent investigates, the attacker may already have mutated the DNS record or shifted IP addresses +If your Change Data Capture (CDC) pipeline only overwrites historical state in the operational database, the agent lacks the context it needs to reconstruct the attack vector. It needs to be able to ask, "What was the exact topology of this DNS route 5 minutes ago?" +Apache Iceberg’s Time Travel feature serves this purpose. However, running sub-second-latency queries on a fast-streaming Lakehouse is challenging due to metadata bloat and commit contention. + +This guide looks at the mechanics of high-frequency streaming CDC. It describes an architecture that uses OLake's thread-coordinated ingestion and background compaction, OLake Fusion. This method transforms a Lakehouse into a low-latency temporal memory layer for autonomous threat detection, threat pattern prediction, correlation and monitoring. + +## Architectural Primitives: Iceberg Time Travel + +To understand why high-velocity CDC breaks traditional ingestion, we must first establish how Apache Iceberg facilitates Time Travel. +Iceberg does not manage data at the folder or directory level; it is a metadata-first architecture. Every write operation generates a linear chain of immutable temporal snapshots. Think of it as an un-erasable version history—a continuous timeline of permanent photographs capturing exactly what the network looked like at any given millisecond. The state of the table is defined through a strict metadata tree: +metadata.json: The root pointer that tracks the current state and all historical snapshots. +Manifest Lists (snap-*.avro): A file listing the manifests included in a specific snapshot. +Manifests (*.avro): Files that track the exact paths, row counts, and column-level statistics (min/max bounds) of the underlying Parquet data files. +When an autonomous agent runs a Time Travel query with FOR SYSTEM_TIME AS OF, the execution engine does not look through the raw data. It only opens metadata.json, finds the snapshot ID that was active at that millisecond, and uses the manifest statistics to quickly remove irrelevant files. + +This metadata structure enables sub-second time-based reasoning. However, keeping this structure clean during the intense demands of real-time streaming is where traditional systems fail. + +![The Mechanics of Iceberg Time Travel](/img/blog/2026/6/mechanics-of-iceberg-timetravel.webp) + +Fig.1 The Mechanics of Iceberg Time Travel. This diagram illustrates how the query engine (ClickHouse or BigQuery) executes a temporal query without scanning the entire data lake. + +## Defining the Streaming Bottleneck +Before detailing the solution, it is necessary to establish the data schema and understand why high-frequency streaming CDC strains traditional query engines. + +### 1. Operational State: The Network Edge Schema +Consider a high-velocity operational database that helps users monitor live network edges. Every device/client/interface connection, disconnection, or resolution triggers an INSERT, UPDATE, or DELETE. + +```sql +CREATE TABLE network_edges ( +edge_id UUID PRIMARY KEY, +source_ip VARCHAR(45), +resolved_ip VARCHAR(45), +target_port INT, +connection_state VARCHAR(20), -- e.g., 'ESTABLISHED', 'DROPPED', 'MUTATED' +updated_at TIMESTAMP +) +PARTITIONED BY (days(updated_at)); +``` +As threat vectors evolve, this schema will also change. Since Apache Iceberg tracks columns by fixed IDs instead of names, OLake can easily pass upstream PostgreSQL ALTER TABLE events, such as adding a tls_cipher_suite column, directly to the Lakehouse. This process does not need pipeline downtime or rewriting old Parquet files. + +### 2. The Direct-to-Storage Ingestion Path + +To use the Agentic Firewall, data is ingested using native CDC. While massive enterprise architectures often utilise an event bus like Kafka to fan out telemetry to multiple distinct consumers, OLake’s architecture allows us to bypass this intermediary on the Lakehouse path. By removing the message bus, we strip out a layer of latency and operational overhead: + +* **Data Source:** PostgreSQL logical replication slot. + +* **Ingestion Engine:** OLake reads directly from the Postgres replication slot and writes directly to an Apache Iceberg table on Google Cloud Storage (GCS), managing the upserts and state natively. + +### 3. Managing Read Amplification: The Physics of Merge-On-Read +In a high-throughput environment (e.g., enterprise /data center networks handling 10,000 state changes per second), we collide with the fundamental limits of object storage and columnar formats. Traditional data lake architectures default to Copy-On-Write (COW), which fails catastrophically under continuous mutation. + +#### The Failure of Copy-On-Write (COW): +Think of COW like rewriting an entire 500-page textbook just to fix a single typo. Under the hood, a single 1KB row mutation forces the execution engine to deserialise an entire 500MB Parquet block, apply the state change in memory, re-encode the columnar structures, and execute a massive PUT request back to Google Cloud Storage. At 10,000 updates a second, this creates a catastrophic Write Amplification Factor (WAF). It saturates network bandwidth, exhausts cloud API quotas, and instantly grinds the ingestion pipeline to a halt. + +#### The Shift to Merge-On-Read (MOR): +To survive this streaming firehose and meet strict ingestion SLAs, the Iceberg table must be configured for Merge-On-Read. Think of MOR like slapping a sticky note with a correction onto the textbook's cover. MOR shifts the compute penalty from write time to read time. During a micro-batch commit, OLake doesn't touch the massive base files. Instead, it converts random database updates into pure sequential APPEND operations—writing new data files (INSERTs) alongside small Equality Delete files (the sticky notes, e.g., "Consider any row where source_ip = '10.0.4.55' as deleted"). + +#### The Read Amplification Wall: +While appending "sticky notes" takes microseconds and saves the write path, it pushes a massive computational burden down to the query execution layer. When a vectorized engine (like BigQuery or chDB or DuckDB) queries the lake, it must broadcast all active Equality Deletes across its worker nodes and maintain them in a RAM-resident hash set. During the scan, the engine must perform a dynamic anti-join against the base data on the fly. As delete files accumulate, evaluating these row-level predicates destroys CPU cache locality and breaks SIMD vectorization efficiency. Left unmanaged, a temporal query SLA that should be 50ms spikes to 15 seconds as the engine exhausts its memory footprint, filtering obsolete rows. This creates the exact read amplification wall that OLake's architecture is engineered to bypass. + +### Architecting the Solution: The Mechanics of an OLake Commit + +![End to End Agentic Data Architecture](/img/blog/2026/6/agentic-data-architecture.webp) + +**Figure 2. The end-to-end Agentic Data Architecture.** Notice how the ingestion tier bypasses traditional message brokers like Kafka to minimise latency. On the left, OLake worker threads ingest CDC data directly from the PostgreSQL replication slot, using internal RAM buffers to fold noisy events before writing Parquet files directly to Google Cloud Storage. While OLake Fusion quietly compacts these files in the background, the Master Process handles the lightweight Iceberg catalog commits. On the right, this clean metadata allows the Vertex AI Control Plane to execute sub-second temporal reasoning via a vectorised execution engine. + +To enable sub-second Time Travel, the ingestion layer must rigorously manage metadata hygiene. OLake achieves this through a highly concurrent, conflict-free architecture. + +#### Phase 1: Lock-Free I/O via Logical Decoding + +Before tailing the WAL for real-time deltas, OLake natively executes a lock-free historical snapshot of the Postgres table to establish the baseline state in Iceberg. Once the baseline is synced, it transitions to Logical Decoding for streaming updates. +During this streaming phase, when OLake processes a batch of UPDATE events, it does not check the Iceberg Catalog to find existing data. Because the Postgres Write-Ahead Log (WAL) records not only the new state but also streams a complete change event containing the pre-update primary key (edge_id), OLake holds the exact unique identifier needed to generate the Equality Delete condition entirely in RAM. +It writes the new state as data-*.parquet files and simultaneously writes the eq-delete-*.parquet files without ever polling the Iceberg catalog. Unlike traditional ETL, which maintains a local lookup table to find record locations, OLake uses the pre-update primary key from the WAL, allowing it to issue a "delete" command blindly. This is what makes it truly horizontally scalable. Workers do not need to coordinate or share state to know what to delete. By completely bypassing the "read-before-write" penalty (the computational tax of locating and loading existing data into memory before applying an update), OLake maintains a decentralized, lock-free I/O phase. + +#### Phase 2: Hybrid Execution & GC Mitigation (Arrow/Java) + +OLake achieves high-throughput ingestion through a pragmatic hybrid execution model, offering two modes for Iceberg writes: + +* **Arrow-Based Mode:** OLake generates Parquet files in Go using Apache Arrow, then submits them to iceberg-java to register them into the table. The manifest and min/max stats are generated efficiently at the Java level. + +* **Java Insert Mode:** OLake pulls data in Golang and submits the rows via a gRPC call to iceberg-java, which builds the Parquet files and the full ingestion, managing conflict-free schema evolution. +While the Arrow-based mode already minimizes Garbage Collection (GC) overhead by delegating Parquet generation to Go, a fully Go-based structure is on the roadmap as iceberg-go matures. + +#### Phase 3: Thread Coordination and the Master Process Lock + +In standard distributed ingestion pipelines, multiple workers often attempt to commit to the catalog simultaneously, resulting in a CommitFailedException (and the "thundering herd" problem). +OLake avoids this catalog contention entirely by utilizing a central Master Process. Because OLake streams directly from the Postgres replication slot, it pulls these high-velocity changes into an internal RAM buffer to form a micro-batch. No matter how many edge changes happen, OLake keeps buffering in memory till the point it reaches a significant file size, so preventing small file problems. +Crucially, the heavy I/O operations—generating the Parquet files and flushing them to Google Cloud Storage or S3—are executed entirely in parallel by the worker threads without any locking. It is only after these parallel flushes succeed that the Master Process takes a lightning-fast internal lock (under 200ms) solely to perform the atomic metadata swap (snapshot) in the Iceberg catalog. This decoupled design sequences the commit cleanly while keeping the data generation path completely unblocked. By coordinating the threads internally, OLake eliminates the CommitFailedException loop and maintains maximum ingestion throughput. + + +![Resolving Catalog Contention](/img/blog/2026/6/resolving-catalog-contention.webp) + +**Figure 3.Resolving catalog contention through Master Process thread coordination.** Instead of multiple worker threads crashing into the Iceberg catalog and causing a "thundering herd" problem, OLake centralizes the commit phase. As shown, individual worker threads process the PostgreSQL WAL stream concurrently, writing the heavy physical Parquet files directly to Cloud Storage. However, the catalog update is strictly sequenced: the Master Process acquires a lightning-fast internal lock (under 200ms) to perform the atomic swap in the Iceberg Catalog, ensuring conflict-free ingestion even under massive throughput. + +#### Phase 4: Repaying Compaction Debt via OLake Fusion + +To address the read amplification caused by Equality Deletes, OLake Fusion runs asynchronously in the background to sanitise the storage layer. It is important to note the architectural boundary here: OLake manages the compaction, ensuring the data is optimally structured for the downstream execution engines (like Trino, DuckDB, or BigQuery) that power the agentic reasoning tier. + +**Lite Compaction (Logical to Physical):** +Resolving broad Equality Deletes on the fly requires the query engine to execute expensive dynamic anti-joins in memory. Fusion converts these logical rules into specific Position Deletes, specifying the exact file paths and row offsets to skip for the query engine. Additionally, it repairs fragmentation by rolling small micro-batches into larger segments (typically close to 1/8th of the target 256MB or 512MB file size). + +**Medium Compaction (The Physical Purge):** +A separate bin-packing process reduces these segments to nearly the final target file size. It does not guarantee that after compaction data file will be of target file size.. Crucially, it physically removes the deleted rows from the Parquet blocks entirely. This restores contiguous memory layouts, allowing the query engine to scan only active data and maintain its SIMD vectorization efficiency. + +**Full Compaction (Global Reorganization):** +While Medium compaction handles localized cleanup, Full Compaction is the heavyweight background process that completely resets a partition's technical debt. Rather than relying on advanced metadata-level pruning or complex clustering, OLake focuses on raw structural efficiency. It comprehensively rewrites all base data and delete files into perfectly sized, pristine Parquet blocks. This eliminates all read-time reconciliation overhead, allowing the downstream execution engines to query pure, unfragmented data. + +Architectural Warning: Merge-On-Read acts like a “computational” loan, and OLake Fusion is the repayment. If the background compaction tier lacks resources and can't keep pace with data ingestion, "delete file bloat" will accumulate. As these uncompacted files grow, the read amplification problem will resurface, leading to Time Travel queries missing their sub-second service-level agreements (SLAs). SRE teams must closely monitor the compaction queue depth to ensure the reasoning tier remains efficient. + +### The Reasoning Tier: Executing Temporal Queries via MCP + +With perfect temporal hygiene maintained by OLake, this historical timeline can be exposed to an autonomous agent. +The AI agent is equipped with an MCP tool. Upon detecting a suspicious event, it invokes the tool, passing the suspected source_ip and the exact timestamp of the anomaly. +```python +# MCP Tool Schema +{ +"name": "query_historical_network_state", +"description": "Queries the exact state of a network node at a specific millisecond in the past.", +"parameters": { +"type": "object", +"properties": { +"source_ip": { "type": "string" }, +"target_timestamp": { "type": "string" } +}, +"required": ["source_ip", "target_timestamp"] +} +} +``` +Behind the MCP server, the Time Travel query runs on a vectorised engine. The type of engine used depends on the specific Agentic SLA. For embedded, serverless agents needing microsecond-level checks, a closely linked engine, such as chDB or DuckDB works best. On the other hand, for large-scale, cross-cloud incident tracing and enterprise-level reasoning, the system can easily shift to Google BigQuery as the control plane engine. It runs the same queries on the same Iceberg tables. + +Vectorised engines process data in blocks of columns using CPU SIMD instructions, enabling them to scan massive datasets with sub-second latency. +When the AI swarm decides to investigate, it outputs the JSON parameters. The MCP Server intercepts this request, compiles it into a parameterised SQL Time Travel query, and pushes it down to the execution engine. +The Vectorized Execution Path: +```sql +-- Map the Iceberg table using native GCS integration +CREATE TABLE security_lake_edges +ENGINE = Iceberg('gcs://storage.googleapis.com/security-data-lake/network_edges', 'access_key', 'secret_key'); +-- Execute the Agentic Time Travel Query +SELECT +resolved_ip, +connection_state +FROM security_lake_edges +FOR SYSTEM_TIME AS OF '2026-05-02 01:15:00.000' +WHERE source_ip = '10.0.4.55'; +``` +* **Catalog Resolution:** The execution engine contacts the Iceberg Catalog (e.g., Google Cloud Lakehouse) to resolve or determine the FOR SYSTEM_TIME AS OF timestamp into the correct metadata.json and active manifest. +* **Direct Parquet Reads:** The execution layer retrieves the specific byte ranges of the remaining Parquet files natively from GCS. Leveraging the Position Deletes generated by OLake Fusion, the engine bypasses obsolete rows with minimal compute overhead. + +### Architectural Tradeoffs: Tuning the Temporal Context Window + +Implementing Iceberg Time Travel requires the users to balance cloud storage costs with the necessary Agentic Context Window. +Aggressively expiring snapshots, such as removing history older than 2/6 hours, improves lakehouse performance. However, this also destroys the agent's temporal memory. If a security swarm needs to trace an Advanced Persistent Threat over 72 hours, an aggressive expiration policy will lead to a failed query. +To support Agentic workflows, we would need to adjust the following table properties to meet the required SLA: +```sql +ALTER TABLE iceberg.security_lake.network_edges SET TBLPROPERTIES ( +'history.expire.max-snapshot-age-ms'='604800000', -- Retain history for 7 days +'history.expire.min-snapshots-to-keep'='1000' +); +``` +While retaining historical Parquet files increases storage costs, it guarantees the availability of a highly indexed, multi-day temporal memory layer. + +### Looking Ahead: The Decoupled Agentic Data Stack + +The move toward autonomous LLM-driven agents calls for a fresh look at data engineering principles. Modern AI agents need to operate at inference speed. They cannot handle the metadata bloat or compaction lag that comes with legacy ingestion pipelines. +This requirement aligns perfectly with the architectural evolution of the Google Cloud Lakehouse and the Google Cloud Next 2026 announcements. By standardising on Apache Iceberg as a zero-copy format, enterprise architecture is entirely decoupling storage from compute. In this model, Google + +Cloud Storage (GCS) provides the scalable substrate, BigQuery acts as the massively parallel execution engine to crunch the historical metadata, and Vertex AI serves as the autonomous control plane reasoning over the results. +However, this sophisticated cross-cloud Agentic vision is physically impossible without the right ingestion plumbing. OLake serves as critical infrastructure in this stack. By utilizing Master Process thread coordination for conflict-free snapshot generation and Fusion for asynchronous compaction, OLake shields the downstream query engines from the complexities of streaming CDC. + +Stateless agents are important for heuristic blocking, whereas historical threat analysis and incident tracking require agents with stateful temporal memory. By combining OLake's accurate CDC with Apache Iceberg's built-in Time Travel, data teams will help transform object storage into a solid foundation for the next generation of autonomous enterprise defense. + +#### Disclaimer + +Opinions expressed are my own in my personal capacity and do not represent the views, policies or positions of my current and/ ex ,or their subsidiaries or affiliates diff --git a/blog/2026-06-12-apache-iceberg-row-lineage.mdx b/blog/2026-06-12-apache-iceberg-row-lineage.mdx new file mode 100644 index 000000000..ece1e17a8 --- /dev/null +++ b/blog/2026-06-12-apache-iceberg-row-lineage.mdx @@ -0,0 +1,311 @@ +--- +slug: apache-iceberg-row-lineage +title: "Apache Iceberg Row Lineage: Tracking Data Lineage at the Row-Level" +description: "How Apache Iceberg v3 row lineage tracks row-level changes for CDC, with a tested look at _row_id preservation across Spark 3.5 and Iceberg 1.9 vs 1.10." +tags: [iceberg, row-lineage, cdc, spark, olake, v3] +authors: [anshika] +image: /img/blog/cover/row_lineage_cover.webp +--- +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import BlogCTA from '@site/src/components/BlogCTA'; + +![Architect's Guide to CDC with Apache Iceberg](/img/blog/cover/row_lineage_cover.webp) + +If you've ever needed to figure out exactly which rows changed in a data lake table in the last hour, you already know the options aren't great. Snapshots and time travel can show you the table at two points; however, comparing them gives you file-level differences, not row-level ones. A row that got updated looks like a brand new row in any file-based difference. You can fall back on an `updated_at` column, but that only works if every upstream job remembers to set it. Or you can stand up a separate CDC pipeline, which then becomes a thing you have to maintain. + +Apache Iceberg row lineage fixes this in the table format itself. In v3, each row carries a stable ID and a marker for when it was last touched, so to pull changes you write a `WHERE` clause. No sidecar log, no mirror table, no timestamp guessing. + +The rest of this post is about how that actually works, why the design ended up the way it did, and the parts that still bite you in practice. + +## What is Row Lineage in Apache Iceberg? +Quick refresher in case you need it: Iceberg is a table format for data lakes, and every commit to an Iceberg table produces a new snapshot with a sequence number that goes up by one. Before v3, that sequence number was the finest grain of tracking you got. You knew something changed in snapshot 47, but not which rows. v3 pushes the tracking down to the row itself. Every row now knows its own identity and the last commit that touched it + +**Row lineage** in Iceberg consists of two special metadata fields that are automatically added to each row in a v3 table. The first is `_row_id`, a unique identifier for the row within the table that stays static across the row's lifetime. Each row is assigned a unique `_row_id` when it is first created. + +### Why use _row_id instead of a primary key? + +Because large analytic tables often have no reliable natural key. Iceberg assigns a system-generated `_row_id` instead of relying on a primary key because it needs a consistent identifier that stays with a row across changes like updates and deletes. A primary key depends on user-defined fields, and large analytic tables in a data lake often have no unique or natural key to use. A system-generated `_row_id` avoids that problem. + +By design, `_row_id` stays stable through copy-on-write updates, merge-on-read updates, and compaction. Getting that behavior depends on the Iceberg version doing the write: for the Spark 3.5 integration, update-time lineage support landed in Iceberg 1.10.0 (PR #12736, covering UPDATE and MERGE). On 1.10 or later, an update keeps the original `_row_id` and only advances `_last_updated_sequence_number`. The experiment below shows this on Spark 3.5 with Iceberg 1.10. + +```sql +CREATE TABLE lineage_test (id BIGINT, name STRING, qty INT) +USING iceberg +TBLPROPERTIES ('format-version' = '3'); + +INSERT INTO lineage_test VALUES (1, 'Widget', 100); + +-- Case 1: copy-on-write update +ALTER TABLE lineage_test SET TBLPROPERTIES ('write.update.mode' = 'copy-on-write'); +UPDATE lineage_test SET qty = 200 WHERE id = 1; + +-- Case 2: merge-on-read update (uses deletion vectors in v3) +ALTER TABLE lineage_test SET TBLPROPERTIES ('write.update.mode' = 'merge-on-read'); +UPDATE lineage_test SET qty = 300 WHERE id = 1; + +-- Case 3: explicit delete then insert +DELETE FROM lineage_test WHERE id = 1; +INSERT INTO lineage_test VALUES (1, 'Widget', 400); +``` +| Snapshot | Operation | qty | `_row_id` | `_last_updated_sequence_number` | Identity preserved? | +| --- | --- | --- | --- | --- | --- | +| 1 | Initial insert | 100 | 0 | 1 | — | +| 2 | Update (copy-on-write) | 200 | 0 | 2 | Yes | +| 3 | Update (merge-on-read) | 300 | 0 | 3 | Yes | +| 4 | Delete | — | (gone) | — | Row removed | +| 5 | Insert new row | 400 | 3 | 5 | No, new row | + +Both updates keep `_row_id = 0` while the sequence number advances, so the row's identity is preserved across the change. The new row inserted in snapshot 5 is a genuinely different row and gets a fresh ID. + +### How does Iceberg figure out the _row_id for any row, especially during updates? + +When a row is first written to the table, Iceberg assigns a unique `_row_id` based on a globally increasing identifier. The `_row_id` is assigned using the row ID of the first row from the snapshot (or data file), plus its position within the file. This ensures every row is assigned a unique and stable ID. + +For updates, the existing `_row_id` is meant to be carried over. When a row is modified through copy-on-write or merge-on-read, an engine that implements lineage writes keeps the same `_row_id`, preserving the row's identity, while only `_last_updated_sequence_number` advances. This means that even if the row's data changes, its identity (`_row_id`) remains the same across all updates, making it possible to track the full history of the row. + +The second field, `_last_updated_sequence_number`, is the sequence number of the snapshot (commit) in which the row was last modified. For a newly inserted row, this will be the sequence number of the insert commit; if the row is updated later, this field is updated to the newer sequence number of the update commit. + +These lineage fields are stored as hidden metadata columns and do not require any changes to your table schema or application code. As long as a table is using format version 3, Iceberg will maintain `_row_id` and `_last_updated_sequence_number` for you automatically. Engines like Spark, Trino, and Flink expose these fields either directly or via system column syntax (for example, Trino uses `$row_id` and `$last_updated_sequence_number` in queries, whereas Spark and Flink refer to them as normal columns `_row_id` and `_last_updated_sequence_number`). The lineage columns enable queries to reason about row versioning and history directly from the data, without custom tracking mechanisms. + +Before row lineage existed, Iceberg could tell you only the overall changes between snapshots (via snapshot diffs or change logs), but it was difficult to pinpoint individual row modifications or reconstruct a detailed change history for a row. With row-level metadata on each row, you can now query an Iceberg table and filter by these lineage fields to get all incremental changes with full fidelity, making it much easier to implement auditing and CDC use cases. In other words, row lineage elevates Iceberg from just tracking files and snapshots to also tracking individual rows through time. + +![Row ID traching sequence](/img/blog/2026/11/row_id_sequence_tracking.webp) + +## How Row Lineage Works Under the Hood + + +The tricky part of row lineage is making it work in a distributed setting. Multiple writers may be inserting concurrently. Files get rewritten during compaction. A row's ID has to stay the same through all of that, and no two rows can ever collide. Iceberg pulls this off by assigning IDs lazily at commit time rather than at write time, which means a retried or failed commit doesn't poison the ID space. Here's how it works at each level. + +**Table-level row ID counter.** Each Iceberg table in v3 maintains a next-row-id counter in its metadata. This counter tracks the next available `_row_id` that can be assigned. When a new snapshot (commit) is about to be created, the table's current next-row-id is used as the starting point for row IDs in that commit. + +**Snapshot commit allocation.** When a writer commits a new snapshot, the snapshot metadata records a first-row-id for that snapshot equal to the table's next-row-id at commit time. All new rows added in this snapshot will have IDs starting from this value. The next-row-id counter is then incremented (during the atomic commit) by the number of new row IDs assigned, ensuring no two snapshots ever reuse the same ID range. + +**Manifest and file ranges.** Within the snapshot, new data is often split across multiple manifest files (which in turn reference the actual data files). Iceberg assigns each new manifest a contiguous range of row IDs. The first new manifest in a commit gets the snapshot's first-row-id; subsequent manifests get starting IDs that are offset by the number of new rows accounted for in previous manifests. This chained assignment guarantees that every manifest covers a unique segment of the ID space, preserving a global ordering of row IDs even if data is partitioned into separate files. + +**Deferred assignment in data files.** The sequence looks like this when a row makes it to disk: +1. The data file is written with `first_row_id = null`. The writer doesn't know its assigned range yet. +2. At commit time, the manifest list is written, and each manifest gets a starting row ID based on its position in the commit. +3. At read time, any file still missing a `first_row_id` inherits one from its manifest, and per-row IDs are computed as `manifest_first_row_id + file_offset + row_position`. +The benefit of this is that data files never need to be rewritten just to assign IDs. If a commit retries, only the manifest list has to change. + +**Row updates and moves.** What happens when a row is updated or migrated to a new file (e.g. due to compaction or clustering)? Iceberg's lineage design dictates that if an engine chooses to model the update as a true update (rather than a delete+insert), it should carry over the existing `_row_id` to the new file. When the updated row is written, its `_last_updated_sequence_number` is left unassigned in the data file and is resolved via inheritance at read time to the sequence number of the commit that wrote it. This is the same inheritance mechanism Iceberg uses for newly inserted rows, and it's why the commit sequence number doesn't need to be known until the snapshot is successfully committed. The `_row_id` remains the same, preserving the row's identity, but its `_last_updated_sequence_number` reflects the latest commit. This allows the system to recognize it as the same logical row that has changed in a new snapshot. + +:::note +If an engine uses delete + insert (an equality model) to implement updates, which Iceberg also supports, the new row will get a new `_row_id` because the original row was deleted without explicitly preserving the ID. In this case, the change will appear as a deletion of one ID and insertion of a different ID. +::: + + +### How to avoid this in practice + +The good news is that most of the time, you don't have to think about this. If you're using normal SQL on a recent engine, things just work. The trickier cases are streaming pipelines and homegrown CDC code. A few things to keep in mind. + +- **Stick with SQL UPDATE and MERGE.** When you write `UPDATE` or `MERGE INTO ... WHEN MATCHED THEN UPDATE`, you're telling the engine "this row is the same row, just with new values." That's the signal that lets it preserve the `_row_id`. The problem starts when application code does its own `DELETE` followed by `INSERT`. At that point, as far as the table knows, those are two completely separate operations on two different rows. + +- **Watch the update mode.** Set `write.update.mode` to copy-on-write or merge-on-read explicitly. On v3 tables, merge-on-read uses deletion vectors by default. Deletion vectors are the delete type the spec preserves lineage through, but whether the original `_row_id` is actually carried forward depends on your engine's Iceberg version (for Spark 3.5, that means 1.10 or later). Equality deletes are the failure mode here, and they mostly show up in streaming writers where the engine can't cheaply look up the original row's position. + +- **Be careful with streaming ingest.** Flink, Kafka Connect, and similar tools sometimes default to equality-delete semantics for performance reasons. They don't want to do a position lookup on every row. If you're streaming CDC from Postgres or MySQL into Iceberg, check your writer's config before assuming `_row_id` survives. There's a real tradeoff here: equality deletes are cheaper to write; position deletes preserve identity. Pick based on whether downstream consumers care about row continuity. + +- **Test your own write path.** This is what bit us on an earlier run: on Spark 3.5 with Iceberg 1.9.0, the same merge-on-read UPDATE that succeeds above didn't preserve _row_id. The spec says it should; the Spark 3.5 implementation didn't catch up until Iceberg 1.10. The only way to know for sure with your stack is to insert a row, update it, and check whether the ID survives. It's a ten-second test that prevents a much bigger problem downstream. + +Row lineage works most effectively when updates are applied in a way that preserves row IDs, similar to how a database would perform an update. + +![Row ID Assignment Flow](/img/blog/2026/11/row_id_assignment_flow.webp) + +By deferring and chaining the assignment of row IDs in this manner, Iceberg ensures global uniqueness and consistency of `_row_id` across the entire table. Even if data is partitioned or written in parallel, no two rows will ever share the same ID, and the IDs roughly track commit order, which isn't quite the same as insert order. Say two writers start at the same time. Whichever one commits first gets the lower ID range, regardless of which one started writing rows first. And even inside a single commit, the per-row ordering depends on how the engine partitions and shuffles data internally. Two rows from the same `INSERT INTO ... VALUES (a), (b)` can land with their `_row_id` values in either order. The `_last_updated_sequence_number` is always set to the commit that last touched the row, which means it will inherit the latest sequence number on any insert or true update (one that modifies the existing row in place rather than treating the change as a delete followed by an insert). Together, these two fields give each row a stable identity and a version history pointer. + +## Benefits and Use Cases of Row Lineage + +- **Efficient incremental data processing.** Perhaps the biggest benefit is making incremental change queries trivial. Rather than diffing entire snapshots or scanning for timestamp differences, an engine can simply query for all rows with `_last_updated_sequence_number` greater than X to get the changes since snapshot X. Because `_last_updated_sequence_number` is stored as metadata, the query engine can often prune files or use metadata-only reads to find these rows, instead of reading the whole dataset. This dramatically speeds up change data capture (CDC) pipelines, where downstream systems only need the new or changed rows since the last checkpoint. As Databricks describes it in their Iceberg v3 announcement, row lineage lets engines find row-level changes by matching versions of rows across commits, so they can process changes selectively and make downstream updates faster and cheaper. ([Databricks: Apache Iceberg v3: Moving the Ecosystem Towards Unification](https://www.databricks.com/blog/apache-icebergtm-v3-moving-ecosystem-towards-unification)) + +- **Fine-grained auditing and compliance.** Row lineage provides a built-in audit trail for data modifications. Because each row carries the sequence number of its last update, you can retrieve a full history of changes by looking at past snapshots and matching the row IDs. For compliance scenarios, you can answer questions like “When was this record last changed and where did it come from?” much more easily. The ([AWS Big Data Blog](https://aws.amazon.com/blogs/big-data/accelerate-data-lake-operations-with-apache-iceberg-v3-deletion-vectors-and-row-lineage/)) notes that lineage gives full fidelity change history for audit and governance – previously, Iceberg’s snapshot diff would only show the net result, but now you can trace every insert/update to each row. And since the IDs are stable, you can even track if a row was deleted: if a previously existing _row_id is no longer present in the latest snapshot, that indicates the row was removed in some commit (though identifying deletions may require comparing snapshots or using Iceberg’s delete files). + +- **Stable keys for downstream systems.** Downstream analytics systems or applications can rely on Iceberg's `_row_id` as a stable primary key for the data. If the same row appears in multiple snapshots (with the same ID), it's the same logical entity evolving over time. If a new row appears with a new ID, it truly represents a new entity. This stability was not available in append-only logs or file-based diffs, where an update might look like a brand new record. With row lineage, it is possible to, for example, ingest changes from Iceberg into an operational database or search index and use `_row_id` as the key to perform upserts, ensuring you update existing entries rather than duplicating them. + +## Querying Changes with Row Lineage: Examples + +One of the best ways to appreciate row lineage is to see it in action with some queries. Let's walk through a simple example using SQL (this could be Spark SQL, Trino, and so on, as long as the engine supports Iceberg v3). + +Suppose we have an Iceberg v3 table named `product_data` that has row lineage enabled. After performing some transactions (inserts, updates, deletes), we want to retrieve all the changes that occurred after a certain point. We can do this by filtering on `_last_updated_sequence_number`. Here is an example depicting row lineage in Iceberg v3: + +```sql +-- Create table with Iceberg v3 + row lineage enabled +CREATE TABLE product_data ( + product_id BIGINT, + name STRING, + quantity INT +) +USING iceberg +TBLPROPERTIES ('format-version' = '3'); + +-- Snapshot 1: insert 4 products +INSERT INTO product_data (product_id, name, quantity) VALUES + (1, 'Thermal Bottle', 123), + (2, 'Desk Mat', 345), + (3, 'USB-C Hub', 567), + (4, 'Notebook', 869); + +-- Snapshot 2: update one row (product_id = 2) +UPDATE product_data +SET name = 'Desk Mat (Revised)' +WHERE product_id = 2; + +-- Snapshot 3: delete one row (delete product_id = 4) +DELETE FROM product_data +WHERE product_id = 4; + +-- Snapshot 4: insert a new row +INSERT INTO product_data (product_id, name, quantity) VALUES + (5, 'Wireless Mouse', 979); + +-- Query: rows inserted/updated after sequence number 0 +SELECT + product_id, name, quantity, + _row_id, _last_updated_sequence_number +FROM product_data +WHERE _last_updated_sequence_number > 0 +ORDER BY _last_updated_sequence_number, _row_id; +``` +In a fresh table where we inserted some rows and then made a couple of updates/deletes, the result might look like this: + +- Rows that were inserted in the first snapshot will appear with `_last_updated_sequence_number = 1`. (If we filtered with `_last_updated_sequence_number > 0`, we'd get those initial inserts as "changes since snapshot 0". In practice, you might filter using a higher number once you have a baseline.) +- If a row was updated in a later snapshot, that row will appear with the higher sequence number. In our example above, the row with `product_id = 2` was updated, and indeed its `_last_updated_sequence_number` became 3 (assuming the update happened in the third snapshot). The `_row_id` for that product remains the same as when it was first inserted, allowing us to correlate it back to the original insert. +- Any row that was deleted will simply not show up in the current snapshot results. In the example, we deleted `product_id = 4`, so that row is absent from the query output. We see the remaining rows that are either inserted or updated. A deleted row's last-known `_row_id` and data would still exist in older snapshots if we needed to audit it via time travel. + +The above query demonstrates an incremental pull of changes. In a production scenario, you might remember the last sequence number you processed (say, 100) and then run a similar query with `WHERE _last_updated_sequence_number > 100` on the next run to get all new changes. Because this is built into Iceberg, no custom CDC mirror tables or extra log files are needed; you are simply querying the table itself. The Iceberg metadata (manifest files) can often satisfy the predicate on `_last_updated_sequence_number` without full table scans, making this very efficient. + +![Sequence Number Filter](/img/blog/2026/11/sequence_number_filter_cdc.webp) + +It also lets you follow a single row across snapshots, which is the part pure snapshot-diff can't really do. Snapshot history has always been there in Iceberg, but it's organized by commit, not by row. If you diff snapshot 5 against snapshot 10 and a row is gone, you know something changed. You just don't know whether it was the same logical row the whole time, or whether it got deleted and reinserted with the same business key somewhere in between, or whether it was updated three times along the way. To answer that with snapshot-diff alone, you'd have to walk every intermediate snapshot and match rows by business keys yourself, hoping nothing collided. + +With `_row_id`, that whole reconstruction goes away. A row inserted in snapshot 5, updated in snapshot 7, and deleted in snapshot 10 leaves a clean trail: the same `_row_id` shows up with increasing `_last_updated_sequence_number` values from snapshot 5 through 9, then disappears in snapshot 10. You can query that history directly instead of stitching it together. + +A single query against the live table gives you the row's current identity and the last time it changed, not its full trail: + +```sql +SELECT _row_id, _last_updated_sequence_number +FROM lineage_test +WHERE _row_id = 0; +``` +To see every snapshot the row changed in, read the table at each snapshot and watch `_last_updated_sequence_number` advance. First pull the snapshot IDs from the metadata table (these are assigned per commit, so the values on your machine will differ from any shown here): + +```sql +SELECT snapshot_id, committed_at +FROM lineage_test.snapshots +ORDER BY committed_at; +``` +Then plug those IDs into a time-travel query, one branch per snapshot. The snapshot IDs below are from our run; substitute your own from the query above (yours will differ, since they're assigned per commit): + + + +```sql +SELECT 1 AS snapshot, _row_id, qty, _last_updated_sequence_number AS last_seq +FROM lineage_test VERSION AS OF 2005109397149177061 WHERE _row_id = 0 +UNION ALL +SELECT 2, _row_id, qty, _last_updated_sequence_number +FROM lineage_test VERSION AS OF 3517781618417195484 WHERE _row_id = 0 +UNION ALL +SELECT 3, _row_id, qty, _last_updated_sequence_number +FROM lineage_test VERSION AS OF 8726901000894316292 WHERE _row_id = 0 +UNION ALL +SELECT 4, _row_id, qty, _last_updated_sequence_number +FROM lineage_test VERSION AS OF 1047375627323916752 WHERE _row_id = 0 +ORDER BY snapshot; +``` + + + +```python +# Reads the snapshot IDs at runtime, so there is nothing to substitute by hand. +# Needs Spark 3.5 with Iceberg 1.10. +snapshots = spark.sql( + "SELECT snapshot_id, committed_at FROM lineage_test.snapshots ORDER BY committed_at" +).collect() + +print(f"{'snapshot':<10}{'_row_id':<9}{'qty':<6}{'last_seq':<9}") +for i, s in enumerate(snapshots, start=1): + rows = spark.sql( + f"SELECT qty, _row_id AS rid, _last_updated_sequence_number AS seq " + f"FROM lineage_test VERSION AS OF {s.snapshot_id} WHERE _row_id = 0" + ).collect() + if rows: + r = rows[0] + print(f"{i:<10}{str(r.rid):<9}{str(r.qty):<6}{str(r.seq):<9}") + else: + print(f"{i:<10}{'(row gone)':<9}{'-':<6}{'-':<9}") +``` + + + + +| snapshot | `_row_id` | qty | last_seq | +| --- | --- | --- | --- | +| 1 | 0 | 100 | 1 | +| 2 | 0 | 200 | 2 | +| 3 | 0 | 300 | 3 | +| 4 | (row gone) | — | — | + +The `_row_id` stays 0 the whole way down, which is what lets you filter on it instead of a business key. `last_seq` advances on every change, and the row drops out at the delete, so the snapshots where `last_seq` moves are the ones the row changed in. Because the filter is the stable `_row_id` rather than `id`, this still works even if a business key gets reused later. + +:::note +This guarantee applies when the insert and delete occur in separate commits. If an insert and delete happen within the same snapshot (i.e. as part of a single atomic commit), Iceberg does not expose the intermediate state since snapshots represent the final committed view of the table. In such cases, the row's transient existence is not observable. +::: + +## What Row Lineage Doesn't Solve + +Row lineage is genuinely useful, but it's not magic. A few things are worth knowing before you build anything important on top of it. + +**Equality-delete updates break it, and your Iceberg version matters too.** The spec says lineage is only guaranteed through copy-on-write updates and merge-on-read updates with position deletes or deletion vectors. Equality deletes are excluded because an equality-delete writer never reads the existing row and can't carry its ID forward. Choosing copy-on-write or merge-on-read with deletion vectors is the correct setting, but the setting only takes effect if your Iceberg version implements the carry-forward. On Spark 3.5 with Iceberg 1.10.0, a merge-on-read UPDATE preserves the ID: `_row_id` stays fixed while `_last_updated_sequence_number` advances. + +| event | qty | `_row_id` | `_last_updated_sequence_number` | +| --- | --- | --- | --- | +| insert | 100 | 0 | 1 | +| update | 200 | 0 | 2 | + +Support for this in the Spark 3.5 integration landed in Iceberg 1.10.0 (PR #12736, covering UPDATE and MERGE). Earlier versions could write the correct delete file without carrying the original `_row_id` forward, because locating the row and copying its ID into the rewritten data file are two separate steps in the writer. The spec requires that second step, that a row moved to a new data file keep its existing `_row_id`, and excludes only equality deletes. AWS ships the working combination in Amazon EMR 7.12 (Spark 3.5.6 with Iceberg 1.10). Check the Iceberg version behind your engine, not only the Spark version, and test before depending on it. + +**You don't get history before the upgrade.** Switching a v2 table to v3 doesn't go back and assign IDs to old data. Your audit trail effectively starts the day you flip the property. If you need lineage going further back than that, you'll need another mechanism in place. Iceberg won't reconstruct it for you. + +**Engine support is uneven, and reading is easier than writing.** Reading lineage works on recent versions of Spark, EMR, Databricks, StarRocks, Dremio, and (as of release 480) Trino. The trickier question is whether your write engine preserves `_row_id` on update. Test it before you depend on it. Write a row, update it, and check whether the ID survives. If it doesn't, you have a delete and an insert pretending to be an update. + +**Same-commit changes are invisible.** Lineage records what each commit did, not what happened inside it. A row inserted and deleted in the same atomic commit was never there, as far as the table is concerned. This is usually what you want, but it's worth knowing if you're counting on full intra-commit fidelity. + +## Enabling and Using Row Lineage (Iceberg v3) + +To take advantage of row lineage, you need to be on Iceberg format version 3. Heads up: most engines still create new tables as v2 by default. Trino and Starburst, for example, won't switch you to v3 unless you ask for it. So plan to set `format-version = 3` explicitly when you create the table. + +The good news is that v3 isn't bleeding-edge anymore. AWS rolled out v3 deletion vectors and row lineage across EMR, Glue, S3 Tables, and SageMaker in late 2025. Databricks supports it through Unity Catalog. Starburst, Dremio, and StarRocks have added v3 support, and Trino added row lineage in release 480 (March 2026), though support there is recent and still maturing. The defaults will almost certainly flip to v3 over the next year, but for now, the opt-in is on you. + +For new tables, specify the table property `format-version = 3` when creating the table. For example, in Spark SQL: + +```sql +CREATE TABLE mydb.my_table ( + -- columns +) +USING iceberg +TBLPROPERTIES ('format-version' = '3'); +``` + +This ensures the table is created as v3, and row lineage will be active from the get-go. +**Upgrading Existing Tables:** If you have an Iceberg v2 table, you can upgrade it in place to v3. This is an atomic metadata operation – no data rewrite is needed. For example: + +```sql +ALTER TABLE mydb.my_old_table +SET TBLPROPERTIES ('format-version' = '3'); +``` + +When you run this, Iceberg will bump the table's spec version to 3 and begin tracking row lineage for all new changes going forward. All existing data files remain as they are (with no `_row_id` fields written in them yet), but the table metadata now knows to treat them as having the lineage columns as null. For snapshots created before the upgrade, `_row_id` will be NULL because first-row-id was not recorded. + +:::info Upgrade caveat +Iceberg will not retroactively assign row IDs to old snapshots. Historical data from before the upgrade will not have meaningful `_row_id` or `_last_updated_sequence_number` values because those were not tracked in v2. Any new writes after the upgrade will have lineage. This is usually fine, as audit requirements often apply from the point of enabling onward. Just be aware that you won't get a full change history from before the upgrade unless you had other mechanisms in place. +::: + +Once on v3, the lineage fields are present for new writes. Reading them is well supported, but whether a write path actually preserves `_row_id` on update depends on the engine and its Iceberg version, as the Spark 3.5 case above showed. For that integration, update-time preservation landed in Iceberg 1.10.0, which AWS ships in EMR 7.12 (Spark 3.5.6 with Iceberg 1.10). Always verify your write engine preserves `_row_id` before depending on it, and don't mix v2 and v3 writers on the same table. + +## Conclusion + +Row lineage is one of those features that looks like a small spec change and ends up shifting what's actually feasible to build on a data lake. The practical test for your setup is simple: pick a non-critical table, switch it to v3, run an update through whatever engine writes to it, and check whether `_row_id` stayed the same. If yes, you have CDC. If no, you have a delete and an insert wearing a costume, and you'll want to figure out why before anything downstream depends on it. + +The spec is here, the engines are mostly here, and defaults will catch up in 2026. The cheapest move right now is to try it on something small so you know what you're working with before something forces you to. + + diff --git a/blog/2026-06-16-olake-vs-aws-dms.mdx b/blog/2026-06-16-olake-vs-aws-dms.mdx new file mode 100644 index 000000000..8865bcad9 --- /dev/null +++ b/blog/2026-06-16-olake-vs-aws-dms.mdx @@ -0,0 +1,187 @@ +--- +slug: olake-vs-aws-dms +title: "AWS DMS vs OLake: Choosing the Right Tool for Your Iceberg Pipeline" +description: "Compare AWS DMS and OLake for database-to-Iceberg pipelines: setup, CDC, schema evolution, scaling, and cost, with benchmark numbers on over 4 billion rows." +tags: [olake, aws, cdc, iceberg, comparison, replication] +authors: [anshika] +image: /img/blog/cover/aws-dms-vs-olake-cover.webp +--- +import BlogCTA from '@site/src/components/BlogCTA'; + +![AWS DMS VS OLake Cover](/img/blog/cover/aws-dms-vs-olake-cover.webp) + +:::info TL;DR +AWS DMS is a database migration service with CDC layered on. OLake is a replication engine built for streaming change data into Apache Iceberg. In a published benchmark on just over 4 billion rows, OLake ran the full load about 4.6x faster, sustained CDC about 1.38x faster, and cost roughly 4.61x less on compute for the full refresh. Memory use was close to a tie. Choose DMS for a finite move into AWS; choose OLake for an always-on pipeline into open table formats. +::: + +If you've ever run AWS DMS as the engine behind a lakehouse pipeline, you know the failure mode: A schema change breaks the task, the logs say almost nothing, and the only resolution anybody trusts is another full reload. That's not a misconfiguration. That's the difference between a migration tool and a replication tool and why most teams comparing AWS DMS and OLake show up with a specific problem rather than out of idle curiosity. + +Both tools move data out of operational databases like PostgreSQL and MySQL into a place where you can analyse it, so at a glance they look interchangeable. What makes them different is what they were created for. AWS DMS is built to do a one-time migration of a database, with ongoing change data capture layered on. OLake is designed for CDC into open table formats such as Apache Iceberg, where the full load is just a step in the job, not the end goal. That first assumption drives almost everything else that follows: how much setup the pipeline needs, how it scales, what a schema change costs you, what the monthly bill looks like. + +This comparison breaks down where each tool fits, across setup, continuous replication, schema evolution, scaling, and cost, with the benchmark numbers behind the claims and an honest look at when AWS DMS is still the right choice. + +![OLake vs AWS DMS Schema Evolution](/img/blog/2026/16/olake-vs-aws-dms-schema-evolution.webp) + +## What each tool is built for + +Both AWS DMS and OLake move data out of operational databases into a place where you can query, but they were built for different jobs, and the architecture of each makes that obvious. + +### AWS DMS: A Managed Database Migration Service + +AWS DMS is a managed service to migrate databases to AWS with minimum downtime. AWS DMS supports both homogeneous moves (MySQL to MySQL) and heterogeneous moves (Oracle to PostgreSQL). The AWS Schema Conversion Tool handles engine-to-engine schema differences. + +Architecturally, DMS uses a replication instance that you size, manage, and pay for as compute. You configure a source endpoint and a target endpoint and then run a replication task that performs a full load and optionally continuous CDC by reading the source transaction logs (binlog, WAL, or redo logs depending on the engine). A serverless option skips the instance sizing step but comes with its own constraints. The design as a whole assumes a finite move: get the data into AWS, and then either decommission the source or run both in parallel through a transition. + +![OLake vs AWS DMS Architecture](/img/blog/2026/16/olake-vs-aws-dms-dms-architecture.webp) + +### OLake: An Open-source Replication Engine for Apache Iceberg + +OLake is an open-source ingestion engine written in Go that replicates PostgreSQL, MySQL, MongoDB, Oracle, DB2 and MSSQL databases, as well as Kafka and S3 into Apache Iceberg or plain Parquet. It works at full load and CDC in a single sync mode, discovers and evolves schemas automatically, and writes to open tables readable by any Iceberg-compatible engine. + +The architecture is modular: a Core for state, concurrency and monitoring and pluggable Drivers for sources and Writers for destinations. Each Writer is integrated with its Driver, pushing records directly to the target, without staging in an intermediary store, keeping latency low. The Core splits large tables into virtual chunks for parallel reads. It has a CDC cursor, so if a sync gets interrupted, it can pick up from the last checkpoint and not start over again. It registers tables in your catalogue, be it AWS Glue, REST, JDBC or Hive Metastore. You run it from the command line or the latest UI. OLake also provides table maintenance through OLake Fusion, which takes care of the compaction and small-file cleanup that Iceberg tables require over time. The project is on [GitHub](https://github.com/datazip-inc/olake), the engine is OLake Go. + +![OLake Architecture](/img/blog/2026/16/olake-architecture.webp) + +## Head-to-head + +| Dimension | AWS DMS | OLake | +| --- | --- | --- | +| Primary design goal | One-time database migration to AWS | CDC replication into open lakehouse formats (Apache Iceberg) | +| Setup for Postgres parallelism | Manual partition-boundary scripting, pglogical configuration | Full Refresh plus CDC sync mode, minimal config | +| Full-load throughput | Baseline | Roughly **4.6x faster** in the published test | +| CDC support | Supported, but one stalled table can block the whole source-to-target flow | Built for sustained CDC, around **1.38x faster** in the same test | +| Schema evolution | DDL changes can break the pipeline; fallback is a full reload | Automatic schema discovery and evolution | +| Output format | Oriented toward Amazon targets | Open Iceberg or Parquet, readable by any engine | +| Compute cost | Baseline | About 4.61x cheaper on compute for the full refresh | +| Licensing | Managed AWS service | Open source | +| Destination targets | Many AWS targets, including databases (RDS, Aurora, Redshift), S3, DynamoDB | Object storage only (S3, ADLS, GCS) as Parquet or Iceberg; not a database target | + +The rows worth expanding are the ones that bite teams after the pipeline is live, not during the demo. + +## Setup and configuration + +There are a lot of moving parts to getting AWS DMS up and running. You provision and size a replication instance, create a source and target endpoint, enable source-side logging that CDC relies on (logical replication or pglogical for PostgreSQL, binlog for MySQL, or supplemental logging for Oracle), and define the table mappings. It's the parallelism that hits. With DMS you have to generate the partition boundaries yourself and if you want real throughput out of a large PostgreSQL table, you usually have to figure out what those boundaries are and script them by hand. None of this is difficult the first time, but each piece is a surface of maintenance someone has to own in the face of changing tables and schemas. + +OLake collapses most of that into a guided setup. The UI walks you through connecting a source and destination, including the Iceberg catalog, then discovering the available streams and running a sync, without scripting any of it. You don't manage parallelism either. OLake splits large tables into virtual chunks and reads them in parallel by itself, and you shape behavior by picking a sync mode like Full-Refresh plus CDC rather than tuning boundaries by hand. Teams that want finer control can drive the same flow from the CLI with `source.json` and `destination.json` config files, but the UI covers the common path end to end. + +![OLake vs AWS DMS Setup Comparison](/img/blog/2026/16/olake-vs-aws-dms-setup-comparison.webp) + +The real difference is the person doing the tuning. DMS asks you to set up and parallelise; OLake does those calls for you and exposes only the few decisions that really matter as configuration. + + +## Data replication + +DMS does fine for a one-time migration. The trouble starts when teams run it as a long-lived CDC pipeline, and the same complaints come up again and again, both in public forums and from OLake's own users. One line sums it up: DMS is the Data Migration Service, not a replication service. + +The pattern these users report is worth spelling out, because it is specific. When a task breaks, the logs give very little away about what failed or why, so root-causing turns into guesswork while the pipeline sits stalled. A single problem on one table can block all replication from the source to the target, which leaves downstream dashboards reading stale data and transaction logs growing inside the source RDS instances. More than one team has resorted to dropping the affected table on the warehouse side and letting DMS repopulate it from scratch, because that was faster than diagnosing the stall. Part of this is the shape of the tool itself. The migrate-then-catch-up model, a point-in-time load followed by replaying buffered changes, fits a finite move better than indefinite replication. + +This isn't hypothetical. Xeno, a customer engagement platform, ran MySQL CDC on AWS DMS and kept hitting broken replication on routine schema changes, with recovery forcing full table reloads that stretched past 17 hours. After moving those pipelines to OLake, schema changes stopped breaking the flow and full-load time dropped from roughly 24 hours to around 13. You can read the full customer story [here](https://olake.io/customer-stories/xeno-aws-dms-alternative-mysql-cdc). + +OLake is built for CDC as its main job, not a feature added on later, so it holds throughput on sustained incremental loads instead of drifting over time. Because streams are handled independently, a problem with one table does not freeze the rest of the pipeline, and the rest of your data keeps flowing while you deal with the single issue. When something does need attention, OLake exposes live sync stats, record counts, throughput, and an estimated finish time so you can see what is happening instead of piecing it together from vague logs. Together, steady throughput, isolated failures, and real visibility make CDC-based replication something you can rely on rather than something that surprises you. + +![OLake vs AWS DMS Failure Isolation](/img/blog/2026/16/olake-vs-aws-dms-failure-isolation.webp) + +## Schema evolution + +A DDL change on the source is where the two tools differ most in daily use. With DMS, a changing schema can break the running pipeline, and the usual fix is a full reload, which on a large table costs real time and downtime. It gets worse for teams that deliberately do not want DDL replicated, since they are left fixing schema drift by hand. Users also report data problems along the way: missing records, null values, and missing columns in the target, with a full reload again being the only reliable fix. There is even a known quirk where DMS, for some targets, stores boolean values as the strings "true" and "false", the kind of thing you find out in production, not in the docs. + +![AWS DMS Schema Change Full Reload](/img/blog/2026/16/aws-dms-schema-change-full-reload.webp) + +OLake handles this differently. It detects schema changes during the discovery step and evolves the target tables on its own, so a new column, a renamed field, or a type change upstream is picked up and applied rather than left to break the run. The pipeline keeps moving when the source moves, and a column added upstream does not turn into a reload. Because the destination is Apache Iceberg, which supports schema evolution at the table-format level, those changes are tracked without rewriting existing data, so old records stay queryable alongside the new shape. You can also set alerts on schema changes, so a shift on the source is something you are told about rather than something you find once the data already looks wrong. + + ![OLake Schema Evolution Iceberg](/img/blog/2026/16/olake-schema-evolution-iceberg.webp) + + +## Scaling and resource use + +DMS ties throughput to the replication instance you size and pay for, so memory and CPU are capped by the instance class you choose, and a large table that outgrows that instance forces you to scale up or split the work. Getting good parallelism out of a big PostgreSQL table means generating partition boundaries by hand and tuning the load, which is more to set up and more to maintain. The initial load also works in stages: it captures each table at a point in time, holds the changes that happen meanwhile, and applies them before switching to CDC. That sequence needs table locks and puts heavy load on the source database at scale. Large tables, on the order of 100GB with heavy update traffic, were a common trigger for repeated full loads in the user reports above. DMS also does not move indexes, triggers, or stored procedures, so those have to be handled separately. + +OLake needs a host too. It runs as a process on compute you provide, a VM, container, or Kubernetes pod, so there is a machine to size on this side as well. The difference is what that sizing involves. OLake is not a managed instance with a fixed class ceiling that you scale up the moment a table outgrows it, and it is built for high-throughput ingestion at terabyte scale, with memory efficiency treated as a goal rather than an afterthought. Instead of asking you to break large tables apart by hand, it splits them into virtual chunks on its own and reads those chunks in parallel, so a big table is a throughput opportunity rather than a problem to work around. It also streams records straight to the destination as they are read, rather than staging them in memory or an intermediary store first, which keeps memory use predictable even on very large loads and avoids the ceilings that force the splitting workarounds in the first place. Scaling is then a question of how many chunks to run in parallel and how much compute you give the host, not which instance class to jump to next. + + +## Connectors, targets, and lock-in + +DMS supports CDC database sources and points naturally at Amazon targets, with a VPC required for those targets and some cross-region paths unsupported, such as parts of DynamoDB. The serverless option trades instance management for further restrictions, so the limitations pages are worth reading closely before committing. If your pipeline needs to reach beyond that footprint, you end up adding tools around DMS to cover the gaps. + +OLake writes to Apache Iceberg or Parquet, open formats that Spark, Trino, Flink, and other engines can query without copying the data again, and it pulls from databases, Kafka, and S3. For teams that want to avoid committing their analytics layer to a single vendor, open output is a structural advantage rather than a feature. + +## The benchmark, with the setup stated + +But a benchmark is only useful if you know how it was run, so here's the setup: + +Both tools moved the same data using the same hardware: an Azure PostgreSQL source (32 vCores and 128 GB RAM) writing Parquet to AWS S3. The dataset was the standard NYC Taxi data, the `trips2` and `fhv_trips` tables, with a little over 4 billion rows in total. The full load test measured the time to move the full data set. Then the CDC test pushed 50 million change records across the two tables to see how each tool fared under sustained change. OLake ran in Full Refresh + CDC sync mode with 32 threads. DMS was running with 40 parallel tasks, and pglogical was to be enabled first. + +Benchmark results: + +- **Full refresh:** OLake ran the full load about 4.6x faster than AWS DMS on just over 4 billion rows. +- **CDC:** OLake was ~1.38x faster on the incremental load. +- **Compute cost:** OLake was around 4.61x cheaper for the full refresh, and that difference only adds up the more frequently the job runs, as a daily sync repeats the difference week after week. +One result can be stated simply rather than flowery. The memory was almost a tie. OLake averaged about 31.76 GB across the full 4 billion rows and DMS about 30 GB, close enough to call it a tie. The difference between the two tools is speed and cost, not memory consumption. This does not need to be taken on trust, as the setup and dataset are open. You can run it again against your own data. The full numbers, including how cost adds up over time, are on the [DMS vs OLake benchmark page](/blog/olake-vs-aws-dms-benchmark). + +One result can be stated simply rather than flowery. The memory was almost a tie. OLake averaged about 31.76 GB across the full 4 billion rows and DMS about 30 GB, close enough to call it a tie. The difference between the two tools is speed and cost, not memory consumption. This does not need to be taken on trust, as the setup and dataset are open. You can run it again against your own data. The full numbers, including how cost adds up over time, are on the [DMS vs OLake benchmark page](/blog/olake-vs-aws-dms-benchmark). + +## When AWS DMS is still the right call + +An honest case for DMS. Because a comparison that praises only one side is not worth reading. It's a good choice for a one-off move into AWS, and it's especially good when the destination is a database engine such as RDS or Aurora, which OLake cannot target, and at heterogeneous migrations where the source and target run different engines. If your database is on the smaller side, can tolerate some downtime, and you are happy with a fully managed service which lands data in AWS rather than open formats, then the simple path is to stay inside the AWS toolchain. DMS can even feed Iceberg near real-time, but it takes some additional tooling on top, which some teams have set up and run well. + +DMS does what it was built to do, well, as long as you follow the habits a migration tool expects. One small example is forgetting to drop a PostgreSQL replication slot once a migration is done, which can cause trouble on the source database later on. The message is not that DMS is a bad tool. DMS was built to migrate, and that is where the problems begin. When a migration tool is left running as permanent replication infrastructure. + +## When OLake is the better fit + +OLake fits well when the work doesn't stop. OLake was built for the job of running a lakehouse pipeline on a daily basis once a migration has been completed. You're streaming changes into Apache Iceberg on an ongoing basis. You need to query that data from more than one engine. You want to keep your compute bill in check as volume grows. You want your analytics layer to stay on open formats instead of being tied to one vendor. OLake is built to satisfy each of those, not to bring additional tools. + +It also takes away the ongoing costs that eat into teams on a long-lived DMS pipeline. Schema changes are absorbed, rather than breaking the run, so there is no manual-restart tax every time the source changes. If there is a failure, it stays local to that stream. It does not freeze the whole flow. And OLake Fusion handles the Iceberg table maintenance, the compaction and the small-file cleanup that prevent query performance from degrading over time, which DMS doesn't address at all. + +And that's the real difference for an always-on database-to-lakehouse pipeline: not which tool is faster in a single run, but whether you spend your week watching the pipeline or trusting it. + +## Choosing between them + +The right call comes down to the shape of the job, not a scorecard. The two tools are good at different things, and most of the regret comes from picking one for work it was never meant to do. The quickest way to place your own situation is to match it against the table below. + +| If your situation looks like this | The better fit | +| --- | --- | +| A one-time migration of a database into AWS | AWS DMS | +| A heterogeneous move, where source and target run different engines (for example Oracle to Aurora) | AWS DMS | +| A fully managed service, staying inside AWS, with no need for open formats | AWS DMS | +| CDC into Apache Iceberg that runs indefinitely | OLake | +| Querying the same data from several engines (Trino, Spark, Snowflake, and others) | OLake | +| Keeping compute cost down at terabyte scale or with frequent syncs | OLake | +| Avoiding vendor lock-in by landing data in open table formats | OLake | +| Schema changes that should not turn into manual reloads | OLake | +| Iceberg table upkeep (compaction and small-file cleanup) handled for you | OLake | + +## Conclusion + +Reach for AWS DMS when the work is finite and the destination is AWS. Its strength is getting a database, including one on a different engine, onto AWS through a managed service with little fuss. If you do not need open formats and the pipeline has an end date, the simplicity of staying inside one ecosystem is worth a lot. + +Reach for OLake when the work does not end. Changes capture into Apache Iceberg, data that several query engines need to read, costs that have to stay flat as volume grows, and a stack you would rather not tie to a single vendor are all jobs it was built for. Its schema handling means a column added upstream does not cost you a reload, and OLake Fusion keeps the Iceberg tables healthy instead of leaving compaction as a chore for later. + +Here is the pattern worth remembering. Most teams that struggle with DMS are not using a bad tool. They are using a migration tool to solve a replication problem, and the gap between those two jobs is where the late nights come from. If that sounds like your setup, the honest next step is to run the numbers on your own data. The [OLake documentation](https://olake.io/docs) is a good place to start, and the comparison holds up best when you test it against the workload you actually have rather than someone else's. + +## FAQs + +### Q1. Has anyone replaced AWS DMS with OLake in production? + +Yes. Xeno moved its MySQL CDC pipelines off AWS DMS to OLake, self-hosted on Kubernetes, after schema changes kept breaking replication. Full-load time fell from about 24 hours to around 13, and schema changes no longer trigger reloads. + +### Q2. Is OLake a good AWS DMS alternative? + +For an ongoing pipeline into Apache Iceberg, yes. DMS is built for a one-time migration with CDC added on, so teams that run it as permanent replication infrastructure tend to hit schema breaks and full reloads. OLake is built for sustained CDC into open table formats. + +### Q3. Can AWS DMS replicate into Apache Iceberg? + +It can, but not on its own. DMS points naturally at Amazon targets and needs extra tooling on top to feed Iceberg in near real time. OLake writes Iceberg or Parquet directly. + +### Q4. Why does a schema change break an AWS DMS pipeline? + +A DDL change on the source can stall the running task, and the fix most teams trust is a full reload. OLake detects schema changes during discovery and evolves the target tables on its own. + +### Q5. Is OLake faster than AWS DMS? + +In the published test, on the same hardware and dataset, OLake ran the full load about 4.6x faster and sustained CDC about 1.38x faster. Memory use was close to a tie. + +### Q6. Can OLake write to a database such as RDS, Aurora, or Redshift? + +No. OLake writes to object storage (S3, ADLS, GCS) as Parquet or Iceberg. If your destination is a database engine, DMS is the better fit. + + \ No newline at end of file diff --git a/blog/2026-07-13-exactly-once-delivery-iceberg.mdx b/blog/2026-07-13-exactly-once-delivery-iceberg.mdx new file mode 100644 index 000000000..7472fd6d4 --- /dev/null +++ b/blog/2026-07-13-exactly-once-delivery-iceberg.mdx @@ -0,0 +1,325 @@ +--- +slug: exactly-once-delivery-iceberg +title: "How OLake Guarantees Exactly-Once Delivery to Apache Iceberg" +description: "How OLake guarantees exactly-once delivery into Apache Iceberg across Full Refresh, Incremental, and CDC syncs using atomic commits and checkpoint recovery. No duplicates, no data loss, no external coordinator." +date: 2026-07-13 +authors: [vaibhav] +tags: [iceberg, cdc, exactly-once, olake, lakehouse, postgres, mysql, mongodb, mssql] +image: /img/blog/cover/exactly-once-delivery-iceberg-cover.webp +--- + +import BlogCTA from '@site/src/components/BlogCTA'; + +![How OLake Guarantees Exactly-Once Delivery to Apache Iceberg](/img/blog/cover/exactly-once-delivery-iceberg-cover.webp) + +[OLake](https://olake.io/) is an open-source tool for replicating databases into Apache Iceberg and S3 compaitable destinations, built for fast, reliable Full Refresh, Incremental, and CDC syncs. To know more about how OLake Go works read, [deep dive into OLake's architecture](/blog/olake-architecture-deep-dive). + +## TL;DR + +Every OLake sync makes two writes that are not atomic with each other: the data commit into Apache Iceberg and the checkpoint save to `state.json`. A crash between the two leaves the destination ahead of the checkpoint, which historically meant duplicated rows on retry or, worse, a recovery that misread the gap as corruption. + +OLake closes this gap by committing data files and a progress marker to Iceberg in a single atomic operation, then comparing that marker against `state.json` on every restart. If the checkpoint matches, the sync runs normally. If Iceberg is ahead, the data already landed, so OLake skips the work and repairs the checkpoint without re-reading the source or re-writing the destination. + +The same comparison runs across Full Refresh, Incremental, and CDC on every driver. No coordinator, no transaction log, no locks: whenever a sync dies, the table ends up correct, with nothing duplicated and nothing missing. + +## Why Exactly-Once Delivery Matters for Iceberg Pipelines + +OLake transfers data from various sources into an Iceberg table that everything downstream treats as ground truth. And long-running syncs might get interrupted at times due to various reasons such as a pod eviction, a dropped connection, an OOM kill, a manual restart. What actually matters is what happens the next time the sync starts up. + +The tricky part is that every sync does two writes that aren't atomic with each other. It commits data into the destination, and separately it updates a checkpoint file that records how far it got, so the next run knows where to resume. + +If the process dies between those two writes, the destination is ahead of what the checkpoint knows about. Resume blindly from the checkpoint and it re-reads data that already landed, adding duplicate records. Skip ahead without checking and it misses records that never got written. And because nothing actually failed, no error ever warns you it happened, you only find out when you go digging through the data yourself. + +This isn't hypothetical, it's what used to happen. Some cases might be: A writer would commit a chunk to Iceberg and then die before the checkpoint was saved, so the retry reprocessed the same chunk and duplicated rows. On the CDC side, a Postgres slot would get acknowledged, the process would crash before the checkpoint caught up, and recovery would misread the gap as corruption and ask the user to wipe the destination and start over. + +Exactly-once delivery fixes both. It adds one check that runs across all sync modes, Full Refresh, Incremental, and CDC, and behaves the same on every driver. Here is how it works. + +## How OLake's Exactly-Once Delivery Mechanism Works + +Every mode and every driver follows the same underlying pattern. Here is the brief idea before we move into the depth of each mode. + +Before a writer commits, OLake generates a **deterministic thread ID** for what it's about to write. It's the stream ID combined with something specific to the mode: the chunk bounds for a backfill, the cursor value for an incremental window (CDC doesn't require deterministic thread id). + +When the write happens, the data files and a progress marker are committed to Iceberg together, in one operation. The marker sits in the table's `olake_2pc` property and carries the thread ID plus the mode's progress: committed chunk IDs, a WAL or binlog position, a resume token, or a cursor value. + +Only after that commit succeeds, OLake saves the same progress to `state.json`. + +The commit and the checkpoint aren't atomic with each other, so a crash can land in between, leaving Iceberg with a marker that `state.json` hasn't caught up to. That's what recovery handles. On the next run, before reading anything from the source, OLake reads each stream's marker back from Iceberg and compares it to `state.json`: + +- **`state.json` matches or is ahead:** nothing to recover, the sync runs normally. +- **Iceberg metadata file is ahead:** that data already committed, so OLake skips it and fixes `state.json` to match, without re-reading the source or re-writing the destination. +- **No marker:** a fresh stream, nothing to compare. + +The middle case is the only one where recovery does real work, and every mode comes down to it. The rest just changes what the marker holds (chunk IDs, a cursor value, or a log position or token) and how many run at once: one table split into chunks, one stream with one cursor, or several CDC streams sharing a single log. The sections below walk through each mode with real payloads. + +## The Commit Itself: How OLake Writes to Iceberg + +OLake writes to Iceberg through a Java process it runs over gRPC, which owns all catalog operations. A finishing writer sends a `COMMIT` carrying its `MetadataState` payload, and Iceberg applies the new Parquet files and the metadata update as a single atomic operation. There's no window where the data exists without its marker, or the reverse, and that guarantee is exactly what lets recovery trust the comparison above, no external coordinator needed. + +Closing a writer also takes a per-stream lock on the Go side, so when several backfill chunks for the same stream finish at once, their metadata commits are serialized instead of racing on the same table. Once the commit lands, OLake writes the same progress into `state.json`, the checkpoint. + +## Full Refresh + +![Full Refresh commit and recovery flow](/img/blog/2026/7/exactly-once-delivery-full-refresh.webp) + +### How It Runs + +OLake splits large tables into chunks for parallel processing and fast transfer. Each chunk is assigned a deterministic thread ID built from the stream ID and the chunk's min and max bounds. Because the ID is derived from the chunk's content rather than a random value, the same chunk always gets the same thread ID across runs, this is the foundation of Full Refresh recovery. + +Chunks are processed concurrently across the configured thread pool. When a chunk finishes: + +- **Commit step:** The writer sends a `COMMIT` to the Iceberg Java server. The Java server atomically writes the chunk's Parquet files and updates the metadata file. Inside the `olake_2pc` property, the `full_refresh_committed_ids` array is extended with the current thread's ID. This is the durable record that this chunk's data is in the table. +- **Checkpoint step:** The chunk is removed from the `chunks` array inside `state.json`. + +### What Happens on Failure + +If the process crashes after the commit step but before the checkpoint step, `state.json` still lists that chunk as pending. On the next run, OLake starts fresh from `state.json`, queues the same chunks, and creates writer threads for them. But before doing any reading, each thread checks the `full_refresh_committed_ids` list read from the Iceberg table's metadata file. If the current thread's deterministic ID is already present in that list, the chunk is silently skipped as the data is already in the table. + +This way, a crash between the commit step and the checkpoint step causes zero data loss and zero duplication. Chunks that fully committed are never re-processed. Chunks where the commit itself failed are re-processed from scratch. + +### Example: Postgres Table public.users + +Say OLake splits the table into two chunks. After chunk 1 commits (commit succeeds, process crashes before the checkpoint): + +Iceberg metadata file `olake_2pc` property: + +```json +{ + "full_refresh_committed_ids": [ + "public.users_min[1]-max[10000]" + ] +} +``` + +On the next run, OLake finds unprocessed chunks saved in `state.json` and creates writer for these chunks. + +When a writer is created for `[1, 10000]`, it generates the same thread ID `public.users_min[1]-max[10000]`, finds it in `full_refresh_committed_ids`, and skips the chunk. As the thread ID created for the second chunk is not present in the `full_refresh_committed_ids`, only `[10001, 20000]` is read and written. + +## CDC + +### Global LSN Sources: Postgres and MySQL + +![Global LSN CDC recovery flow for Postgres and MySQL](/img/blog/2026/7/exactly-once-delivery-cdc-global-lsn.webp) + +Postgres and MySQL represent all change events from a single, ordered stream, the WAL replication slot (Postgres) or binlog (MySQL). Every change, regardless of which table it affects, has a position in this global sequence. OLake reads this stream sequentially: one replication connection covers all selected streams in a single pass. + +#### The Commit Sequence + +A writer thread is created per selected stream at the start of CDC. As the replication stream is consumed, change records are routed to the appropriate writer. When the sync window closes: + +- **Commit Step:** Each stream's writer sends a `COMMIT` to the Iceberg Java server. The commit payload carries the final WAL LSN (Postgres) or binlog file-and-position (MySQL) as the `MetadataState`. The Java server atomically writes data files and embeds this position in the stream's `olake_2pc` table property. +- **Checkpoint Step:** For Postgres, PostCDC first acknowledges the replication slot to inform the server it can reclaim WAL space, then saves the final LSN to `state.json`. For MySQL, PostCDC saves the final binlog position to `state.json`. + +#### What Happens on Failure + +Consider this scenario: Stream A commits to Iceberg (Commit succeeds), but the process crashes before PostCDC runs Checkpointing. Now `state.json` still holds the old LSN, but Stream A's Iceberg table has metadata that records the newer LSN it committed. + +On the next run, before reading a single event from the replication stream, OLake compares each stream's metadata LSN against the global state LSN: + +- If a stream's metadata LSN is ahead of the state LSN, that stream has already committed its data and does not need to re-read those WAL events. It is excluded from the replication session. +- The remaining streams, those whose metadata LSN matches the state LSN, need to catch up. OLake starts a bounded replication sync: reading from the current state LSN up to the metadata LSN, but only emitting events for those remaining streams. Once this bounded sync completes, all streams are at the same LSN and `state.json` is updated normally. + +This guarantees that no stream misses events that fell in the gap between the last committed LSN and where the process crashed. + +#### Example: Postgres with Two Streams: public.orders and public.users + +`state.json` before CDC sync (global position): + +```json +{ + "type": "GLOBAL", + "global": { + "state": { "lsn": "0/1A2B3C4D" }, + "streams": ["public.orders", "public.users"] + } +} +``` + +CDC runs. `public.orders` commits first (Commit phase succeeds). Process crashes before Checkpoint phase runs. + +`public.orders` Iceberg metadata file `olake_2pc` after Committing: + +```json +{ + "id": "public.orders_", + "state": "{\"lsn\":\"0/2B3C4D5E\"}" +} +``` + +`public.users` Iceberg metadata file `olake_2pc` (unchanged - crash happened before its commit): + +```json +{ + "id": "public.users_", + "state": "{\"lsn\":\"0/1A2B3C4D\"}" +} +``` + +On the next run, OLake reads both metadata files: + +- orders metadata LSN `0/2B3C4D5E` > state LSN `0/1A2B3C4D` → skip orders, it is already committed +- users metadata LSN `0/1A2B3C4D` = state LSN → include users in a bounded replay up to `0/2B3C4D5E` + +After the bounded sync and Checkpoint step: + +`state.json` updated: + +```json +{ + "type": "GLOBAL", + "global": { + "state": { "lsn": "0/2B3C4D5E" }, + "streams": ["public.orders", "public.users"] + } +} +``` + +### Per-Stream LSN Sources: MSSQL and MongoDB + +![Per-stream LSN CDC recovery flow for MSSQL and MongoDB](/img/blog/2026/7/exactly-once-delivery-cdc-per-stream-lsn.webp) + +MSSQL and MongoDB do not share a global change position across tables. Each collection or table has its own independent LSN or resume token. OLake runs a concurrent change stream per selected stream, each stream is processed independently and in parallel. + +Because there is no shared global position, recovery is handled per-stream. At the start of `StreamChanges` for a given stream, the code compares the stream's position stored in `state.json` against the position stored in that stream's Iceberg metadata file. If the metadata position is strictly ahead, meaning Commit Step committed but Checkpoint Step (state save) did not, the stream returns immediately with the metadata position as its result. No CDC log reading occurs. PostCDC then saves this position to `state.json`, completing the recovery in Checkpoint Step. The next regular sync picks up from that position onwards. + +#### Example: MongoDB Collection mydb.orders + +`state.json` before CDC sync: + +```json +{ + "type": "STREAM", + "streams": [ + { + "stream": "orders", + "namespace": "mydb", + "state": { "_cdc_resume_token": "826A7B3C..." } + } + ] +} +``` + +CDC runs. `mydb.orders` commits to Iceberg (Commit Step succeeds). Process crashes before Checkpoint Step. + +`mydb.orders` Iceberg metadata file `olake_2pc` after Commit Step: + +```json +{ + "state": "826B8C4D..." +} +``` + +On the next run, OLake compares state token `826A7B3C...` against metadata token `826B8C4D...`. Metadata is ahead, Commit Step committed but Checkpoint Step did not. The stream returns immediately without reading any change events. PostCDC writes the metadata token to `state.json`. + +`state.json` after recovery (Checkpoint Step): + +```json +{ + "type": "STREAM", + "streams": [ + { + "stream": "orders", + "namespace": "mydb", + "state": { "_cdc_resume_token": "826B8C4D..." } + } + ] +} +``` + +The next sync then reads change events starting after `826B8C4D...`. + +## Incremental + +![Incremental sync cursor recovery flow](/img/blog/2026/7/exactly-once-delivery-incremental.webp) + +Incremental syncs advance a cursor column (such as `updated_at`) forward with each run. OLake records the maximum cursor value seen at the start of the sync, reads all rows up to that value, and then saves the new high bookmark. + +### The Commit Sequence + +Before reading any data, OLake captures the maximum cursor value currently in the source table. A deterministic thread ID is generated from the stream ID combined with that maximum cursor value. Because the cursor value is baked into the thread ID, the same sync window always produces the same thread ID. + +- **Commit Step:** When the sync completes, the writer commits to Iceberg, carrying the cursor values as the `MetadataState`. The Java server atomically writes data files and embeds the cursor values in the `olake_2pc` table property. +- **Checkpoint Step:** The new cursor value is written to `state.json`. + +### What Happens on Failure + +If the process crashes after Commit Step but before Checkpoint Step, `state.json` still holds the old cursor value. On the next run, OLake fetches the same maximum cursor value from the source (since nothing has changed), generates the same thread ID, and creates a writer thread. + +Before reading data, it reads `prevMetadataState` from the Iceberg table. If the stored metadata ID matches the current thread ID, confirming this is the same sync window that already committed, OLake reads the cursor values out of the metadata, updates `state.json` to reflect the committed position, and exits without re-reading any source data. + +The result: the committed data stays in Iceberg exactly once, and `state.json` is corrected to match, ready for the next run to advance the cursor further. + +### Example: Postgres Table public.orders with Cursor updated_at + +`state.json` before incremental sync: + +```json +{ + "type": "STREAM", + "streams": [ + { + "stream": "orders", + "namespace": "public", + "state": { "updated_at": "2024-06-01T00:00:00Z" } + } + ] +} +``` + +Thread ID is generated from this `state.json` cursor value: `public.orders_2024-06-01T00:00:00Z_...`. Incremental reads all rows with `updated_at > 2024-06-01T00:00:00Z`, stores the max seen value of updated at for that sync `2024-06-01T05:00:00Z` and commits to Iceberg. Commit Step succeeds. Process crashes before Checkpoint Step. + +`public.orders` Iceberg metadata file `olake_2pc` after Commit Step: + +```json +{ + "id": "public.orders_2024-06-01T00:00:00Z_...", + "state": "{\"updated_at\":\"2024-06-01T05:00:00Z\"}" +} +``` + +On the next run, OLake reads the cursor from `state.json`, fetches `updated_at` still `2024-06-01T00:00:00Z`. The same thread ID is generated. OLake reads the metadata file, sees the ID matches, reads the cursor out of the metadata, and exits without reading the source again and updates the `state.json` value to `2024-06-01T05:00:00Z`. Checkpoint Step confirms `state.json` is consistent. + +`state.json` after recovery: + +```json +{ + "type": "STREAM", + "streams": [ + { + "stream": "orders", + "namespace": "public", + "state": { "updated_at": "2024-06-01T05:00:00Z" } + } + ] +} +``` + +## Conclusion + +Every mode comes down to one comparison: the position saved in `state.json` against the position committed to Iceberg. When they agree, or `state.json` is ahead, OLake just moves forward. When Iceberg is ahead, a crash landed between the commit and the checkpoint, so OLake reads the committed position back out of the metadata and repairs `state.json`, without re-reading or re-writing any data. No coordinator, no transaction log, no locks, just Iceberg's atomic commit and few comparison logics. + +Whenever a sync dies, the table always ends up correct, with nothing duplicated and nothing missing. And that is what really counts: you might be running your dashboards, reports, and models straight off this Iceberg table, and with exactly-once delivery you can trust those numbers are right, every single time. + +## FAQ + +### Q1. Does OLake guarantee exactly-once delivery to Apache Iceberg? + +Yes, across all three sync modes: Full Refresh, Incremental, and CDC. OLake commits data files and a progress marker to Iceberg in a single atomic operation, then compares that marker against its `state.json` checkpoint on every restart, so a crashed sync resumes without duplicating or skipping records. + +### Q2. What happens if a sync crashes between the Iceberg commit and the checkpoint save? + +On the next run, before reading anything from the source, OLake reads the progress marker back from the Iceberg table's metadata and compares it to `state.json`. If the metadata is ahead, that data already committed, so OLake skips re-reading and re-writing it and just repairs `state.json` to match. If they already agree, the sync runs normally. + +### Q3. How does OLake make the data write and progress marker atomic? + +OLake writes to Iceberg through a Java process that owns all catalog operations. A finishing writer sends a commit carrying its progress payload, and Iceberg applies the new Parquet files and the metadata update as a single atomic operation. There is no window where the data exists without its marker or the marker without its data, which is what lets recovery trust the comparison against `state.json` after a crash. + +### Q4. Does OLake need an external coordinator or transaction log for this? + +No. The guarantee comes entirely from Iceberg's atomic commit plus the comparison between the committed metadata and `state.json`. No coordinator, no transaction log, no locks. + +### Q5. Does exactly-once delivery work the same way across all sources? + +Yes, broadly. Every source runs the same core check, comparing the committed marker in metadata file against `state.json` before doing any work. The differences are in how each sync mode saves and recovers that marker, and [CDC](#cdc) in particular parses and stores it a bit differently across drivers. See the [Full Refresh](#full-refresh), [Incremental](#incremental), and [CDC](#cdc) sections above for the exact mechanics per mode. + + diff --git a/blog/2026-07-21-schema-evolution-without-breaking-pipelines.mdx b/blog/2026-07-21-schema-evolution-without-breaking-pipelines.mdx new file mode 100644 index 000000000..f15a95306 --- /dev/null +++ b/blog/2026-07-21-schema-evolution-without-breaking-pipelines.mdx @@ -0,0 +1,154 @@ +--- +slug: schema-evolution-without-breaking-pipelines +title: "How OLake Handles Schema Evolution Without Breaking Your Pipeline" +description: "How OLake handles schema evolution into Apache Iceberg: automatic column adds, retained drops, safe type promotions, and explicit failures only where data would corrupt. Built on Iceberg's field-ID tracking, across Postgres, MySQL, MongoDB, and Kafka." +date: 2026-07-21 +authors: [anshika] +tags: [iceberg, schema-evolution, cdc, olake, lakehouse, postgres, mysql, mongodb, kafka] +image: /img/blog/cover/schema-evolution-apache-iceberg-cover.webp +--- +import BlogCTA from '@site/src/components/BlogCTA'; + +![How OLake Handles Schema Evolution Without Breaking Your Pipeline](/img/blog/cover/schema-evolution-apache-iceberg-cover.webp) + +## TL;DR: +Source databases are constantly evolving. Columns are added, renamed, and dropped. Data types are widened. Usually without warning. When that happens, most replication pipelines break or lose data silently. OLake is built to absorb these changes: it runs schema discovery at the start of every sync, adds new columns automatically, retains dropped columns so downstream consumers are never broken, applies safe type promotions on the fly, and fails loudly only on the small set of changes that would genuinely corrupt data. It can do all of this cheaply because it writes to Apache Iceberg, which tracks columns by unique field IDs instead of names or positions, making structural changes pure metadata operations with no file rewrites. This post walks through exactly what OLake does with each type of change, and where the type mapping rules live so you can check them yourself. + +## Introduction +Anyone who has run a production database replication pipeline long enough has run into this problem. Business requirements evolve, and that means columns get added, fields get renamed, or data types get widened (e.g., INT → BIGINT, FLOAT → DOUBLE). When that happens, the next sync crashes outright, or worse, silently writes wrong data downstream. This isn't a rare edge case; it's a routine, expected part of running a production database. Your pipeline has to take whatever comes down it. + +This is the problem that schema evolution is meant to solve. Get it wrong and the pipeline needs constant babysitting. Get it right and it quietly survives a real, living production database. This post covers what schema evolution means when replicating into a data lakehouse, what OLake does with each type of change, the exact type mapping rules that decide whether a sync keeps running, and the Iceberg design that makes all of it cheap. + +![How OLake's replication pipeline detects schema changes, applies evolution rules, and writes safely to an Apache Iceberg lakehouse](/img/blog/2026/21/schema-evolution-olake-iceberg-pipeline-diagram.webp) + +## What Is Schema Evolution + +No production database stays the same shape for long. + +A new feature ships, and a `last_login_at` column appears that didn't exist yesterday. A cleanup effort retires three deprecated fields. Two years of steady growth push a column past what `INT` can hold, so it has to be widened to `BIGINT`. None of these are edge cases. This is simply what happens when a production application runs on a database. + +Schema evolution is the term for handling all such schema changes without breaking your pipeline. + +There are two flavors of change worth separating out, because they behave differently and need different handling. + +The first is **structural change**, changes to which fields exist on the table. A new column appears. An old one disappears. Someone renames `user_name` to `username_display`. A table gets renamed entirely. What's inside each field stays the same; it's the table's makeup that's different. + +The second is **data type change**; the same column exists, but what it holds has changed. An `INT` column starts receiving values that don't fit in 32 bits anymore, so it needs to widen to `BIGINT`. A `FLOAT` column gets bumped to `DOUBLE` for more precision. Or business requirements shift and a field that used to hold numbers now needs to hold strings, to accommodate edge cases like "N/A" or legacy IDs. These changes are not all equal: some can be applied safely at the destination, while others, like that numbers-to-strings change, are incompatible with Iceberg's type system. The sections below cover how OLake handles each case. + +![Schema changes split into structural changes and data type changes, both flowing through the replication pipeline to either continue the sync or break it](/img/blog/2026/21/schema-evolution-change-types-and-outcomes-diagram.webp) + +Both categories happen routinely as a product evolves, a new signup flow adds fields to capture more user data, a pricing change adds a currency column, a growing user base pushes an ID column past its original range. The question isn't whether they'll hit your pipeline. It's whether your pipeline survives them when they do or whether you get paged at 2am because a sync broke. + +## Why Traditional Formats Struggle + +When it comes to schema change safety, it's all about one thing: the table format's ability to reliably track which column is which. Get this wrong and stale data can silently end up in the wrong column. This is precisely the failure Iceberg's and OLake's design eliminates. + +Formats that identify columns by name seem intuitive until a column gets deleted and its name is later reused for something different. Old data from the deleted column then reads as if it belongs to the new column with the same name, silent corruption with no errors raised. Formats that identify columns by position have the mirror problem: remove a column from the middle of a table, and every column after it shifts by one, so existing files written before the deletion now read with the wrong mappings. Fixing that usually means a full table rewrite. + +This is why schema changes have historically been expensive and risky, not because the logic is complicated, but because name-based and position-based identification make them fragile by design. Iceberg removes the fragility at its root by identifying columns a third way, which is what the next section covers. + +## The Foundation: How Iceberg Makes This Possible + +OLake writes to Apache Iceberg, and Iceberg solves the identification problem differently from traditional formats. Instead of identifying columns by name or position, every column in an Iceberg table gets a unique field ID when it is first created. That ID is stored in both the table metadata and the Parquet file metadata, and when Iceberg reads a data file, it matches columns by ID, not by name or position. + +Because of this, Iceberg supports adding, dropping, renaming, and reordering columns, plus safe type widening, as pure metadata operations. No data files get rewritten for any of them. The guarantees follow directly from the ID mechanism: a new column gets an ID no previous file ever used, so old files return null for it instead of misreading other data; dropping a column shifts nothing because positions are not used for lookups; and IDs are never reused, so old data can never be accidentally read as belonging to a newer column that happens to share a name. Every change produces a new versioned schema in metadata, and old versions are retained. + +These field IDs live inside the Iceberg table. They protect how Iceberg reads its own data files. How OLake maps source database columns to destination columns is a separate mechanism, covered in the next section, and the distinction matters for how renames behave. + +## How OLake Handles Schema Evolution + +This section covers what OLake does when the source schema changes, using Iceberg's operations as the building blocks. When a sync runs, OLake compares incoming record fields and types against the current destination schema. If a compatible change is detected, OLake decides what to do based on what kind of change it is. The exact behavior for every case below is documented in the [OLake schema evolution doc](https://olake.io/docs/features/schema). + +### Column-Level Changes + +**New column added at source:** When a sync receives a record with a new column, OLake adds that column to the Iceberg schema with a new field ID and writes its values. Whether new columns are included automatically or wait for manual selection is controlled by the "Sync new columns automatically" setting. This is safe because an added Iceberg column can never read existing values from another column: old files simply return null for it, so historical rows show null until the source backfills them. One nuance worth knowing: if the new column is sparse, meaning no row in the current sync has a non-null value for it, it will not appear at the destination until at least one row does. Iceberg stores data column-wise in Parquet, so there is no point materializing a column of nulls. + +**Column dropped at source:** When a source column is dropped, the destination column is deliberately retained, and new rows simply carry null for it. Downstream consumers of the Iceberg table may still depend on it, and old snapshots stay queryable, so new rows simply carry null for the column instead of it disappearing. This is the safer default, and it costs nothing because Iceberg matches columns by ID, so a retained column never interferes with any other column's values. If the storage footprint matters, a rewrite manifest job can drop the dead column later. + +**Column renamed at source:** Iceberg's field IDs exist only inside the Iceberg table; a source database column carries no such ID. OLake therefore maps source columns to destination columns by name, which means a renamed source column looks identical to one column being dropped and a new one being added, and that is how OLake handles it: the old column stays in the destination with its full history but stops receiving values, and a new column with the new name is created and starts receiving data. No data is lost, but history stays under the old column name, so downstream SQL should be updated to the new name. Iceberg itself supports true in-place renames on the same field ID; OLake does not currently use this for source renames. + +### Table-Level Changes + +**New table added at source:** Newly detected source tables appear in the OLake UI, and you choose which ones to sync. Tables not selected are ignored, and pipelines for existing tables run as usual. Once a new table is enabled, OLake applies whichever sync mode is configured, and initial full loads run in parallel. + +**Table renamed at source** When a table is renamed at the source, OLake treats it as a new table, so it does not carry the old table's identity over to the new name. A new table is created at the destination, and the old one stays in place with its historical data. Since the renamed table is treated as a new stream, no manual action is needed. It syncs from scratch automatically, just like any newly added stream. + +**Table deleted at source:** No new data is added to the destination table, but existing data and metadata remain queryable, so downstream queries on historical data continue to work. If a table with the same name is later recreated at the source, note that OLake does not currently support reusing that name at the destination: the recreated table's name collides with the existing destination table's name, and the sync fails. Since the original destination table still holds the deleted table's historical data, give the recreated table a different name. That preserves the old history and avoids the collision. + +### Schema-less Sources (MongoDB, Kafka JSON) + +Some sources have no fixed schema at all. MongoDB documents in the same collection can have entirely different fields, and JSON messages on a Kafka topic carry no enforced structure either. For these sources, OLake cannot read a fixed schema at discovery time the way it can for PostgreSQL or MySQL. Instead it infers the schema from the records themselves, establishing an initial structure from the first records it reads and then evolving it as new fields appear. New keys are added to the destination schema as they appear, and removed keys stop showing up in new rows, the same add and drop behavior as any Iceberg column. + +When normalization is turned on for these schema-less sources, OLake performs Level-0 flattening, expanding top-level nested fields into their own columns, and handles the resulting schema evolution automatically. New and changed fields are picked up from the records and written per the Iceberg v2 spec, with no manual intervention. + +### Applied Automatically: Widening Promotions + +OLake follows Iceberg's type promotion rules directly. These widening changes are applied automatically because Iceberg supports them natively, with no manual intervention: + +| Source type change | Destination behavior | +| --- | --- | +| INT → BIGINT | Column type widened automatically | +| FLOAT → DOUBLE | Column type widened automatically | + +### Handled Without Failing: Compatible Conversions + +Two categories of change that look risky are handled without interrupting the pipeline: + +**Value-fits narrowing.** When the destination column has a wider type than the incoming values, for example a BIGINT column receiving INT values, OLake validates that every incoming value fits within the destination type's range and stores it without error. The pipeline continues without interruption. + +### Sync Fails: Unsupported Changes + +Some changes are unsupported in Iceberg v2 and in OLake. Attempting them fails the sync explicitly with a clear error rather than writing corrupted data. Two common examples: + +| Attempted change | Why it fails | +| --- | --- | +| FLOAT value into an INT column | Would silently lose the fractional part | +| STRING value into an INT / DOUBLE / LONG / FLOAT column | Not all string values are numeric | + +A sync that fails loudly is recoverable. A sync that silently mangles values is the kind of data quality issue that does not surface until someone notices their dashboard numbers are wrong weeks later. + +## Type Mapping Reference + +This table covers some of the data type changes OLake handles, grouped by outcome, so you can check a specific conversion before it hits your pipeline. All behavior here follows Apache Iceberg v2's type promotion rules as implemented by OLake Go. + +| From | To | Outcome | Why / notes | +| --- | --- | --- | --- | +| INT | BIGINT (LONG) | Applied automatically | Widening promotion; destination expands to the larger range, no data affected | +| FLOAT | DOUBLE | Applied automatically | Widening promotion to higher precision, no data loss | +| BIGINT | INT | Handled without failing | Value-fits narrowing: OLake validates each value fits the destination range, then stores it | + +## Conclusion + +OLake is built for a constantly changing source schema. Safe changes apply on their own, dropped columns are kept so nothing downstream breaks, and the sync stops only for the handful of type changes that would genuinely corrupt data. Apache Iceberg is what makes this cheap, since tracking columns by unique field IDs turns operations that used to require full table rewrites into metadata updates that take milliseconds. + +## FAQ + +**Q1. Does OLake support column renames without losing data?** + +No data is lost, but it is not an in-place rename today. When a source column is renamed, OLake treats it as the old column being retired and a new one added: the old column stays in the destination with its history but stops receiving values, and a new column with the new name starts receiving data. Iceberg itself supports true in-place renames on the same field ID; OLake does not currently use this for source renames. + +**Q2. What exactly happens when a sync hits an unsupported type change today?** + +The sync fails explicitly with a clear error instead of writing corrupted data. Some examples of unsupported changes are a FLOAT value arriving into an INT column and a STRING value arriving into a numeric column. Once the DLQ column feature ships, unsupported values will be routed to the DLQ column instead of stopping the sync. + +**Q3. Why doesn't a new sparse column show up immediately at the destination?** + +Iceberg stores data in columnar Parquet format. A column with no non-null values anywhere in the current sync has nothing to write. OLake waits until at least one row has a real value before creating the column in the destination. + +**Q4. Does schema evolution work the same way across all supported sources?** + +Destination behavior is the same across sources because everything goes through the Iceberg writer. What differs is how the initial schema is built: structured sources like PostgreSQL and MySQL expose a fixed catalog schema at discover time, while schema-less sources like MongoDB and Kafka JSON infer structure from sampled records. During sync, new fields and type changes are picked up from the records themselves in both cases. + +**Q5. Can I recover a dropped column at the destination?** + +OLake does not automatically drop destination columns when a source column is dropped; it leaves the column in place so downstream consumers that still depend on it are not broken, and old snapshots remain queryable. If you want the column removed from the Iceberg table, that is a manual operation. + +**Q6. Does schema evolution add latency to syncs?** + +Schema evolution in Iceberg is a metadata-only operation; no data files get rewritten. The overhead is the time to update the table's schema metadata, which is milliseconds regardless of table size, and OLake only does this work when a change is actually detected. + +**Q7. What happens when a new table appears at the source?** + +It is detected on the next scheduled run and listed in the OLake UI, but nothing syncs until you explicitly enable it for the job. Once enabled, OLake creates the destination table and starts syncing under your configured sync mode, with the initial full load running in parallel. + + diff --git a/blog/authors.yml b/blog/authors.yml index 396aae8e7..0cfdbedd2 100644 --- a/blog/authors.yml +++ b/blog/authors.yml @@ -153,3 +153,21 @@ anshika: email: hello@olake.io socials: linkedin: anshika + +siddharth: + page: true + name: Siddharth Chevella + title: OLake Maintainer + image_url: /img/authors/siddharth.webp + email: siddharth@olake.io + socials: + linkedin: siddharth-ch05 + +shuva: + page: true + name: Shuva Jyoti Kar + title: Senior Principal Engineer + image_url: /img/authors/shuva-jyoti-kar.webp + email: shuva.jyoti.kar.87@gmail.com + socials: + linkedin: shuva-jyoti-kar diff --git a/blog/tags.yml b/blog/tags.yml index b83ed1c9f..95ba382e1 100644 --- a/blog/tags.yml +++ b/blog/tags.yml @@ -124,9 +124,9 @@ analytics: description: 'Blogs on the topic Data Analytics' amoro: - label: 'Apache Amoro' + label: 'Apache Amoro™' permalink: '/amoro' - description: 'Blogs on the topic Apache Amoro' + description: 'Blogs on the topic Apache Amoro™' ml: label: 'Machine Learning' @@ -353,3 +353,92 @@ metrics: permalink: '/metrics' description: 'Blogs on the topic Data Metrics and Performance Monitoring' +fusion: + label: 'OLake Fusion' + permalink: '/fusion' + description: 'Blogs on the topic OLake Fusion for Iceberg table maintenance' + +compaction: + label: 'Compaction' + permalink: '/compaction' + description: 'Blogs on the topic Iceberg table compaction' + +optimization: + label: 'Optimization' + permalink: '/optimization' + description: 'Blogs on the topic data and query optimization' + +iceberg-tables: + label: 'Iceberg Tables' + permalink: '/iceberg-tables' + description: 'Blogs on the topic Apache Iceberg table management' + +iceberg-maintenance: + label: 'Iceberg Maintenance' + permalink: '/iceberg-maintenance' + description: 'Blogs on the topic Apache Iceberg table maintenance' + +small-files: + label: 'Small Files' + permalink: '/small-files' + description: 'Blogs on the topic small files problem in data lakehouses' + +binpack-compaction: + label: 'Binpack Compaction' + permalink: '/binpack-compaction' + description: 'Blogs on Binpack Compaction' + +sort-compaction: + label: 'Sort Compaction' + permalink: '/sort-compaction' + description: 'Blogs on Sort Compaction' + +manifest-rewrite: + label: 'Manifest Rewrite' + permalink: '/manifest-rewrite' + description: 'Blogs on Manifest Rewrite' + +metadata-optimization: + label: 'Metadata Optimization' + permalink: '/metadata-optimization' + description: 'Blogs on Metadata Optimization' + +tpch: + label: 'TPC-H' + permalink: '/tpch' + description: 'Blogs on TPC-H benchmarks' + +benchmark: + label: 'Benchmark' + permalink: '/benchmark' + description: 'Blogs on benchmarking' + +spark: + label: 'Apache Spark' + permalink: '/spark' + description: 'Blogs on Apache Spark' + +agentic-ai: + label: 'Agentic AI' + permalink: '/agentic-ai' + description: 'Blogs on Agentic AI' + +google-cloud-lakehouse: + label: 'Google Cloud Lakehouse' + permalink: '/google-cloud-lakehouse' + description: 'Blogs on Google Cloud Lakehouse' + +mcp: + label: 'MCP' + permalink: '/mcp' + description: 'Blogs on Model Context Protocol (MCP)' + +row-lineage: + label: 'Row Lineage' + permalink: '/row-lineage' + description: 'Blogs on the topic Row Lineage in data lakehouse formats' + +v3: + label: 'Iceberg v3' + permalink: '/v3' + description: 'Blogs on the topic Apache Iceberg v3 format' diff --git a/customer-stories/2026-06-14-xeno-aws-dms-alternative-mysql-cdc.mdx b/customer-stories/2026-06-14-xeno-aws-dms-alternative-mysql-cdc.mdx new file mode 100644 index 000000000..08cb8a592 --- /dev/null +++ b/customer-stories/2026-06-14-xeno-aws-dms-alternative-mysql-cdc.mdx @@ -0,0 +1,156 @@ +--- +title: "Zero Pipeline Failures, 50% Faster Loads: How Xeno Rebuilt Their Data Foundation on OLake" +description: "How Xeno replaced AWS DMS with OLake for MySQL CDC, cutting full-load time nearly 50% and ending schema-change pipeline failures, all self-hosted on Kubernetes." +authors: [merlyn] +slug: xeno-aws-dms-alternative-mysql-cdc +tags: [customer-stories, customers, b2b, cdc, mysql, mongodb, kubernetes, real-time, data-sync] +image: /img/customers/xeno/cover-image-xeno.webp +date: 2026-06-14 +--- + +# Zero Pipeline Failures, 50% Faster Loads: How Xeno Rebuilt Their Data Foundation on OLake + +
    + +![Xeno Cover Image](/img/customers/xeno/cover-image-xeno.webp) + +
    + +:::info TL;DR + +Xeno, an AI-powered customer engagement platform for retailers, ran its MySQL CDC replication on Stitch and then AWS DMS, but kept hitting broken replication whenever its schema changed, with recovery often forcing full table reloads. After ruling out Fivetran on row-based pricing, Xeno migrated its MySQL CDC pipelines to OLake, self-hosted on Kubernetes via Helm. + +- Full-load time dropped from approximately 24 hours on AWS DMS to around 13 hours on OLake, a nearly 50% reduction. +- Schema changes no longer break pipelines or trigger full reloads. +- OLake processes 40 to 50 GB daily, with most pipelines syncing hourly and high-priority pipelines every five minutes. +- Every MySQL CDC pipeline that previously ran on DMS now runs on OLake; MongoDB CDC is next. +::: + +
    + +![AWS DMS vs OLake Before After](/img/customers/xeno/aws-dms-vs-olake-before-after.webp) + +
    + +## About Xeno + +Xeno is an AI-powered customer engagement platform built for retailers and consumer brands. Operating across India, Xeno helps hundreds of fashion, beauty, QSR, and retail brands unify customer data, run personalized campaigns, and drive repeat purchases across WhatsApp, SMS, email, Instagram, and Facebook. + +## The Problem: AWS DMS Pipeline That Couldn't Keep Up With Schema Changes + +Xeno, an AI-powered customer engagement platform for retailers, ran its MySQL CDC replication on Stitch and then AWS DMS, but kept hitting broken replication whenever its schema changed, with recovery often forcing full table reloads. After ruling out Fivetran on row-based pricing, Xeno migrated its MySQL CDC pipelines to OLake, self-hosted on Kubernetes via Helm. + +The breaking point wasn't data volume. It was change. + +Every fast-moving engineering team makes routine schema changes: adding a column, modifying a table structure, updating indexes. On AWS DMS, these changes frequently broke replication pipelines. What should have taken minutes of database maintenance turned into hours of incident response. + +The recovery process compounded the pain: + +- **No visibility into root causes.** DMS offered limited logging, leaving engineers to comb through opaque errors with no clear path to resolution. +- **Full table reloads as the only fix.** For larger tables, these reloads stretched past 17 hours, spiking infrastructure costs and leaving downstream dashboards displaying stale data. +- **Workarounds that created more problems.** Duplicate tables, custom views, and manual interventions kept data flowing, but added fragility and complexity to an already brittle system. +The engineering team found themselves spending more time keeping pipelines alive than building the data products the business actually needed. + +
    + +![AWS DMS Change Pipeline Breaks](/img/customers/xeno/aws-dms-schema-change-pipeline-breaks.webp) + +
    + +## Evaluating the Obvious Alternatives + +Fivetran was the natural first alternative to evaluate: managed, reliable, and well-regarded. It addressed the technical problems with AWS DMS. But when the team modeled future data volumes and growth, the row-based pricing became a long-term cost concern. Solving an operational problem while introducing a financial one wasn't the trade-off they were looking for. + +They needed a self-hosted alternative that delivered managed-platform reliability without unpredictable scaling costs, and without handing over control of their infrastructure. + +## From POC to Production: Evaluating OLake as an AWS DMS Alternative + +Xeno wasn't looking for another replication tool. They'd already been through two. What they needed was a platform that could handle the specific failures that had cost them months of engineering time: schema changes breaking pipelines, opaque errors with no recovery path, and full reloads that stretched into days. + +Their requirements going into the OLake evaluation were clear: + +- Reliable CDC replication that survives routine schema changes +- Visibility into pipeline failures and logs +- Elimination of full-table reloads during recovery +- Lower operational overhead and infrastructure costs +Getting started was unexpectedly smooth. Deploying OLake via Helm on Kubernetes took minimal effort, the UI handled most configuration without custom scripting, and the documentation was clear enough that the team moved quickly from setup to testing real workloads. + +The POC wasn't without hiccups. They hit issues with larger partitioned tables and performance tuning early on. But what stood out wasn't that problems arose, it was how fast they were resolved. The OLake team engaged directly over Slack, diagnosed issues, and shipped fixes within one to two business days. For a team that had spent months navigating slow AWS support cycles, this was a meaningful contrast. + +By the end of the POC, Xeno had confidence in both the platform and the team behind it. + +The migration followed a deliberate, staged approach, transitioning MySQL workloads from AWS DMS to OLake incrementally and validating reliability and performance at each step. Today, every MySQL CDC pipeline that previously ran on DMS runs on OLake. + +The results have been immediate and measurable. OLake now processes 40 to 50 GB of data daily, with most pipelines syncing hourly and high-priority pipelines ingesting every five minutes. + +For the engineering team, the impact is straightforward: they've stopped managing pipelines and started building with data. + +## Why OLake Won as an AWS DMS Alternative + +OLake addressed both sides of the equation that other tools couldn't solve simultaneously: operational reliability and cost efficiency. Four things stood out during evaluation. + +**1. Schema evolution that just works.** The core failure mode with DMS, routine schema changes breaking pipelines, was OLake's strongest differentiator. The platform handles DDL and DML changes gracefully, without triggering reloads or requiring manual intervention. For a team whose source systems were constantly evolving, this was the deciding factor. + +**2. Self-hosted, predictable costs.** Unlike row-based SaaS pricing, OLake deploys on Xeno's own Kubernetes environment via Helm. There's no per-row cost, no cross-region transfer markup, and no pricing surprises as data volumes grow, giving the team cost visibility that scaled with the business, not against it. + +**3. Significantly faster full loads.** Full-load performance improved from approximately 24 hours on AWS DMS to around 13 hours on OLake, a nearly 50% reduction. This directly cut recovery times, reduced infrastructure costs during load windows, and meant downstream dashboards waited far less for fresh data. + +**4. Support that moved as fast as they needed.** During the POC, Xeno hit issues with partitioned tables and performance tuning. What mattered wasn't that problems arose, it was that the OLake team diagnosed and resolved them within a few days. For a team that had spent months navigating slow AWS support cycles, this responsiveness was a meaningful signal of what production support would look like. + +
    + +![Schema Evolution AWS DMS VS OLake](/img/customers/xeno/schema-evolution-aws-dms-vs-olake.webp) + +
    + +## In Their Own Words + + + +## What's Next + +With MySQL pipelines stable, Xeno now views OLake as the strategic ingestion layer for their broader data ecosystem, not just a DMS replacement. + +MongoDB migration is the immediate next step. Xeno plans to replicate the MySQL playbook for MongoDB CDC replication. + +Data archival on S3 is the longer-term opportunity, using OLake not just for replication but as a foundation for cost-effective historical storage. + +The question has shifted from "Can OLake handle our workloads?" to "How far can we take it?" + +## Conclusion + +For Xeno's data team, the measure of success was never uptime percentages or load times. It was whether they could trust their pipelines enough to stop thinking about them. + +Three months into production, that trust has been earned. OLake handles the complexity of a constantly evolving data environment so the engineering team doesn't have to, and that shift, more than any individual metric, is what makes it a long-term foundation for Xeno's data strategy. + +## Frequently Asked Questions + +### Q1. Why do schema changes break AWS DMS pipelines? + +AWS DMS maps each source table to a fixed target structure when a task starts, and its CDC has limited support for propagating DDL. So when a column is added, a table is altered, or an index changes, the source no longer matches what the task expects, and replication errors out instead of adapting. Limited logging then makes the failure hard to diagnose, and the usual way back is a full table reload, which stretches past 17 hours on large tables. + +### Q2. Is OLake a good AWS DMS alternative for MySQL CDC? + +For Xeno it was. OLake handles DDL and DML schema changes without triggering full reloads or manual intervention, and every MySQL CDC pipeline that previously ran on AWS DMS now runs on OLake, processing 40 to 50 GB daily. + +### Q3. How does OLake handle schema evolution differently from AWS DMS? + +When a column is added, a table is altered, or an index changes, OLake applies the change and keeps replicating with no full reload and no manual intervention. AWS DMS often interrupts replication on those same routine changes, and recovery typically means a full table reload, which is slow and costly on large tables. + +### Q4. How does OLake pricing compare to Fivetran? + +Fivetran uses row-based pricing, which Xeno found difficult to forecast as data volumes grew. OLake is open-source, self-hosted on the customer's own Kubernetes environment via Helm, with no per-row cost and no cross-region transfer markup, giving more predictable costs at scale. + +### Q5. How much faster are full loads on OLake versus AWS DMS? + +In a benchmark moving 4.0 billion rows from PostgreSQL to Parquet files in S3, OLake completed the full load in 1 hour 59 minutes versus 9 hours 8 minutes on AWS DMS, about 4.6 times faster and roughly 78% less time. OLake sustained around 558,765 rows/sec against 122,000 rows/sec on AWS DMS. + +### Q6. Does OLake support MongoDB CDC as well as MySQL? + +Yes. Xeno started with MySQL CDC and plans MongoDB migration as the immediate next step, applying the same playbook used for its MySQL pipelines. \ No newline at end of file diff --git a/customer-stories/authors.yml b/customer-stories/authors.yml index 401afc267..8c977c26e 100644 --- a/customer-stories/authors.yml +++ b/customer-stories/authors.yml @@ -43,3 +43,11 @@ abhishek-sinha: socials: linkedin: abhisheksinha598 +merlyn: + page: true + name: Merlyn Mathew + title: Product Manager, OLake + image_url: /img/authors/merlyn.webp + bio: Product Manager at OLake, focused on customer stories, product positioning, and helping data teams move to open lakehouse architectures. + socials: + linkedin: merlynm \ No newline at end of file diff --git a/docs/api/olake-ui-api.mdx b/docs/api/olake-ui-api.mdx new file mode 100644 index 000000000..b9e1b5b9e --- /dev/null +++ b/docs/api/olake-ui-api.mdx @@ -0,0 +1,66 @@ +--- +title: "OLake UI API" +description: "Session-authenticated API reference for OLake UI endpoints." +sidebar_label: OLake UI API +--- + +import Heading from "@theme/Heading"; + + + + +This page contains the official Swagger documentation for the OLake UI API, including endpoint reference details, authentication instructions, and guidance for testing requests. + +## Accessing the OLake UI API + +- **OLake UI API**: [http://localhost:8000/swagger](http://localhost:8000/swagger) + +![OLake UI Swagger interface](/img/docs/api/swagger_ui.webp) + +## Authentication + +This API uses **Session-based Authentication** (Cookies). + +To access protected endpoints: + +1. **Login**: Send a POST request to `/login` with your credentials. +2. **Cookie**: A session cookie will be set in your browser automatically upon successful login. +3. **Requests**: Subsequent requests will automatically include this cookie. + +There is no need to manually handle tokens or headers. + +### Using cURL with Cookies + +When using cURL, you must explicitly save and send cookies between requests: + +**Step 1: Login and save the cookie** + +```bash +curl -L 'http://localhost:8000/login' \ + -H 'Content-Type: application/json' \ + -c cookies.txt \ + -d '{"username": "admin", "password": "password"}' +``` + +**Step 2: Use the saved cookie for subsequent requests** + +```bash +curl -L 'http://localhost:8000/api/v1/project/123/jobs' \ + -H 'Accept: */*' \ + -b cookies.txt +``` + +The `-c cookies.txt` flag saves the session cookie, and `-b cookies.txt` sends it with subsequent requests. + +## Response Format + +The API uses standard HTTP response codes to indicate success or failure: + +- **200 OK**: Request was successful. +- **400 Bad Request**: The request was invalid. +- **401 Unauthorized**: Authentication failed or session expired. +- **500 Internal Server Error**: Something went wrong on the server side. diff --git a/docs/benchmarks/ingestion.mdx b/docs/benchmarks/ingestion.mdx index 60e5be884..aac6fc8d3 100644 --- a/docs/benchmarks/ingestion.mdx +++ b/docs/benchmarks/ingestion.mdx @@ -14,7 +14,8 @@ Use the tabs below to view detailed benchmarks per connector. Each tab has a uni { label: 'MongoDB', value: 'mongodb' }, { label: 'MySQL', value: 'mysql' }, { label: 'Oracle', value: 'oracle' }, - { label: 'Kafka', value: 'kafka' } + { label: 'Kafka', value: 'kafka' }, + { label: 'MSSQL', value: 'mssql' } ]} queryString="tab" > @@ -29,18 +30,20 @@ Use the tabs below to view detailed benchmarks per connector. Each tab has a uni - The original repo ingests data into local postgres, so we have modified it to ingest into remote cloud postgres (Azure Flexible DB) [NYC Taxi Data](https://github.com/datazip-inc/nyc-taxi-data-benchmark/tree/remote-postgres) - Total rows **4,008,587,913 rows** including both tables. - The average row size is **144 bytes** for `trips` and **121 bytes** for `fhv_trips`. -- OLake & Debezium were run on **Azure Standard D64ls v5 VM (64 vCPUs, 128 GiB memory)**, other platforms are used as a cloud offering (Fivetran, Estuary, Airbyte) +- OLake Go & Debezium were run on **Azure Standard D64ls v5 VM (64 vCPUs, 128 GiB memory)**, other platforms are used as a cloud offering (Fivetran, Estuary, Airbyte) - Database instance: **Azure Standard_D32ads_v5 (32 vCores, 128 GiB Memory, 51200 max IOPS)** -
    +::::note +We used AWS Glue as Iceberg catalog and AWS S3 as the storage layer on the destination side for this benchmarks. +:::: -_(OLake vs. Popular Data-Movement Tools)_ +_(OLake Go vs. Popular Data-Movement Tools)_ #### 1. Speed Comparison – **Full-Load Performance** | Tool | Rows Synced | Throughput (rows / sec) | Relative to OLake | |----------------------------------------------------|-------------|-------------------------|--------------------| -| **OLake**
    (as of 30th Jan 2026) | **4.01 B** | **5,80,113 RPS** | – | +| **OLake Go**
    (as of 30th Jan 2026) | **4.01 B** | **5,80,113 RPS** | – | | Fivetran
    (as of 30th Apr 2025) | 4.01 B | 46,395 RPS | **12.5 × slower** | | [Debezium](/blog/debezium-vs-olake) (memiiso)
    (as of 30th Apr 2025) | 1.28 B | 14,839 RPS | **39.1 × slower** | | Estuary
    (as of 30th Apr 2025) | 0.34 B | 3,982 RPS | **146 × slower** | @@ -49,7 +52,7 @@ _(OLake vs. Popular Data-Movement Tools)_ ¹ _Estuary ran the same 24-hour window but processed a ~10× smaller dataset, so its throughput looks even lower when normalized._ -**Memory usage (OLake)** - `Standard D64ls v5 (64 vcpus, 128 GiB Memory)` +**Memory usage (OLake Go)** - `Standard D64ls v5 (64 vcpus, 128 GiB Memory)` | Memory Stats | Usage (GB) | |--------|----| @@ -57,19 +60,19 @@ _(OLake vs. Popular Data-Movement Tools)_ | Max | 74.95 | | Mean | 60.01 | -> OLake maintains high throughput while keeping memory usage efficient. +> OLake Go maintains high throughput while keeping memory usage efficient. ::::info -1. The time elapsed for all the tools was 24 hours, but OLake and Fivetran were able to process the entire dataset in that time. Airbyte failed with a sync after 7.5 hours, so we only have throughput for the first part of the test. +1. The time elapsed for all the tools was 24 hours, but OLake Go and Fivetran were able to process the entire dataset in that time. Airbyte failed with a sync after 7.5 hours, so we only have throughput for the first part of the test. :::: -**Key takeaway:** **OLake** now delivers upto **12.5x faster bulk-load** performance than **Fivetran**, while outpacing every other open-source alternative by **35x** to over **1000x**. +**Key takeaway:** **OLake Go** now delivers upto **12.5x faster bulk-load** performance than **Fivetran**, while outpacing every other open-source alternative by **35x** to over **1000x**. #### 2. Speed Comparison – **[Change-Data-Capture (CDC)](/blog/how-to-set-up-postgresql-cdc-on-aws-rds)** | Tool | CDC Window | Throughput (rows / sec) | Relative to OLake | | ------------------ | -----------: | ----------------------: | ----------------- | -| **OLake**
    (as of 30th Jan 2026) | **15 min** | **55,555 RPS** | – | +| **OLake Go**
    (as of 30th Jan 2026) | **15 min** | **55,555 RPS** | – | | Fivetran
    (as of 30th Apr 2025) | 31 min | 26,910 RPS | **2 × slower** | | Debezium (memiiso)
    (as of 30th Apr 2025) | 60 min | 13,808 RPS | **4 × slower** | | Estuary
    (as of 30th Apr 2025) | 4.5 h | 3,085 RPS | **18 × slower** | @@ -79,13 +82,13 @@ _(OLake vs. Popular Data-Movement Tools)_ The rows synced in the CDC test were the same 50 million changes that OLake processed in 15 minutes. The other tools were tested on the same dataset, but they had different CDC windows (timings). :::: -**Key takeaway:** For incremental workloads OLake leads the pack, moving 50 million PostgreSQL changes into Iceberg **106 % faster than Fivetran** and **10-95× faster than other OSS connectors**. +**Key takeaway:** For incremental workloads OLake Go leads the pack, moving 50 million PostgreSQL changes into Iceberg **106 % faster than Fivetran** and **4-95× faster than other OSS connectors**. #### 3. Cost Comparison (Vendor List Prices) | Tool | Scenario | Spend (USD) | Rows Synced | | ------------- | --------------- | ------------------------------------------------------------------------------------: | -----------: | -| **OLake** | Full Load / CDC | Cost of a `Standard D64ls v5 (64 vcpus, 128 GiB memory)` running for 1.91 hours **< $ 6** | 4.01 B / 50M | +| **OLake Go** | Full Load / CDC | Cost of a `Standard D64ls v5 (64 vcpus, 128 GiB memory)` running for 1.91 hours **< $ 6** | 4.01 B / 50M | | Fivetran | Full Load | $ 0 (free full sync) | 4.01 B | | Estuary | Full Load | $ 1,668 | 0.34 B | | Airbyte Cloud | Full Load | $ 5,560 | 12.7 M | @@ -93,7 +96,7 @@ The rows synced in the CDC test were the same 50 million changes that OLake proc | Estuary | CDC | $ 17.63 | 50 M | | Airbyte Cloud | CDC | $ 148.95 | 50 M | -- **OLake** is open-source and can be deployed on your own Kubernetes cluster or cloud VMs; you pay only for the compute and storage you provision. +- **OLake Go** is open-source and can be deployed on your own Kubernetes cluster or cloud VMs; you pay only for the compute and storage you provision. ### Dataset and Table Schemas Please refer to [this GitHub repository](https://github.com/datazip-inc/nyc-taxi-data-benchmark/tree/remote-postgres) for the dataset we used to conduct these benchmarks. @@ -174,11 +177,7 @@ CREATE TABLE fhv_trips ( ); ``` -::::note -We used AWS Glue as Iceberg catalog and AWS S3 as the storage layer on the destination side for this benchmarks. -:::: - -> **Bottom line:** If you need to land terabytes of [PostgreSQL](/blog/how-to-set-up-postgres-apache-iceberg) data into Apache Iceberg quickly—and keep it continually up-to-date—OLake delivers enterprise-grade speed without the enterprise-grade bill. +> **Bottom line:** If you need to land terabytes of [PostgreSQL](/blog/how-to-set-up-postgres-apache-iceberg) data into Apache Iceberg quickly—and keep it continually up-to-date—OLake Go delivers enterprise-grade speed without the enterprise-grade bill. @@ -186,7 +185,7 @@ We used AWS Glue as Iceberg catalog and AWS S3 as the storage layer on the desti ### Oracle → Apache Iceberg Connector Benchmark -Oracle data powers your business—so migrations should be fast and seamless. OLake helps you move massive Oracle datasets to Apache Iceberg at high speed, with predictable performance and no vendor lock-in. +Oracle data powers your business—so migrations should be fast and seamless. OLake Go helps you move massive Oracle datasets to Apache Iceberg at high speed, with predictable performance and no vendor lock-in. **Benchmark Environment** @@ -194,20 +193,22 @@ Oracle data powers your business—so migrations should be fast and seamless. OL - Since the original repo supports only PostgreSQL, we first ingested the **NYC Taxi Data** in the cloud PostgreSQL database (Azure Flexible DB), and then transferred the tables from there to our Oracle database. - Total rows **4,008,587,913 rows** including both tables. - The average row size is **144 bytes** for `trips` and **121 bytes** for `fhv_trips`. -- OLake was run on **Azure Standard D64ls v5 VM (64 vCPUs, 128 GiB memory)** +- OLake Go was run on **Azure Standard D64ls v5 VM (64 vCPUs, 128 GiB memory)** - Database instance: **AWS RDS db.r6i.4xlarge (8 vCPUs, 32 GiB Memory)** -
    +:::::note +We used AWS Glue as Iceberg catalog and AWS S3 as the storage layer on the destination side for this benchmarks. +::::: -_(OLake's Performance with Oracle Database)_ +_(OLake Go's Performance with Oracle Database)_ #### 1. Speed Test – **Full-Load Performance** | Tool | Rows Synced | Throughput (rows / sec) | |--------------------------------------|-------------|-------------------------| -| **OLake**
    (as of 30th Jan 2026) | **4.01 B** | **5,26,337 RPS** | +| **OLake Go**
    (as of 30th Jan 2026) | **4.01 B** | **5,26,337 RPS** | -**Memory usage (OLake)** - `Standard D64ls v5 (64 vCPUs, 128 GiB Memory)` +**Memory usage (OLake Go)** - `Standard D64ls v5 (64 vCPUs, 128 GiB Memory)` | Memory Stats | Usage (GB) | |--------|----| @@ -215,13 +216,13 @@ _(OLake's Performance with Oracle Database)_ | Max | 93.16 | | Mean | 73.18 | -> OLake maintains high throughput while keeping memory usage efficient. +> OLake Go maintains high throughput while keeping memory usage efficient. #### 2. Cost at a Glance | Tool | Scenario | Spend (USD) | Rows Synced | | ------------- | --------------- | ------------------------------------------------------------------------------------: | -----------: | -| **OLake** | Full Load | Cost of a `Standard D64ls v5 (64 vCPUs, 128 GiB memory)` running for 2.11 hours **< $ 6** | 4.01 B | +| **OLake Go** | Full Load | Cost of a `Standard D64ls v5 (64 vCPUs, 128 GiB memory)` running for 2.11 hours **< $ 6** | 4.01 B | ### Dataset and Table Schemas Please refer to [this GitHub repository](https://github.com/datazip-inc/nyc-taxi-data-benchmark/tree/remote-postgres) for the dataset we used to conduct these benchmarks. @@ -302,11 +303,7 @@ CREATE TABLE fhv_trips ( ); ``` -:::::note -We used AWS Glue as Iceberg catalog and AWS S3 as the storage layer on the destination side for this benchmarks. -::::: - -> **Bottom line:** If you need to land terabytes of Oracle data into Apache Iceberg quickly—OLake delivers enterprise-grade speed without the enterprise-grade bill. +> **Bottom line:** If you need to land terabytes of Oracle data into Apache Iceberg quickly—OLake Go delivers enterprise-grade speed without the enterprise-grade bill. @@ -320,21 +317,23 @@ We used AWS Glue as Iceberg catalog and AWS S3 as the storage layer on the desti - This benchmark uses standard Twitter data `tweets` table. - Total **233,955,436 rows** for the `tweets` tables. - The average row size for `tweets` table is **3655 bytes** -- OLake was run on **AWS EC2 c6i.16xlarge (64 vCPUs, 128 GiB memory)** +- OLake Go was run on **Standard D64ls v5 (64 vCPUs, 128 GiB Memory)** - Database instance: **3 x Standard D16as v5 (16 vcpus, 64 GiB memory)** -
    +:::::note +We used AWS Glue as Iceberg catalog and AWS S3 as the storage layer on the destination side for this benchmarks. +::::: -_(OLake's Performance with MongoDB Database)_ +_(OLake Go vs. Popular Data-Movement Tools)_ #### 1. Speed Test – **Full-Load Performance** | Tool | Rows Synced | Throughput (rows / sec) | Relative to OLake | |--------------------------------------|-------------|-------------------------|--------------------| -| **OLake**
    (as of 5th Feb 2026) | **233 M** | **37,879 RPS** | – | +| **OLake Go**
    (as of 5th Feb 2026) | **233 M** | **37,879 RPS** | – | | **Fivetran**
    (as of 15th Feb 2026) | **233 M** | **14,997 RPS** | **2.5 × slower** | -**Memory usage (OLake)** - `Standard D64ls v5 (64 vCPUs, 128 GiB Memory)` +**Memory usage (OLake Go)** - `Standard D64ls v5 (64 vCPUs, 128 GiB Memory)` | Memory Stats | Usage (GB) | |--------|----| @@ -342,20 +341,20 @@ _(OLake's Performance with MongoDB Database)_ | Max | 112.22 | | Mean | 71.18 | -> OLake maintains high throughput while keeping memory usage efficient. +> OLake Go maintains high throughput while keeping memory usage efficient. #### 2. Speed Comparison – **Change-Data-Capture (CDC)** | Tool | CDC Window | Throughput (rows / sec) | Relative to OLake | | ------------------ | -----------: | ----------------------: | ----------------- | -| **OLake** | **38.96 mins** | **10,692 RPS** | – | +| **OLake Go** | **38.96 mins** | **10,692 RPS** | – | | **Fivetran** | **72 mins** | **5,787 RPS** | **1.85 × slower** | #### 3. Cost at a Glance | Tool | Scenario | Spend (USD) | Rows Synced | | ------------- | --------------- | ------------------------------------------------------------------------------------: | -----------: | -| **OLake** | Full Load | Cost of a `Standard D64ls v5 (64 vCPUs, 128 GiB memory)` running for 1.71 hours **< $ 5** | 233 M | +| **OLake Go** | Full Load | Cost of a `Standard D64ls v5 (64 vCPUs, 128 GiB memory)` running for 1.71 hours **< $ 5** | 233 M | ### Dataset and Table Schemas @@ -407,10 +406,6 @@ CREATE TABLE trips ( ); ``` -:::::note -We used AWS Glue as Iceberg catalog and AWS S3 as the storage layer on the destination side for this benchmarks. -::::: - @@ -423,50 +418,52 @@ We used AWS Glue as Iceberg catalog and AWS S3 as the storage layer on the desti - Since the original repo supports only PostgreSQL, we first ingested the NYC Taxi Data in the cloud PostgreSQL database (Azure Flexible DB), and then transferred the tables from there to our MySQL database. - Total rows **4,001,991,536 rows** including both tables. - The average row size is **144 bytes** for `trips` and **121 bytes** for `fhv_trips`. -- OLake & Debezium were run on **AWS EC2 c6i.16xlarge (64 vCPUs, 128 GiB memory)** +- OLake Go was run on **Azure Standard D64ls v5 VM (64 vCPUs, 128 GiB memory)** - Database instance: **Azure Standard D32as v6 (32 vCPUs, 128 GiB Memory)** -
    +:::::note +We used AWS Glue as Iceberg catalog and AWS S3 as the storage layer on the destination side for this benchmarks. +::::: -_(OLake vs. Popular Data-Movement Tool)_ +_(OLake Go vs. Popular Data-Movement Tool)_ #### 1. Speed Comparison – **Full-Load Performance** | Tool | Rows Synced | Throughput (rows / sec) | Relative to OLake | |----------------------------------------------------|-------------|-------------------------|--------------------| -| **OLake**
    (as of 14th Nov 2025) | **4.0 B** | **3,38,005 RPS** | – | -| Fivetran
    (as of 14th Nov 2025) | 4.0 B | 119,106 RPS | **2.83 × slower** | +| **OLake Go**
    (as of 30th May 2026) | **4.0 B** | **1,39,773 RPS** | – | +| Fivetran
    (as of 30th May 2026) | 4.0 B | 73,087 RPS | **1.91 × slower** | -**Memory usage (OLake)** - `c6i.16xlarge (64 vCPUs, 128 GiB memory)` +**Memory usage (OLake Go)** - `Standard D64ls v5 (64 vcpus, 128 GiB Memory)` | Memory Stats | Usage (GB) | |--------|----| | Min | 3.24 | -| Max | 75.1 | -| Mean | 48.95 | +| Max | 83.6 | +| Mean | 53.25 | -> OLake maintains high throughput while keeping memory usage efficient. +> OLake Go maintains high throughput while keeping memory usage efficient. #### 2. Speed Comparison – **Change-Data-Capture (CDC)** | Tool | CDC Window | Throughput (rows / sec) | Relative to OLake | | ------------------ | -----------: | ----------------------: | ----------------- | -| **OLake** | **16.06 min** | **51,867 RPS** | – | -| Fivetran | 29.86 min | 27,901 RPS | **1.85 × slower** | +| **OLake Go** | **13.9 min** | **59,951 RPS** | – | +| Fivetran | 21.15 min | 39,374 RPS | **1.52 × slower** | -**Key takeaway:** For incremental workloads OLake leads the pack, moving 50 million MySQL changes into Iceberg **85.9 % faster than Fivetran** +**Key takeaway:** For incremental workloads OLake Go leads the pack, moving 50 million MySQL changes into Iceberg **52.3 % faster than Fivetran** #### 3. Cost Comparison (Vendor List Prices) | Tool | Scenario | Spend (USD) | Rows Synced | | ------------- | --------------- | ------------------------------------------------------------------------------------: | -----------: | -| **OLake** | Full Load / CDC | Cost of a `c6i.16xlarge (64 vCPUs, 128 GiB memory)` running for 3.3 hours **< $ 11** | 4.0 B / 50 M | +| **OLake Go** | Full Load / CDC | Cost of a `Standard D64ls v5 (64 vcpus, 128 GiB Memory)` running for 7.95 hours **< $ 22** | 4.0 B / 50 M | | Fivetran | Full Load | $ 0 (free full sync) | 4.0 B | | Fivetran | CDC | $ 2, 375.80 | 50 M | -- **OLake** is open-source and can be deployed on your own Kubernetes cluster or cloud VMs; you pay only for the compute and storage you provision. +- **OLake Go** is open-source and can be deployed on your own Kubernetes cluster or cloud VMs; you pay only for the compute and storage you provision. ### Dataset and Table Schemas Please refer to [this GitHub repository](https://github.com/datazip-inc/nyc-taxi-data-benchmark/tree/remote-postgres) for the dataset we used to conduct these benchmarks. @@ -546,11 +543,8 @@ CREATE TABLE fhv_trips ( PRIMARY KEY (id) ); ``` -:::::note -We used AWS Glue as Iceberg catalog and AWS S3 as the storage layer on the destination side for this benchmarks. -::::: -> **Bottom line:** If you need to land terabytes of MySQL data into Apache Iceberg quickly—OLake delivers enterprise-grade speed without the enterprise-grade bill. +> **Bottom line:** If you need to land terabytes of MySQL data into Apache Iceberg quickly—OLake Go delivers enterprise-grade speed without the enterprise-grade bill.     @@ -567,41 +561,42 @@ We used AWS Glue as Iceberg catalog and AWS S3 as the storage layer on the desti - The average message size is **144 bytes** (same as the original `trips` table row size). - Kafka Cluster: **AWS MSK [3 broker nodes each m7g.4xlarge (16 vCPUs, 64 GiB Memory)]** - Topic Configuration: **1 topic (`trips`) with 5 partitions** -- OLake was run on **Azure Standard D64ls v6 VM (64 vCPUs, 128 GiB memory)** - -
    +- OLake Go was run on **Azure Standard D64ls v6 VM (64 vCPUs, 128 GiB memory)** +:::::note +We used AWS Glue as Iceberg catalog and AWS S3 as the storage layer on the destination side for this benchmarks. +::::: -_(OLake vs. Popular Data-Movement Tools)_ +_(OLake Go vs. Popular Data-Movement Tools)_ #### 1. Speed Comparison – **Batch-Load Performance** | Tool | Rows Synced | Throughput (rows / sec) | Relative to OLake | |----------------------------------------------------|-------------|-------------------------|--------------------| -| **OLake**
    (as of 27th Feb 2026) | **1.01 B** | **2,09,065 MPS** | – | +| **OLake Go**
    (as of 27th Feb 2026) | **1.01 B** | **2,09,065 MPS** | – | | Flink
    (as of 27th Feb 2026) | 1.01 B | 2,56,410 MPS | **1.23 × faster** | -**Memory usage (OLake vs Flink)** - `Standard D64ls v6 (64 vcpus, 128 GiB Memory)` +**Memory usage (OLake Go vs Flink)** - `Standard D64ls v6 (64 vcpus, 128 GiB Memory)` -| Memory Stats | OLake Usage (GB) | Flink Usage (GB) | +| Memory Stats | OLake Go Usage (GB) | Flink Usage (GB) | |--------|----|----| | Min | 2.55 | 16.34 | | Max | 12.86 | 40.92 | | Mean | 12.04 | 39.89 | -> OLake is more memory-efficient than Flink, using roughly 3× less mean memory usage—making it a lighter option for resource constrained environments. +> OLake Go is more memory-efficient than Flink, using roughly 3× less mean memory usage—making it a lighter option for resource constrained environments. #### 2. Cost Comparison (Vendor List Prices) | Tool | Scenario | Compute (USD) | Brokers (USD) | Total (USD) | Rows Synced | |--------|-------------|---------------|---------------|-------------|-------------| -| OLake | Batch Load | $4.85 `(D64ls v6 running for 1.34 hours)` | $5.92 `(3× m7g.4xlarge running for 1.34 hours)` | **$10.77** | 1.01 B | +| OLake Go | Batch Load | $4.85 `(D64ls v6 running for 1.34 hours)` | $5.92 `(3× m7g.4xlarge running for 1.34 hours)` | **$10.77** | 1.01 B | | Flink | Batch Load | $3.90 `(D64ls v6 running for 1.08 hours)` | $4.77 `(3× m7g.4xlarge running for 1.08 hours)` | **$8.67** | 1.01 B | :::info -Both **OLake** and **Flink** are open-source and can be deployed on your own Kubernetes cluster or cloud VMs; you pay only for the compute and storage you provision. +Both **OLake Go** and **Flink** are open-source and can be deployed on your own Kubernetes cluster or cloud VMs; you pay only for the compute and storage you provision. ::: ### Dataset and Table Schemas @@ -644,10 +639,6 @@ CREATE TABLE trips ( ); ``` -:::::note -We used AWS Glue as Iceberg catalog and AWS S3 as the storage layer on the destination side for this benchmarks. -::::: -
    Flink setup for this benchmark @@ -972,6 +963,138 @@ Verify that Flink is running properly by accessing the Flink Web UI at `http://l
    + + + + +### MSSQL → Apache Iceberg Connector Benchmark + +**Benchmark Environment** + +- This benchmark uses standard [NYC Taxi Data](https://github.com/toddwschneider/nyc-taxi-data) `trips` and `fhv_trips` tables. +- Since the original repo supports only PostgreSQL, we first ingested the NYC Taxi Data in the cloud PostgreSQL database (Azure Flexible DB), and then transferred the tables from there to our MSSQL database. +- Total rows **4,008,587,913 rows** including both tables. +- The average row size is **215 bytes** for `trips` and **177 bytes** for `fhv_trips`. +- OLake Go was run on **GCP Custom N2 VM (64 vCPUs, 128 GiB memory)** +- Database instance: **GCP Standard N2 VM (32 vCPUs, 128 GiB memory)** + +:::::note +We used AWS Glue as Iceberg catalog and AWS S3 as the storage layer on the destination side for this benchmarks. +::::: + +
    + +_(OLake Go vs. Popular Data-Movement Tool)_ + +#### 1. Speed Comparison – **Full-Load Performance** + +| Tool | Rows Synced | Throughput (rows / sec) | Relative to OLake | +|----------------------------------------------------|-------------|-------------------------|--------------------| +| **OLake Go**
    (as of 09th June 2026) | **4.0 B** | **3,45,866 RPS** | – | +| Fivetran
    (as of 09th June 2026) | 4.0 B | 79,982 RPS | **4.32 × slower** | + + +**Memory usage (OLake Go)** - `GCP Custom N2 VM (64 vCPUs, 128 GiB memory)` + +| Memory Stats | Usage (GB) | +|--------|----| +| Min | 4.51 | +| Max | 82.97 | +| Mean | 66.02 | + +> OLake Go maintains high throughput while keeping memory usage efficient. + +#### 2. Cost Comparison (Vendor List Prices) + +| Tool | Scenario | Spend (USD) | Rows Synced | +| ------------- | --------------- | ------------------------------------------------------------------------------------: | -----------: | +| **OLake Go** | Full Load | Cost of a `GCP Custom N2 VM (64 vCPUs, 128 GiB memory)` running for 3.21 hours **< $ 11** | 4.0 B | +| Fivetran | Full Load | $ 0 (free full sync) | 4.0 B | + +- **OLake Go** is open-source and can be deployed on your own Kubernetes cluster or cloud VMs; you pay only for the compute and storage you provision. + +### Dataset and Table Schemas +Please refer to [this GitHub repository](https://github.com/datazip-inc/nyc-taxi-data-benchmark/tree/remote-postgres) for the dataset we used to conduct these benchmarks. + +:::::note +For MSSQL, we performed a full-load of the `trips` and `fhv_trips` tables into Apache Iceberg. +::::: + +### `trips` table + +```sql +CREATE TABLE trips ( + id BIGINT NOT NULL, + cab_type_id INT NULL, + vendor_id INT NULL, + pickup_datetime DATETIME2 NULL, + dropoff_datetime DATETIME2 NULL, + store_and_fwd_flag BIT NULL, + rate_code_id INT NULL, + pickup_longitude DECIMAL NULL, + pickup_latitude DECIMAL NULL, + dropoff_longitude DECIMAL NULL, + dropoff_latitude DECIMAL NULL, + passenger_count INT NULL, + trip_distance DECIMAL NULL, + fare_amount DECIMAL NULL, + extra DECIMAL NULL, + mta_tax DECIMAL NULL, + tip_amount DECIMAL NULL, + tolls_amount DECIMAL NULL, + ehail_fee DECIMAL NULL, + improvement_surcharge DECIMAL NULL, + congestion_surcharge DECIMAL NULL, + airport_fee DECIMAL NULL, + total_amount DECIMAL NULL, + payment_type INT NULL, + trip_type INT NULL, + pickup_nyct2010_gid INT NULL, + dropoff_nyct2010_gid INT NULL, + pickup_location_id INT NULL, + dropoff_location_id INT NULL, + PRIMARY KEY (id) +); +``` + +### `fhv_trips` table + +```sql +CREATE TABLE fhv_trips ( + id BIGINT NOT NULL, + hvfhs_license_num VARCHAR NULL, + dispatching_base_num VARCHAR NULL, + originating_base_num VARCHAR NULL, + request_datetime DATETIME2 NULL, + on_scene_datetime DATETIME2 NULL, + pickup_datetime DATETIME2 NULL, + dropoff_datetime DATETIME2 NULL, + pickup_location_id INT NULL, + dropoff_location_id INT NULL, + trip_miles DECIMAL NULL, + trip_time DECIMAL NULL, + base_passenger_fare DECIMAL NULL, + tolls DECIMAL NULL, + black_car_fund DECIMAL NULL, + sales_tax DECIMAL NULL, + congestion_surcharge DECIMAL NULL, + airport_fee DECIMAL NULL, + tips DECIMAL NULL, + driver_pay DECIMAL NULL, + shared_request BIT NULL, + shared_match BIT NULL, + access_a_ride BIT NULL, + wav_request BIT NULL, + wav_match BIT NULL, + legacy_shared_ride INT NULL, + affiliated_base_num VARCHAR NULL, + PRIMARY KEY (id) +); +``` + +> **Bottom line:** If you need to land terabytes of MSSQL data into Apache Iceberg quickly—OLake Go delivers enterprise-grade speed without the enterprise-grade bill. + +     diff --git a/docs/benchmarks/optimization.mdx b/docs/benchmarks/optimization.mdx deleted file mode 100644 index 945250c7c..000000000 --- a/docs/benchmarks/optimization.mdx +++ /dev/null @@ -1,8 +0,0 @@ ---- -title: Optimization Benchmarks -sidebar_label: Optimization Benchmarks ---- - -# Optimization Benchmarks - -Coming Soon..... \ No newline at end of file diff --git a/docs/community/commands-and-flags.mdx b/docs/community/commands-and-flags.mdx index d4a79205f..799c062dc 100644 --- a/docs/community/commands-and-flags.mdx +++ b/docs/community/commands-and-flags.mdx @@ -5,15 +5,15 @@ sidebar_label: Commands and Flags sidebar_position: 2 --- -# OLake Commands and Flags +# OLake Go Commands and Flags -OLake provides a set of CLI commands, each designed for specific use cases.
    +OLake Go provides a set of CLI commands, each designed for specific use cases.
    **Commands can be executed in three ways:** - Using the `build.sh` script - Running the generated binary -- Through the [OLake Docker CLI](/docs/install/docker-cli). +- Through the [OLake Go Docker CLI](/docs/install/docker-cli). ```bash title="build.sh command" ./build.sh driver-[SOURCE-TYPE] [COMMAND] [FLAG] @@ -23,7 +23,7 @@ The `./build.sh` script is a Unix shell script. It runs natively on Linux and ma For Windows: - Use Git Bash, WSL (Windows Subsystem for Linux), or another Unix-like shell. -- Alternatively, use the OLake Docker CLI, which works consistently across platforms. +- Alternatively, use the OLake Go Docker CLI, which works consistently across platforms. ::: **Explanation of placeholders:** @@ -105,11 +105,19 @@ All of these flags need to be specified: ### 5. Clear Destination ``` -./build.sh driver-[SOURCE-TYPE] [COMMAND] --clear-destination +./build.sh driver-[SOURCE-TYPE] clear-destination [FLAG] ``` -**Description:** +#### Description: - Clears data in the destination, only for the selected streams defined in `streams.json`. -- Resets the state file for those streams. + +#### Required flags: +All of these flags need to be specified: +- [**`--streams`**](/docs/community/commands-and-flags#3-streams) → Specifies path to the `streams.json` file (produced by the discover command). +- [**`--destination`**](/docs/community/commands-and-flags#4-destination) → Specifies path to the destination configuration file. + +#### Optional flags: +- [**`--state`**](/docs/community/commands-and-flags#5-state) → Used to reset the state file for selected streams. + --- @@ -309,21 +317,21 @@ MongoDB stream state includes a `_data` field that stores the resume token: ``` **Description:** - Used with the [`spec`](#2-spec) command to generate JSON Schema and UI Schema for the specified destination. -- `TYPE_OF_DESTINATION` can be any OLake supported destination, for example: iceberg or parquet. +- `TYPE_OF_DESTINATION` can be any OLake Go supported destination, for example: iceberg or parquet. ### 8. Decryption of configuration files ``` ./build.sh driver-[SOURCE-TYPE] [COMMAND] --encryption-key [DECRYPTION_KEY] ``` **Description:** -- Provides a key for OLake to decrypt encrypted configuration files during execution. +- Provides a key for OLake Go to decrypt encrypted configuration files during execution. - Supported values include KMS keys, UUIDs, or custom strings. - The flag must follow the encrypted file in the command.
    Example: ``` ./build.sh driver-mysql check config [PATH_TO_SOURCE_CONFIG_FILE] --encryption-key hello-world ``` - In this case, if the source config file is encrypted, OLake uses the provided key (`hello-world`) to decrypt and parse it. + In this case, if the source config file is encrypted, OLake Go uses the provided key (`hello-world`) to decrypt and parse it. ### 9. No Save diff --git a/docs/community/contributing.mdx b/docs/community/contributing.mdx index 888dbf192..3a4b3ac46 100644 --- a/docs/community/contributing.mdx +++ b/docs/community/contributing.mdx @@ -1,16 +1,16 @@ --- title: "Contributing Guide & Developer Setup | OLake Open Source" description: Contributing to OLake guide -sidebar_label: Contribute to OLake +sidebar_label: Contribute to OLake Go sidebar_position: 1 --- -# Contributing to OLake +# Contributing to OLake Go Hi there! We're thrilled that you'd like to contribute to this project, thank you for your interest. Whether it's a bug report, new feature, correction, or additional documentation, we greatly value feedback and contributions from our community. If you are a student contributor and new to data engineering domain, finish the below materials and then proceed to pick up issues: -- [General Terminologies](../../understanding/terminologies/general) -- [OLake Terminologies](../../understanding/terminologies/olake) +- [General Terminologies](../../understanding/terminologies/general/?terminology-type=general) +- [OLake Go Terminologies](../../understanding/terminologies/general/?terminology-type=olake) Core contributors and community members engage in the following public channels: @@ -20,12 +20,12 @@ Core contributors and community members engage in the following public channels: ## Orientation -Here's a list of repositories that contain OLake-related packages: +Here's a list of repositories that contain OLake Go-related packages: -- [datazip-inc/olake](https://github.com/datazip-inc/olake) is the main repository containing the core OLake replication engine and CLI tools -- [datazip-inc/olake-ui](https://github.com/datazip-inc/olake-frontend) contains the web-based user interface for managing OLake jobs and configurations +- [datazip-inc/olake](https://github.com/datazip-inc/olake) is the main repository containing the core OLake Go replication engine and CLI tools +- [datazip-inc/olake-ui](https://github.com/datazip-inc/olake-frontend) contains the web-based user interface for managing OLake Go jobs and configurations - [datazip-inc/olake-docs](https://github.com/datazip-inc/olake-docs) contains the documentation website and guides -- [datazip-inc/olake-helm](https://github.com/datazip-inc/olake-helm) contains official Helm charts for deploying OLake on Kubernetes. +- [datazip-inc/olake-helm](https://github.com/datazip-inc/olake-helm) contains official Helm charts for deploying OLake Go on Kubernetes. ## Types of Contributions @@ -43,7 +43,7 @@ Before making any significant changes and before filing a new issue, please chec The best way to report a bug is to file an issue on GitHub. Please include: - Your operating system name and version -- OLake version +- OLake Go version - Source and destination database details - Detailed steps to reproduce the bug - Any modifications you have made relevant to the bug @@ -125,7 +125,7 @@ If you’d like to contribute code or docs, check existing GitHub issues and pic ### A good start will be to explore: **Issues to work on:** -- [first good issues](https://github.com/datazip-inc/olake/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22good%20first%20issue%22) +- [good first issues](https://github.com/datazip-inc/olake/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22good%20first%20issue%22) - [good second issues](https://github.com/datazip-inc/olake/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22good%20second%20issue%22) **Getting started guides:** @@ -136,21 +136,21 @@ See the issues and if its looks interesting enough to you, comment on it and one ## Types of contributors -OLake recognizes two primary contributor roles, each with distinct responsibilities and privileges within the project: +OLake Go recognizes two primary contributor roles, each with distinct responsibilities and privileges within the project: ### Project Maintainers -Project Maintainers guide OLake by sorting and prioritizing issues, reviewing and merging pull requests in all repositories, and coordinating releases. +Project Maintainers guide OLake Go by sorting and prioritizing issues, reviewing and merging pull requests in all repositories, and coordinating releases. ### Contributors -Contributors include anyone who engages with OLake by submitting code, reporting bugs, writing documentation, proposing improvements, or helping on community channels. All contributions—large or small—are welcome. +Contributors include anyone who engages with OLake Go by submitting code, reporting bugs, writing documentation, proposing improvements, or helping on community channels. All contributions—large or small—are welcome. ## Goodies We know how much you love swags and stickers, for each PR merged (again, not typo fixes or documentation fixes) we will send over a Tshirt and stickers to you. Take a sneak peak below. -**Note:** PR qualifying for goodies is subject to value it adds and the OLake team approval. +**Note:** PR qualifying for goodies is subject to value it adds and the OLake Go team approval.
    {/* On mobile: single column; on md and above: three columns */} @@ -178,5 +178,5 @@ We know how much you love swags and stickers, for each PR merged (again, not typ :::info -Refer [here](https://olake.io/docs/writers/iceberg/troubleshooting#vscode-debug-configuration) for more details on how to debug OLake's Iceberg writer using VSCode. +Refer [here](https://olake.io/docs/writers/iceberg/troubleshooting#vscode-debug-configuration) for more details on how to debug OLake Go's Iceberg writer using VSCode. ::: diff --git a/docs/community/issues-and-prs.mdx b/docs/community/issues-and-prs.mdx index 0091ca613..276b7dd60 100644 --- a/docs/community/issues-and-prs.mdx +++ b/docs/community/issues-and-prs.mdx @@ -12,7 +12,7 @@ Please read through this document before submitting any issues or pull requests :::warning -- All contributions must target the OLake [staging branch](https://github.com/datazip-inc/olake/tree/staging). Create feature branches from staging and open PRs against staging. +- All contributions must target the OLake Go [staging branch](https://github.com/datazip-inc/olake/tree/staging). Create feature branches from staging and open PRs against staging. ::: @@ -117,6 +117,18 @@ Next, take a pull (in case any new code has been added, this will help you avoid - Describe how you tested your change. - Check the Preview tab to make sure the Markdown is correctly rendered and that all tags and references are linked. If not, go back and edit the Markdown. +### 5. Attach a validation video + +For every pull request, please record and attach a short video demo of your change. This helps reviewers quickly understand the behavior and verify that it works end-to-end with OLake Go, and also ensures contributors actually run OLake Go with their changes. + +Your video should: + +- Quickly walk through the relevant code changes. +- Execute a sync using OLake Go that exercises those changes. +- Highlight logs or UI views confirming that the run completes successfully. + +**PRs without a validation video will not be reviewed!** + ### Request Review - Once your PR is ready, remove the "[WIP]" from the title and/or change it from a draft PR to a regular PR. - If a specific reviewer is not assigned automatically, please request a review from the project maintainer. diff --git a/docs/community/learning-modules.mdx b/docs/community/learning-modules.mdx index 5555c195e..3c909a9ab 100644 --- a/docs/community/learning-modules.mdx +++ b/docs/community/learning-modules.mdx @@ -7,17 +7,17 @@ sidebar_position: 99 ## Learning Modules -Before jumping in and setting up a development environment and contributing to OLake, if you have limited knowledge of the required technologies and tools, this is the section where you should learn everything important that's required. This page hosts **learning modules** covering essential concepts and technologies that will help you get started with OLake development. +Before jumping in and setting up a development environment and contributing to OLake Go, if you have limited knowledge of the required technologies and tools, this is the section where you should learn everything important that's required. This page hosts **learning modules** covering essential concepts and technologies that will help you get started with OLake Go development. --- ## Required Learning Modules -These modules are **mandatory** and should be completed before you start contributing to OLake. They are listed in the **recommended order of priority**. +These modules are **mandatory** and should be completed before you start contributing to OLake Go. They are listed in the **recommended order of priority**. ### 1. Golang (Go) -OLake is primarily written in **Golang**, so having a solid grasp of Go makes it much easier to understand the codebase, contribute features, and debug issues. +OLake Go is primarily written in **Golang**, so having a solid grasp of Go makes it much easier to understand the codebase, contribute features, and debug issues. - **If you prefer videos**: Watch this YouTube playlist for a step‑by‑step Go learning path (from basics to advanced topics): - [Golang Video Playlist](https://www.youtube.com/watch?v=yyUHQIec83I) @@ -32,8 +32,8 @@ Focus on: ### 2. Docker -OLake heavily uses **Docker** for local development and the Docker CLI workflow. Understanding Docker helps you: -- Run OLake containers correctly +OLake Go heavily uses **Docker** for local development and the Docker CLI workflow. Understanding Docker helps you: +- Run OLake Go containers correctly - Understand volumes, networks, and images used in the examples - Debug environment‑related issues @@ -44,7 +44,7 @@ Recommended video: ### 3. ETL and ELT -OLake is fundamentally a **data migration / ETL tool**, so understanding ETL/ELT concepts is critical: +OLake Go is fundamentally a **data migration / ETL tool**, so understanding ETL/ELT concepts is critical: - What it means to **extract**, **transform**, and **load** data - How data flows from operational systems into warehouses or lakehouses - Why transformation might happen before (ETL) or after loading (ELT) @@ -56,8 +56,8 @@ Recommended reading: ### 4. Apache Iceberg -OLake writes data into **Apache Iceberg** tables, which is the core table format powering many OLake destinations. Understanding Iceberg helps you reason about: -- How OLake writes snapshots, data files, and metadata +OLake Go writes data into **Apache Iceberg** tables, which is the core table format powering many OLake Go destinations. Understanding Iceberg helps you reason about: +- How OLake Go writes snapshots, data files, and metadata - How schema evolution and partitioning work - Why Iceberg is chosen over traditional table formats @@ -71,12 +71,12 @@ Key concepts to grasp: --- -### 5. OLake Docs +### 5. OLake Go Docs -Familiarizing yourself with **OLake's official documentation** is essential for understanding how to set up, configure, and work with OLake effectively. The documentation provides comprehensive guides on: +Familiarizing yourself with **OLake Go's official documentation** is essential for understanding how to set up, configure, and work with OLake Go effectively. The documentation provides comprehensive guides on: - Setting up a development environment -- Understanding OLake's architecture and data flow +- Understanding OLake Go's architecture and data flow - Configuring sources, destinations, and sync modes - Using OLake CLI commands and flags - Debugging and troubleshooting @@ -88,8 +88,8 @@ Recommended starting point: ### 6. Change Data Capture (CDC) -OLake supports **CDC (Change Data Capture)** and **Incremental syncs**. Knowing CDC concepts helps you understand: -- How OLake tracks inserts, updates, and deletes over time +OLake Go supports **CDC (Change Data Capture)** and **Incremental syncs**. Knowing CDC concepts helps you understand: +- How OLake Go tracks inserts, updates, and deletes over time - Why resume tokens, log positions, and state files (`state.json`) are important - How incremental syncs differ from full refreshes @@ -104,11 +104,11 @@ Focus on: ## Additional Resources (Optional) -The following tutorials and resources are **not mandatory** but are recommended for a deeper understanding of related technologies and concepts that can enhance your OLake development experience. +The following tutorials and resources are **not mandatory** but are recommended for a deeper understanding of related technologies and concepts that can enhance your OLake Go development experience. ### 1. Data Lakehouse Architecture -Understanding the **data lakehouse** concept helps you appreciate how OLake fits into modern data architectures. A lakehouse combines the best of data lakes and data warehouses, enabling both structured and unstructured data processing. +Understanding the **data lakehouse** concept helps you appreciate how OLake Go fits into modern data architectures. A lakehouse combines the best of data lakes and data warehouses, enabling both structured and unstructured data processing. Recommended video: - [Introduction to Data Lakehouse](https://www.youtube.com/watch?v=9R2z-mzzX0M) @@ -117,9 +117,9 @@ Recommended video: ### 2. Apache Parquet File Format -OLake supports **Parquet** as a destination format. Understanding Parquet helps you understand: +OLake Go supports **Parquet** as a destination format. Understanding Parquet helps you understand: - How columnar storage works and why it's efficient for analytics -- How OLake writes data in Parquet format +- How OLake Go writes data in Parquet format - The relationship between Parquet and Iceberg (Iceberg can use Parquet files) Recommended video: @@ -129,7 +129,7 @@ Recommended video: ### 3. PostgreSQL Fundamentals -While not required if you're only working with other sources, understanding **PostgreSQL** is valuable since it's one of the most commonly used sources with OLake. This knowledge helps you: +While not required if you're only working with other sources, understanding **PostgreSQL** is valuable since it's one of the most commonly used sources with OLake Go. This knowledge helps you: - Understand source database concepts and structures - Better configure PostgreSQL connections and CDC settings - Debug source-related issues diff --git a/docs/community/setting-up-a-dev-env.mdx b/docs/community/setting-up-a-dev-env.mdx index 345da7411..d67dfb70d 100644 --- a/docs/community/setting-up-a-dev-env.mdx +++ b/docs/community/setting-up-a-dev-env.mdx @@ -13,7 +13,7 @@ If you're new to Data Engineering, Docker, or Golang, start with the [**Learning # Setting up a Development Environment -Watch this comprehensive video tutorial that walks you through the entire process of setting up OLake, syncing data, and debugging with a live demonstration: +Watch this comprehensive video tutorial that walks you through the entire process of setting up OLake Go, syncing data, and debugging with a live demonstration: @@ -21,7 +21,7 @@ Watch this comprehensive video tutorial that walks you through the entire proces **Pre-requisites:** -Before setting up and running OLake, ensure you have all the following installed on your system: +Before setting up and running OLake Go, ensure you have all the following installed on your system: @@ -184,14 +184,14 @@ docker compose -f docker-compose.source.yml --profile postgres -f docker-compose ## 2. OLake CLI Setup -The diagram below illustrates the high-level architecture of OLake, showing how data flows from various sources to different destinations: +The diagram below illustrates the high-level architecture of OLake Go, showing how data flows from various sources to different destinations: -![OLake data flow from MongoDB, Postgres, MySQL sources to Iceberg and Parquet destinations](/img/community/setting-up-a-dev-env/olake_diagram.webp) +![OLake Go data flow from MongoDB, Postgres, MySQL sources to Iceberg and Parquet destinations](/img/community/setting-up-a-dev-env/olake_diagram.webp)
    -How this local OLake architecture works +How this local OLake Go architecture works -The diagram represents a **complete local OLake pipeline** that you can run on your laptop using Docker Compose and the OLake CLI. +The diagram represents a **complete local OLake Go pipeline** that you can run on your laptop using Docker Compose and the OLake CLI. On the **left side**, you have your **operational databases**. In this setup, we provide three options: - **Postgres** @@ -201,30 +201,30 @@ On the **left side**, you have your **operational databases**. In this setup, we You choose which one to run by switching the Docker Compose `--profile` flag. Each source container exposes a database with a **dummy table pre-populated with sample data** (for example, simple customer/order-style records), so that you can immediately run initial loads and incremental or change-data-capture (CDC) syncs without having to insert data manually. -In the **middle of the diagram** is **OLake**, which acts as the **replication and transformation engine**: +In the **middle of the diagram** is **OLake Go**, which acts as the **replication and transformation engine**: - It connects to the selected source using the settings you define in `source.json` (host, port, credentials, CDC configuration, etc.). - It reads **new rows and changes** (inserts/updates/deletes) from the source in case of cdc or incremental syncs. - It applies transformations on the data such as **normalization** and **partitioning** if applied by the user. - It then routes that data to the configured destination using `destination.json`, handling schema mapping and partitioning as needed. -On the **right side of the diagram** are the **analytics destinations**, where OLake writes the data: +On the **right side of the diagram** are the **analytics destinations**, where OLake Go writes the data: - **Iceberg destination** - - OLake writes rows into **Apache Iceberg tables**. + - OLake Go writes rows into **Apache Iceberg tables**. - In local mode, the Iceberg catalog is backed by **MinIO** (an S3-compatible object storage running locally), and in real deployments it can use **Amazon S3** or any S3-compatible storage. - - OLake handles creating and updating Iceberg table files and metadata, so you can query these tables via the **Spark + Iceberg** service that is part of this local setup. + - OLake Go handles creating and updating Iceberg table files and metadata, so you can query these tables via the **Spark + Iceberg** service that is part of this local setup. - **Parquet destination** - - OLake writes data as **plain Parquet files** directly into a bucket. In local mode, the bucket is in **MinIO** (an S3-compatible object storage running locally), and in real deployments it can use **Amazon S3** or any S3-compatible storage. + - OLake Go writes data as **plain Parquet files** directly into a bucket. In local mode, the bucket is in **MinIO** (an S3-compatible object storage running locally), and in real deployments it can use **Amazon S3** or any S3-compatible storage. - This is useful when you want a simple folder of Parquet files for lightweight analytics or debugging, without the additional table-layer that Iceberg provides. -Putting it together, the image shows that **OLake sits between your transactional databases (Postgres/MySQL/MongoDB) and your analytical storage (Iceberg tables or Parquet files on MinIO/S3)**. -You can swap sources and destinations just by changing the Docker profile and your `source.json`/`destination.json`, while OLake consistently takes care of **moving and organizing the data from source to lakehouse**. +Putting it together, the image shows that **OLake Go sits between your transactional databases (Postgres/MySQL/MongoDB) and your analytical storage (Iceberg tables or Parquet files on MinIO/S3)**. +You can swap sources and destinations just by changing the Docker profile and your `source.json`/`destination.json`, while OLake Go consistently takes care of **moving and organizing the data from source to lakehouse**.
    ### Fork and Clone -Clone the OLake repository and navigate to the project directory: +Clone the OLake Go repository and navigate to the project directory: ```bash git clone git@github.com:datazip-inc/olake.git && cd olake @@ -356,7 +356,7 @@ You can use your own Kafka configuration. The docker compose is **WIP**.     -## 3. Commands to run OLake code +## 3. Commands to run OLake Go code :::important You **must** review the **Discover** and **Sync** commands in the [Commands and Flags](/docs/community/commands-and-flags) page before proceeding. These sections contain essential information about all available flags and options: @@ -380,7 +380,7 @@ If you want to learn more about `streams.json` file and how to modify it. Refer ### Sync Command -The sync command is used to sync data from the source to the destination. OLake supports three different sync modes: +The sync command is used to sync data from the source to the destination. OLake Go supports three different sync modes: @@ -393,13 +393,13 @@ For the first full refresh, run the sync command without the `--state` flag: ./build.sh driver-postgres sync --config $(pwd)/source.json --catalog $(pwd)/streams.json --destination $(pwd)/destination.json ``` -After this initial sync completes, a `state.json` and `stats.json` file are automatically generated. The `state.json` file contains the necessary resume tokens and metadata that OLake uses for CDC (Change Data Capture) or Incremental sync operations. Essentially, it tells OLake from where to resume or start the next sync. +After this initial sync completes, a `state.json` and `stats.json` file are automatically generated. The `state.json` file contains the necessary resume tokens and metadata that OLake Go uses for CDC (Change Data Capture) or Incremental sync operations. Essentially, it tells OLake Go from where to resume or start the next sync. -**Incremental Sync** is an **append-only operation** that syncs only new records based on cursor fields. You configure a primary cursor field (and optionally a secondary cursor field) in your `streams.json`. On the first run, a full refresh occurs automatically. After that, OLake only syncs records where the cursor field value is greater than the maximum value stored from the previous sync. +**Incremental Sync** is an **append-only operation** that syncs only new records based on cursor fields. You configure a primary cursor field (and optionally a secondary cursor field) in your `streams.json`. On the first run, a full refresh occurs automatically. After that, OLake Go only syncs records where the cursor field value is greater than the maximum value stored from the previous sync. :::note Modifications or deletions to existing records in the source will **not** be reflected in the destination. @@ -411,7 +411,7 @@ Run the sync command with the `--state` flag enabled: ./build.sh driver-postgres sync --config $(pwd)/source.json --catalog $(pwd)/streams.json --destination $(pwd)/destination.json --state $(pwd)/state.json ``` -To learn how Incremental sync works in OLake, including how it tracks changes and resumes from previous sync points, watch the following video: +To learn how Incremental sync works in OLake Go, including how it tracks changes and resumes from previous sync points, watch the following video:
    @@ -421,7 +421,7 @@ To learn how Incremental sync works in OLake, including how it tracks changes an **CDC (Change Data Capture)** is an upsert mode operation that tracks and replicates all changes made in the source database to the destination. It checks for any inserts, updates, or deletes in the source and performs similar changes in the destination, ensuring the destination stays in sync with the source. -This ensures that OLake continues from where it left off, only syncing new or changed data since the last sync operation. +This ensures that OLake Go continues from where it left off, only syncing new or changed data since the last sync operation. Run the sync command with the `--state` flag enabled: @@ -429,7 +429,7 @@ Run the sync command with the `--state` flag enabled: ./build.sh driver-postgres sync --config $(pwd)/source.json --catalog $(pwd)/streams.json --destination $(pwd)/destination.json --state $(pwd)/state.json ``` -To learn how CDC sync works in OLake, including how it tracks changes and resumes from previous sync points, watch the [comprehensive video tutorial](#setup-video-tutorial) above. +To learn how CDC sync works in OLake Go, including how it tracks changes and resumes from previous sync points, watch the [comprehensive video tutorial](#setup-video-tutorial) above.     @@ -453,7 +453,7 @@ You can follow these steps to try out CDC or Incremental sync functionality: :::info - **For Incremental Sync**: Only run the **Insert Records** operation (Step 1) below, as incremental sync only handles new records and does not track updates or deletes. - **For CDC**: You can try out all operations below (Insert, Update, and Delete) as CDC tracks all changes including inserts, updates, and deletes. -- If you want to know more about OLake generated variables like `op_type`, check out the [OLake Generated Columns](/docs/understanding/terminologies/olake#olake-generated-columns) section. +- If you want to know more about OLake Go generated variables like `op_type`, check out the [OLake Go Generated Columns](/docs/understanding/terminologies/olake#olake-generated-columns) section. ::: When running sync with state mode enabled, you can verify Change Data Capture/Incremental functionality by following these example steps: @@ -828,12 +828,75 @@ Now, set up debug points in the codebase and click "Launch Go Code". ![VS Code debugger paused on a breakpoint in Go code for OLake, with locals, call stack, and debug console outputs visible](/img/docs/getting-started/debug.webp) -### A Deep Dive into How to Debug OLake: +### A Deep Dive into How to Debug OLake Go: ## 7. OLake UI Setup -**UI setup**: Please follow the setup instructions at [https://github.com/datazip-inc/olake-ui/](https://github.com/datazip-inc/olake-ui/) +**UI setup**: Please follow the setup instructions at [Quick Start (Docker Compose)](https://olake.io/docs/getting-started/quickstart/#quick-start-docker-compose) +### Local Testing with Custom Drivers + +This guide explains how to test changes made in the [olake-cli](https://github.com/datazip-inc/olake) repository within the [olake-ui](https://github.com/datazip-inc/olake-ui) environment. This is useful for testing new features, or JSON schema changes + +#### 1. Build Custom Images + +Before running the tests, build custom Docker images for the driver under test. + +Execute the following commands from the root of the [olake-cli](https://github.com/datazip-inc/olake) repository on your current working branch. +Build the image for the desired source driver and also build the MySQL image, as MySQL is used as the default driver for destination-related operations. + +Before building the images, make sure the JAR file exists: + +```bash +mvn -f destination/iceberg/olake-iceberg-java-writer/pom.xml clean package -DskipTests +``` + +```bash +# Build source image for the selected driver +docker build \ + --build-arg DRIVER_NAME= \ + -t olakego/source-: \ + . +``` + +```bash +# Build image for MySQL (required for destination related operations) +docker build \ + --build-arg DRIVER_NAME=mysql \ + -t olakego/source-mysql: \ + . +``` + + +#### 2. Configure Environment Variables + +The UI server needs to be informed to use these custom images and bypass the standard version compatibility checks. + +Set the following environment variables to the `olake-ui` service in the `docker-compose-v1.yml` file: + +| Variable | Value | Description | +| :--- | :--- | :--- | +| `APP_ENV` | `development` | Enables developement environment. | +| `CUSTOM_DRIVER_VERSION` | `` | The specific tag you used when building your custom image. | + +#### Docker compose: +```bash title="docker-compose-v1.yml" +olake-ui: + image: ${CONTAINER_REGISTRY_BASE:-registry-1.docker.io}/olakego/ui:latest + pull_policy: always + container_name: olake-ui + environment: + APP_ENV: development + CUSTOM_DRIVER_VERSION: +``` + +#### 3. How It Works + +`CUSTOM_DRIVER_VERSION` takes effect only when `APP_ENV` is set to `development`. + +When both are set: +- The specified version is added to the driver version dropdown in the UI. +- Semver compatibility checks are bypassed. diff --git a/docs/connectors/db2/index.mdx b/docs/connectors/db2/index.mdx index 19e852c18..6249c623d 100644 --- a/docs/connectors/db2/index.mdx +++ b/docs/connectors/db2/index.mdx @@ -5,17 +5,17 @@ sidebar_label: DB2 LUW --- import Head from '@docusaurus/Head'; - +import TOCTabLinker from '@site/src/components/TOCTabLinker'; import DateTimeHandling from '@site/src/components/DateTimeHandling'; # Overview {#overview} -The OLake DB2 LUW (Linux Unix Windows) Source connector supports multiple synchronization modes. It offers features like parallel chunking, checkpointing, and automatic resume for failed full loads. This connector can be used within the OLake UI or run locally via Docker for open-source workflows. +The OLake Go DB2 LUW (Linux Unix Windows) Source connector supports multiple synchronization modes. It offers features like parallel chunking, checkpointing, and automatic resume for failed full loads. This connector can be used within the OLake UI or run locally via Docker for open-source workflows. ## Sync Modes Supported - **Full Refresh** -- **Incremental** +- **Full Refresh + Incremental** ## Prerequisites @@ -38,13 +38,21 @@ The OLake DB2 LUW (Linux Unix Windows) Source connector supports multiple synchr ## Configuration {#configuration} - + + + - + ### 1. Navigate to the Source Configuration Page -1. Complete the [OLake UI Setup Guide](/docs/getting-started/olake-ui) +1. Complete the [OLake UI Setup Guide](/docs/getting-started/quickstart/) 2. After logging in to the OLake UI, select the `Sources` tab from the left sidebar. 3. Click **`Create Source`** on the top right corner. 4. Select **DB2** from the connector dropdown. @@ -65,9 +73,9 @@ The OLake DB2 LUW (Linux Unix Windows) Source connector supports multiple synchr | Password `required` | The password corresponding to the provided username for authentication. | `db2pwd` | | Username `required` | Database user used to authenticate the connection. | `db2-user` | | Max Threads | Maximum number of worker threads the connector can spin up for parallel tasks. | `10` | -| JDBC URL Parameters | Extra [JDBC URL parameters](https://jdbc.postgresql.org/documentation/use/) for fine-tuning the connection. | `{"connectTimeout":"20"}` | +| JDBC URL Parameters | Extra [JDBC URL parameters](https://www.ibm.com/docs/en/db2/11.5.x?topic=information-properties-data-server-driver-jdbc-sqlj) for fine-tuning the connection. | `{"connectTimeout":"20"}` | | SSL Mode | SSL configuration for the database connection. Contains details such as the SSL mode. | `disable` | -| SSH Config | Configure OLake to connect through an SSH tunnel. See [SSH Config details](/docs/understanding/terminologies/olake#1-ssh-configuration) for the list of supported parameters. |
    • `No Tunnel`
    • `SSH Key Authentication`
    • `SSH Password Authentication`
    | +| SSH Config | Configure OLake Go to connect through an SSH tunnel. |
    • `No Tunnel`
    • `SSH Key Authentication`
    • `SSH Password Authentication`
    | | Retry Count | Number of times the retry will take place incase of timeout before failing the sync. | `3` | ### 3. Test Connection @@ -78,7 +86,7 @@ The OLake DB2 LUW (Linux Unix Windows) Source connector supports multiple synchr     - + ### 1. Create Configuration File @@ -94,16 +102,16 @@ An example `source.json` file will look like this: | Field | Description | Example Value | Type | | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | ------ | -| DB2 Host `required` | Hostname or IP address of the DB2 database server. | `DB2 LUW-host` | String | -| DB2 Port `required` | TCP port on which the DB2 listener is accepting connections. | `50000` | Integer | -| Database Name `required` | The name of the target database to connect to. | `olake-db` | String | -| Password `required` | The password corresponding to the provided username for authentication. | `db2pwd` | String | -| Username `required` | Database user used to authenticate the connection. | `db2-user` | String | -| Max Threads | Maximum number of worker threads the connector can spin up for parallel tasks. | `10` | Integer | -| JDBC URL Parameters | Extra [JDBC URL parameters](https://jdbc.postgresql.org/documentation/use/) for fine-tuning the connection. | `{"connectTimeout":"20"}` | Object | -| SSL Mode | SSL configuration for the database connection. Contains details such as the SSL mode. | `disable` | Object | -| SSH Config | Configure OLake to connect through an SSH tunnel. See [SSH Config details](/docs/understanding/terminologies/olake#1-ssh-configuration) for the list of supported parameters. |
    • `No Tunnel`
    • `SSH Key Authentication`
    • `SSH Password Authentication`
    | Object | -| Retry Count | Number of times the retry will take place incase of timeout before failing the sync. | `3` | Integer | +| host `required` | Hostname or IP address of the DB2 database server. | `DB2 LUW-host` | String | +| port `required` | TCP port on which the DB2 listener is accepting connections. | `50000` | Integer | +| database `required` | The name of the target database to connect to. | `olake-db` | String | +| password `required` | The password corresponding to the provided username for authentication. | `db2pwd` | String | +| username `required` | Database user used to authenticate the connection. | `db2-user` | String | +| max_threads | Maximum number of worker threads the connector can spin up for parallel tasks. | `10` | Integer | +| jdbc_url_params | Extra [JDBC URL parameters](https://www.ibm.com/docs/en/db2/11.5.x?topic=properties-connectionstring) for fine-tuning the connection. | `{"connectTimeout":"20"}` | Object | +| ssl | SSL configuration for the database connection. Contains details such as the SSL mode. | `disable` | Object | +| ssh_config | Configure OLake Go to connect through an SSH tunnel. | `{"host": "my-tunnel-host", "port": 22 , "username": "my-tunnel-user", "password": "tunnel-password"}` | Object | +| retry_count | Number of times the retry will take place incase of timeout before failing the sync. | `3` | Integer | Similarly, `destination.json` file can be created inside this folder. For more information, see destination documentation. @@ -120,10 +128,10 @@ check \ ``` :::note -When you run any OLake commands using `.buildsh`, the IBM Data Server ODBC and CLI driver gets automatically installed on your system. +When you run any OLake Go commands using `.buildsh`, the IBM Data Server ODBC and CLI driver gets automatically installed on your system. ::: -- If OLake is able to connect with DB2 `{"connectionStatus":{"status":"SUCCEEDED"},"type":"CONNECTION_STATUS"}` response is returned. +- If OLake Go is able to connect with DB2 `{"connectionStatus":{"status":"SUCCEEDED"},"type":"CONNECTION_STATUS"}` response is returned. - In case of connection failure, refer to the [Troubleshooting section](#troubleshooting). @@ -137,9 +145,9 @@ When you run any OLake commands using `.buildsh`, the IBM Data Server ODBC and C :::info -- OLake always ingests timestamp data in UTC format, independent of the source timezone. -- Run this command to collect table and index statistics. This populates the system catalog tables with metadata such as npages, page size, and average row size, which OLake requires during sync: - ```sql +- OLake Go always ingests timestamp data in UTC format, independent of the source timezone. +- Run this command to collect table and index statistics. This populates the system catalog tables with metadata such as npages, page size, and average row size, which OLake Go requires during sync: + ```sql CALL SYSPROC.ADMIN_CMD('RUNSTATS ON TABLE schema_name.table_name AND INDEXES ALL'); ``` ::: diff --git a/docs/connectors/kafka/index.mdx b/docs/connectors/kafka/index.mdx index 6c7e8ffc2..b20b5e909 100644 --- a/docs/connectors/kafka/index.mdx +++ b/docs/connectors/kafka/index.mdx @@ -5,27 +5,30 @@ sidebar_label: Kafka --- import Head from '@docusaurus/Head'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import TOCTabLinker from '@site/src/components/TOCTabLinker'; import DateTimeHandling from '@site/src/components/DateTimeHandling'; # Overview {#overview} -The OLake Kafka Source connector syncs messages from Kafka topics directly to the destination. It supports only one synchronization mode and offers features like parallel partition processing and checkpointing. This connector can be used within the OLake UI or run locally via Docker for open-source workflows. +The OLake Go Kafka Source connector syncs messages from Kafka topics directly to the destination. It supports only one synchronization mode and offers features like parallel partition processing and checkpointing. This connector can be used within the OLake UI or run locally via Docker for open-source workflows. :::warning Important: Kafka Retention Period -Kafka topics have a configured retention period. Once messages exceed this retention period, they are automatically deleted from the topic. **OLake cannot access or sync deleted messages.** Ensure your sync schedule runs frequently enough to capture data before it expires. +Kafka topics have a configured retention period. Once messages exceed this retention period, they are automatically deleted from the topic. **OLake Go cannot access or sync deleted messages.** Ensure your sync schedule runs frequently enough to capture data before it expires. ::: :::note **Kafka Connector Requirements** -- Kafka as a source connector is available in **OLake v0.3.0 and above** on both the source and destination side. -- The OLake Kafka connector has been **tested specifically with Amazon MSK** (Amazon Managed Streaming for Apache Kafka), AWS's fully managed Apache Kafka service. The connector uses standard Kafka protocols and should work with other Kafka distributions. +- Kafka as a source connector is available in **OLake Go v0.3.0 and above** on both the source and destination side. +- The OLake Go Kafka connector has been **tested specifically with Amazon MSK** (Amazon Managed Streaming for Apache Kafka), AWS's fully managed Apache Kafka service. The connector uses standard Kafka protocols and should work with other Kafka distributions. ::: ## Authentication {#authentication} -OLake supports three authentication methods for connecting to Kafka brokers: +OLake Go supports three authentication methods for connecting to Kafka brokers: ### 1. PLAINTEXT @@ -78,11 +81,11 @@ Authentication with SSL/TLS encryption. This is the most secure option, providin ### Version Prerequisites -Kafka Version 0.10.1.0 or higher. +Kafka Version 2.4.0 or higher. ### Connection Prerequisites -- OLake and the Kafka broker servers must be accessible within the same network. +- OLake Go and the Kafka broker servers must be accessible within the same network. **After initial Prerequisites are fulfilled, the configurations for Kafka can be configured.** @@ -90,13 +93,21 @@ Kafka Version 0.10.1.0 or higher. ## Configuration {#configuration} - + + + ### 1. Navigate to the Source Configuration Page -1. Complete the [OLake UI Setup Guide](/docs/getting-started/olake-ui) +1. Complete the [OLake UI Setup Guide](/docs/getting-started/quickstart/) 2. After logging in to the OLake UI, select the `Sources` tab from the left sidebar. 3. Click **`Create Source`** on the top right corner. 4. Select **Kafka** from the connector dropdown @@ -107,20 +118,19 @@ Kafka Version 0.10.1.0 or higher. - Enter Kafka credentials.
    -
    ![OLake Kafka source setup with endpoint and SSH options](/img/docs/sources/kafka/kafka-config.webp)
    +
    ![OLake Go Kafka source setup with endpoint and SSH options](/img/docs/sources/kafka/kafka-config.webp)
    | Field | Description | Example Value | |-----------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------| | Kafka BootstrapServers `required` | Comma-separated list of Kafka broker addresses (host:port) for establishing the initial connection to the Kafka cluster. | `"broker:9092, broker:9093"` | -| Consumer Group ID | Unique identifier for the consumer group used to track messages and coordinate with consumer members. If not provided, OLake automatically generates one. | `example-consumer-group` | -| Protocol `required` | Configuration object containing Kafka security settings and authentication configurations for connecting to Kafka. Sub-parameters: `security_protocol`, `sasl_mechanism`, `sasl_jaas_config`. | `{`
    ` "security_protocol": "SASL_SSL",`
    ` "sasl_mechanism": "SCRAM-SHA-512",`
    ` "sasl_jaas_config": "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"YOUR_KAFKA_USERNAME\" password=\"YOUR_KAFKA_PASSWORD\";"`
    `}` | +| Consumer Group ID | Unique identifier for the consumer group used to track messages and coordinate with consumer members. If not provided, OLake Go automatically generates one. | `example-consumer-group` | | Security Protocol `required` | Protocol used for communication with Kafka.
    Supported options: `PLAINTEXT`, `SASL_PLAINTEXT`, or `SASL_SSL`. | `SASL_SSL` | | SASL Mechanism | Specifies the type of SASL authentication protocol used to verify client identity when connecting to Kafka.
    Supported options: `PLAIN`, `SCRAM-SHA-512`. | `"SCRAM-SHA-512"` | | SASL JAAS Config | JAAS configuration string containing the login module and authentication credentials **(username and password)** for SASL authentication. | If SASL mechanism is `PLAIN`:
    `"org.apache.kafka.common.security.plain.PlainLoginModule required username="username" password="password";"`
    If SASL mechanism is `SCRAM-SHA-512`:
    `"org.apache.kafka.common.security.scram.ScramLoginModule required username="username" password="password";"` | | Threads Equal Total Partitions | When `true`, the number of consumer thread is equal to total number of partitions for most optimal parallel processing. When `false`, consumer threads are set to the `max_threads` value. | `false` | | | Max Threads | Maximum number of parallel threads for processing or syncing data. | `3` | -| Backoff Retry Count | Number of retry attempts for establishing sync with exponential backoff. | `3` | +| Retry Count | Number of retry attempts for establishing sync with exponential backoff. | `3` | | Schema Registry Endpoint | Endpoint of the Confluence Schema Registry. For AVRO-based topics, schema registry is mandatory. | `http://localhost:8081` | | Authentication Type | Authentication method for the Schema Registry.
    Supported options:
    - `No Authentication`
    - `Username & Password`
    - `Bearer Token` | 1. `No Authentication` - nothing is required
    2. `Username & Password` - username and password is required (e.g., `username` = dummy, `password` = password)
    3. `Bearer Token` - bearer token is required (e.g., dummy_token) | @@ -131,7 +141,7 @@ Kafka Version 0.10.1.0 or higher. - In case of connection failure, refer to the [Troubleshooting section](#troubleshooting).     - + ### 1. Create Configuration File @@ -147,16 +157,16 @@ An example `source.json` file will look like this: | Field | Description | Example Value | Type | |-----------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------| ------- | -| Kafka BootstrapServers `required` | Comma-separated list of Kafka broker addresses **(host:port)** for establishing the initial connection to the Kafka cluster. | `"broker:9092, broker:9093"` | String | -| Consumer Group ID | Unique identifier for the consumer group used to track message and coordinate with other consumer members. If not provided, OLake automatically generates one. | `example-consumer-group` | String | -| Protocol `required` | Configuration object containing Kafka security settings and authentication configurations for connecting to Kafka. Sub-parameters: `security_protocol`, `sasl_mechanism`, `sasl_jaas_config`. | `{`
    ` "security_protocol": "SASL_SSL",`
    ` "sasl_mechanism": "SCRAM-SHA-512",`
    ` "sasl_jaas_config": "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"YOUR_KAFKA_USERNAME\" password=\"YOUR_KAFKA_PASSWORD\";"`
    `}` | Object | -| Security Protocol `required` | Protocol used for communication with Kafka.
    Supported options: `PLAINTEXT`, `SASL_PLAINTEXT`, or `SASL_SSL`. | `"SASL_SSL"` | String | -| SASL Mechanism | Specifies the type of SASL authentication protocol used to verify client identity when connecting to Kafka. .
    Supported options: `PLAIN`, `SCRAM-SHA-512`. | `"SCRAM-SHA-512"` | String | -| SASL JAAS Config | JAAS configuration string containing the login module and authentication credentials **(username and password)** for SASL authentication. | If SASL mechanism is `PLAIN`:
    `"org.apache.kafka.common.security.plain.PlainLoginModule required username="username" password="password";"`
    If SASL mechanism is `SCRAM-SHA-512`:
    `"org.apache.kafka.common.security.scram.ScramLoginModule required username="username" password="password";"` | String | -| Threads Equal Total Partitions | When `true`, the number of consumer thread is equal to total number of partitions for most optimal parallel processing. When `false`, consumer threads are set to the `max_threads` value. | `false` | Boolean | | -| Max Threads | Maximum number of parallel threads for processing or syncing data. | `3` | Integer | -| Backoff Retry Count | Number of retry attempts for establishing sync with exponential backoff. | `3` | Integer | -| Schema Registry (Optional) | If schema registry is configured, at least schema registry `endpoint` has to be provided. If authentication is also present, then either authenticate using `username` and `password` or `bearer_token`. For AVRO-based topics, schema registry is mandatory. | `"schema_registry": {`
    ` "endpoint": "http://localhost:8081",`
    ` "username": "dummy",`
    ` "password": "dummy-password"`
    `}` | Object | +| bootstrap_servers `required` | Comma-separated list of Kafka broker addresses **(host:port)** for establishing the initial connection to the Kafka cluster. | `"broker:9092, broker:9093"` | String | +| consumer_group_id | Unique identifier for the consumer group used to track message and coordinate with other consumer members. If not provided, OLake Go automatically generates one. | `example-consumer-group` | String | +| protocol `required` | Configuration object containing Kafka security settings and authentication configurations for connecting to Kafka. Sub-parameters: `security_protocol`, `sasl_mechanism`, `sasl_jaas_config`. | `{`
    ` "security_protocol": "SASL_SSL",`
    ` "sasl_mechanism": "SCRAM-SHA-512",`
    ` "sasl_jaas_config": "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"YOUR_KAFKA_USERNAME\" password=\"YOUR_KAFKA_PASSWORD\";"`
    `}` | Object | +| security_protocol `required` | Protocol used for communication with Kafka.
    Supported options: `PLAINTEXT`, `SASL_PLAINTEXT`, or `SASL_SSL`. | `"SASL_SSL"` | String | +| sasl_mechanism | Specifies the type of SASL authentication protocol used to verify client identity when connecting to Kafka. .
    Supported options: `PLAIN`, `SCRAM-SHA-512`. | `"SCRAM-SHA-512"` | String | +| sasl_jaas_config | JAAS configuration string containing the login module and authentication credentials **(username and password)** for SASL authentication. | If SASL mechanism is `PLAIN`:
    `"org.apache.kafka.common.security.plain.PlainLoginModule required username="username" password="password";"`
    If SASL mechanism is `SCRAM-SHA-512`:
    `"org.apache.kafka.common.security.scram.ScramLoginModule required username="username" password="password";"` | String | +| threads_equal_total_partitions | When `true`, the number of consumer thread is equal to total number of partitions for most optimal parallel processing. When `false`, consumer threads are set to the `max_threads` value. | `false` | Boolean | | +| max_threads | Maximum number of parallel threads for processing or syncing data. | `3` | Integer | +| backoff_retry_count | Number of retry attempts for establishing sync with exponential backoff. | `3` | Integer | +| schema_registry `Optional` | If schema registry is configured, at least schema registry `endpoint` has to be provided. If authentication is also present, then either authenticate using `username` and `password` or `bearer_token`. For AVRO-based topics, schema registry is mandatory. | `"schema_registry": {`
    ` "endpoint": "http://localhost:8081",`
    ` "username": "dummy",`
    ` "password": "dummy-password"`
    `}` | Object | Similarly, `destination.json` file can be created inside this folder. For more information, see destination documentation. @@ -172,7 +182,7 @@ check \ --config /mnt/config/source.json ``` -- If OLake is able to connect with Kafka `{"connectionStatus":{"status":"SUCCEEDED"},"type":"CONNECTION_STATUS"}` response is returned. +- If OLake Go is able to connect with Kafka `{"connectionStatus":{"status":"SUCCEEDED"},"type":"CONNECTION_STATUS"}` response is returned. - In case of connection failure, refer to the [Troubleshooting section](#troubleshooting). @@ -183,8 +193,12 @@ check \ ## Kafka General Information -When OLake syncs data from Kafka topics to Iceberg, it transfers the complete message structure along with Kafka metadata. This ensures full traceability and context for each message. +When OLake Go syncs data from Kafka topics to Iceberg, it transfers the complete message structure along with Kafka metadata. This ensures full traceability and context for each message. +:::note Consumer Group Requirements +- The consumer group configured for OLake Go must be **dedicated exclusively to OLake Go**. Also each job must use a **unique consumer group ID**. This consumer group should not be shared with any other process. +- If a consumer group rebalance occurs while a sync is in progress, the sync will shut down gracefully. This means that all messages read up to that point are written to the destination, and the next sync resumes from where it left off. +::: ### Columns Transferred @@ -199,24 +213,24 @@ The following columns are automatically included when syncing Kafka messages to :::warning Message Format Requirement - The Kafka messages **must contain data in JSON or AVRO format**. -- If messages are null or empty, OLake will read the messages but **skip writing them to the destination**. For AVRO format, a **Confluence Schema Registry must be configured**—schema registry is mandatory only for AVRO data and if not provided OLake will consider the messages as JSON format. +- If messages are null or empty, OLake Go will read the messages but **skip writing them to the destination**. For AVRO format, a **Confluence Schema Registry must be configured**—schema registry is mandatory only for AVRO data and if not provided OLake Go will consider the messages as JSON format. ::: ### Normalization -OLake provides flexibility in how message values are stored: +OLake Go provides flexibility in how message values are stored: - **Normalization = `true`**: The payload in the `value` column is Level 0 flattened, and each nested field becomes a separate column in the destination table. This makes querying individual fields easier. - **Normalization = `false`**: The payload is stored as-is in the `value` column without flattening. The entire JSON object remains in a single column. - + --- ## Troubleshooting {#troubleshooting} ### 1. Consumer Group Corruption -If a consumer group becomes corrupted, you'll need to use a different consumer group to continue syncing data. The resolution depends on how you're running OLake. +If a consumer group becomes corrupted, you'll need to use a different consumer group to continue syncing data. The resolution depends on how you're running OLake Go. #### Using OLake UI diff --git a/docs/connectors/mongodb/index.mdx b/docs/connectors/mongodb/index.mdx index c123eab66..d4fbb2f7c 100644 --- a/docs/connectors/mongodb/index.mdx +++ b/docs/connectors/mongodb/index.mdx @@ -5,17 +5,20 @@ sidebar_label: MongoDB --- import DateTimeHandling from '@site/src/components/DateTimeHandling'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import TOCTabLinker from '@site/src/components/TOCTabLinker'; # Overview {#overview} -The OLake MongoDB Source connector supports multiple synchronization modes. It offers features like parallel chunking, checkpointing, and automatic resume for failed full loads. This connector can be used within the OLake UI or run locally via Docker for open-source workflows. +The OLake Go MongoDB Source connector supports multiple synchronization modes. It offers features like parallel chunking, checkpointing, and automatic resume for failed full loads. This connector can be used within the OLake UI or run locally via Docker for open-source workflows. ## Sync Modes Supported - **Full Refresh** - **Full Refresh + Incremental** - **Full Refresh + CDC** -- [**CDC Only**](/docs/features/#2-sync-modes-supported) +- **CDC Only** ## Prerequisites @@ -25,15 +28,15 @@ MongoDB Version 4.0 or higher ### CDC Prerequisites -For [Change Data Capture (CDC)](/docs/understanding/terminologies/general/#39-change-data-capture) mode, MongoDB must meet the following requirements: +For [Change Data Capture (CDC)](/docs/understanding/terminologies/general/?terminology-type=general#39-change-data-capture) mode, MongoDB must meet the following requirements: - MongoDB must be running in **replica set mode** (`--replSet rs0`) -- [**oplog**](/docs/understanding/terminologies/general/#26-oplog-mongodb) must be enabled (automatic in replica sets) +- [**oplog**](/docs/understanding/terminologies/general/?terminology-type=general#26-oplog-mongodb) must be enabled (automatic in replica sets) :::info -1. CDC in OLake is not a continuous always-on process. It requires execution through the **Orchestrator**. +1. CDC in OLake Go is not a continuous always-on process. It requires execution through the **Orchestrator**. -2. If you don’t have access to enable CDC (replica sets + oplog), OLake also supports **Incremental sync**. +2. If you don’t have access to enable CDC (replica sets + oplog), OLake Go also supports **Incremental sync**. ::: @@ -51,13 +54,21 @@ For local setup, follow **[MongoDB via Docker Compose](/docs/connectors/mongodb/ ## Configuration {#configuration} - + - + + + ### 1. Navigate to the Source Configuration Page -1. Complete the [OLake UI Setup Guide](/docs/getting-started/olake-ui) +1. Complete the [OLake UI Setup Guide](/docs/getting-started/quickstart/) 2. After logging in to the OLake UI, select the `Sources` tab from the left sidebar. 3. Click **`Create Source`** on the top right corner. 4. Select **MongoDB** from the connector dropdown @@ -66,23 +77,26 @@ For local setup, follow **[MongoDB via Docker Compose](/docs/connectors/mongodb/ ### 2. Provide Configuration Details - Enter MongoDB credentials. -![OLake MongoDB source setup form with connection and auth fields](/img/docs/sources/mongodb/source-config.webp) +![OLake Go MongoDB source setup form with connection and auth fields](/img/docs/sources/mongodb/source-config.webp) | Field | Description | Example Value | | ------------------------------------------------------ | ------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------- | -| Hosts
    `required` | List of MongoDB hosts. Use DNS SRV format if `srv = true` | `x.xxx.xxx.120:27017`, `x.xxx.xxx.133:27017` (multiple hosts supported) | -| Username
    ^ | MongoDB authentication username | `mongo-user` | -| Password
    ^ | MongoDB authentication password | `mongo-pwd` | -| Auth DB
    ^ | Authentication database name | `admin-db` | +| Hosts `required` | List of MongoDB hosts. Use DNS SRV format if `srv = true` | `x.xxx.xxx.120:27017`, `x.xxx.xxx.133:27017` (multiple hosts supported) | +| Username `required` | MongoDB authentication username | `mongo-user` | +| Password `required` | MongoDB authentication password | `mongo-pwd` | +| Auth DB `required` | Authentication database name | `admin-db` | | Replica Set | Name of the replica set (if applicable) | `rs0` | | Read Preference | MongoDB read preference setting | `secondaryPreferred` | | Use SRV | Enable DNS SRV connection strings. When `true`, only one host allowed in `hosts` field | `false` | -| Database Name
    `required` | Target MongoDB database name to replicate | `my-db` | +| Database Name `required` | Target MongoDB database name to replicate | `my-db` | | Max Threads | Maximum parallel threads for chunk-based snapshotting | `3` | | Retry Count | Number of retry attempts with exponential backoff. | `3` | | Chunking Strategy | Data chunking strategy: `timestamp`, `splitVector`. Defaults to `splitVector` if empty | `splitVector` | | IAM Authentication | Turn on to use IAM credentials stored in host machine instead of `Username` and `Password` | `off` | -| Additional Connection Parameters | Additional MongoDB connection string parameters (e.g., `authMechanism`).
    Each parameter will be added to the connection URI | `authMechanism=SCRAM-SHA-256` | +| TLS CA Certificate | CA certificate content in PEM format for verifying the MongoDB server. Use this instead of `tlsCAFile` in Additional Connection Parameters — file paths are not supported in UI/Docker deployments. | `-----BEGIN CERTIFICATE-----\n...` | +| TLS Client Certificate & Key | Client certificate and private key in PEM format (combined in one field). Required for mutual TLS or MONGODB-X509 authentication. Leave empty for SCRAM + TLS. | `-----BEGIN CERTIFICATE-----\n...\n-----BEGIN PRIVATE KEY-----\n...` | +| Additional Connection Parameters | Additional MongoDB connection string parameters (e.g., `tls`, `connectTimeoutMS`). Each parameter will be added to the connection URI. File-path TLS parameters (`tlsCAFile`, `tlsCertificateKeyFile`) are not supported in UI/Docker — use the TLS fields above instead. | `connectTimeoutMS=5000` | +| SSH Config | Configure OLake Go to connect through an SSH tunnel. |
    • `No Tunnel`
    • `SSH Key Authentication`
    • `SSH Password Authentication`
    | **^ Not required when using IAM Authentication** @@ -93,7 +107,7 @@ For local setup, follow **[MongoDB via Docker Compose](/docs/connectors/mongodb/     - + ### 1. Create Configuration File @@ -116,25 +130,33 @@ An example `source.json` file will look like this: "max_threads": 5, "backoff_retry_count": 4, "chunking_strategy": "", - "use_iam": false + "use_iam": false, + "tls_ca_cert": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----\n", + "tls_certificate_key": "", + "additional_params": { + "connectTimeoutMS": "5000" + } } ``` | Field | Description | Example Value | Type | | --------------------------------------------------- | -------------------------------------------------------------------------------------- | ------------------------------------------------ | -------- | -| `hosts` `required` | List of MongoDB hosts. Use DNS SRV format if `srv = true` | `["x.xxx.xxx.120:27017", "x.xxx.xxx.133:27017"]` | STRING[] | -| `username` ^ | MongoDB authentication username | `"mongo-user"` | STRING | -| `password` ^ | MongoDB authentication password | `"mongo-pwd"` | STRING | -| `authdb` ^ | Authentication database name | `"admin-db"` | STRING | -| `replica_set` | Name of the replica set (if applicable) | `"rs0"` | STRING | -| `read_preference` | MongoDB read preference setting | `"secondaryPreferred"` | STRING | -| `srv` | Enable DNS SRV connection strings. When `true`, only one host allowed in `hosts` field | `false` | BOOLEAN | -| `database` `required` | Target MongoDB database name to replicate | `"my-db"` | STRING | -| `max_threads` | Maximum parallel threads for chunk-based snapshotting | `3` | INTEGER | -| `backoff_retry_count` | Number of retry attempts with exponential backoff. | `3` | INTEGER | -| `chunking_strategy` | Data chunking strategy: `timestamp`, `splitVector`. Defaults to `splitVector` | `"splitVector"` | STRING | -| `use_iam` | Use IAM credentials stored in host machine instead of `username` and `pasword` | `false` | BOOLEAN | -| `Additional Connection Parameters` | Additional MongoDB connection string parameters (e.g., `authMechanism`).
    Each parameter will be added to the connection URI | `authMechanism=SCRAM-SHA-256` | DICTIONARY | +| hosts `required` | List of MongoDB hosts. Use DNS SRV format if `srv = true` | `["x.xxx.xxx.120:27017", "x.xxx.xxx.133:27017"]` | STRING[] | +| username `required` | MongoDB authentication username | `"mongo-user"` | STRING | +| password `required` | MongoDB authentication password | `"mongo-pwd"` | STRING | +| authdb `required` | Authentication database name | `"admin-db"` | STRING | +| replica_set | Name of the replica set (if applicable) | `"rs0"` | STRING | +| read_preference | MongoDB read preference setting | `"secondaryPreferred"` | STRING | +| srv | Enable DNS SRV connection strings. When `true`, only one host allowed in `hosts` field | `false` | BOOLEAN | +| database `required` | Target MongoDB database name to replicate | `"my-db"` | STRING | +| max_threads | Maximum parallel threads for chunk-based snapshotting | `3` | INTEGER | +| backoff_retry_count | Number of retry attempts with exponential backoff. | `3` | INTEGER | +| chunking_strategy | Data chunking strategy: `timestamp`, `splitVector`. Defaults to `splitVector` | `"splitVector"` | STRING | +| use_iam | Use IAM credentials stored in host machine instead of `username` and `pasword` | `false` | BOOLEAN | +| tls_ca_cert | CA certificate PEM content for verifying the MongoDB server. Use instead of `tlsCAFile` in `additional_params` — file paths are not supported in UI/Docker deployments. | `"-----BEGIN CERTIFICATE-----\n..."` | STRING | +| tls_certificate_key | Combined client certificate and private key PEM for mutual TLS or MONGODB-X509 authentication. Optional for SCRAM + TLS. | `"-----BEGIN CERTIFICATE-----\n...\n-----BEGIN PRIVATE KEY-----\n..."` | STRING | +| additional_params | Additional MongoDB connection string parameters (e.g., `tls`, `connectTimeoutMS`). File-path TLS parameters (`tlsCAFile`, `tlsCertificateKeyFile`) are not supported in UI/Docker — use `tls_ca_cert` and `tls_certificate_key` instead. | `{"connectTimeoutMS": "5000"}` | DICTIONARY | +| ssh_config | Configure OLake Go to connect through an SSH tunnel. | `{"host": "my-tunnel-host", "port": 22 , "username": "my-tunnel-user", "password": "tunnel-password"}` | Object | **^ Not required when `use_iam` is set to `true`** @@ -151,7 +173,7 @@ check \ --config /mnt/config/source.json ``` -- If OLake is able to connect with MongoDB `{"connectionStatus":{"status":"SUCCEEDED"},"type":"CONNECTION_STATUS"}` response is returned. +- If OLake Go is able to connect with MongoDB `{"connectionStatus":{"status":"SUCCEEDED"},"type":"CONNECTION_STATUS"}` response is returned. - In case of connection failure, refer to the [Troubleshooting section](/docs/connectors/mongodb#troubleshooting). @@ -159,6 +181,16 @@ check \     +### TLS / SSL Configuration + +Use the **TLS CA Certificate** and **TLS Client Certificate & Key** fields when connecting to a TLS-enabled MongoDB instance from the UI or Docker. The connector runs in an ephemeral container with only the config file mounted, so certificate file paths such as `tlsCAFile` or `tlsCertificateKeyFile` in `additional_params` will fail with a file-not-found error. + +- **`tls_ca_cert`**: Paste the CA certificate PEM to verify the MongoDB server certificate. When set, `tls=true` is added to the connection URI automatically. +- **`tls_certificate_key`**: Paste the client certificate and private key PEM blocks in one field. Required only for mutual TLS or MONGODB-X509 authentication. +- **`additional_params`**: Use for other URI options (`connectTimeoutMS`, `readPreference`, etc.). Do not use `tlsCAFile` or `tlsCertificateKeyFile` when inline PEM fields are set. + +For CLI usage with locally mounted certificate files, you can still pass `tlsCAFile` in `additional_params` if the inline TLS fields are left empty. + --- ## Data Type Mapping {#data-type-mapping} @@ -173,7 +205,7 @@ check \ | string, object, objectId, binData (binary), code, regex (BSONRegExp), decimal128, maxKey, minKey, array, undefined | string | :::info timestamptz timezone -OLake always ingests timestamp data in UTC format, independent of the source timezone. +OLake Go always ingests timestamp data in UTC format, independent of the source timezone. ::: --- @@ -198,13 +230,19 @@ OLake always ingests timestamp data in UTC format, independent of the source tim **Solution**: Verify replica set is active by running `rs.status()` -### 3. File not found (CLI): +### 3. TLS certificate file not found (UI/Docker): + +**Cause**: `tlsCAFile` or `tlsCertificateKeyFile` was set in `additional_params`, but the connector container cannot access local filesystem paths. + +**Solution**: Paste the certificate content into `tls_ca_cert` and/or `tls_certificate_key` instead of using file-path parameters. + +### 4. File not found (CLI): **Cause**: Not in correct directory while running commands **Solution**: Make sure both source.json is present in correct directory and the commands are executed while inside the directory -### 4. file name too long & FATAL error occurred while reading records: failed to finish backfill chunk 381: main writer closed: +### 5. file name too long & FATAL error occurred while reading records: failed to finish backfill chunk 381: main writer closed: **Cause**: The generated file or directory name exceeded the Linux limit of 255 bytes (often happens when partitioning on very long string values). ```bash diff --git a/docs/connectors/mssql/index.mdx b/docs/connectors/mssql/index.mdx index 4d197c78c..36d077afd 100644 --- a/docs/connectors/mssql/index.mdx +++ b/docs/connectors/mssql/index.mdx @@ -5,18 +5,36 @@ sidebar_label: MSSQL --- import DateTimeHandling from '@site/src/components/DateTimeHandling'; +import TOCTabLinker from '@site/src/components/TOCTabLinker'; # Overview {#overview} -The OLake MSSQL Source connector supports multiple synchronization modes. It offers features like parallel chunking, checkpointing, and automatic resume for failed full loads. This connector can be used within the OLake UI or run locally via Docker for open-source workflows. +The OLake Go MSSQL Source connector supports multiple synchronization modes. It offers features like parallel chunking, checkpointing, and automatic resume for failed full loads. This connector can be used within the OLake UI or run locally via Docker for open-source workflows. ## Sync Modes Supported - **Full Refresh** -- [**Full Refresh + CDC**](/docs/features/#2-sync-modes-supported) -- [**CDC Only**](/docs/features/#2-sync-modes-supported) +- **Full Refresh + CDC** +- **CDC Only** - **Full Refresh + Incremental** +:::info **CHUNKING PERFORMANCE FOR TABLES WITHOUT PRIMARY KEYS** + +This is optional and not mandatory. If you want faster chunking for tables without primary keys, you can grant the following permission: + +**SQL Server 2016-2019:** + +```sql +GRANT VIEW DATABASE STATE TO ; +``` + +**SQL Server 2022 or later:** + +```sql +GRANT VIEW DATABASE PERFORMANCE STATE TO ; +``` +::: + ## Prerequisites ### Version Prerequisites @@ -25,7 +43,7 @@ The OLake MSSQL Source connector supports multiple synchronization modes. It off ### CDC Prerequisites -To use CDC with OLake, you must enable CDC on both the database and individual tables you want to capture. +To use CDC with OLake Go, you must enable CDC on both the database and individual tables you want to capture. :::info CDC change tables retain events for **3 days** by default (configurable via `sys.sp_cdc_change_job`). Reducing retention raises the risk of losing unconsumed events during downtime; if events expire before processing, a **full backfill** is required to restore consistency. It is recommended to **increase or adjust the retention period** based on your workload and recovery requirements. @@ -133,7 +151,7 @@ EXEC sys.sp_cdc_enable_table ``` :::info -OLake also provides an option to automate this process. Instead of performing these steps manually, user can simply enable the **Manage Capture Instance** toggle in the UI and OLake will handle capture instance management. For this to work, the capture user must have `db_owner` role membership on the source database. This can be achieved by running: +OLake Go also provides an option to automate this process. Instead of performing these steps manually, user can simply enable the **Manage Capture Instance** toggle in the UI and OLake Go will handle capture instance management. For this to work, the capture user must have `db_owner` role membership on the source database. This can be achieved by running: ```sql ALTER ROLE db_owner ADD MEMBER [username]; @@ -145,6 +163,30 @@ ALTER ROLE db_owner ADD MEMBER [username]; - **Columnstore indexes**: CDC cannot be enabled on tables with a clustered columnstore index. Starting with SQL Server 2016, it can be enabled on tables with a nonclustered columnstore index. - **Computed columns**: CDC doesn't support values for computed columns, even if defined as persisted. Computed columns included in a capture instance will always have a value of `NULL`. This is intended behavior, not a bug. +#### 5. Using a Read-Only Secondary Replica + +To connect the MSSQL source to a read-only secondary replica, set the following JDBC URL parameter in the source configuration: + +```json +"jdbc_url_params": { + "ApplicationIntent": "ReadOnly" +} +``` + +In **OLake UI**, add this under **JDBC URL Parameters** as a key-value pair: `ApplicationIntent` (key) and `ReadOnly` (value). + +:::info +CDC must be enabled on the primary database. +::: + +If you connect to a read-only secondary replica and enable **Manage Capture Instance**, OLake Go prompts for **primary database credentials** so it can create and manage capture instances on the primary. Without primary configuration, capture instances cannot be managed automatically. + +If you prefer not to provide primary database details, you can still sync from the secondary replica by managing capture instances manually on the primary and leaving **Manage Capture Instance** disabled. + +:::note SSH Tunnel +When SSH tunneling is enabled, both the primary and secondary database connections must be reachable through the same bastion host. +::: + ### Connection Prerequisites - Read access to the tables for the MSSQL user. @@ -155,13 +197,21 @@ ALTER ROLE db_owner ADD MEMBER [username]; ## Configuration {#configuration} - + - + + + ### 1. Navigate to the Source Configuration Page -1. Complete the [OLake UI Setup Guide](/docs/getting-started/olake-ui) +1. Complete the [OLake UI Setup Guide](/docs/getting-started/quickstart/) 2. After logging in to the OLake UI, select the `Sources` tab from the left sidebar. 3. Click **`Create Source`** on the top right corner. 4. Select **MSSQL** from the connector dropdown @@ -185,6 +235,8 @@ ALTER ROLE db_owner ADD MEMBER [username]; | SSL Mode | SSL configuration for the database connection. Contains details such as the SSL mode. | `disable` | | Retry Count | Number of times the retry will take place incase of timeout before failing the sync. | `3` | | Manage Capture Instance | When enabled, it automatically creates and manages CDC capture instances on schema evolution. | `true/false` | +| JDBC URL Parameters | Extra [JDBC URL parameters](https://learn.microsoft.com/en-us/sql/connect/jdbc/setting-the-connection-properties?view=sql-server-ver17#properties) for fine-tuning the connection. | `{"connectTimeout":"20"}` | +| SSH Config | Configure OLake Go to connect through an SSH tunnel. |
    • `No Tunnel`
    • `SSH Key Authentication`
    • `SSH Password Authentication`
    | ### 3. Test Connection @@ -208,15 +260,17 @@ An example `source.json` file will look like this: | Field | Description | Example Value | Type | | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | ------ | -| MSSQL Host `required` | Hostname or IP address of the MSSQL database server. | `MSSQL-host` | String | -| MSSQL Port `required` | TCP port on which the MSSQL listener is accepting connections. | `1433` | Integer | -| Database Name `required` | The name of the target database to connect to. | `olake-db` | String | -| Password `required` | The password corresponding to the provided username for authentication. | `mssqlpwd` | String | -| Username `required` | Database user used to authenticate the connection. | `mssql-user` | String | -| Max Threads | Maximum number of worker threads the connector can spin up for parallel tasks. | `10` | Integer | -| SSL Mode | SSL configuration for the database connection. Contains details such as the SSL mode. | `disable` | Object | -| Retry Count | Number of times the retry will take place incase of timeout before failing the sync. | `3` | Integer | -| Manage Capture Instance | When enabled, it automatically creates and manages CDC capture instances on schema evolution. | `true/false` | Boolean | +| host `required` | Hostname or IP address of the MSSQL database server. | `MSSQL-host` | String | +| port `required` | TCP port on which the MSSQL listener is accepting connections. | `1433` | Integer | +| database `required` | The name of the target database to connect to. | `olake-db` | String | +| password `required` | The password corresponding to the provided username for authentication. | `mssqlpwd` | String | +| username `required` | Database user used to authenticate the connection. | `mssql-user` | String | +| max_threads | Maximum number of worker threads the connector can spin up for parallel tasks. | `10` | Integer | +| ssl | SSL configuration for the database connection. Contains details such as the SSL mode. | `disable` | Object | +| retry_count | Number of times the retry will take place incase of timeout before failing the sync. | `3` | Integer | +| manage_capture_instances | When enabled, it automatically creates and manages CDC capture instances on schema evolution. | `true/false` | Boolean | +| jdbc_url_params | Extra [JDBC URL parameters](https://learn.microsoft.com/en-us/sql/connect/jdbc/setting-the-connection-properties?view=sql-server-ver17#properties) for fine-tuning the connection. | `{"connectTimeout":"20"}` | Object | +| ssh_config | Configure OLake Go to connect through an SSH tunnel. | `{"host": "my-tunnel-host", "port": 22 , "username": "my-tunnel-user", "password": "tunnel-password"}` | Object | Similarly, `destination.json` file can be created inside this folder. For more information, see destination documentation. @@ -232,7 +286,7 @@ check \ --config /mnt/config/source.json ``` -- If OLake is able to connect with MSSQL `{"connectionStatus":{"status":"SUCCEEDED"},"type":"CONNECTION_STATUS"}` response is returned. +- If OLake Go is able to connect with MSSQL `{"connectionStatus":{"status":"SUCCEEDED"},"type":"CONNECTION_STATUS"}` response is returned.     @@ -253,3 +307,13 @@ check \ --- +## Troubleshooting {#troubleshooting} + +### 1. High Database CPU usage during Full Refresh + +Jobs syncing tables without a primary key can consume more CPU because rowid computation is done for the rows. + +**Solution:** For non-primary-key table jobs, if CPU usage is high, reduce `max_threads` in the source configuration or set it to the default value. + +**If the issue is not listed here, post the query on Slack to get it resolved within a few hours.** + diff --git a/docs/connectors/mysql/index.mdx b/docs/connectors/mysql/index.mdx index 2c861964f..0dacbdad0 100644 --- a/docs/connectors/mysql/index.mdx +++ b/docs/connectors/mysql/index.mdx @@ -5,17 +5,20 @@ sidebar_label: MySQL --- import DateTimeHandling from '@site/src/components/DateTimeHandling'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import TOCTabLinker from '@site/src/components/TOCTabLinker'; # Overview {#overview} -In addition to MySQL, OLake also supports MariaDB flavor as CDC source. It supports multiple sync modes and offers features like parallel chunking, checkpointing, and automatic resume for failed full loads. This connector can be used within the OLake UI or run locally via Docker for open-source workflows. +In addition to MySQL, OLake Go also supports MariaDB flavor as CDC source. It supports multiple sync modes and offers features like parallel chunking, checkpointing, and automatic resume for failed full loads. This connector can be used within the OLake UI or run locally via Docker for open-source workflows. ## Sync Modes Supported - **Full Refresh** -- [**Full Refresh + CDC**](/docs/features/#2-sync-modes-supported) -- [**CDC Only**](/docs/features/#2-sync-modes-supported) -- [**Full Refresh + Incremental**](/docs/features/#2-sync-modes-supported) +- **Full Refresh + CDC** +- **CDC Only** +- **Full Refresh + Incremental** ## Prerequisites @@ -50,13 +53,21 @@ In addition to MySQL, OLake also supports MariaDB flavor as CDC source. It suppo ## Configuration {#configuration} - + - + + + ### 1. Navigate to the Source Configuration Page -1. Complete the [OLake UI Setup Guide](/docs/getting-started/olake-ui) +1. Complete the [OLake UI Setup Guide](/docs/getting-started/quickstart/) 2. After logging in to the OLake UI, select the `Sources` tab from the left sidebar. 3. Click **`Create Source`** on the top right corner. 4. Select **MySQL** from the connector dropdown @@ -82,9 +93,9 @@ In addition to MySQL, OLake also supports MariaDB flavor as CDC source. It suppo | Skip TLS Verification | Indicates whether to skip TLS certificate verification. | `false` | | Max Threads | Maximum number of parallel threads for processing or syncing data. | `3` | | Backoff Retry Count | Number of retry attempts for establishing sync with exponential backoff. | `3` | -| SSH Config | Configure OLake to connect through an SSH tunnel. See [SSH Config details](/docs/understanding/terminologies/olake#1-ssh-configuration) for the list of supported parameters. |
    • `No Tunnel`
    • `SSH Key Authentication`
    • `SSH Password Authentication`
    | +| SSH Config | Configure OLake Go to connect through an SSH tunnel. |
    • `No Tunnel`
    • `SSH Key Authentication`
    • `SSH Password Authentication`
    | | SSL Configuration | SSL configuration for the database connection. Contains details such as the SSL mode. | `disable` | -| JDBC URL Parameters | Extra [JDBC URL parameters](https://jdbc.postgresql.org/documentation/use/) for fine-tuning the connection. | `{"connectTimeout":"10000"}` | +| JDBC URL Parameters | Extra [JDBC URL parameters](https://dev.mysql.com/doc/connector-j/en/connector-j-reference-configuration-properties.html) for fine-tuning the connection. | `{"connectTimeout":"10000"}` | ### 3. Test Connection @@ -94,7 +105,7 @@ In addition to MySQL, OLake also supports MariaDB flavor as CDC source. It suppo     - + ### 1. Create Configuration File @@ -120,9 +131,9 @@ An example `source.json` file will look like this: | tls_skip_verify | Indicates whether to skip TLS certificate verification. | `false` | bool | | max_threads | Maximum number of parallel threads for processing or syncing data. | `3` | integer | | backoff_retry_count | Number of retry attempts for establishing sync with exponential backoff. | `3` | integer | -| ssh_config | Configure OLake to connect through an SSH tunnel. See [SSH Config details](/docs/understanding/terminologies/olake#1-ssh-configuration) for parameter descriptions. | `{"host": "my-tunnel-host", "port": 22 , "username": "my-tunnel-user", "password": "tunnel-password"}` | object | +| ssh_config | Configure OLake Go to connect through an SSH tunnel. | `{"host": "my-tunnel-host", "port": 22 , "username": "my-tunnel-user", "password": "tunnel-password"}` | object | | ssl | SSL configuration for the database connection. Contains details such as the SSL mode. | `{"mode": "disable"}` | object | -| jdbc_url_params | Extra [JDBC URL parameters](https://jdbc.postgresql.org/documentation/use/) for fine-tuning the connection. | `{"connectTimeout":"10000"}` | object | +| jdbc_url_params | Extra [JDBC URL parameters](https://dev.mysql.com/doc/connector-j/en/connector-j-reference-configuration-properties.html) for fine-tuning the connection. | `{"connectTimeout":"10000"}` | object | Similarly, `destination.json` file can be created inside this folder. For more information, see destination documentation. @@ -138,7 +149,7 @@ check \ --config /mnt/config/source.json ``` -- If OLake is able to connect with MySQL `{"connectionStatus":{"status":"SUCCEEDED"},"type":"CONNECTION_STATUS"}` response is returned. +- If OLake Go is able to connect with MySQL `{"connectionStatus":{"status":"SUCCEEDED"},"type":"CONNECTION_STATUS"}` response is returned. - In case of connection failure, refer to the [Troubleshooting section](#troubleshooting). diff --git a/docs/connectors/oracle/index.mdx b/docs/connectors/oracle/index.mdx index 7dc2bc8bb..88ae6f9a6 100644 --- a/docs/connectors/oracle/index.mdx +++ b/docs/connectors/oracle/index.mdx @@ -5,10 +5,14 @@ sidebar_label: Oracle --- import DateTimeHandling from '@site/src/components/DateTimeHandling'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import TOCTabLinker from '@site/src/components/TOCTabLinker'; + # Overview {#overview} -The OLake Oracle Source connector supports two synchronization modes. It offers features like parallel chunking, checkpointing, and automatic resume for failed full loads. This connector can be used within the OLake UI or run locally via Docker for open-source workflows. +The OLake Go Oracle Source connector supports two synchronization modes. It offers features like parallel chunking, checkpointing, and automatic resume for failed full loads. This connector can be used within the OLake UI or run locally via Docker for open-source workflows. ## Sync Modes Supported @@ -20,7 +24,7 @@ The OLake Oracle Source connector supports two synchronization modes. It offers ### Version Prerequisites Oracle Database 11g or above (tested with Oracle **19c**, and **23ai**) ### Connection Prerequisites - - Following permissions need to be granted to the user which will be used to connect Oracle DB with OLake. + - Following permissions need to be granted to the user which will be used to connect Oracle DB with OLake Go. **After initial Prerequisites are fulfilled, the configurations for Oracle can be configured.** @@ -29,13 +33,21 @@ The OLake Oracle Source connector supports two synchronization modes. It offers ## Configuration {#configuration} - + + + - + ### 1. Navigate to the Source Configuration Page -1. Complete the [OLake UI Setup Guide](/docs/getting-started/olake-ui) +1. Complete the [OLake UI Setup Guide](/docs/getting-started/quickstart/) 2. After logging in to the OLake UI, select the `Sources` tab from the left sidebar. 3. Click **`Create Source`** on the top right corner. 4. Select **Oracle** from the connector dropdown @@ -53,14 +65,15 @@ The OLake Oracle Source connector supports two synchronization modes. It offers | Host `required` | Hostname or IP address of the Oracle database server. | `oracle-host` | | Port `required` | TCP port on which the Oracle listener is accepting connections. | `1521` | | Connection Type `required` | Method used to establish a connection between the Oracle database server. It can be either Service Name or SID | `Service Name` | -| Service Name^ | Oracle service name that identifies the specific database service to connect to. | `oracle-service-name` | -| SID^ | Oracle DB sid that uniquely identifies the database instance. | `oracle-sid` | +| Service Name `required` | Oracle service name that identifies the specific database service to connect to. | `oracle-service-name` | +| SID `required` | Oracle DB sid that uniquely identifies the database instance. | `oracle-sid` | | Username `required` | Database user used to authenticate the connection. | `oracle-user` | | Password | Password for the specified user. | `oracle-password` | | Max Threads | Maximum number of worker threads the connector can spin up for parallel tasks. | `10` | | Retry Count | Number of times the connector will retry a failed operation before giving up. | `0` | | JDBC URL Parameters | Extra [JDBC URL parameters](https://docs.oracle.com/en/database/oracle/oracle-database/21/jajdb/oracle/jdbc/OracleDriver.html) for fine-tuning the connection. | `{"TIMEOUT": "86400"}` | | SSL Mode | SSL configuration for the database connection. Contains details such as the SSL mode. | `disable` | +| SSH Config | Configure OLake Go to connect through an SSH tunnel. |
    • `No Tunnel`
    • `SSH Key Authentication`
    • `SSH Password Authentication`
    | :::warning Oracle Username Case Sensitivity Oracle Database automatically converts unquoted lowercase usernames to uppercase. If a connection error occurs, try changing the username to all uppercase. @@ -74,7 +87,7 @@ Oracle Database automatically converts unquoted lowercase usernames to uppercase     - + ### 1. Create Configuration File @@ -92,14 +105,15 @@ An example `source.json` file will look like this: | ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | ------- | | host `required` | Hostname or IP address of the Oracle database server. | `"oracle-host"` | String | | port `required` | TCP port on which the Oracle listener is accepting connections. | `1521` | Integer | -| service_name^ | Oracle service name that identifies the specific database service to connect to. | `"oracle-service-name"` | String | -| sid^ | Oracle DB sid that uniquely identifies the database instance. | `"oracle-sid"` | String | +| service_name `required` | Oracle service name that identifies the specific database service to connect to. | `"oracle-service-name"` | String | +| sid `required` | Oracle DB sid that uniquely identifies the database instance. | `"oracle-sid"` | String | | username `required` | Database user used to authenticate the connection. | `"oracle-user"` | String | | password | Password for the specified user. | `"oracle-password"` | String | | max_threads | Maximum number of worker threads the connector can spin up for parallel tasks. | `10` | Integer | | backoff_retry_count | Number of times the retry will take place incase of timeout before failing the sync. | `3` | Integer | | jdbc_url_params | Extra [JDBC URL parameters](https://docs.oracle.com/en/database/oracle/oracle-database/21/jajdb/oracle/jdbc/OracleDriver.html) for fine-tuning the connection. | `{"TIMEOUT": "86400"}` | Object | | ssl | SSL configuration for the database connection. Contains details such as the SSL mode. | `{"mode": "disable"}` | Object | +| ssh_config | Configure OLake Go to connect through an SSH tunnel. | `{"host": "my-tunnel-host", "port": 22 , "username": "my-tunnel-user", "password": "tunnel-password"}` | Object | **^ Only one among `service_name` or `sid` is required** @@ -121,7 +135,7 @@ check \ --config /mnt/config/source.json ``` -- If OLake is able to connect with Oracle DB `{"connectionStatus":{"status":"SUCCEEDED"},"type":"CONNECTION_STATUS"}` response is returned. +- If OLake Go is able to connect with Oracle DB `{"connectionStatus":{"status":"SUCCEEDED"},"type":"CONNECTION_STATUS"}` response is returned. - In case of connection failure, refer to the [Troubleshooting section](#troubleshooting). diff --git a/docs/connectors/postgres/index.mdx b/docs/connectors/postgres/index.mdx index ad5261879..2a5b12d15 100644 --- a/docs/connectors/postgres/index.mdx +++ b/docs/connectors/postgres/index.mdx @@ -5,20 +5,23 @@ sidebar_label: Postgres --- import DateTimeHandling from '@site/src/components/DateTimeHandling'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import TOCTabLinker from '@site/src/components/TOCTabLinker'; # Overview {#overview} -The OLake Postgres Source connector supports multiple synchronization modes. It offers features like parallel chunking, checkpointing, and automatic resume for failed full loads. This connector can be used within the OLake UI or run locally via Docker for open-source workflows. +The OLake Go Postgres Source connector supports multiple synchronization modes. It offers features like parallel chunking, checkpointing, and automatic resume for failed full loads. This connector can be used within the OLake UI or run locally via Docker for open-source workflows. ## Sync Modes Supported - **Full Refresh** -- [**Full Refresh + CDC**](/docs/features/#2-sync-modes-supported) -- [**CDC Only**](/docs/features/#2-sync-modes-supported) +- **Full Refresh + CDC** +- **CDC Only** - **Full Refresh + Incremental** :::danger **wal2json for CDC Deprecated** -OLake has deprecated the `wal2json`-based CDC method. Please use the native `pgoutput` plugin, which is faster and more reliable. +OLake Go has deprecated the `wal2json`-based CDC method. Please use the native `pgoutput` plugin, which is faster and more reliable. For users who previously set up jobs using the wal2json plugin: [wal2json plugin documentation](/docs/connectors/postgres/wal2json_plugin.mdx) ::: @@ -45,13 +48,21 @@ For users who previously set up jobs using the wal2json plugin: [wal2json plugin ## Configuration {#configuration} - + - + + + ### 1. Navigate to the Source Configuration Page -1. Complete the [OLake UI Setup Guide](/docs/getting-started/olake-ui) +1. Complete the [OLake UI Setup Guide](/docs/getting-started/quickstart/) 2. After logging in to the OLake UI, select the `Sources` tab from the left sidebar. 3. Click **`Create Source`** on the top right corner. 4. Select **Postgres** from the connector dropdown @@ -61,7 +72,7 @@ For users who previously set up jobs using the wal2json plugin: [wal2json plugin - Enter Postgres credentials.
    -
    ![Form for creating a Postgres source in OLake, showing fields for endpoint configuration, authentication, and connection options](/img/docs/sources/postgres/postgres-config.webp)
    +
    ![Form for creating a Postgres source in OLake Go, showing fields for endpoint configuration, authentication, and connection options](/img/docs/sources/postgres/postgres-config.webp)
    | Field | Description | Example Value | @@ -76,12 +87,13 @@ For users who previously set up jobs using the wal2json plugin: [wal2json plugin | Max Threads | Maximum number of worker threads the connector can spin up for parallel tasks. | `10` | | Replication Slot `required for CDC` | Logical replication slot (pgoutput) that retains WAL segments until read, ensuring consistent CDC. | `olake-repl-slot` | | Publication `required for CDC` | Name of the PostgreSQL publication that defines which tables and operations (INSERT, UPDATE, DELETE, TRUNCATE) to replicate. Must exist in the same database as the replication slot and include all tables for CDC capture. | `postgres_pub` | +| Schemas | List of schemas for discovery. Helps avoid reading all schemas by limiting discovery to only the specified ones. When omitted, all accessible non-system schemas are discovered. | `public,analytics` | | JDBC URL Parameters | Extra [JDBC URL parameters](https://jdbc.postgresql.org/documentation/use/) for fine-tuning the connection. | `{"connectTimeout":"20"}` | -| SSL Mode | SSL configuration for the database connection. Contains details such as the SSL mode. | `disable` | -| SSH Config | Configure OLake to connect through an SSH tunnel. See [SSH Config details](/docs/understanding/terminologies/olake#1-ssh-configuration) for the list of supported parameters. |
    • `No Tunnel`
    • `SSH Key Authentication`
    • `SSH Password Authentication`
    | +| SSL Mode | SSL configuration for the database connection. Contains details such as the SSL mode. |
    • `require`
    • `disable`
    • `verify-ca`
    • `verify-full`
    | +| SSH Config | Configure OLake Go to connect through an SSH tunnel. |
    • `No Tunnel`
    • `SSH Key Authentication`
    • `SSH Password Authentication`
    | :::warning Important: Unique Replication Slot Required -Do not reuse the same `replication slot` and `publication` across multiple OLake jobs. Sharing them will result in data loss and inconsistencies. +Do not reuse the same `replication slot` and `publication` across multiple OLake Go jobs. Sharing them will result in data loss and inconsistencies. ::: ### 3. Test Connection @@ -92,7 +104,7 @@ Do not reuse the same `replication slot` and `publication` across multiple OLake     - + ### 1. Create Configuration File @@ -117,11 +129,12 @@ An example `source.json` file will look like this: | initial_wait_time `required for CDC` | Idle timeout for pgoutput log reading | `120` | Integer | | replication_slot `required for CDC` | Logical replication slot (pgoutput) that retains WAL segments until read, ensuring consistent CDC. | `"olake-repl-slot"` | String | | publication `required for CDC` | Name of the PostgreSQL publication that defines which tables and operations (INSERT, UPDATE, DELETE, TRUNCATE) to replicate. Must exist in the same database as the replication slot and include all tables for CDC capture. | `"postgres_pub"` | String | +| Schemas | Comma-separated list of schemas for discovery. Helps avoid reading all schemas by limiting discovery to only the specified ones. When omitted, all accessible non-system schemas are discovered. | [`public`,`analytics`] | String [] | | max_threads | Maximum number of worker threads the connector can spin up for parallel tasks. | `10` | Integer | | retry_count | Number of times the retry will take place incase of timeout before failing the sync. | `3` | Integer | | jdbc_url_params | Extra [JDBC URL parameters](https://jdbc.postgresql.org/documentation/use/) for fine-tuning the connection. | `{"connectTimeout":"20"}` | Object | -| ssl | SSL configuration for the database connection. Contains details such as the SSL mode. | `{"mode": "disable"}` | Object | -| ssh_config | Configure OLake to connect through an SSH tunnel. See [SSH Config details](/docs/understanding/terminologies/olake#1-ssh-configuration) for parameter descriptions. | `{"host": "my-tunnel-host", "port": 22 , "username": "my-tunnel-user", "password": "tunnel-password"}` | Object | +| ssl | SSL configuration for the database connection. Contains details such as the SSL mode. |
    • `require`
    • `disable`
    • `verify-ca`
    • `verify-full`
    | Object | +| ssh_config | Configure OLake Go to connect through an SSH tunnel. | `{"host": "my-tunnel-host", "port": 22 , "username": "my-tunnel-user", "password": "tunnel-password"}` | Object | Similarly, `destination.json` file can be created inside this folder. For more information, see destination documentation. @@ -138,7 +151,7 @@ check \ --config /mnt/config/source.json ``` -- If OLake is able to connect with Postgres `{"connectionStatus":{"status":"SUCCEEDED"},"type":"CONNECTION_STATUS"}` response is returned. +- If OLake Go is able to connect with Postgres `{"connectionStatus":{"status":"SUCCEEDED"},"type":"CONNECTION_STATUS"}` response is returned. - In case of connection failure, refer to the [Troubleshooting section](#troubleshooting). @@ -150,7 +163,7 @@ check \ ## Data Type Mapping {#data-type-mapping} - + --- @@ -173,7 +186,7 @@ g_hba.conf entry for host "4.240.65.100", user "product_team", database "postgre ### 2. Connecting to Supabase -OLake can connect seamlessly to [Supabase](https://supabase.com) Postgres databases using any of the three connection modes that Supabase provides: +OLake Go can connect seamlessly to [Supabase](https://supabase.com) Postgres databases using any of the three connection modes that Supabase provides: - **Direct Connection** – Connects directly to the Postgres instance. _Note: By default, Supabase's direct connection supports only **IPv6**. IPv4 support can be enabled via the paid IPv4 add-on._ @@ -188,7 +201,7 @@ The Direct Connection mode is required for CDC (Change Data Capture) operations, #### IP Version Compatibility -OLake fully supports both **IPv4 and IPv6** connections. However, Supabase’s IP compatibility depends on the connection mode and account configuration. +OLake Go fully supports both **IPv4 and IPv6** connections. However, Supabase’s IP compatibility depends on the connection mode and account configuration. To check your machine or cloud environment’s IP version, you can run: @@ -218,7 +231,7 @@ By default, Docker containers support only IPv4, which may block access to Supab #### RLS (Row-Level Security) Considerations -When using a read-only database role for syncing with OLake, ensure that **Row-Level Security (RLS)** is either disabled or properly configured. If RLS policies restrict access, sync operations may complete without error but result in **zero rows being replicated**. +When using a read-only database role for syncing with OLake Go, ensure that **Row-Level Security (RLS)** is either disabled or properly configured. If RLS policies restrict access, sync operations may complete without error but result in **zero rows being replicated**. To allow a specific role to bypass RLS entirely, execute the following command (replace `` with the actual role name): @@ -226,6 +239,38 @@ To allow a specific role to bypass RLS entirely, execute the following command ( ALTER USER WITH BYPASSRLS; ``` +### 3. Failed with non retryable error: LSN not updated after `X` mins + +```logs +Failed with non retryable error: LSN not updated after `X` mins +``` + +This issue occurs when `wal_sender_timeout` is set too low (for example, `30s`). If CDC processing takes longer than that, PostgreSQL closes the replication connection before LSN acknowledgment is completed. + +It is more likely to happen on databases with concurrent writes or heavier CDC load. + +**Solution**: Set `wal_sender_timeout` to `0` (recommended) or a sufficiently large value. + +### 4. Delete Records Only Retain the Primary Key + +By default, WAL only includes the primary key on deletes. In OLake Go, delete records (`_op_type = 'd'`) will therefore only retain the primary key and OLake Go metadata columns and other columns appear blank or null. + +**Solution**: To retain the full row values for deleted records, run the following on the affected table: + +```sql +ALTER TABLE REPLICA IDENTITY FULL; +``` + +### 5. TOAST Columns Return Null on updates to Non-TOAST Columns + +When a table has TOAST columns and an `UPDATE` modifies a non-TOAST column, PostgreSQL does not include the unchanged TOAST column value in the WAL entry. As a result, OLake Go receives no value for that column and writes `null` to the destination. + +**Solution**: To ensure TOAST column values are always included in the WAL output, run the following on the affected table: + +```sql +ALTER TABLE REPLICA IDENTITY FULL; +``` + **If the issue is not listed here, post the query on Slack to get it resolved within a few hours.** ## Changelog diff --git a/docs/connectors/postgres/setup/aurora.mdx b/docs/connectors/postgres/setup/aurora.mdx index 421c4bacd..a566568f3 100644 --- a/docs/connectors/postgres/setup/aurora.mdx +++ b/docs/connectors/postgres/setup/aurora.mdx @@ -29,7 +29,7 @@ Configure these key parameters: - **`rds.logical_replication = 1`** - Enables logical replication (required for pgoutput) - **`wal_level = logical`** - May be set automatically by the above parameter - **`max_wal_senders`** and **`max_replication_slots`** - Set to accommodate your CDC connections - - **`wal_sender_timeout = 0`** - *(Optional)* Prevents timeouts during long snapshots + - **`wal_sender_timeout = 0`** - Prevents timeouts during long snapshots Save the changes after configuration. @@ -199,6 +199,14 @@ ALTER PUBLICATION olake_publication ADD TABLE test_table; INSERT INTO test_table (data) VALUES ('PostgreSQL pgoutput CDC test - INSERT'); ``` +:::note Delete records and full row values +By default, WAL only includes the primary key on deletes. In OLake, delete records (`_op_type = 'd'`) will therefore only retain the primary key and OLake metadata columns — other columns appear blank or null. To retain the full deleted row run the following query: + +```sql +ALTER TABLE test_table REPLICA IDENTITY FULL; +``` +::: + **Check for changes using pg_logical_slot_peek_binary_changes:** ```sql diff --git a/docs/connectors/postgres/setup/azure.mdx b/docs/connectors/postgres/setup/azure.mdx index c5b259e36..7e3d4b39b 100644 --- a/docs/connectors/postgres/setup/azure.mdx +++ b/docs/connectors/postgres/setup/azure.mdx @@ -35,7 +35,7 @@ In the Azure Portal, navigate to your PostgreSQL Flexible Server. Go to `Setting | `wal_level` | `logical` | Enable logical replication | | `max_replication_slots` | `≥5` | Number of concurrent CDC connections | | `max_wal_senders` | `≥7` | Should exceed replication slots | - | `wal_sender_timeout` | `0` | *(Optional)* Prevents snapshot timeouts | + | `wal_sender_timeout` | `0` | Prevents snapshot timeouts | ![azure-server-parameters](/img/docs/cdc/postgres/azure-server-parameters.webp) @@ -225,6 +225,14 @@ ALTER PUBLICATION olake_publication ADD TABLE test_table; INSERT INTO test_table (data) VALUES ('PostgreSQL pgoutput CDC test - INSERT'); ``` +:::note Delete records and full row values +By default, WAL only includes the primary key on deletes. In OLake, delete records (`_op_type = 'd'`) will therefore only retain the primary key and OLake metadata columns — other columns appear blank or null. To retain the full deleted row run the following query: + +```sql +ALTER TABLE test_table REPLICA IDENTITY FULL; +``` +::: + **Check for changes using pg_logical_slot_peek_binary_changes:** ```sql diff --git a/docs/connectors/postgres/setup/gcp.mdx b/docs/connectors/postgres/setup/gcp.mdx index 455d7ce43..e2ba58ebc 100644 --- a/docs/connectors/postgres/setup/gcp.mdx +++ b/docs/connectors/postgres/setup/gcp.mdx @@ -205,6 +205,14 @@ ALTER PUBLICATION olake_publication ADD TABLE test_table; INSERT INTO test_table (data) VALUES ('PostgreSQL pgoutput CDC test - INSERT'); ``` +:::note Delete records and full row values +By default, WAL only includes the primary key on deletes. In OLake, delete records (`_op_type = 'd'`) will therefore only retain the primary key and OLake metadata columns — other columns appear blank or null. To retain the full deleted row run the following query: + +```sql +ALTER TABLE test_table REPLICA IDENTITY FULL; +``` +::: + **Check for changes using pg_logical_slot_peek_binary_changes:** ```sql diff --git a/docs/connectors/postgres/setup/generic.mdx b/docs/connectors/postgres/setup/generic.mdx index aced37c16..af29a2249 100644 --- a/docs/connectors/postgres/setup/generic.mdx +++ b/docs/connectors/postgres/setup/generic.mdx @@ -49,7 +49,7 @@ wal_level = logical max_replication_slots = 10 max_wal_senders = 10 -# Optional: Prevent timeouts during long snapshots +# Prevent timeouts during long snapshots wal_sender_timeout = 0 # Optional: Control WAL retention @@ -151,6 +151,10 @@ ALTER DEFAULT PRIVILEGES IN SCHEMA schema1 GRANT SELECT ON TABLES TO cdc_user; - `INDEX`: Uses a specific unique index - `NOTHING`: Only INSERT operations are replicated +:::note Delete records and full row values +By default, WAL only includes the primary key on deletes. In OLake, delete records (`_op_type = 'd'`) will therefore only retain the primary key and OLake metadata columns — other columns appear blank or null. To retain the full deleted row, use **Option 2** below. +::: + **Set Replica Identity for Tables:** ```sql diff --git a/docs/connectors/postgres/setup/rds.mdx b/docs/connectors/postgres/setup/rds.mdx index 5e27bab88..3e5070be3 100644 --- a/docs/connectors/postgres/setup/rds.mdx +++ b/docs/connectors/postgres/setup/rds.mdx @@ -32,7 +32,7 @@ In the AWS RDS console, go to `Parameter Groups` and create a new parameter grou | `wal_level` | `logical` | Usually set automatically by above | | `max_replication_slots` | `≥5` | Number of concurrent CDC connections | | `max_wal_senders` | `≥7` | Should exceed replication slots | -| `wal_sender_timeout` | `0` | *(Optional)* Prevents snapshot timeouts | +| `wal_sender_timeout` | `0` | Prevents snapshot timeouts | Save the parameter group changes. @@ -200,6 +200,14 @@ ALTER PUBLICATION olake_publication ADD TABLE test_table; INSERT INTO test_table (data) VALUES ('PostgreSQL pgoutput CDC test - INSERT'); ``` +:::note Delete records and full row values +By default, WAL only includes the primary key on deletes. In OLake, delete records (`_op_type = 'd'`) will therefore only retain the primary key and OLake metadata columns — other columns appear blank or null. To retain the full deleted row run the following query: + +```sql +ALTER TABLE test_table REPLICA IDENTITY FULL; +``` +::: + **Check for changes using pg_logical_slot_peek_binary_changes:** ```sql diff --git a/docs/connectors/s3/index.mdx b/docs/connectors/s3/index.mdx index 4af2c5f7f..1bb5e063b 100644 --- a/docs/connectors/s3/index.mdx +++ b/docs/connectors/s3/index.mdx @@ -5,10 +5,11 @@ sidebar_label: S3 --- import DateTimeHandling from '@site/src/components/DateTimeHandling'; +import TOCTabLinker from '@site/src/components/TOCTabLinker'; # Overview {#overview} -The OLake S3 Source connector ingests data from Amazon S3 or S3-compatible storage (MinIO, LocalStack). It offers features like parallel chunking, checkpointing, and automatic resume for failed full loads. This connector can be used within the OLake UI or run locally via Docker for open-source workflows. +The OLake Go S3 Source connector ingests data from Amazon S3 or S3-compatible storage (MinIO, LocalStack). It offers features like parallel chunking, checkpointing, and automatic resume for failed full loads. This connector can be used within the OLake UI or run locally via Docker for open-source workflows. ## Key Features @@ -35,7 +36,7 @@ The OLake S3 Source connector ingests data from Amazon S3 or S3-compatible stora ## Sync Modes Supported - **Full Refresh** -- **Incremental** +- **Full Refresh + Incremental**
    How Incremental Sync Works @@ -105,7 +106,7 @@ The OLake S3 Source connector ingests data from Amazon S3 or S3-compatible stora } ``` - Replace `` with your actual S3 bucket name. - - This policy provides read-only access required for the OLake S3 source connector. + - This policy provides read-only access required for the OLake Go S3 source connector. **After initial Prerequisites are fulfilled, the configurations for S3 can be configured.** @@ -113,14 +114,22 @@ The OLake S3 Source connector ingests data from Amazon S3 or S3-compatible stora ## Configuration {#configuration} - + - + + + ### 1. Navigate to the Source Configuration Page -1. Complete the [OLake UI Setup Guide](/docs/install/olake-ui) -2. After logging in to the OlakeUI, select the `Sources` tab from the left sidebar +1. Complete the [OLake UI Setup Guide](/docs/getting-started/quickstart/) +2. After logging in to the OLake UI, select the `Sources` tab from the left sidebar 3. Click **`Create Source`** on the top right corner 4. Select **S3** from the connector dropdown 5. Provide a name for this source @@ -209,7 +218,7 @@ No additional configuration required for Parquet files. ### 1. Create Configuration File - - Once the Olake CLI is setup, create a folder to store configuration files such as `source.json` and `destination.json`. + - Once the OLake CLI is setup, create a folder to store configuration files such as `source.json` and `destination.json`. The `source.json` file for postgres must contain these mandatory fields. @@ -276,7 +285,7 @@ check \ --config /mnt/config/source.json ``` -- If OLake is able to connect with S3 `{"connectionStatus":{"status":"SUCCEEDED"},"type":"CONNECTION_STATUS"}` response is returned. +- If OLake Go is able to connect with S3 `{"connectionStatus":{"status":"SUCCEEDED"},"type":"CONNECTION_STATUS"}` response is returned. - In case of connection failure, refer to the [Troubleshooting section](#troubleshooting). diff --git a/docs/core/architecture.mdx b/docs/core/architecture.mdx index 8ace8aba7..15dc6fdbe 100644 --- a/docs/core/architecture.mdx +++ b/docs/core/architecture.mdx @@ -5,21 +5,14 @@ sidebar_label: Architecture sidebar_position: 1 --- -# Architecture +# OLake Go Architecture -## OLake Ingestion Architecture - -For an in-depth look at OLake's architecture, including [chunking strategies](/blog/what-makes-olake-fast), [concurrency models](/blog/what-makes-olake-fast), and [state management](/blog/what-makes-olake-fast), explore our blog: [**A Deep Dive into OLake Architecture and Inner Workings**](/blog/olake-architecture-deep-dive). +For an in-depth look at OLake Go's architecture, including chunking strategies, concurrency models, and state management, explore our blog: [**A Deep Dive into OLake Go Architecture and Inner Workings**](/blog/olake-architecture-deep-dive).
    ![OLake architecture diagram with connectors between user, database, and lakehouse](/img/docs/architecture.webp)
    -## OLake Optimization Architecture - -
    -![OLake fusion architecture diagram](/img/docs/fusion-architecture.webp) -
    \ No newline at end of file diff --git a/docs/core/use-cases.mdx b/docs/core/use-cases.mdx index 13d504522..744273b75 100644 --- a/docs/core/use-cases.mdx +++ b/docs/core/use-cases.mdx @@ -6,12 +6,12 @@ sidebar_position: 3 --- -# Use Cases for OLake +# Use Cases for OLake Go ### 1. Offloading OLTP Databases for Analytics Running complex analytical queries directly on production **OLTP (Online Transaction Processing) databases** can degrade performance and affect transactional workloads. -OLake addresses this by replicating data from **MySQL**, **PostgreSQL**, **Oracle**, and **MongoDB** into an [**Apache Iceberg**](/iceberg/why-iceberg) based data lake. +OLake Go addresses this by replicating data from **MySQL**, **PostgreSQL**, **Oracle**, and **MongoDB** into an [**Apache Iceberg**](/iceberg/why-iceberg) based data lake. This approach provides: @@ -23,12 +23,12 @@ This approach provides: - **Resilience** → Dead Letter Queue (DLQ) ensures schema changes don’t break pipelines. -With OLake, you can maintain stable transactional systems while enabling scalable and reliable analytics on **Apache Iceberg**. +With OLake Go, you can maintain stable transactional systems while enabling scalable and reliable analytics on **Apache Iceberg**. ### 2. Building Open Data Stacks and Scaling Data Engineering -Organizations looking to reduce reliance on proprietary ETL and data warehousing tools can use **OLake** as part of an [**open-source data stack**](/blog/building-open-data-lakehouse-with-olake-presto). By standardizing on **Apache Iceberg** as the table format, OLake ensures broad compatibility with query engines like **Trino**, **Presto**, **Spark**, **Dremio**, and **DuckDB**. +Organizations looking to reduce reliance on proprietary ETL and data warehousing tools can use **OLake Go** as part of an [**open-source data stack**](/blog/building-open-data-lakehouse-with-olake-presto). By standardizing on **Apache Iceberg** as the table format, OLake Go ensures broad compatibility with query engines like **Trino**, **Presto**, **Spark**, **Dremio**, and **DuckDB**. -With its open-source approach, OLake helps teams: +With its open-source approach, OLake Go helps teams: - **Replace managed ETL/replication services** with a community-driven alternative. @@ -43,7 +43,7 @@ Support multiple query engines across different use cases and teams. This enables a **flexible**, **scalable**, and **future-proof data architecture** built on open standards. ### 3. Enabling Near-Real-Time Analytics -Modern applications need fresh data within minutes, not hours. **OLake** enables near-real-time analytics by continuously replicating data from transactional databases using [**CDC**](/docs/understanding/terminologies/general/#39-change-data-capture), often achieving **sub-minute** latency for updates to appear in **Iceberg**. +Modern applications need fresh data within minutes, not hours. **OLake Go** enables near-real-time analytics by continuously replicating data from transactional databases using [**CDC**](/docs/understanding/terminologies/general/#39-change-data-capture), often achieving **sub-minute** latency for updates to appear in **Iceberg**. Key benefits: @@ -56,7 +56,7 @@ Key benefits: This allows teams to run **fast**, **cost-efficient analytics** on frequently updated data. ### 4. Cost-Effective Data Retention and Compliance -Storing historical data for compliance, audits, or analysis can be costly in traditional data warehouses. **OLake** addresses this by replicating data into **Apache Iceberg**, which stores it on cost-efficient object storage (e.g., S3, GCS). +Storing historical data for compliance, audits, or analysis can be costly in traditional data warehouses. **OLake Go** addresses this by replicating data into **Apache Iceberg**, which stores it on cost-efficient object storage (e.g., S3, GCS). With Iceberg, data remains **immediately queryable** across compatible engines—no rehydration needed. Its built-in schema evolution ensures that structural changes over time don’t break access to historical data. @@ -71,7 +71,7 @@ Key benefits: Adapt to schema changes seamlessly with Iceberg. ### 5. Powering AI and ML Data Pipelines -Building effective AI and ML models requires **fresh**, **reliable**, and **structured data**. **OLake** automates the ingestion of transactional data into an **Iceberg-based lakehouse**, ensuring that pipelines always have access to the latest information. +Building effective AI and ML models requires **fresh**, **reliable**, and **structured data**. **OLake Go** automates the ingestion of transactional data into an **Iceberg-based lakehouse**, ensuring that pipelines always have access to the latest information. With continuous updates, [ML feature stores](/blog/apache-iceberg-vs-delta-lake-guide) and training datasets stay current, while Iceberg's compatibility with engines like **PySpark** and **DuckDB** makes it easy to plug into existing data science workflows. This supports faster model development and iteration. @@ -84,7 +84,7 @@ Key benefits: - **Seamlessly integrate** with ML processing engines (e.g., PySpark, DuckDB). ### 6. Simplifying Change Data Capture -Setting up scalable CDC pipelines is often complex. **OLake** makes this easier by providing an open-source solution purpose-built for **database-to-Iceberg replication**. +Setting up scalable CDC pipelines is often complex. **OLake Go** makes this easier by providing an open-source solution purpose-built for **database-to-Iceberg replication**. It uses **log-based CDC** for minimal impact on source databases, supports schema evolution to handle structural changes, and includes a **Dead Letter Queue (DLQ)** for reliable error handling. The design aligns with open-source streaming concepts, ensuring flexibility and robustness. @@ -97,7 +97,7 @@ Key benefits: - Dead Letter Queue for dependable error management. ### 7. Reducing Cloud Data Warehouse Costs -Cloud data warehouses can become expensive due to storage and compute costs. **OLake** helps reduce these expenses by offloading raw, historical, or less frequently used data into an [**Iceberg lakehouse**](/iceberg/move-to-iceberg) on cost-effective object storage. +Cloud data warehouses can become expensive due to storage and compute costs. **OLake Go** helps reduce these expenses by offloading raw, historical, or less frequently used data into an [**Iceberg lakehouse**](/iceberg/move-to-iceberg) on cost-effective object storage. This lets teams keep their warehouse optimized for active data, while still retaining full access to complete datasets in Iceberg. diff --git a/docs/dmsvsolake.mdx b/docs/dmsvsolake.mdx index 7a5bcc410..661a04b46 100644 --- a/docs/dmsvsolake.mdx +++ b/docs/dmsvsolake.mdx @@ -1,18 +1,18 @@ --- title: "OLake vs AWS DMS | Data Migration Performance & Cost 2025" description: "Compare OLake and AWS DMS on PostgreSQL migration speed, resource use, and cost. OLake offers faster sync, lower memory use, and cost savings." -sidebar_label: AWS DMS vs OLake +sidebar_label: AWS DMS vs OLake Go sidebar_position: 3 --- import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -## OLake vs AWS DMS Benchmark +## OLake Go vs AWS DMS Benchmark In today's data-driven world, organizations are migrating **massive datasets** from traditional databases to cloud storage. But here's the million-dollar question: **Which tool can handle multi billion records efficiently without breaking the bank?** -The following benchmark evaluates **performance**, **environment** configuration, and **cost** considerations for migrating PostgreSQL data to Parquet (AWS S3) using **AWS Database Migration Service (DMS)** and **OLake**. +The following benchmark evaluates **performance**, **environment** configuration, and **cost** considerations for migrating PostgreSQL data to Parquet (AWS S3) using **AWS Database Migration Service (DMS)** and **OLake Go**. ## Workload @@ -37,16 +37,16 @@ The full-load test evaluates the time and throughput required to transfer the co | Tool | Rows Processed | Total Time | Avg Throughput (rows/sec) | | ------ | --------------: | ---------: | -------------------------: | -| OLake | 4,008,587,913 | 1 h 59 m | 558,765 | +| OLake Go | 4,008,587,913 | 1 h 59 m | 558,765 | | AWS DMS| 4,008,587,913 | 9 h 8 m | 122,000 | **Key observations:** -- **OLake** achieved a **4.6× faster** full refresh performance compared to **AWS DMS**. -- OLake eliminates the need for manual partition-boundary scripting, while DMS relies on manual boundary generation to achieve PostgreSQL parallelization. +- **OLake Go** achieved a **4.6× faster** full refresh performance compared to **AWS DMS**. +- OLake Go eliminates the need for manual partition-boundary scripting, while DMS relies on manual boundary generation to achieve PostgreSQL parallelization. **Parallelism used:** -- OLake ran with **32 threads**. +- OLake Go ran with **32 threads**. - DMS ran with: **40 parallel tasks**. ## CDC Results (Insert-only; 50M rows) @@ -56,45 +56,45 @@ The CDC test evaluates the sustained ingestion of incremental changes from Postg | Tool | Rows Processed | Total Time | Avg Throughput (rows/sec) | | ------ | --------------: | ---------: | -------------------------: | | AWS DMS| 50,000,000 | 22 m 41 s | 36,738 | -| OLake | 50,000,000 | 16 m 24 s | 50,812 | +| OLake Go | 50,000,000 | 16 m 24 s | 50,812 | **Key Observations:** -- **OLake** performed the CDC workload approximately **1.38× faster** than **AWS DMS**. -- OLake requires minimal configuration , selecting the Full Refresh + CDC Sync Mode option is sufficient, whereas DMS requires enabling pglogical via Azure extensions, performing a detailed pglogical setup, and specifying the starting LSN for replication tasks. +- **OLake Go** performed the CDC workload approximately **1.38× faster** than **AWS DMS**. +- OLake Go requires minimal configuration , selecting the Full Refresh + CDC Sync Mode option is sufficient, whereas DMS requires enabling pglogical via Azure extensions, performing a detailed pglogical setup, and specifying the starting LSN for replication tasks. **Parallelism used:** -- OLake ran with **32 threads**. +- OLake Go ran with **32 threads**. - DMS ran with: **40 parallel tasks**. ## Resource Utilization We’re focusing on resource utilization for the Full Refresh process, as it’s significantly more resource-intensive than CDC. -The memory utilization shown corresponds to a transfer of **~4 billion records**. As the data volumes increase, being memory efficiency becomes highly crucial — an area where **OLake** excels. +The memory utilization shown corresponds to a transfer of **~4 billion records**. As the data volumes increase, being memory efficiency becomes highly crucial — an area where **OLake Go** excels. -| Memory Stats | OLake | DMS | +| Memory Stats | OLake Go | DMS | |--------------|-------|-----| | Min | 4.19 GB | 20 GB | | Max | 50.55 GB | 40 GB | | Mean | 31.76 GB | 30 GB | -**OLake's** memory usage ranged from 4.19 GB to 50.55 GB, **averaging 31.76 GB** throughout the transfer. For **DMS**, the memory usage ranged from 20 GB to 40 GB, **averaging 30 GB** throughout the transfer. +**OLake Go's** memory usage ranged from 4.19 GB to 50.55 GB, **averaging 31.76 GB** throughout the transfer. For **DMS**, the memory usage ranged from 20 GB to 40 GB, **averaging 30 GB** throughout the transfer. ## Cost Comparison (Compute Only) -Compute costs scale linearly with runtime at a given instance class for both OLake and DMS. +Compute costs scale linearly with runtime at a given instance class for both OLake Go and DMS. Here is the cost comparison for **Full Refresh** process. | Scenario | Instance | Runtime | Approx. Cost | | --------------------- | ---------------- | -------: | -----------: | | DMS Full Refresh | c6i.16xlarge | 9h 08m | ~$28.03 | -| OLake Full Refresh | c6i.16xlarge | 1h 59m | ~$6.08 | +| OLake Go Full Refresh | c6i.16xlarge | 1h 59m | ~$6.08 | **Important-** -**OLake** is delivering **4.61x cost savings** as compared to **DMS** on compute alone. When you're moving terabytes monthly, those savings add up fast! +**OLake Go** is delivering **4.61x cost savings** as compared to **DMS** on compute alone. When you're moving terabytes monthly, those savings add up fast! **Scaling Impact-** @@ -105,19 +105,19 @@ Assume that the same dataset is migrated **"once daily"** for the durations spec | Tool | 1 Month Cost | 2 Months Cost | 6 Months Cost | |------|--------------|---------------|-------------| | DMS | ~$840 | ~$1,681 | ~$5045 | -| OLake| ~$182 | ~$364 | ~$1094 | +| OLake Go| ~$182 | ~$364 | ~$1094 | The cost trends over time make it evident which tool is more cost-efficient. Considering, typical workflows involve **more than one daily sync**, so the costs would increase from the baseline shown here. :::info[Point to Remember] -OLake is open source and incurs no licensing fees ; costs depend solely on the user’s infrastructure and storage consumption +OLake Go is open source and incurs no licensing fees ; costs depend solely on the user’s infrastructure and storage consumption ::: ## Dataset and Table Schemas -The OLake benchmarks page provides NYC Taxi table schemas designed to support both bulk transfer and CDC scenarios at scale, ensuring comparability across different tools. +The OLake Go benchmarks page provides NYC Taxi table schemas designed to support both bulk transfer and CDC scenarios at scale, ensuring comparability across different tools. The dataset and reproducible setup are available in the GitHub repository: [NYC Taxi Data Benchmark](https://github.com/datazip-inc/nyc-taxi-data-benchmark/tree/remote-postgres). @@ -203,7 +203,7 @@ CONSTRAINT fhv_trips_pkey PRIMARY KEY (id) - DMS PostgreSQL full loads may require manual task mapping with partition boundaries to achieve higher parallelism. - DMS CDC for Azure PostgreSQL required enabling `pglogical` via `azure.extensions` and configuring replication sets with aligned start LSN. -- OLake parallelization and schema handling are automatic; no manual boundary generation was required in this benchmark. +- OLake Go parallelization and schema handling are automatic; no manual boundary generation was required in this benchmark. diff --git a/docs/features/index.mdx b/docs/features/index.mdx deleted file mode 100644 index b4c55b87f..000000000 --- a/docs/features/index.mdx +++ /dev/null @@ -1,275 +0,0 @@ ---- -title: "Data Sync Features & Schema Evolution | OLake Platform Guide" -description: "Explore OLake features: parallel chunking, stateful sync, CDC, deduplication, partitioning, and schema evolution for resilient data pipelines." -sidebar_label: Overview ---- - -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; - - - - - -## Source Level Capabilities - -### 1. [Parallelised Chunking](/blog/what-makes-olake-fast) - -Parallel chunking is a technique that splits large datasets or collections into smaller virtual chunks, allowing them to be read and processed simultaneously. It is used in sync modes such as Full Refresh, Full Refresh + CDC, and Full Refresh + Incremental. - -**What it does**: -- Splits big collections into manageable pieces without altering the underlying data. -- Each chunk can be processed parallely & independently. - -**Benefit**: -- Enables **parallel reads**, dramatically reducing the time needed to perform full snapshots or scans of large datasets. -- Improves ingestion speed, scalability, and overall system performance. - -### 2. [Sync Modes](/blog/what-makes-olake-fast) Supported - -OLake supports following sync modes to provide flexibility across use cases: - -- **Full Refresh** → Loads the complete table from the source. -- **Full Refresh + Incremental** → Performs an initial full load, then captures subsequent changes using incremental logic (Primary or Fallback Cursor). -- **Full Refresh + CDC** → Performs an initial full load, then continuously captures inserts, updates, and deletes in near real-time via Change Data Capture (CDC). -- **Strict CDC** → Only captures changes from the source database logs (inserts, updates, deletes), without performing an initial full load. - -### 3. Stateful, Resumable Syncs - -OLake ensures that data syncs resume automatically from the last checkpoint after interruptions. Applicable for Full Refresh + CDC and Full Refresh + Incremental & Strict CDC sync modes. - -**What it does**: -- Maintains **state** of in-progress syncs. -- Automatically **resumes** after crashes, network failures, or manual pauses. -- Eliminates the need for restarting jobs from scratch. - -**Benefit**: -- Reduces data duplication and processing time. -- Ensures **reliable**, **fault-tolerant** pipelines. -- Minimizes manual intervention for operational teams - -### 4. Configurable Max Connections - -OLake allows configuring the maximum number of database connections per source, helping prevent overload and ensuring stable performance on the source system. - -### 5. Exact Source Data Type Mapping - -OLake guarantees accurate mapping of source database types to Iceberg, maintaining schema integrity and ensuring reliable data replication. - -### 6. Data Filter - -Data Filters let you replicate only the rows you need, based on specified column values, during Full Refreshes syncs (Full Refresh, Full Refresh + Incremental and Full Refresh + CDC). By filtering at the source, they reduce database load, save storage and processing resources, and make downstream queries faster and more efficient. - -
    - -## Destination Level Capabilities - -### 1. Data Deduplication - -Data Deduplication ensures that only unique records are stored and processed : saving space, reducing costs, and improving data quality. OLake automatically deduplicates data using the primary key from the source tables, guaranteeing that each primary key maps to a single row in the destination along with its corresponding olake_id. - -### 2. Hive Style Partitioning - -Partitioning is the process of dividing large datasets into smaller, more manageable segments based on specific column values (e.g., date, region, or category), improving query performance, scalability, and data organization - -- [**Iceberg partitioning**](/docs/writers/iceberg/partitioning/) → Metadata-driven, no need for directory-based partitioning; enables efficient pruning and schema evolution. -- **S3-style partitioning** → Traditional folder-based layout (e.g., `year=2025/month=08/day=22/`) for compatibility with external tools. -- **Normalization** → Automatically expands **level-1 nested JSON fields** into top-level columns. - -**What it does**: - -- Converts nested JSON objects into **flat columns** for easier querying. -- Preserves all data while simplifying structure. - -**Benefit**: - -- Makes **SQL queries simpler and faster**. -- Reduces the need for complex JSON parsing in queries. -- Improves readability and downstream analytics efficiency. - -### 3. [Schema Evolution](/blog/2025/10/03/iceberg-metadata) & Data Types Changes - -OLake automatically handles changes in your table's schema without breaking downstream jobs. Read More [Schema Evolution in OLake](/docs/features?tab=schema-evolution) - -**What it does**: -- Detects **column additions**, **deletions**, or **renames**. -- Supports **data type promotions as of Iceberg v2** (e.g., `int → long`, `float → double`). -- Updates table metadata seamlessly. - -**Benefit**: - -- Ensures pipeline stability even as source schemas evolve. -- Eliminates costly manual migrations or pipeline rewrites. -- Keeps data consistent and queries reliable at scale. - -### 4. Append Mode - -Adds all incoming data from the source to the destination table without performing deduplication. Full load always runs in append mode, and for CDC or incremental syncs, it can be used to disable upsert behavior. Upsert mode ensures no duplicate records by writing delete entries for existing rows before inserting new ones. - -### 5. Dead Letter Queue Columns (WIP) - -The DLQ column handles values with data type changes not supported by Iceberg / Parquet destinations type promotions, safely storing them without loss. This prevents sync failures and ensures downstream models remain stable. By isolating incompatible values, it allows users to continue syncing data seamlessly while addressing type mismatches at their convenience, improving reliability and reducing manual intervention. - -     - - - -# Schema Evolution and Data Type Changes - -This document explains how OLake handles schema changes and data type changes in your data pipelines. It covers two distinct features that help maintain pipeline resilience when your source data structures evolve. - -## Schema Evolution - -Schema evolution refers to changes in your database structure like adding, removing, or renaming columns and tables. OLake handles these changes to prevent pipeline failures and data loss. - -### Schema Evolution — Column-Level Changes - -| Change Type | How OLake Detects & Handles It | Typical Pipeline Impact | Extra Details & Tips | -|------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **Adding a column** | OLake runs a *schema discovery* at the start of every sync. When a new source column appears, it is **automatically** added to the Iceberg schema (new field-ID) and starts receiving values immediately. If the source back-fills historical rows, CDC registers them as updates. **No user action required.** | **No breakage.** Historical rows show `NULL` until back-filled. | • Monitor write throughput if a back-fill is large. | -| **Deleting a column** | Schema discovery also detects when a source column has been removed.
    After discovery confirms the column is no longer in the source, OLake updates the Iceberg schema accordingly.
    The deleted column still exists in the destination schema, so old snapshots remain queryable. | **No breakage.** ETL continues with a “virtual” column (null-filled). | • BI tools won’t break, but may show the column full of nulls — communicate schema changes.
    • Run a *rewrite manifests* job later to drop the dead column if storage footprint matters. | -| **Renaming a column** | Column renames are also detected during *schema discovery*.

    When a source column is renamed, OLake interprets this as:
    → The old column remains in the destination (but no new values are written).
    → A new column with the updated name is added and starts receiving incoming data.

    *WIP:* Because Iceberg keeps immutable field IDs, OLake can also just update the column’s name on the same field ID (e.g., `customer_id → client_id`) — avoiding data migration entirely. | **No breakage.** | • Renames are instant — no file rewrites.
    • Update SQL queries downstream to use the new column name. | -| **JSON / Semi-structured key add / remove / rename** | OLake flattens keys to a canonical path inside a single JSON column (or keeps raw JSON).
    • Added keys appear automatically.
    • Removed keys vanish from new rows.
    • Renamed keys are treated as “remove + add” because JSON has no intrinsic field ID. | **No breakage.** | | - - -:::info -- A sparse new column (will not be synced to destination unless there is atleast 1 non `NULL` value). Because Iceberg stores data column-wise (Parquet). -::: - - -### Schema Evolution — Table-Level Changes - -| Change Type | How OLake Detects & Handles It | Typical Pipeline Impact | Extra Details & Tips | -|------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------| -| **Adding a table** | Newly detected source tables appear in the OLake UI list. **You choose which ones to sync.** Once added, OLake applies whichever sync mode you’ve configured. Tables not selected to sync are ignored. | **No breakage.** Pipelines for existing tables run as usual; disabled tables simply do not sync. | Initial full loads run in parallel. | -| **Deleting a table** | No new data will get added to the deleted table. Existing table data and metadata remain queryable. | **No breakage.** Downstream queries on historic data still work; new inserts stop. | If the table is recreated later with the same name but different structure, treat it as a brand-new stream to avoid field-ID collisions. | -| **Renaming a table** | When a source table is renamed, OLake treats it as a new table. It is discovered as a new stream in schema discovery. Once you enable sync for this table, OLake applies the configured sync mode.

    • The old Iceberg table keeps historical data. | **No breakage**, but post-rename data lands in a separate table unless you merge histories. | For continuous history, enable the new table quickly and (optionally) set an alias so both names map to the same Iceberg table. | - - - -## Schema Data Type Changes - -Schema data type changes refer to modifications to the data type of existing columns (e.g., changing `INT` to `BIGINT`). OLake leverages `Apache Iceberg v2` tables' type promotion capabilities to handle compatible changes automatically. - -### Supported Data Type Promotions - -OLake supports Iceberg v2’s widening promotions, where the destination column type expands to accommodate a larger range or higher precision without data loss. These include: - -| From | To | Notes | -| ------------- | --------------------------- | ----------------------------------------------------- | -| INT | LONG (BIGINT) | Widening integers is safe | -| FLOAT | DOUBLE | Promoting to higher precision works without data loss | -| DATE | TIMESTAMP, TIMESTAMP_NS | Dates can be safely converted to timestamps | -| DECIMAL(P, S) | DECIMAL(P', S) where P' > P | Only widening precision is supported | - -:::info -Writing an `INT` value into a `FLOAT` or `DOUBLE` column is supported in Iceberg, as it is considered a safe numeric conversion rather than a schema change.
    Similarly, writing a `BIGINT` value into a `DOUBLE` is also supported. -::: - -:::caution -- Iceberg v2 supports widening type changes only. Narrowing changes (e.g., `BIGINT` to `INT`) along with any other data type changes will result in an errror as are not supported. -::: - -### Handling Incompatible Data Type Changes - -For data type changes not supported by Iceberg v2: -1. **(INT, LONG, FLOAT, DOUBLE) ➡ STRING** - OLake provides enhanced type conversion handling. At the destination if the data type is string, and incoming values are numeric, then the values are converted and stored as string. -2. Narrowing Type Conversions like: - - **BIGINT to INT** - - **DOUBLE to FLOAT** - - When a source value’s data type has a smaller range or precision than the destination column’s type, OLake treats this as a narrowing conversion and handles it seamlessly. For example, if the destination column is defined as BIGINT but the incoming values are INT, OLake recognizes that every INT value falls within BIGINT’s range and simply stores the values without error. This validation step ensures that compatible, smaller-range types are accepted even when Iceberg v2 would flag a mismatch. - -:::tip -Any other incompatible data type changes will be captured by OLake using a Dead Letter Queue column (DLQ) (feature coming soon). -::: - -### Unsupported Data Type Conversions - -The data type changes listed below are unsupported in Iceberg v2 and OLake. Attempting these will result in sync failure. - -1. Attempting to write a `FLOAT` value into an `INT` column. -2. Attempting to write a `STRING` value into `INT/DOUBLE/LONG/FLOAT` column. - -## Example Scenarios - -### Scenario 1: Adding a Column in Source - -When a new column appears in your source data: - -- OLake automatically detects the new column -- The column is added to your destination schema -- New data includes values for this column -- Historical data has null values for this column - -### Scenario 2: Adding a Table / Collection in Source - -When a new table appears in your source database: - -- OLake automatically detects the new table in the next scheduled run -- The table is added to your destination schema -- A New table gets created. - -### Scenario 3: Table Name Change - -When a table name changes in your source database: -- OLake automatically detects the new table name -- The table is added to your destination schema -- A New table gets created -- The old table name is retained in the destination schema but will not be populated with new data - -### Scenario 4: Widening Type Conversion - -The following conversions are handled: -- **INT → BIGINT** -- **FLOAT → DOUBLE** - -When a column data type is `INT` and it encounters `BIGINT` type or when a `FLOAT` column encounters a `DOUBLE` type: - -- OLake detects the widening type change -- Column type is updated in the destination -- All values are properly converted -- Pipeline continues without interruption - -### Scenario 5: Narrowing Type Conversion (BIGINT to INT Conversion) - -The following conversions are handled: -- **BIGINT → INT** -- **DOUBLE → FLOAT** - -When a column data type is `BIGINT` and it encounters `INT` type or when a `DOUBLE` column encounters a `FLOAT` type: - -- OLake detects the narrowing type change -- Column type is validated in the destination making sure it fits in the target type's range -- All values are properly approved -- Pipeline continues without interruption - -### Scenario 6: Incompatible Type Change - -The following conversions are handled: -- **INT → STRING** -- **BIGINT → STRING** -- **FLOAT → STRING** -- **DOUBLE → STRING** - -When a column data type is `STRING` and it encounters `NUMERIC` type: - -- OLake detects the incompatible type change -- Numeric values are converted to their string representation -- Converted values are stored as strings in the destination -- Pipeline continues without interruption - -### Scenario 7: Unsupported Type Change - -When the column data type is `INT` and it encounters `FLOAT` or when a `NUMERIC` column encounters a `STRING` type: - -- OLake detects the type mismatch -- The write is not allowed and the **sync fails** - -For more detailed information on Iceberg's schema evolution capabilities, refer to the [Apache Iceberg documentation](https://iceberg.apache.org/spec/#schema-evolution). - -     - -     - - diff --git a/docs/features/schema.mdx b/docs/features/schema.mdx index d297390fd..518ed791a 100644 --- a/docs/features/schema.mdx +++ b/docs/features/schema.mdx @@ -1,13 +1,187 @@ --- -title: Schema Evolution and Datatype Change (Moved) -description: OLake Schema Evolution and Datatype Change description +title: "Schema Evolution & Data Type Changes | OLake" +description: "Learn how OLake handles schema changes and data type changes in your data pipelines using Apache Iceberg v2." +sidebar_label: Schema Evolution sidebar_position: 2 --- -import Head from '@docusaurus/Head'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; - - - +# Schema Evolution and Data Type Changes -Redirecting to [Features → Schema evolution](/docs/features?tab=schema-evolution)… +This document explains how OLake Go handles schema changes and data type changes in your data pipelines. It covers two distinct features that help maintain pipeline resilience when your source data structures evolve. + +## Schema Evolution + +Schema evolution refers to changes in your database structure like adding, removing, or renaming columns and tables. OLake Go handles these changes to prevent pipeline failures and data loss. + +### Schema Evolution — Column-Level Changes + +| Change Type | How OLake Go Detects & Handles It | Typical Pipeline Impact | Extra Details & Tips | +|------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **Adding a column** | OLake Go runs a *schema discovery* at the start of every sync. When a new source column appears, it is **automatically** added to the Iceberg schema (new field-ID) and starts receiving values immediately. If the source back-fills historical rows, CDC registers them as updates. **No user action required.** | **No breakage.** Historical rows show `NULL` until back-filled. | • Monitor write throughput if a back-fill is large. | +| **Deleting a column** | Schema discovery also detects when a source column has been removed.
    After discovery confirms the column is no longer in the source, OLake Go updates the Iceberg schema accordingly.
    The deleted column still exists in the destination schema, so old snapshots remain queryable. | **No breakage.** ETL continues with a "virtual" column (null-filled). | • BI tools won't break, but may show the column full of nulls — communicate schema changes.
    • Run a *rewrite manifests* job later to drop the dead column if storage footprint matters. | +| **Renaming a column** | Column renames are also detected during *schema discovery*.

    When a source column is renamed, OLake Go interprets this as:
    → The old column remains in the destination (but no new values are written).
    → A new column with the updated name is added and starts receiving incoming data.

    *WIP:* Because Iceberg keeps immutable field IDs, OLake Go can also just update the column's name on the same field ID (e.g., `customer_id → client_id`) — avoiding data migration entirely. | **No breakage.** | • Renames are instant — no file rewrites.
    • Update SQL queries downstream to use the new column name. | +| **JSON / Semi-structured key add / remove / rename** | OLake Go flattens keys to a canonical path inside a single JSON column (or keeps raw JSON).
    • Added keys appear automatically.
    • Removed keys vanish from new rows.
    • Renamed keys are treated as "remove + add" because JSON has no intrinsic field ID. | **No breakage.** | | + + +:::info +- A sparse new column is not synced to the destination until at least one row has a non `NULL` value. Iceberg stores data column-wise in Parquet, so empty columns are omitted. +::: + + +### Schema Evolution — Table-Level Changes + +| Change Type | How OLake Go Detects & Handles It | Typical Pipeline Impact | Extra Details & Tips | +|------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------| +| **Adding a table** | Newly detected source tables appear in the OLake UI list. **You choose which ones to sync.** Once added, OLake Go applies whichever sync mode you've configured. Tables not selected to sync are ignored. | **No breakage.** Pipelines for existing tables run as usual; disabled tables simply do not sync. | Initial full loads run in parallel. | +| **Deleting a table** | No new data will get added to the deleted table. Existing table data and metadata remain queryable. | **No breakage.** Downstream queries on historic data still work; new inserts stop. | If the table is recreated later with the same name but different structure, treat it as a brand-new stream to avoid field-ID collisions. | +| **Renaming a table** | When a source table is renamed, OLake Go treats it as a new table. It is discovered as a new stream in schema discovery. Once you enable sync for this table, OLake Go applies the configured sync mode.

    • The old Iceberg table keeps historical data. | **No breakage**, but post-rename data lands in a separate table unless you merge histories. | For continuous history, enable the new table quickly and (optionally) set an alias so both names map to the same Iceberg table. | + + + +## Schema Data Type Changes + +Schema data type changes refer to modifications to the data type of existing columns (e.g., changing `INT` to `BIGINT`). OLake Go leverages `Apache Iceberg v2` tables' type promotion capabilities to handle compatible changes automatically. + +### Supported Data Type Promotions + +OLake Go supports Iceberg v2's widening promotions, where the destination column type expands to accommodate a larger range or higher precision without data loss. These include: + +| From | To | Notes | +| ------------- | --------------------------- | ----------------------------------------------------- | +| INT | LONG (BIGINT) | Widening integers is safe | +| FLOAT | DOUBLE | Promoting to higher precision works without data loss | +| DATE | TIMESTAMP, TIMESTAMP_NS | Dates can be safely converted to timestamps | +| DECIMAL(P, S) | DECIMAL(P', S) where P' > P | Only widening precision is supported | + +:::info +Writing an `INT` value into a `FLOAT` or `DOUBLE` column is supported in Iceberg, as it is considered a safe numeric conversion rather than a schema change.
    Similarly, writing a `BIGINT` value into a `DOUBLE` is also supported. +::: + +:::caution +- Iceberg v2 supports widening type changes only. Narrowing changes (e.g., `BIGINT` to `INT`) along with any other data type changes will result in an errror as are not supported. +- Parquet files are immutable. Existing files cannot be modified in place, so in-place data type changes and schema evolution at the file level are not supported. +::: + +### Handling Incompatible Data Type Changes + + + + For data type changes not supported by Iceberg v2: + 1. **(INT, LONG, FLOAT, DOUBLE) ➡ STRING** - OLake Go provides enhanced type conversion handling. At the destination if the data type is string, and incoming values are numeric, then the values are converted and stored as string. + 2. Narrowing Type Conversions like: + - **BIGINT to INT** + - **DOUBLE to FLOAT** + + When a source value's data type has a smaller range or precision than the destination column's type, OLake Go treats this as a narrowing conversion and handles it seamlessly. For example, if the destination column is defined as BIGINT but the incoming values are INT, OLake Go recognizes that every INT value falls within BIGINT's range and simply stores the values without error. This validation step ensures that compatible, smaller-range types are accepted even when Iceberg v2 would flag a mismatch. + + :::tip + Any other incompatible data type changes will be captured by OLake Go using a Dead Letter Queue column (DLQ) (feature coming soon). + ::: + + + For Parquet destinations, OLake Go supports widening type conversions in this order: + + **BOOLEAN** → **INT64** → **FLOAT64** → **STRING** + + This means a type can be converted to any type on its right. For example: + - **BOOLEAN** can be converted to **INT64**, **FLOAT64**, or **STRING** + - **INT64** can be converted to **FLOAT64** or **STRING** + - **FLOAT64** can be converted to **STRING** + + + +### Unsupported Data Type Conversions + + + + The data type changes listed below are unsupported in Iceberg v2 and OLake Go. Attempting these will result in sync failure. + + 1. Attempting to write a `FLOAT` value into an `INT` column. + 2. Attempting to write a `STRING` value into `INT/DOUBLE/LONG/FLOAT` column. + + + The data type changes listed below are unsupported in Parquet and OLake Go. Attempting these will result in sync failure. + - `STRING` cannot be converted to `FLOAT64`, `INT64`, or `BOOLEAN` + + + +## Example Scenarios + +### Scenario 1: Adding a Column in Source + +When a new column appears in your source data: + +- OLake Go automatically detects the new column +- New columns are synced automatically only when **Sync new columns automatically** is enabled. If not user has to manually select those columns to be included in the sync +- The column is added to your destination schema +- New data includes values for this column +- Historical data has null values for this column + +### Scenario 2: Adding a Table / Collection in Source + +When a new table appears in your source database: + +- OLake Go automatically detects the new table on the next scheduled run and lists it in the UI +- The table is synced only when you explicitly enable it for the job +- Once enabled, OLake Go creates the destination table and starts syncing using your configured sync mode + +### Scenario 3: Table Name Change + +When a table name changes in your source database: +- OLake Go automatically detects the new table name +- The table is added to your destination schema +- A New table gets created +- The old table name is retained in the destination schema but will not be populated with new data + +### Scenario 4: Widening Type Conversion + +The following conversions are handled: +- **INT → BIGINT** +- **FLOAT → DOUBLE** + +When a column data type is `INT` and it encounters `BIGINT` type or when a `FLOAT` column encounters a `DOUBLE` type: + +- OLake Go detects the widening type change +- Column type is updated in the destination +- All values are properly converted +- Pipeline continues without interruption + +### Scenario 5: Narrowing Type Conversion (BIGINT to INT Conversion) + +The following conversions are handled: +- **BIGINT → INT** +- **DOUBLE → FLOAT** + +When a column data type is `BIGINT` and it encounters `INT` type or when a `DOUBLE` column encounters a `FLOAT` type: + +- OLake Go detects the narrowing type change +- Column type is validated in the destination making sure it fits in the target type's range +- All values are properly approved +- Pipeline continues without interruption + +### Scenario 6: Incompatible Type Change + +The following conversions are handled: +- **INT → STRING** +- **BIGINT → STRING** +- **FLOAT → STRING** +- **DOUBLE → STRING** + +When a column data type is `STRING` and it encounters `NUMERIC` type: + +- OLake Go detects the incompatible type change +- Numeric values are converted to their string representation +- Converted values are stored as strings in the destination +- Pipeline continues without interruption + +### Scenario 7: Unsupported Type Change + +When the column data type is `INT` and it encounters `FLOAT` or when a `NUMERIC` column encounters a `STRING` type: + +- OLake Go detects the type mismatch +- The write is not allowed and the **sync fails** + +For more detailed information on Iceberg's schema evolution capabilities, refer to the [Apache Iceberg documentation](https://iceberg.apache.org/spec/#schema-evolution). diff --git a/docs/fusion/community/channels.mdx b/docs/fusion/community/channels.mdx new file mode 100644 index 000000000..dc77a2aa2 --- /dev/null +++ b/docs/fusion/community/channels.mdx @@ -0,0 +1,8 @@ +--- +title: Channels +sidebar_label: Channels +--- + +# Channels + +Coming soon. diff --git a/docs/fusion/community/code-of-conduct.mdx b/docs/fusion/community/code-of-conduct.mdx new file mode 100644 index 000000000..4ed0164e6 --- /dev/null +++ b/docs/fusion/community/code-of-conduct.mdx @@ -0,0 +1,8 @@ +--- +title: Code of Conduct +sidebar_label: Code of Conduct +--- + +# Code of Conduct + +Coming soon. diff --git a/docs/fusion/community/contributing.mdx b/docs/fusion/community/contributing.mdx new file mode 100644 index 000000000..5d13a4d45 --- /dev/null +++ b/docs/fusion/community/contributing.mdx @@ -0,0 +1,8 @@ +--- +title: Contributing +sidebar_label: Contributing +--- + +# Contributing + +Coming soon. diff --git a/docs/fusion/community/issues-and-prs.mdx b/docs/fusion/community/issues-and-prs.mdx new file mode 100644 index 000000000..2a1b16bf2 --- /dev/null +++ b/docs/fusion/community/issues-and-prs.mdx @@ -0,0 +1,8 @@ +--- +title: How to Raise a PR +sidebar_label: How to Raise a PR +--- + +# How to Raise a PR + +Coming soon. diff --git a/docs/fusion/compaction/configuration.mdx b/docs/fusion/compaction/configuration.mdx new file mode 100644 index 000000000..d8d9e1f91 --- /dev/null +++ b/docs/fusion/compaction/configuration.mdx @@ -0,0 +1,64 @@ +--- +title: Configuration +sidebar_position: 2 +--- + +# Configuring Table Compaction + +Each table in OLake Fusion can have its **own compaction schedule and advanced settings**. + +## 1. Table Level Configuration + +### Step 1. Click the Configure Button + +Click the **Configure** button next to the table you want to compact. +This opens a modal where you can schedule **Lite, Medium, and Full compactions**. + +![Configure button](pathname:///img/docs/iceberg-maintenance/compaction/configure-button.webp) + + +### Step 2. Set the Compaction Schedule + +- Select a schedule from the **predefined dropdown options** or choose **Custom** to specify your own cron expression. +- Compaction will run automatically according to the schedule set for that table. + +![Compaction Schedule](pathname:///img/docs/iceberg-maintenance/compaction/configuration.webp) + +### Step 3. Advanced Config: Target File Size + +- Expand the **Advanced Config** panel in the modal. +- Specify the **Target File Size** for the table (default **512 MB** if you leave it unchanged). + +For how **target file size** affects **Lite**, **Medium**, and **Full** compaction, see [Types of Compaction Supported in OLake Fusion](/docs/fusion/compaction/types-of-compaction/#types-of-compaction). + +> **Tip:** Choose a target size based on your query patterns and table size. Larger files can improve scan efficiency based on the query but may increase the cost of rewriting files. + +![Target File Size](pathname:///img/docs/iceberg-maintenance/compaction/target-file-size.webp) + +### Step 4. Save the Configuration + +- Click **Save**. + +![Save Configuration](pathname:///img/docs/iceberg-maintenance/compaction/save-configuration.webp) + +- A dialog box confirms that the configuration was successful. + +![Save successful](/img/docs/iceberg-maintenance/compaction/configuration-successful.webp) + +- Once saved, the **Status** toggle for that table turns on automatically and compaction runs on the schedule you configured. Confirm this in the **Status** column as the toggle should appear active. + +![status active](/img/docs/iceberg-maintenance/compaction/active-status-single.webp) + +## 2. Bulk Configuration + +Bulk configuration lets you apply the same compaction schedule and settings to multiple tables in one go, instead of configuring each table individually. + +To get started, select the tables you want to configure from the **Tables** page using their checkboxes, then click **Bulk Configure**. The same configuration modal opens — set the schedule, target file size, and save, exactly as described in Steps 2–4 of [Single Table Configuration](#1-single-table-configuration). + +![bulk select tables](/img/docs/iceberg-maintenance/compaction/bulk-select-tables.webp) + +:::note Default Configuration + +If any configuration is left unset, the following defaults apply: **24 hours** as the frequency for each compaction type and **512 MB** as the target file size. + +::: diff --git a/docs/fusion/compaction/types-of-compaction.mdx b/docs/fusion/compaction/types-of-compaction.mdx new file mode 100644 index 000000000..a2cdda627 --- /dev/null +++ b/docs/fusion/compaction/types-of-compaction.mdx @@ -0,0 +1,45 @@ +--- +title: Types of Compaction +sidebar_label: Types of Compaction +--- + +## Types of Compaction + +Before diving into the types of compaction, it helps to understand how OLake Fusion categorizes files in an Iceberg table: + +- **Small files (fragments)** — Tiny files well below the target size. These pile up quickly when data is written frequently, such as in streaming or high-frequency CDC scenarios. +- **Medium-sized files (segments)** — Files bigger than fragments but still not at the ideal size. These are partially compacted files that haven't yet reached the target. +- **Optimally sized files** — Files at exactly the configured target file size. These are what compaction aims to produce. + +The goal of compaction is to turn fragments and segments into optimally sized files. + +OLake supports three types of compaction: + +### 1. Lite Compaction + +Lite compaction is the lightest and most frequently run type. It focuses on two things, **merging fragment into larger ones** and **converting Equality Delete Files into Position Delete Files**. Position Delete Files are cheaper for query engines to process, so this conversion alone improves read performance without doing a heavy rewrite. Since streaming writes and high-frequency CDC constantly produce small fragment files, Lite compaction is typically scheduled to run frequentlly to keep the table tidy before the clutter builds up. + +### 2. Medium Compaction + +Medium compaction goes a step further. It **merges segment files up to the target file size**, and when too many Position Delete Files have accumulated, it merges them directly into the corresponding Data Files that is physically removing deleted rows from the table. This is more thorough than Lite compaction but still does not rewrite the entire table. The Medium Compaction is typically scheduled less frequently than the Lite Compaction to keep the table efficient and not spend too much compute. + +### 3. Full Compaction + +Full compaction is the deepest and most comprehensive type. It rewrites all data files fragments, segments, and delete files into optimally sized files that exactly match the configured target file size. Because it rewrites the entire table, it is the most compute-intensive option and is typically run less frequently. Use it when tables have accumulated heavy fragmentation over time or when you need the best possible query performance. + +:::info Compaction precedence + +When more than one type is scheduled at the same time, only the highest-priority type runs: + +**Full** > **Medium** > **Lite** + +::: + +### Choosing the Right Compaction Type + +| Compaction Type | Output | What it Does | Cost Incurred | When to Use | +|-------------------|--------|--------------|------|-------------| +| **Lite** | Equality delete files are converted to positional delete files and small files are merged | Improves query engine compatibility without rewriting data files | **Low** | Use when the table has too many small files and you want lightweight maintenance with low compute.| +| **Medium** | Deletes are applied and data files are merged; output sizes fall between 1/8 of target file size and the target file size itself | Reduces fragmentation by merging data files into larger files up to the target size | **Medium** | Use when you need more than Lite: deletes fully applied and files merged toward the target size without a full table rewrite. | +| **Full** | Data files are completely rewritten into files aligned with the target file size | Performs a full copy-on-write rewrite of the table to produce the most best file layout | **High** | Use when tables are heavily fragmented or when maximum query performance and best file layout are required. | + diff --git a/docs/fusion/core/architecture.mdx b/docs/fusion/core/architecture.mdx new file mode 100644 index 000000000..3555c1c3e --- /dev/null +++ b/docs/fusion/core/architecture.mdx @@ -0,0 +1,10 @@ +--- +title: Architecture +sidebar_label: Architecture +--- + +# OLake Fusion Architecture + +
    +![OLake fusion architecture diagram](/img/docs/fusion-architecture.webp) +
    \ No newline at end of file diff --git a/docs/fusion/core/compatibility/query-engines.mdx b/docs/fusion/core/compatibility/query-engines.mdx new file mode 100644 index 000000000..4d01ae384 --- /dev/null +++ b/docs/fusion/core/compatibility/query-engines.mdx @@ -0,0 +1,8 @@ +--- +title: Query Engine Compatibility +sidebar_label: Compatibility with Query Engines +--- + +# Query Engine Compatibility + +Coming soon. diff --git a/docs/fusion/core/terminologies.mdx b/docs/fusion/core/terminologies.mdx new file mode 100644 index 000000000..438e12d5f --- /dev/null +++ b/docs/fusion/core/terminologies.mdx @@ -0,0 +1,8 @@ +--- +title: Terminologies +sidebar_label: Terminologies +--- + +# Terminologies + +Coming soon. diff --git a/docs/fusion/core/use-cases.mdx b/docs/fusion/core/use-cases.mdx new file mode 100644 index 000000000..46c8d2f97 --- /dev/null +++ b/docs/fusion/core/use-cases.mdx @@ -0,0 +1,50 @@ +--- +title: Use Cases +sidebar_label: Use Cases +--- + +# Use Cases for OLake Fusion + +### 1. Improving Query Performance on Iceberg Tables + +As Iceberg tables grow through continuous ingestion, updates, and deletes, they accumulate **small files** and **delete files** which silently degrade query performance over time. + +OLake Fusion addresses this by compacting these files and rewriting your **Iceberg tables** keeping them compact and query-ready at all times. + +This approach provides: + +- **Faster query execution** → Fewer files per scan means less I/O overhead for query engines. + +- **Reduced planning time** → Cleaner metadata speeds up query planning in engines like Trino, Spark, and DuckDB. + +- **Delete file resolution** → Positional and equality delete files are applied and removed, eliminating redundant overhead at read time. + +With OLake Fusion, your Iceberg tables stay performant as they scale — without requiring manual maintenance scripts or custom tooling. + +### 2. Observability of Iceberg Tables + +Understanding the health and state of your Iceberg tables is critical for diagnosing performance issues and planning maintenance. Without visibility into table internals, teams are often left guessing why queries are slow or why storage costs are climbing. + +OLake Fusion provides deep **observability into your Iceberg table**, exposing key metrics that help you understand the exact state of the table. + +Key benefits: + +- **File-level insights** → Track the number of data files, delete files, total file count and average file size per table. + +- **Health Score** → Use it to quickly identify which tables need compaction without having to dig into individual file-level metrics. + +This gives data engineers and platform teams the clarity needed to make informed decisions about when and where to run maintenance — instead of running it blindly on a schedule. + +### 3. Table-Level Optimization + +Different Iceberg tables have different maintenance needs. A high-frequency CDC table accumulates small files rapidly, while a large historical table might only need periodic snapshot cleanup. A one-size-fits-all maintenance strategy wastes compute and can introduce unnecessary churn. + +OLake Fusion enables **fine-grained, table-level optimization**, allowing teams to configure and apply maintenance operations independently for individual tables. + +Key benefits: + +- **Per-table configuration** → Define compaction strategies and configurations independently for each table. + +- **Selective operation control** → Choose which type of compaction to apply, not all tables need a complete rewrite always. + +This granular control ensures that compute is spent where it matters most, keeping critical tables optimized without over-maintaining stable ones. \ No newline at end of file diff --git a/docs/fusion/getting-started/compaction.mdx b/docs/fusion/getting-started/compaction.mdx new file mode 100644 index 000000000..5adc69c34 --- /dev/null +++ b/docs/fusion/getting-started/compaction.mdx @@ -0,0 +1,928 @@ +--- +title: Compaction Benchmarks +sidebar_label: Benchmarks +--- + +# Compaction Benchmarks + +The following benchmark evaluates performance, environment configuration, and operational considerations for compacting Apache Iceberg tables using **Apache Spark** `rewrite_data_files` and **OLake Fusion** compaction on a CDC-like TPCH workload. + +### Benchmark Environment + +- **Dataset:** This benchmark uses **TPC-H 300 GB** data generated with TPC-H `dbgen`. +- **Total rows:** At TPC-H 300 GB scale, the `lineitem` table contains approximately **1.8 billion rows**. +- The average row size for `lineitem` is **120 bytes**. +- **Destination data size:** The Iceberg dataset size on Google Cloud Storage after ingestion was approximately **85 GB**. +- **Source database instance:** Azure `Standard_D8ads_v5` (**8 vCores, 32 GiB memory**). +- **OLake Go ingestion compute:** Azure `Standard D64ls v5` (**64 vCPUs, 128 GiB memory**). +- **TPCH query execution resources:**
    Master: GCP `c4a-standard-8` (**8 vCPU, 32 GB RAM**)
    Workers: 2 x GCP `c4a-standard-32` (**32 vCPU, 128 GB RAM each**). +- **Fusion and Spark compaction resources:**
    Master: GCP `c4a-standard-8` (**8 vCPU, 32 GB RAM**)
    Workers: 2 x GCP `c4a-highmem-16` (**16 vCPU, 128 GB RAM each**). +- **Compaction runtime configuration (used for both Spark and Fusion):** + | Setting | Value | + |---|---| + | `spark-conf.spark.executor.instances` | 4 | + | `spark-conf.spark.executor.cores` | 7 | + | `spark-conf.spark.executor.memory` | 45g | + | `spark-conf.spark.executor.memoryOverhead` | 12g | + | `spark-conf.spark.driver.memory` | 18g | + | `spark-conf.spark.driver.memoryOverhead` | 10g | + +:::note +We used REST Lakekeeper as the Iceberg catalog and GCP as the storage layer on the destination side for this benchmark. +::: + +### Compaction Results + +#### 1. Speed Comparison + +This compaction comparison evaluates end-to-end compaction runtime for Apache Spark and OLake Fusion on the same Iceberg workload, with average TPC-H Query 6 runtime kept comparable across both setups to ensure a fair baseline for total time and relative execution speed. + +| Engine | Total Compaction Time | Relative to Fusion | +|---|---|---| +| OLake Fusion | 27 mins 2 secs | - | +| Apache Spark | 55 mins 47 secs | **2.06 x slower** | + +**Key takeaway:** OLake Fusion delivered about **2.06x faster compaction than Apache Spark**. + +#### 2. Cost Comparison + +| Instance type | Cost per hour (USD) | +| --- | --- | +| `c4a-standard-8` | $0.38 | +| `c4a-highmem-16` | $0.99 | + +Using the compaction infrastructure and total compaction durations from this benchmark: + +| Engine | Resources included for compaction cost | Hourly infrastructure cost | Total time | Total compaction cost | +| --- | --- | --- | --- | --- | +| Fusion | `c4a-standard-8`
    2 x `c4a-highmem-16` | $2.36/hour | 27 mins 02 secs | **$1.06** | +| Spark | `c4a-standard-8`
    2 x `c4a-highmem-16` | $2.36/hour | 55 mins 47 secs | **$2.19** | + +:::info +**OLake Fusion** is open-source and can be deployed on Docker or Kubernetes; you pay only for the compute and storage you provision. +::: + +### Benchmark Workflow + +- **Data preparation:** Generated TPCH data at 300 GB scale, loaded it into Azure PostgreSQL server, and ingested only the `lineitem` table into a GCP Iceberg bucket using OLake (full load). +- **CDC-like change generation:** Used a Python script to run periodic `UPDATE` statements on `lineitem`, changing `l_comment` for approximately **200,000 random rows every 2 minutes** to simulate continuous CDC pressure. + +
    + View CDC simulation script + + ```python + import psycopg2 + import time + + HOST = "" + PORT = 5432 + USER = "" + PASSWORD = "" + DATABASE = "" + SCHEMA = "tpch" + TABLE = "lineitem" + + RUNS = 120 # total number of runs + INTERVAL_SECONDS = 120 # 2 min between scheduled run starts (see loop below) + TABLESAMPLE_PCT = 0.0112 # ~200k rows out of 1.8B + + print(f"Connecting to {HOST}/{DATABASE} ...") + + conn = None + cursor = None + + try: + conn = psycopg2.connect( + host=HOST, + port=PORT, + user=USER, + password=PASSWORD, + dbname=DATABASE, + sslmode="require", + ) + conn.autocommit = False + cursor = conn.cursor() + print("Connected successfully.") + print(f"Total runs : {RUNS}") + print(f"Interval : every {INTERVAL_SECONDS}s") + print(f"Rows per run : ~200k (TABLESAMPLE SYSTEM {TABLESAMPLE_PCT}%)") + + update_sql = f""" + UPDATE {SCHEMA}.{TABLE} t + SET l_comment = substring(l_comment, 1, char_length(l_comment) - 1) + || chr(65 + (random() * 25)::int) + FROM ( + SELECT ctid + FROM {SCHEMA}.{TABLE} TABLESAMPLE SYSTEM ({TABLESAMPLE_PCT}) + ) s + WHERE t.ctid = s.ctid + """ + + next_run_monotonic = time.monotonic() + + for run_idx in range(RUNS): + # First run starts immediately; subsequent runs are anchored to the + # original start time so interval drift never accumulates. + if run_idx > 0: + sleep_seconds = next_run_monotonic - time.monotonic() + if sleep_seconds > 0: + print(f" Sleeping {round(sleep_seconds, 1)}s until next run ...") + time.sleep(sleep_seconds) + next_run_monotonic += INTERVAL_SECONDS + + print(f"\n[{run_idx + 1}/{RUNS}] Running UPDATE on {SCHEMA}.{TABLE} - ~200k random rows ...") + start = time.time() + cursor.execute(update_sql) + rows_affected = cursor.rowcount + conn.commit() + elapsed = round(time.time() - start, 2) + print(f"[{run_idx + 1}/{RUNS}] Done. Rows updated: {rows_affected} | Time taken: {elapsed}s") + + except Exception as e: + print(f"ERROR: {e}") + if conn: + conn.rollback() + print("Transaction rolled back.") + raise + + finally: + if cursor: + cursor.close() + if conn: + conn.close() + print("\nConnection closed.") + ``` + +
    +- **Analytical query execution:** After the first **2 update cycles**, we started running **TPC-H Query 6** on the `lineitem` Iceberg table; the query was executed continuously without any pause between consecutive runs. + +
    + View TPCH Query script + + ```python + #!/usr/bin/env python3 + import logging + import time + import datetime + from pyspark.sql import SparkSession + + # Logging Setup + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s [%(levelname)s] %(message)s", + datefmt="%Y-%m-%d %H:%M:%S", + ) + log = logging.getLogger("tpch-benchmark") + + CATALOG = "benchmark" + # Iceberg namespace in Lakekeeper (REST catalog); must hold `lineitem`. + DATABASE = "spark_compact" + # Lakekeeper REST catalog warehouse id (matches spark.sql.catalog.benchmark.warehouse). + LAKEKEEPER_WAREHOUSE = "benchmarking" + LAKEKEEPER_URI = "http://10.20.0.64:30081/catalog" + + # Run Config + TOTAL_RUNS = 200 + BREAK_SECONDS = 0 # no pause between runs + + # TPC-H Q6 - Forecasting Revenue Change + # Single table scan on lineitem only. No joins, no GROUP BY. + # Pure sequential scan - best query to measure compaction impact. + TPCH_Q6 = f""" + SELECT + SUM(l_extendedprice * l_discount) AS revenue + FROM + {CATALOG}.{DATABASE}.lineitem + WHERE + l_shipdate >= DATE '1994-01-01' + AND l_shipdate < DATE '1995-01-01' + AND l_discount BETWEEN 0.05 AND 0.07 + AND l_quantity < 24 + """ + + # SparkSession + log.info("Initializing SparkSession...") + spark = ( + SparkSession.builder + .appName("tpch-q6-benchmark") + .config("spark.sql.extensions", + "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") + .config("spark.sql.catalog.benchmark", + "org.apache.iceberg.spark.SparkCatalog") + .config("spark.sql.catalog.benchmark.catalog-impl", + "org.apache.iceberg.rest.RESTCatalog") + .config("spark.sql.catalog.benchmark.uri", + LAKEKEEPER_URI) + .config("spark.sql.catalog.benchmark.warehouse", + LAKEKEEPER_WAREHOUSE) + .config("spark.sql.catalog.benchmark.io-impl", + "org.apache.iceberg.aws.s3.S3FileIO") + .config("spark.sql.catalog.benchmark.s3.endpoint", + "https://storage.googleapis.com/") + .config("spark.sql.catalog.benchmark.s3.path-style-access", "true") + .config("spark.sql.catalog.benchmark.client.region", "ap-south-1") + .config("spark.hadoop.fs.s3a.endpoint", "https://storage.googleapis.com") + .config("spark.hadoop.fs.s3a.path.style.access", "true") + .config("spark.hadoop.fs.s3a.endpoint.region", "ap-south-1") + .config("spark.sql.shuffle.partitions", "128") + .config("spark.sql.defaultCatalog", "benchmark") + .getOrCreate() + ) + + spark.conf.set("spark.sql.iceberg.vectorization.enabled", "false") + spark.sparkContext.setLogLevel("INFO") + + log.info("SparkSession initialized successfully.") + log.info(f" Spark version : {spark.version}") + log.info(f" App name : {spark.sparkContext.appName}") + log.info(f" Master : {spark.sparkContext.master}") + log.info(f" Default parallelism : {spark.sparkContext.defaultParallelism}") + log.info(f" Lakekeeper URI : {LAKEKEEPER_URI}") + log.info(f" Catalog warehouse : {LAKEKEEPER_WAREHOUSE}") + log.info(f" Namespace (db) : {DATABASE}") + log.info(f" Total Q6 runs : {TOTAL_RUNS}") + log.info(f" Break between runs : {BREAK_SECONDS}s") + + run_results = [] # {run, started_at, finished_at, elapsed_s, revenue, status} + + # BENCHMARK LOOP - Q6 x TOTAL_RUNS, no pause between runs + for run_number in range(1, TOTAL_RUNS + 1): + log.info("") + log.info("#" * 70) + log.info(f"#{'':^68}#") + log.info(f"#{'RUN ' + str(run_number) + ' OF ' + str(TOTAL_RUNS):^68}#") + log.info(f"#{'':^68}#") + log.info("#" * 70) + + log.info("") + log.info("=" * 70) + log.info(" TPC-H Q6 - Forecasting Revenue Change") + log.info(f" Table : {CATALOG}.{DATABASE}.lineitem") + log.info(" Type : Single table scan | No joins | No GROUP BY") + log.info("=" * 70) + + run_start_ts = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") + log.info(f" Query started at : {run_start_ts}") + + q_start = time.time() + try: + result = spark.sql(TPCH_Q6) + rows = result.collect() + q_elapsed = round(time.time() - q_start, 2) + run_end_ts = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") + + revenue_val = rows[0]["revenue"] if rows else None + + log.info(f" Query finished at : {run_end_ts}") + log.info(f" Query time : {q_elapsed}s ({round(q_elapsed / 60, 2)} min)") + log.info("") + log.info(" Result:") + log.info(f" revenue = {revenue_val}") + + run_results.append({ + "run": run_number, + "started_at": run_start_ts, + "finished_at": run_end_ts, + "elapsed_s": q_elapsed, + "revenue": revenue_val, + "status": "OK", + }) + + except Exception as e: + q_elapsed = round(time.time() - q_start, 2) + run_end_ts = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") + log.error("=" * 70) + log.error(f" Q6 FAILED after {q_elapsed}s") + log.error(f" Error: {e}") + log.error("=" * 70) + run_results.append({ + "run": run_number, + "started_at": run_start_ts, + "finished_at": run_end_ts, + "elapsed_s": q_elapsed, + "revenue": None, + "status": f"FAILED: {e}", + }) + + log.info("") + log.info("-" * 70) + log.info(f" END OF RUN {run_number} OF {TOTAL_RUNS}") + log.info(f" Elapsed : {q_elapsed}s ({round(q_elapsed / 60, 2)} min)") + remaining = TOTAL_RUNS - run_number + if remaining > 0: + log.info(f" Remaining runs : {remaining}") + else: + log.info(" All runs completed.") + log.info("-" * 70) + + if remaining > 0 and BREAK_SECONDS > 0: + log.info(f" Cooling down for {BREAK_SECONDS}s before next run...") + time.sleep(BREAK_SECONDS) + + # FINAL SUMMARY + log.info("") + log.info("=" * 70) + log.info(" BENCHMARK COMPLETE - ALL RUNS SUMMARY") + log.info("=" * 70) + log.info(f" {'RUN':<6} {'STARTED AT':<22} {'ELAPSED (s)':<14} {'STATUS'}") + log.info(f" {'-'*4:<6} {'-'*19:<22} {'-'*11:<14} {'-'*10}") + + ok_times = [] + for r in run_results: + elapsed_str = str(r["elapsed_s"]) if r["status"] == "OK" else "FAILED" + log.info(f" {r['run']:<6} {r['started_at']:<22} {elapsed_str:<14} {r['status']}") + if r["status"] == "OK": + ok_times.append(r["elapsed_s"]) + + log.info("") + if ok_times: + log.info(f" Successful runs : {len(ok_times)} / {TOTAL_RUNS}") + log.info(f" Min query time : {min(ok_times)}s") + log.info(f" Max query time : {max(ok_times)}s") + log.info(f" Avg query time : {round(sum(ok_times) / len(ok_times), 2)}s") + log.info(f" Total query time : {round(sum(ok_times), 2)}s " + f"({round(sum(ok_times) / 60, 2)} min)") + else: + log.error(" No successful runs to summarize.") + + log.info("=" * 70) + log.info("") + log.info("All done. Stopping SparkSession.") + spark.stop() + ``` + +
    + +
    + View spark-submit command for TPCH Query execution + + ```bash + gcloud dataproc jobs submit pyspark gs://dz-benchmark/tpch_dataproc.py \ + --cluster=dz-olake-tpch-21042026 \ + --region=asia-south1 \ + --properties="^#^spark.jars.packages=org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.7.2,org.apache.iceberg:iceberg-aws-bundle:1.7.2#spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions#spark.sql.catalog.benchmark=org.apache.iceberg.spark.SparkCatalog#spark.sql.catalog.benchmark.catalog-impl=org.apache.iceberg.rest.RESTCatalog#spark.sql.catalog.benchmark.uri=http://10.20.0.64:30081/catalog#spark.sql.catalog.benchmark.warehouse=benchmarking#spark.sql.catalog.benchmark.io-impl=org.apache.iceberg.aws.s3.S3FileIO#spark.sql.catalog.benchmark.s3.endpoint=https://storage.googleapis.com/#spark.sql.catalog.benchmark.s3.path-style-access=true#spark.sql.catalog.benchmark.client.region=ap-south-1#spark.sql.defaultCatalog=benchmark#spark.hadoop.fs.s3a.endpoint=https://storage.googleapis.com#spark.hadoop.fs.s3a.path.style.access=true#spark.hadoop.fs.s3a.endpoint.region=ap-south-1#spark.dynamicAllocation.enabled=false#spark.executor.instances=12#spark.executor.cores=5#spark.executor.memory=15g#spark.executor.memoryOverhead=4g#spark.driver.memory=18g#spark.driver.memoryOverhead=10g#spark.sql.parquet.enableVectorizedReader=false#spark.sql.iceberg.vectorization.enabled=false#spark.hadoop.io.native.lib.available=false#spark.executor.extraJavaOptions=-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35 -XX:G1ReservePercent=20 -XX:UseAVX=0 -Darrow.enable_unsafe_memory_access=false -Darrow.enable_null_check_for_get=true -Dhadoop.io.native.lib.available=false#spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35 -Xss8m -XX:UseAVX=0 -Darrow.enable_unsafe_memory_access=false -Darrow.enable_null_check_for_get=true" + ``` + +
    + +- **Parallel compaction phase:** After completing **5 ingestion cycles**, we ran compaction jobs for a total of **2 hours** while both CDC-style updates and repeated TPC-H Query 6 executions continued in parallel, specifically to measure how concurrent compaction impacts query latency and runtime stability. + +### Apache Spark + +Spark compaction was executed using Apache Iceberg's `rewrite_data_files` procedure through a scheduled Spark job. + +The compaction job was scheduled with each run triggered at a fixed interval of **20 minutes**, allowing compaction to operate alongside ongoing ingestion activity and repeated analytical query execution. + +The Spark `rewrite_data_files` job is configured with the following compaction parameters: + +- `strategy`: Binpack (Rewrites many small files into fewer, size-balanced files to improve scan efficiency) +- `target-file-size-bytes`: 512 MB (Preferred output file size for rewritten data files) +- `max-file-size-bytes`: 614.4 MB (Upper size limit allowed for generated output files) +- `min-file-size-bytes`: 384 MB (Lower size threshold used to select smaller files for rewrite) +- `max-concurrent-file-group-rewrites`: 28 (Maximum number of file groups rewritten in parallel in one run) +- `partial-progress.enabled`: False (Requires a full rewrite attempt instead of committing partial-progress batches) +- `delete-file-threshold`: 1 (Triggers rewrite when at least one delete file is associated with a file group) + +
    +View Spark compaction script (Python) + +```python +#!/usr/bin/env python3 +import logging +import time +import datetime +from pyspark.sql import SparkSession + + +# ─── Logging Setup ───────────────────────────────────────────────────────────── +logging.basicConfig( + level=logging.INFO, + format="%(asctime)s [%(levelname)s] %(message)s", + datefmt="%Y-%m-%d %H:%M:%S", +) +log = logging.getLogger("iceberg-compaction") + + +CATALOG = "benchmark" +DATABASE = "spark_compact" +TABLES = [ + "lineitem", +] + + +# ─── Scheduler Config ────────────────────────────────────────────────────────── +TOTAL_RUNS = 20 +INTERVAL_MINUTES = 20 +INTERVAL_SECONDS = INTERVAL_MINUTES * 60 + + +# ─── TPC-H Q6 — Forecasting Revenue Change ───────────────────────────────────── +def build_tpch_q6(catalog, db): + return { + "Q6 - Forecasting Revenue Change": f""" + SELECT + SUM(l_extendedprice * l_discount) AS revenue + FROM + {catalog}.{db}.lineitem + WHERE + l_shipdate >= date '1994-01-01' + AND l_shipdate < date '1995-01-01' + AND l_discount BETWEEN 0.05 AND 0.07 + AND l_quantity < 24 + """ + } + + +# ─── Helper: Per-table file stats (optionally pinned to a snapshot) ───────────── +def get_table_file_stats(spark, catalog, table_full, snapshot_id=None): + """ + content = 0 -> data files + content = 2 -> equality delete files + snapshot_id -> if provided, reads file state at that exact snapshot (time travel) + if None, reads current state + """ + version_clause = f"VERSION AS OF {snapshot_id}" if snapshot_id else "" + + try: + row = spark.sql( + f"SELECT COUNT(*) AS file_count, SUM(file_size_in_bytes) AS total_bytes " + f"FROM {catalog}.{table_full}.files {version_clause} WHERE content = 0" + ).collect()[0] + data_count = row["file_count"] + data_mb = round(row["total_bytes"] / (1024 * 1024), 2) if row["total_bytes"] else 0 + except Exception as e: + log.warning(f" Could not fetch data file stats: {e}") + data_count, data_mb = "N/A", "N/A" + + try: + eq_count = spark.sql( + f"SELECT COUNT(*) AS file_count " + f"FROM {catalog}.{table_full}.files {version_clause} WHERE content = 2" + ).collect()[0]["file_count"] + except Exception as e: + log.warning(f" Could not fetch equality delete file stats: {e}") + eq_count = "N/A" + + return data_count, data_mb, eq_count + + +# ─── Helper: Find the snapshot ID that compaction created ─────────────────────── +def get_compaction_snapshot(spark, catalog, table_full, compaction_started_at): + """ + After rewrite_data_files completes, find the snapshot with operation='replace' + that was committed after compaction started. This is the exact compaction snapshot. + Returns (snapshot_id, committed_at) or (None, None) if nothing was rewritten. + """ + try: + rows = spark.sql( + f"SELECT snapshot_id, committed_at " + f"FROM {catalog}.{table_full}.snapshots " + f"WHERE operation = 'replace' " + f"AND committed_at >= TIMESTAMP '{compaction_started_at}' " + f"ORDER BY committed_at ASC " + f"LIMIT 1" + ).collect() + if rows: + snap_id = rows[0]["snapshot_id"] + snap_ts = str(rows[0]["committed_at"]) + log.info(f" Compaction snapshot found : id={snap_id} | committed_at={snap_ts}") + return snap_id, snap_ts + else: + log.warning(f" No replace snapshot found after {compaction_started_at} — " + f"compaction may have found nothing to rewrite.") + return None, None + except Exception as e: + log.warning(f" Could not find compaction snapshot: {e}") + return None, None + + +# ─── Helper: Count OLake ingestion appends during compaction window ───────────── +def count_ingestions_during_compaction(spark, catalog, table_full, + pre_committed_at, compaction_committed_at): + """ + Counts append snapshots committed strictly AFTER pre_committed_at and + up to AND INCLUDING compaction_committed_at. + + Each OLake ingestion commit adds exactly: 1 data file + 1 equality-delete file. + We subtract this count from the raw POST snapshot counts to isolate the + pure compaction effect (no ingestion noise). + + Returns 0 safely if either timestamp is None. + """ + if not pre_committed_at or not compaction_committed_at: + log.warning(" Skipping ingestion count — missing pre or compaction timestamp.") + return 0 + try: + row = spark.sql( + f"SELECT COUNT(*) AS cnt " + f"FROM {catalog}.{table_full}.snapshots " + f"WHERE operation IN ('append', 'overwrite') " + f"AND committed_at > TIMESTAMP '{pre_committed_at}' " + f"AND committed_at <= TIMESTAMP '{compaction_committed_at}'" + ).collect()[0] + cnt = row["cnt"] + log.info(f" OLake ingestion commits during compaction window : {cnt} " + f"(each = +1 data file, +1 eq-delete file)") + return cnt + except Exception as e: + log.warning(f" Could not count ingestion snapshots during compaction: {e}") + return 0 + + +# ─── Helper: Run TPC-H Q6 and return elapsed time ─────────────────────────────── +def run_tpch_q6(spark, label, tpch_queries): + log.info("=" * 70) + log.info(f" TPC-H Query Benchmark [{label}]") + log.info("=" * 70) + results = {} + for qname, qsql in tpch_queries.items(): + try: + q_start = time.time() + spark.sql(qsql).collect() + q_elapsed = round(time.time() - q_start, 2) + log.info(f" {qname:<40} : {q_elapsed}s ({round(q_elapsed / 60, 2)} min)") + results[qname] = q_elapsed + except Exception as e: + log.warning(f" {qname:<40} : FAILED — {e}") + results[qname] = "FAILED" + log.info("=" * 70) + return results + + +# ─── Core compaction logic for a single run ───────────────────────────────────── +def run_compaction(spark, run_number, tpch_queries): + overall_start_time = time.time() + overall_start_ts = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") + + log.info("=" * 70) + log.info(f" OVERALL COMPACTION RUN STARTED AT : {overall_start_ts}") + log.info(f" Tables to compact : {TABLES}") + log.info("=" * 70) + + for table_name in TABLES: + TABLE = f"{DATABASE}.{table_name}" + FULL_TABLE = f"{CATALOG}.{TABLE}" + + log.info("") + log.info("=" * 70) + log.info(f" TABLE: {FULL_TABLE}") + log.info("=" * 70) + + # Last 5 snapshots — pre + try: + snaps = spark.sql( + f"SELECT snapshot_id, committed_at, operation " + f"FROM {CATALOG}.{TABLE}.snapshots " + f"ORDER BY committed_at DESC LIMIT 5" + ).collect() + log.info(" [PRE] Last 5 snapshots:") + for row in snaps: + log.info(f" snapshot_id={row['snapshot_id']} | " + f"committed_at={row['committed_at']} | " + f"operation={row['operation']}") + except Exception as e: + log.warning(f" Could not fetch pre-compaction snapshots: {e}") + + # File stats — pre + data_cnt, data_mb, eq_del_cnt = get_table_file_stats(spark, CATALOG, TABLE) + log.info(f" [PRE] Data files (content=0) : {data_cnt} files | {data_mb} MB") + log.info(f" [PRE] Eq-delete files (content=2) : {eq_del_cnt} files") + + # Capture PRE boundary timestamp inline — same snapshot the PRE stats just read from. + # Used as the left edge of the ingestion-counting window. + pre_committed_at = None + try: + pre_row = spark.sql( + f"SELECT committed_at FROM {CATALOG}.{TABLE}.snapshots " + f"ORDER BY committed_at DESC LIMIT 1" + ).collect() + pre_committed_at = str(pre_row[0]["committed_at"]) if pre_row else None + log.info(f" [PRE] Snapshot boundary : {pre_committed_at}") + except Exception as e: + log.warning(f" Could not fetch PRE snapshot boundary: {e}") + + # Disable vectorization at table level + try: + spark.sql( + f"ALTER TABLE {FULL_TABLE} " + f"SET TBLPROPERTIES ('read.parquet.vectorization.enabled' = 'false')" + ) + log.info(f" Disabled vectorization for {table_name}.") + except Exception as e: + log.warning(f" Could not set vectorization property: {e}") + + # Compaction config banner + log.info(" Compaction config:") + log.info(" strategy : binpack") + log.info(" target-file-size-bytes : 536870912 (512 MB)") + log.info(" max-file-size-bytes : 644245094 (614.4 MB)") + log.info(" min-file-size-bytes : 402653184 (384 MB)") + log.info(" max-concurrent-file-group-rewrites: 28") + log.info(" partial-progress.enabled : false") + log.info(" delete-file-threshold : 1") + + table_start_time = time.time() + table_start_ts = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") + log.info(f" Compaction started at : {table_start_ts}") + + try: + result = spark.sql( + f""" + CALL {CATALOG}.system.rewrite_data_files( + table => '{TABLE}', + strategy => 'binpack', + options => map( + 'target-file-size-bytes', '536870912', + 'max-file-size-bytes', '644245094', + 'min-file-size-bytes', '402653184', + 'max-concurrent-file-group-rewrites', '28', + 'partial-progress.enabled', 'false', + 'delete-file-threshold', '1' + ) + ) + """ + ) + rows = result.collect() + table_elapsed = round(time.time() - table_start_time, 2) + table_end_ts = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") + + log.info(f" Compaction finished at : {table_end_ts}") + log.info(f" Compaction time for this table: {table_elapsed}s " + f"({round(table_elapsed / 60, 2)} min)") + + for row in rows: + rewritten_files = row["rewritten_data_files_count"] + added_files = row["added_data_files_count"] + rewritten_bytes = ( + row["rewritten_bytes_count"] + if "rewritten_bytes_count" in row.__fields__ + else "N/A" + ) + rewritten_mb = ( + round(rewritten_bytes / (1024 * 1024), 2) + if isinstance(rewritten_bytes, int) + else "N/A" + ) + log.info(" Compaction Result:") + log.info(f" Files rewritten (input) : {rewritten_files}") + log.info(f" Files added (output) : {added_files}") + log.info(f" Total bytes rewritten : {rewritten_bytes} ({rewritten_mb} MB)") + + except Exception as e: + elapsed = round(time.time() - table_start_time, 2) + log.error(f" Compaction FAILED for {table_name} after {elapsed}s") + log.error(f" Error: {e}") + continue + + # ── Step 1: Find the exact compaction snapshot (operation=replace) ──────── + compaction_snapshot_id, compaction_committed_at = get_compaction_snapshot( + spark, CATALOG, TABLE, table_start_ts + ) + + # ── Step 2: Count OLake ingestions that committed DURING compaction ─────── + # Window: strictly after PRE snapshot → up to and including compaction snapshot + # Each ingestion = +1 data file, +1 eq-delete file (OLake guarantee) + ingestions_during_compaction = count_ingestions_during_compaction( + spark, CATALOG, TABLE, pre_committed_at, compaction_committed_at + ) + + # ── Step 3: Read POST file counts pinned to the compaction snapshot ─────── + data_cnt_p, data_mb_p, eq_del_cnt_p = get_table_file_stats( + spark, CATALOG, TABLE, snapshot_id=compaction_snapshot_id + ) + + # ── Step 4: Subtract ingestion noise → pure compaction POST counts ──────── + # The compaction snapshot inherits ingested files via Iceberg's linear chain. + # Subtracting ingestions_during_compaction isolates the compaction-only effect. + true_data_cnt_p = ( + data_cnt_p - ingestions_during_compaction + if isinstance(data_cnt_p, int) + else data_cnt_p + ) + true_eq_del_cnt_p = ( + eq_del_cnt_p - ingestions_during_compaction + if isinstance(eq_del_cnt_p, int) + else eq_del_cnt_p + ) + + if compaction_snapshot_id: + log.info(f" [POST] File stats pinned to compaction snapshot {compaction_snapshot_id}:") + else: + log.info(f" [POST] File stats (no replace snapshot found — showing current state):") + + log.info(f" [POST] Data files (content=0) at snapshot : {data_cnt_p} files") + log.info(f" [POST] Eq-delete (content=2) at snapshot : {eq_del_cnt_p} files") + log.info(f" [POST] Minus ingestion files during window : -{ingestions_during_compaction} (data), -{ingestions_during_compaction} (eq-delete)") + log.info(f" [POST] True post-compaction data files : {true_data_cnt_p} files | {data_mb_p} MB") + log.info(f" [POST] True post-compaction eq-delete files : {true_eq_del_cnt_p} files") + + # Delta — pure compaction effect only + if isinstance(data_cnt, int) and isinstance(true_data_cnt_p, int): + log.info(f" [DELTA] Data files : {data_cnt} → {true_data_cnt_p} " + f"(change: {true_data_cnt_p - data_cnt:+d})") + if isinstance(eq_del_cnt, int) and isinstance(true_eq_del_cnt_p, int): + log.info(f" [DELTA] Eq-del files : {eq_del_cnt} → {true_eq_del_cnt_p} " + f"(change: {true_eq_del_cnt_p - eq_del_cnt:+d})") + + # Last 3 snapshots — post + try: + snaps_after = spark.sql( + f"SELECT snapshot_id, committed_at, operation " + f"FROM {CATALOG}.{TABLE}.snapshots " + f"ORDER BY committed_at DESC LIMIT 3" + ).collect() + log.info(" [POST] Latest snapshots:") + for row in snaps_after: + log.info(f" snapshot_id={row['snapshot_id']} | " + f"committed_at={row['committed_at']} | " + f"operation={row['operation']}") + except Exception as e: + log.warning(f" Could not fetch post-compaction snapshots: {e}") + + # ─── Per-run Summary ─────────────────────────────────────────────────────── + overall_elapsed = round(time.time() - overall_start_time, 2) + overall_end_ts = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") + + log.info("") + log.info("=" * 70) + log.info(f" RUN {run_number} — COMPACTION SUMMARY") + log.info("=" * 70) + log.info(f" Started at : {overall_start_ts}") + log.info(f" Ended at : {overall_end_ts}") + log.info(f" Total time : {overall_elapsed}s ({round(overall_elapsed / 60, 2)} min)") + log.info("=" * 70) + + +# ══════════════════════════════════════════════════════════════════════════════ +# SPARK SESSION +# ══════════════════════════════════════════════════════════════════════════════ +log.info("Initializing SparkSession...") +spark = ( + SparkSession.builder + .appName("iceberg-compaction-tpch-spark_compact") + .config("spark.sql.extensions", + "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") + .config("spark.sql.catalog.benchmark", + "org.apache.iceberg.spark.SparkCatalog") + .config("spark.sql.catalog.benchmark.catalog-impl", + "org.apache.iceberg.rest.RESTCatalog") + .config("spark.sql.catalog.benchmark.uri", + "http://10.20.0.64:30081/catalog") + .config("spark.sql.catalog.benchmark.warehouse", + "benchmarking") + .config("spark.sql.catalog.benchmark.io-impl", + "org.apache.iceberg.aws.s3.S3FileIO") + .config("spark.sql.catalog.benchmark.s3.endpoint", + "https://storage.googleapis.com/") + .config("spark.sql.catalog.benchmark.s3.path-style-access", "true") + .config("spark.sql.catalog.benchmark.client.region", "ap-south-1") + .config("spark.hadoop.fs.s3a.endpoint", "https://storage.googleapis.com") + .config("spark.hadoop.fs.s3a.path.style.access", "true") + .config("spark.hadoop.fs.s3a.endpoint.region", "ap-south-1") + .config("spark.sql.shuffle.partitions", "128") + .config("spark.sql.defaultCatalog", "benchmark") + .getOrCreate() +) + +spark.conf.set("spark.sql.iceberg.vectorization.enabled", "false") +spark.sparkContext.setLogLevel("INFO") + +log.info("SparkSession initialized successfully.") +log.info(f" Spark version : {spark.version}") +log.info(f" App name : {spark.sparkContext.appName}") +log.info(f" Master : {spark.sparkContext.master}") +log.info(f" Default parallelism : {spark.sparkContext.defaultParallelism}") + +TPCH_QUERIES = build_tpch_q6(CATALOG, DATABASE) + + +# ══════════════════════════════════════════════════════════════════════════════ +# SCHEDULED LOOP +# ══════════════════════════════════════════════════════════════════════════════ +schedule_anchor = time.time() + +for run_number in range(1, TOTAL_RUNS + 1): + + scheduled_start = schedule_anchor + (run_number - 1) * INTERVAL_SECONDS + now = time.time() + wait_seconds = scheduled_start - now + + if wait_seconds > 0: + next_run_ts = datetime.datetime.fromtimestamp(scheduled_start).strftime("%Y-%m-%d %H:%M:%S") + log.info("") + log.info("~" * 70) + log.info(f" Waiting {round(wait_seconds, 1)}s until next scheduled run at {next_run_ts} ...") + log.info("~" * 70) + time.sleep(wait_seconds) + + log.info("") + log.info("#" * 70) + log.info(f"#{'':^68}#") + log.info(f"#{'RUN ' + str(run_number) + ' OF ' + str(TOTAL_RUNS):^68}#") + log.info(f"#{'':^68}#") + log.info("#" * 70) + + run_compaction(spark, run_number, TPCH_QUERIES) + + log.info("") + log.info("-" * 70) + log.info(f" END OF RUN {run_number} OF {TOTAL_RUNS}") + log.info("-" * 70) + + remaining = TOTAL_RUNS - run_number + if remaining > 0: + log.info(f" Remaining runs : {remaining}") + for future_run in range(run_number + 1, TOTAL_RUNS + 1): + future_ts = datetime.datetime.fromtimestamp( + schedule_anchor + (future_run - 1) * INTERVAL_SECONDS + ).strftime("%Y-%m-%d %H:%M:%S") + log.info(f" Run {future_run} scheduled at : {future_ts}") + else: + log.info(" All scheduled runs completed. No further runs.") + log.info("-" * 70) + + +# ══════════════════════════════════════════════════════════════════════════════ +# ALL RUNS DONE +# ══════════════════════════════════════════════════════════════════════════════ +log.info("") +log.info("=" * 70) +log.info(" ALL COMPACTION RUNS FINISHED") +log.info(f" Total runs executed : {TOTAL_RUNS}") +log.info(f" Interval : every {INTERVAL_MINUTES} minute(s)") +log.info("=" * 70) +log.info("") +log.info("All done. Stopping SparkSession.") +spark.stop() +``` + +
    + +
    +View spark-submit command for Spark compaction + +```bash +gcloud dataproc jobs submit pyspark gs://dz-benchmark/dataproc_compaction.py \ + --cluster=dz-olake-compaction-21042026 \ + --region=asia-south1 \ + --properties="^#^spark.jars.packages=org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.7.2,org.apache.iceberg:iceberg-aws-bundle:1.7.2#spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions#spark.sql.catalog.benchmark=org.apache.iceberg.spark.SparkCatalog#spark.sql.catalog.benchmark.catalog-impl=org.apache.iceberg.rest.RESTCatalog#spark.sql.catalog.benchmark.uri=http://10.20.0.64:30081/catalog#spark.sql.catalog.benchmark.warehouse=benchmarking#spark.sql.catalog.benchmark.io-impl=org.apache.iceberg.aws.s3.S3FileIO#spark.sql.catalog.benchmark.s3.endpoint=https://storage.googleapis.com/#spark.sql.catalog.benchmark.s3.path-style-access=true#spark.sql.catalog.benchmark.client.region=ap-south-1#spark.sql.defaultCatalog=benchmark#spark.hadoop.fs.s3a.endpoint=https://storage.googleapis.com#spark.hadoop.fs.s3a.path.style.access=true#spark.hadoop.fs.s3a.endpoint.region=ap-south-1#spark.dynamicAllocation.enabled=false#spark.executor.instances=4#spark.executor.cores=7#spark.executor.memory=45g#spark.executor.memoryOverhead=12g#spark.driver.memory=18g#spark.driver.memoryOverhead=10g#spark.sql.parquet.enableVectorizedReader=false#spark.sql.iceberg.vectorization.enabled=false#spark.hadoop.io.native.lib.available=false#spark.executor.extraJavaOptions=-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35 -XX:G1ReservePercent=20 -XX:UseAVX=0 -Darrow.enable_unsafe_memory_access=false -Darrow.enable_null_check_for_get=true -Dhadoop.io.native.lib.available=false#spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35 -Xss8m -XX:UseAVX=0 -Darrow.enable_unsafe_memory_access=false -Darrow.enable_null_check_for_get=true" +``` + +
    + +### OLake Fusion + +The compaction was executed using Fusion's scheduled compaction flow across three compaction levels. + +The compaction job was configured with two trigger tiers: + +- `Lite`: every 20 minutes +- `Medium`: every 40 minutes + +:::info +In this benchmark, the destination dataset size was below 100 GB, so running `Full` compaction was not necessary. It is more useful as a periodic deep-clean step for much larger datasets (for instance, terabyte-scale tables) where long-term small-file buildup is higher. +::: + +The Fusion compaction setup is configured with the following parameters: + +- `target-size`: 512 MB (Target file size after compaction) + +### Dataset and Table Schemas + +#### TPCH `lineitem` table + +```sql +CREATE TABLE lineitem ( + l_orderkey BIGINT NOT NULL, + l_partkey BIGINT NOT NULL, + l_suppkey BIGINT NOT NULL, + l_linenumber INTEGER NOT NULL, + l_quantity DECIMAL(15,2) NOT NULL, + l_extendedprice DECIMAL(15,2) NOT NULL, + l_discount DECIMAL(15,2) NOT NULL, + l_tax DECIMAL(15,2) NOT NULL, + l_returnflag CHAR(1) NOT NULL, + l_linestatus CHAR(1) NOT NULL, + l_shipdate DATE NOT NULL, + l_commitdate DATE NOT NULL, + l_receiptdate DATE NOT NULL, + l_shipinstruct CHAR(25) NOT NULL, + l_shipmode CHAR(10) NOT NULL, + l_comment VARCHAR(44) NOT NULL +); +``` + +#### TPCH Query 6 used in this benchmark + +```sql +SELECT + SUM(l_extendedprice * l_discount) AS revenue +FROM lineitem +WHERE l_shipdate >= DATE '1994-01-01' + AND l_shipdate < DATE '1995-01-01' + AND l_discount BETWEEN 0.05 AND 0.07 + AND l_quantity < 24; +``` + +> **Bottom line:** If you need to compact Iceberg tables quickly and keep them highly queryable as data changes continuously, OLake Fusion delivers faster compaction cycles with lower infrastructure cost. + diff --git a/docs/getting-started/configure-first-optimization.mdx b/docs/fusion/getting-started/configure-first-compaction.mdx similarity index 60% rename from docs/getting-started/configure-first-optimization.mdx rename to docs/fusion/getting-started/configure-first-compaction.mdx index ae434dbcf..0f3b9991e 100644 --- a/docs/getting-started/configure-first-optimization.mdx +++ b/docs/fusion/getting-started/configure-first-compaction.mdx @@ -1,31 +1,26 @@ --- -title: Configure Your First optimization -sidebar_label: Configure Your First Optimization +title: Configure Your First Compaction +sidebar_label: Configure Your First Compaction --- -# Configure Your First Optimization +# Configure Your First Compaction -## Prerequisites - -Follow the [Quickstart Setup Guide](/docs/install/olake-ui/) to ensure the OLAKE UI is running at [localhost:8000](http://localhost:8000). - -- You have at least one destination configured in Ingestion **or** -- You are ready to add a catalog manually in the optimization section. +This guide walks through configuring your **first compaction** for a table. -:::info Optimization in the OLake UI - -Iceberg maintenance (**Optimization**) is available starting from v0.4.0. Upgrade OLake UI to access the **Maintenance** module. +## Prerequisites -- **Existing users:** If you are already using OLake for Ingestion follow the [upgrade guide](/docs/install/olake-ui/#updating-olake-ui-version) to accesss Maintenance module. -- **New users:** Follow the [quickstart guide](/docs/install/olake-ui/#quick-start) to get started. +- **Existing users (Docker):** If you are already using OLake Go for Ingestion, follow the [upgrade guide](/docs/fusion/install/olake-ui/?setup-mode=configuration#updating-olake-ui-version) to access the Maintenance module. +- **Existing users (Helm / Kubernetes):** If you are running OLake Go on Kubernetes, follow the [chart upgrade guide](/docs/fusion/install/kubernetes-compaction/#upgrading-chart-version) to access the Maintenance module. +- **New Users (Docker):** Follow the [quickstart guide](/docs/fusion/install/olake-ui/#one-command-setup) to get started. +- **New Users (Helm / Kubernetes):** Follow the [quickstart guide](/docs/fusion/install/kubernetes-compaction/#quick-start) to get started. +:::warning +For conflict free compaction of OLake Go-Ingested tables, upgrade **OLake Go** (Ingestion) driver version to **v0.7.0 or higher**. ::: -This guide walks through configuring your **first optimization** for a table. - ## Step 1: Add a Catalog -Catalogs tell OLake where your Iceberg tables live. +Catalogs tell OLake Fusion where your Iceberg tables live. 1. In the OLake UI sidebar, open the **Maintenance** dropdown and go to the **Catalogs** page. 2. Select **New Catalog**. @@ -37,7 +32,7 @@ Catalogs tell OLake where your Iceberg tables live. ![Catalog connected view](pathname:///img/docs/iceberg-maintenance/catalogs/connect-catalog.webp) -For more details on Catalogs, see the [Catalogs documentation](/docs/iceberg-maintenance/catalogs/). +For more details on Catalogs, see the [Catalogs documentation](/docs/fusion/maintenance/catalogs/). After the catalog is saved successfully, a **Catalog Added Successfully** modal appears with two actions: @@ -52,20 +47,20 @@ Click **View Tables** to go to the **Tables** page for Step 2. 1. Use the **Select Catalog** dropdown and select the catalog you just configured. -![Select Catalog](pathname:///img/docs/getting-started/configure-your-first-optimization/select-catalog.webp) +![Select Catalog](pathname:///img/docs/getting-started/configure-your-first-compaction/select-catalog.webp) 2. After selecting the catalog, use the **Select Database** dropdown to choose a database (Iceberg DB) from that catalog. -![Select Database](pathname:///img/docs/getting-started/configure-your-first-optimization/select-database.webp) +![Select Database](pathname:///img/docs/getting-started/configure-your-first-compaction/select-database.webp) Only after you select both a catalog and a database will the list of tables in that Iceberg database appear on the page. -## Step 3: Configure Optimization For Your Table +## Step 3: Configure Compaction for your Table -1. In the tables list, find the table you want to optimize. +1. In the tables list, find the table you want to compact.
    - Tip: Click View Metrics to open table metrics. **Health Score** and **target file size** (and related size signals in the metrics view) help decide whether optimization is required for that table. + Tip: Click View Metrics to open table metrics. **Health Score** and **target file size** (and related size signals in the metrics view) help decide whether compaction is required for that table. ![Table metrics view](pathname:///img/docs/iceberg-maintenance/metrics/view-table-metrics-button.webp) @@ -73,15 +68,15 @@ Only after you select both a catalog and a database will the list of tables in t 2. Click on the **Configure** button. -![Configuration Button](pathname:///img/docs/iceberg-maintenance/optimization/configure-button.webp) +![Configuration Button](pathname:///img/docs/iceberg-maintenance/compaction/configure-button.webp) -This opens a configuration modal (cron modal) where you can set schedules for **Lite**, **Medium**, and **Full** Optimization. +This opens a configuration modal (cron modal) where you can set schedules for **Lite**, **Medium**, and **Full** compaction. -![Configuration Cron](pathname:///img/docs/iceberg-maintenance/optimization/configuration.webp) +![Configuration Cron](pathname:///img/docs/iceberg-maintenance/compaction/configuration.webp) ### Frequency Presets -When configuring optimization for a table, each optimization type has a **Frequency** dropdown with common schedules, such as: +When configuring compaction for a table, each compaction type has a **Frequency** dropdown with common schedules, such as: - Never - Every 30 min @@ -90,11 +85,11 @@ When configuring optimization for a table, each optimization type has a **Freque - Every 12 hours - Every 24 hours -You can configure these frequencies independently for Lite, Medium, and Full Optimization. +You can configure these frequencies independently for Lite, Medium, and Full compaction. **Default schedules are applied automatically** for each table, so there is no need to open the configuration modal and set frequencies on every table unless a different cadence is required. -**Defaults schedules for each type of optimization:** +**Default schedules for each type of compaction:** **Lite** — every 1 hour @@ -108,54 +103,46 @@ If you choose **Custom** in the Frequency dropdown, a **Cron Expression** field You can enter a standard cron expression here. For example: -- `0 0 * * *` – run the optimization once every day at midnight. +- `0 0 * * *` – run compaction once every day at midnight. ## Step 4: (Advanced) Target File Size Under the **Advanced Config** dropdown in the modal, you can configure **Target file size**. -![Target file size](pathname:///img/docs/iceberg-maintenance/optimization/target-file-size.webp) +![Target file size](pathname:///img/docs/iceberg-maintenance/compaction/target-file-size.webp) -**Full** Optimization uses **target file size** directly: rewritten data files are aligned toward that size. +**Full** compaction uses **target file size** directly: rewritten data files are aligned toward that size. -**Lite** and **Medium** use it **indirectly**. Their merge and output bounds are derived from the same setting. How each type relates to this value is explained in the [Optimization overview](/docs/iceberg-maintenance/optimization/overview/). +**Lite** and **Medium** use it **indirectly**. Their merge and output bounds are derived from the same setting. How each type relates to this value is explained in the [Types of Compaction](/docs/fusion/compaction/types-of-compaction/). In general, a **larger** target tends toward **fewer, bigger** files; a **smaller** target tends toward **more, smaller** files. If unsure, start with the default (**512 MB**) and tune later based on query-engine behavior. -## Step 5: Save the Optimization Configuration +## Step 5: Save the Compaction Configuration After configuring the cron: 1. In the modal, click on **Save**. 2. The configuration for that table is saved. -![Save Configuration](pathname:///img/docs/iceberg-maintenance/optimization/save-configuration.webp) +![Save Configuration](pathname:///img/docs/iceberg-maintenance/compaction/save-configuration.webp) After saving, a **Configuration Successful** modal appears. It **closes automatically after 3 seconds**. -![Configuration Successful modal](pathname:///img/docs/iceberg-maintenance/optimization/configuration-successful.webp) +![Configuration Successful modal](pathname:///img/docs/iceberg-maintenance/compaction/configuration-successful.webp) +Once saved, the **Status** toggle for that table turns on automatically and compaction runs on the schedule you configured. Confirm this in the **Status** column as the toggle should appear active. -## Step 6: Enable the Optimization - -Saving the configuration does not start optimization automatically. You must enable it: - -1. On the **Tables** page, locate the **Status** column next to the **Configure** button for your table. -2. Use the **toggle** in the **Status** column to enable the optimization configuration. - -![Enable Configuration](pathname:///img/docs/iceberg-maintenance/optimization/enable-optimization.webp) - -Once enabled, OLake will start running optimization for that table according to the schedule you configured. +![Enable Configuration](pathname:///img/docs/iceberg-maintenance/compaction/enable-compaction.webp) ## Health Score and Last Run Status -With a catalog and database selected, the **Tables** page shows one row per table. The sections below explain **Health Score** (overall table health) and **Last Run status** (per-type status for Lite, Medium, and Full optimization). +With a catalog and database selected, the **Tables** page shows one row per table. The sections below explain **Health Score** (overall table health) and **Last Run status** (per-type status for Lite, Medium, and Full compaction). ### Health Score -**Health Score** is a single number that summarises how “healthy” the table looks from OLake’s perspective. It is **calculated** as: +**Health Score** is a single number that summarises how “healthy” the table looks from OLake Fusion’s perspective. It is **calculated** as: **Health Score** = Small Files Score + Eq Delete Score + Pos Delete Score @@ -165,7 +152,7 @@ With a catalog and database selected, the **Tables** page shows one row per tabl Together, these three parts are **weighted 40% / 40% / 20%**: **Small Files Score** and **Eq Delete Score** each contribute **40%** of the Health Score, and **Pos Delete Score** contributes **20%**. -Higher scores generally mean the table is in better shape for reads; lower scores suggest running or tuning optimization more often. +Higher scores generally mean the table is in better shape for reads; lower scores suggest running or tuning compaction more often. ![Health Score column on the Tables page](pathname:///img/docs/iceberg-maintenance/metrics/health-score.webp) @@ -174,35 +161,33 @@ Higher scores generally mean the table is in better shape for reads; lower score **Last Run status** always shows three badges—**L (Lite)**, **M (Medium)**, **F (Full)**. Each badge is that type’s **latest** outcome: **running**, **success**, **failed**, **cancelled**, **skipped**, or **never run**. - **Letters** — Typically you see three badges together: - - **L** — Lite Optimization - - **M** — Medium Optimization - - **F** — Full Optimization + - **L** — Lite compaction + - **M** — Medium compaction + - **F** — Full compaction - **Colours (quick read, per badge)** - **Green** — that type’s last run **succeeded** - **Red** — that type’s last run **failed** or was **cancelled** - **Yellow** — a run of **that type** is **running** right now - **Gray** with **ⓘ** — that type’s last run was **skipped** - - **Gray** with **◌** — that optimization type has **never** run for this table + - **Gray** with **◌** — that compaction type has **never** run for this table -- **Not Optimized** — Shown **only** when **no** optimization has run yet for that table—**neither** Lite, **nor** Medium, **nor** Full. +- **Not compacted** — Shown **only** when **no** compaction has run yet for that table—**neither** Lite, **nor** Medium, **nor** Full. :::info Hover When you **hover** over a table's Last Run Status, a small **card** opens which includes: -- **Name** — Lite, Medium, or Full +- **Name** — Lite, Medium, or Full - **Last run** — relative time, such as “2 hours ago” - **Status** — plain text such as **Success**, **Failed**, **Cancelled**, **Skipped**, or **Running** ::: -![Last Run status](pathname:///img/docs/iceberg-maintenance/runs-and-Logs/last-run-status-hover.webp) +![Last Run status](pathname:///img/docs/iceberg-maintenance/runs-and-logs/last-run-status-hover.webp) ## Next Steps -After your first optimization runs, you can: - -- **View Logs & Runs** – see each optimization run and its detailed logs: - [Logs & Runs](/docs/iceberg-maintenance/runs-and-logs) -- **View Metrics** – understand how optimization affecme size as ts file counts, sizes, and health score for your table: - [Metrics](/docs/iceberg-maintenance/metrics) +After your first compaction runs, you can: +- **View [Logs & Runs](/docs/fusion/maintenance/runs-and-logs/)** – see each compaction run and its detailed logs: +- **View [Metrics](/docs/fusion/maintenance/metrics/)** – understand how compaction affects file counts, sizes, and health score for your table: + diff --git a/docs/fusion/getting-started/overview.mdx b/docs/fusion/getting-started/overview.mdx new file mode 100644 index 000000000..ad2643fd9 --- /dev/null +++ b/docs/fusion/getting-started/overview.mdx @@ -0,0 +1,42 @@ +--- +title: Overview +sidebar_position: 1 +--- + +## What is OLake Fusion? + +OLake Fusion is an automated Iceberg table maintenance platform that keeps your lakehouse tables efficient, compact, and query-ready as data continuously grows. As Iceberg tables evolve through ingestion, updates, and deletes, they accumulate small files, delete files, and excess metadata — all of which degrade query performance and increase storage overhead over time. + +OLake Fusion takes care of all of this, ensuring your destination Iceberg tables stay maintained, performant, and efficiently queriable as they scale. + +## When is Table Maintenance Required? + +In Apache Iceberg, table maintenance should be performed periodically to ensure consistent query performance and efficient storage as data evolves. + +You should consider running maintenance in the following scenarios: + +- **Frequent Data Ingestion or Updates** + +- **Accumulation of Small Files** + +- **Presence of Delete Files** + +- **High Partition Cardinality** + +- **Degrading Query Performance** + +- **Growing Table Size Over Time** + + Regular maintenance ensures that Iceberg tables remain compacted, scalable, and performant for analytical workloads. + +:::info Maintenance in the OLake UI + +Iceberg Maintenance is available starting from v0.4.0. Upgrade OLake UI to access the **Maintenance** module. + +- **Existing users (Docker):** If you are already using OLake Go for Ingestion, follow the [upgrade guide](/docs/fusion/install/olake-ui/?setup-mode=configuration#updating-olake-ui-version) to access the Maintenance module. +- **Existing users (Helm / Kubernetes):** If you are running OLake Go on Kubernetes, follow the [chart upgrade guide](/docs/fusion/install/kubernetes-compaction/?setup-mode=installation#upgrading-chart-version) to access the Maintenance module. +- **New Users (Docker):** Follow the [quickstart guide](/docs/fusion/install/olake-ui/?setup-mode=installation&quick-start=Ingestion+%2B+Maintenance#one-command-setup) to get started. +- **New Users (Helm / Kubernetes):** Follow the [quickstart guide](/docs/fusion/install/kubernetes-compaction/?setup-mode=installation#quick-start) to get started. + +::: + diff --git a/docs/fusion/getting-started/quickstart.mdx b/docs/fusion/getting-started/quickstart.mdx new file mode 100644 index 000000000..a5ee06539 --- /dev/null +++ b/docs/fusion/getting-started/quickstart.mdx @@ -0,0 +1,60 @@ +--- +title: Quickstart +sidebar_label: Quickstart +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# How to get started with OLake Fusion +This QuickStart guide helps get started with [OLake UI](/docs/fusion/install/olake-ui/#quick-start), a web-based interface designed to simplify the Iceberg table maintenance that is view table health scores, track compaction history, monitor table metrics, and configure compaction settings across your tables. + +## Prerequisites + +The following requirements must be met before starting: +- [Docker](https://docs.docker.com/get-docker/) installed (Docker Desktop recommended) +- [Docker Compose](https://docs.docker.com/compose/) (included with Docker Desktop) +- At least 4GB RAM available for Docker +- Port 8000 available on the system + +## Quick Start (Docker Compose) + +### One-Command Setup +The fastest way to get OLake UI running is with a single command: + +```bash +curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose-v1.yml | ENABLE_OPTIMIZATION="true" docker compose --profile fusion -f - up -d +``` + +*This setup uses Postgres for both metadata and Temporal visibility.* + +This command will: +- Download the latest docker-compose.yml file +- Pull all required Docker images +- Start all services in the background +- Create a default admin user automatically + +### Access the Application + +- **OLake UI**: [http://localhost:8000](http://localhost:8000) + +### Login + +The default credentials are: +- **Username**: `admin` +- **Password**: `password` + +![Login page with fields for admin username and password](/img/docs/jobs/olake-job-0.webp) + +![After login](/img/docs/getting-started/olake-job.webp) + +### Updating OLake UI + +To update OLake UI to the latest version, use the following commands based on your setup: + +```bash +curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose-v1.yml \ +| ENABLE_OPTIMIZATION="true" docker compose --profile fusion -f - down && \ +curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose-v1.yml \ +| ENABLE_OPTIMIZATION="true" docker compose --profile fusion -f - up -d +``` diff --git a/docs/fusion/install/kubernetes-compaction.mdx b/docs/fusion/install/kubernetes-compaction.mdx new file mode 100644 index 000000000..3c02d978e --- /dev/null +++ b/docs/fusion/install/kubernetes-compaction.mdx @@ -0,0 +1,622 @@ +--- +title: "Deploy OLake Maintenance on Kubernetes" +description: Deploy OLake Maintenance on Kubernetes with Helm. Enable Maintenance services, configure Spark optimizer (compactor) settings, and run compaction workflows for Iceberg tables. +sidebar_position: 2 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import BrowserOnly from '@docusaurus/BrowserOnly'; + +# OLake Fusion Kubernetes Installation with Helm + +This guide details the process for deploying OLake ingestion and Fusion maintenance together on Kubernetes using the official Helm chart. Fusion is installed alongside the core OLake ingestion stack and provides Iceberg table maintenance and ingestion services. + +
    +Components & Architecture + +- **OLake UI**: Main web interface for job management and configuration +- **OLake Worker**: Background worker for processing data replication jobs +- **PostgreSQL**: Primary database for storing job data, configurations, sync state, and Temporal visibility data +- **Temporal**: Workflow orchestration engine for managing job execution +- **Signup Init**: One-time initialization service that creates the default admin user +- **Fusion**: Maintenance control plane and API server +- **Fusion Spark Optimizer**: Spark-on-Kubernetes execution layer for Maintenance jobs + +
    + +![OLake UI, Postgres, Temporal orchestrating OLake-Go ingestion, and Fusion performing maintenance against the data catalog](/img/docs/fusion-architecture.webp) + +
    + +
    + +## Prerequisites + +Ensure the following requirements are met before proceeding: + +- **Kubernetes 1.19+**: Administrative access to a Kubernetes cluster +- **Helm 3.2.0+**: Helm client installed and configured. [Installation Guide](https://helm.sh/docs/intro/install/) +- **kubectl**: Configured `kubectl` command-line tool. [Installation Guide](https://kubernetes.io/docs/tasks/tools/install-kubectl/) +- **StorageClass**: A StorageClass is required by the chart to provision persistent volumes for PostgreSQL and the shared storage volume + ```bash + # List available StorageClass + kubectl get storageclass + ``` +- **System Requirements**: Minimum resources per workload: + + | Component | RAM | vCPU | + | --- | --- | --- | + | OLake UI | 4 GB | 2 | + | Temporal | 4 GB | 2 | + | OLake Worker | 4 GB | 2 | + | PostgreSQL | 4 GB | 2 | + | Sync pod (OLake Go) | 8 GB | 4 | + | Fusion AMS | 8 GB | 4 | + | Optimizer | 1 GB | — | + | Spark executor | 4 GB | — | + +:::note +Sync pod sizing depends on the data volume to be synced. Use [JobID-Based Scheduling](/docs/install/kubernetes#jobid-based-scheduling) to map jobs to nodes with sufficient capacity. +::: + + + +OLake will run on mixed capacity, but scheduling the components below on stable, non-preemptible (on-demand) nodes lowers the chance of spot or preemptible churn interrupting core services and compaction work: + +- nfs-server (if enabled) +- PostgreSQL (if enabled) +- Temporal +- OLake workers +- OLake UI +- Fusion and Spark optimizer workloads + + + + +## Quick Start + +### 1. Add OLake Helm Repository + +```bash +helm repo add olake https://datazip-inc.github.io/olake-helm +helm repo update +``` + +### 2. Install the Chart + + + + +**Ingestion + Maintenance:** +```bash +helm install olake olake/olake --set global.storageClass="gp2" --set fusion.enabled=true +``` + + + + + +**Ingestion + Maintenance:** +```bash +helm install olake olake/olake --set fusion.enabled=true +``` + + + + + +### 3. Access OLake UI + +Forward the UI service port to local machine: + +```bash +kubectl port-forward svc/olake-ui 8000:8000 +``` + +Open browser and navigate to: `http://localhost:8000` + +**Default Credentials:** +- Username: `admin` +- Password: `password` + +:::tip +If OLake is installed with **Ingress enabled**, port-forwarding is not necessary. Access the application using the configured Ingress hostname. +::: + +With Fusion enabled, follow [Configure your first compaction](https://olake.io/docs/getting-started/configure-first-compaction) for step-by-step guidance on adding catalogs and scheduling maintenance on the Iceberg tables in the OLake UI. + +## Configuration Options + +#### Customizing Fusion Spark Optimizer Configuration + +The Fusion Spark optimizer can be customized by providing additional Spark configurations via `fusion.optimizer.spark.extraConfig` and other container-specific properties via `fusion.optimizer.spark.properties`. + +For example, to set `spark.sql.shuffle.partitions` and `export.JAVA_HOME`: + +```yaml +fusion: + optimizer: + spark: + extraConfig: + spark.sql.shuffle.partitions: "200" + properties: + export.JAVA_HOME: "/usr/lib/jvm/java-17-openjdk" +``` + +Refer to the [Spark documentation](https://spark.apache.org/docs/latest/configuration.html) for a comprehensive list of configurable properties for Spark containers. + +
    +Compaction Scheduling + +Configure `fusion.nodeSelector`, `fusion.tolerations`, and `fusion.affinity` in `values.yaml` to define scheduling behavior for compaction workloads. +These settings are applied to the Fusion pod and to Spark driver/executor pods created by Fusion. The following example shows the expected values structure: + +```yaml +fusion: + nodeSelector: + workload: "fusion" + tolerations: [] + affinity: {} +``` + +
    + +### Updating OLake UI Version +Pull the latest images and restart the deployments without downtime: + +```bash +# Restart OLake components +kubectl rollout restart deployment/olake-ui +kubectl rollout restart deployment/olake-workers +``` + +### Initial User Setup + +Create a Kubernetes secret to replace default credentials: + +```bash +kubectl create secret generic olake-admin-credentials \ + --from-literal=username='superadmin' \ + --from-literal=password='a-very-secure-password' \ + --from-literal=email='admin@mycompany.com' +``` + +Then configure in `values.yaml`: + +```yaml +olakeUI: + initUser: + existingSecret: "olake-admin-credentials" + secretKeys: + username: "username" + password: "password" + email: "email" +``` + +Apply the configuration: + +```bash +helm upgrade olake olake/olake -f values.yaml --set fusion.enabled=true +``` + +### Cloud IAM Integration + +OLake Fusion's spark optimizer pods can be allowed to securely access cloud resources(AWS Glue or S3) using IAM roles. + +```yaml +global: + jobServiceAccount: + create: true + name: "olake-job-sa" + + # Cloud provider IAM role associations + annotations: + # AWS IRSA + eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/olake-job-role" + + # GCP Workload Identity + iam.gke.io/gcp-service-account: "olake-job@project.iam.gserviceaccount.com" + + # Azure Workload Identity + azure.workload.identity/client-id: "12345678-1234-1234-1234-123456789012" +``` +:::note +Note: For detailed instructions on the creation of IAM roles and service accounts, the official documentation for AWS IRSA, GCP Workload Identity, or Azure Workload Identity should be referred to. For a minimal Glue and S3 IAM access policy please refer [here](https://olake.io/docs/writers/iceberg/catalog/glue#required-iam-permissions). +::: + +### Ingress Configuration + +To expose OLake through an ingress controller, create a custom values file: + +```yaml +# values.yaml +olakeUI: + ingress: + enabled: true + className: "nginx" + hosts: + - host: olake.example.com + paths: + - path: / + pathType: Prefix + tls: + - secretName: olake-tls + hosts: + - olake.example.com +``` + +### Fusion Compaction Scheduling + +Fusion runs two kinds of pods with very different resource profiles: + +- The **Fusion pod** and the **optimizer** pod — a lightweight, always-on orchestrator. +- The **executor** pod(s) — created on-demand when compaction runs, and typically need much more memory/CPU (larger tables = bigger executors). + +Running both on the same node pool means keeping a large node around even when compaction isn't active. To avoid that, the Fusion pod and optimizer can be scheduled independently from the executor pods: + +```yaml +fusion: + nodeSelector: + node-type: "olake-standard" + tolerations: + - key: "service" + operator: "Equal" + value: "olake" + effect: "NoSchedule" + affinity: {} + + optimizer: + # If left empty, falls back to fusion.nodeSelector / tolerations / affinity above. + nodeSelector: {} + tolerations: [] + affinity: {} + + spark: + executor: + # If left empty, falls back to fusion.nodeSelector / tolerations / affinity above. + nodeSelector: + node-type: "olake-compaction-large" + tolerations: + - key: "compaction" + operator: "Equal" + value: "true" + effect: "NoSchedule" + affinity: {} +``` + +### Persistent Storage Configuration + +The OLake application components (UI, Worker, and Activity Pods) require a shared ReadWriteMany (RWX) volume for coordinating pipeline state and metadata. + +For production, a robust, highly-available RWX-capable storage solution such as AWS EFS, GKE Filestore, or Azure Files must be used. This is achieved by disabling the built-in NFS server and providing an existing Kubernetes StorageClass that is backed by a managed storage service. An example for using StorageClass is given below: + +```yaml +nfsServer: + # 1. The development NFS server is disabled + enabled: false + + # 2. An existing ReadWriteMany PersistentVolumeClaim is specified + external: + storageClass: "efs-csi" +``` +:::note +For development and quick starts, a simple NFS server is included and enabled by default. This provides an out-of-the-box shared storage solution without any external dependencies. However, because this server runs as a single pod, it represents a single point of failure and is not recommended for production use. +::: + +:::warning +**Bottlerocket OS on AWS EKS:** The built-in NFS server is incompatible with Bottlerocket OS worker nodes. For such AWS EKS configurations, AWS EFS must be used as an alternative, which requires setting `nfsServer.enabled: false` and configuring the EFS CSI driver as shown above. +::: + +### External PostgreSQL Configuration + +External PostgreSQL databases can be used instead of the built-in postgresql deployment. It is the primary database for storing job data, configurations, and sync state. + +**Requirements:** +- PostgreSQL 12+ with `btree_gin` extension enabled +- This can be enabled with `CREATE EXTENSION IF NOT EXISTS btree_gin;`, then run `\dx` to verify if its enabled. +- Separate OLake, Temporal and Fusion databases created on the PostgreSQL instance +- Network connectivity from Kubernetes cluster to PostgreSQL instance + +There are two ways to configure an external PostgreSQL database: + +#### Option 1: Using `existingSecret` + +Reference a pre-existing Kubernetes Secret containing the database credentials. The secret must be created manually before installing the chart. + +**1. Create the database secret:** +```bash +kubectl create secret generic external-postgres-secret \ + --from-literal=host="postgres-host" \ + --from-literal=port="5432" \ + --from-literal=olake_database="olakeDB" \ + --from-literal=temporal_database="temporalDB" \ + --from-literal=fusion_database="fusionDB" \ + --from-literal=username="username" \ + --from-literal=password="password" \ + --from-literal=ssl_mode="require" +``` + +**2. Configure values.yaml:** +```yaml +postgresql: + enabled: false + external: + existingSecret: "external-postgres-secret" +``` + +#### Option 2: Using `properties` (Recommended for ArgoCD/GitOps) + +Specify the database connection details directly in `values.yaml`. The chart automatically creates a Kubernetes Secret from these values at template time. This approach is fully compatible with ArgoCD and other GitOps tools that use `helm template` for rendering. + +```yaml +postgresql: + enabled: false + external: + properties: + host: "postgres-host" + port: 5432 + username: "username" + password: "password" + olake_database: "olakeDB" + temporal_database: "temporalDB" + fusion_database: "fusionDB" + ssl_mode: "require" +``` + +### Global Environment Variables + +Environment variables defined in `global.env` are automatically propagated to OLake UI, OLake Workers, and Activity Pods: + +```yaml +global: + env: + OLAKE_SECRET_KEY: "your-secret-encryption-key" + RUN_MODE: "production" + # Add any custom environment variables here +``` + +### Private Container Registry + +OLake supports pulling all images from a private or self-hosted container registry. + +#### Step 1 — Point OLake at the registry + +Setting `CONTAINER_REGISTRY_BASE` in `global.env` causes every image reference in the chart to be automatically prefixed with that value — no per-image overrides are needed. + +```yaml +global: + env: + CONTAINER_REGISTRY_BASE: "registry.example.com/myproject" +``` + +The following images must be mirrored to the registry before deploying: + +| Image | Source | +|---|---| +| `library/busybox:latest` | Docker Hub | +| `curlimages/curl:8.1.2` | Docker Hub | +| `olakego/ui:latest`, `olakego/ui-worker:latest` | Docker Hub | +| `olakego/source-*` (e.g. `olakego/source-mongodb:v0.7.0`) | Docker Hub | +| `temporalio/auto-setup:1.22.3`, `temporalio/ui:2.16.2` | Docker Hub | +| `library/postgres:14-alpine` | Docker Hub | +| `olakego/fusion:latest`, `olakego/fusion-spark:latest` | Docker Hub | +| `sig-storage/nfs-provisioner:v4.0.8` | `registry.k8s.io` (built-in NFS only) | + +#### Step 2 — Authenticate with the registry + +If the registry requires credentials, a Kubernetes docker-registry secret must be created: + +```bash +kubectl create secret docker-registry my-registry-secret \ + --docker-server=registry.example.com \ + --docker-username= \ + --docker-password= +``` + +The secret is then referenced in `values.yaml`: + +```yaml +global: + imagePullSecrets: + - name: my-registry-secret + + env: + CONTAINER_REGISTRY_BASE: "registry.example.com/myproject" +``` + +:::note +- For cloud-managed registries (Amazon ECR, Google Artifact Registry, Azure ACR), IAM-based authentication (IRSA / Workload Identity) is preferred over static credentials. See [Cloud IAM Integration](/docs/install/kubernetes/#cloud-iam-integration). +::: + +### Global Pod Annotations + +Annotations defined in `global.podAnnotations` are applied to every pod managed by this chart — all Deployment pods (OLake UI, OLake Workers, PostgreSQL, Temporal, Elasticsearch, Fusion), Fusion Spark optimizer pods, and connector activity pods (sync, discover, check) spawned by olake-workers. + +This is useful for service mesh sidecar injection (Istio, Linkerd) or any admission-webhook-based tooling that requires pod-level annotations. + +```yaml +global: + podAnnotations: + sidecar.istio.io/inject: "true" + linkerd.io/inject: enabled +``` + +## Upgrading Chart Version + +:::info +Currently ES Stack does not have an option to upgrade to maintenance Stack, To use maintenance feature upgrade legacy to latest stack first. +::: + +### Upgrade to Latest Version + +```bash +helm repo update +helm upgrade olake olake/olake --set fusion.enabled=true +``` + +### Upgrade with New Configuration + +```bash +helm upgrade olake olake/olake -f new-values.yaml --set fusion.enabled=true +``` + +:::warning +For conflict free compaction of OLake-Ingested tables, upgrade **OLake Go** (Ingestion) driver version to **v0.7.0 or higher**. +::: + +### Post-Upgrade: Resume Stopped Syncs + +After upgrading, perform a rollout restart of the olake-workers: + +```bash +kubectl rollout restart deployment/olake-workers +``` + +## Troubleshooting + +### Check Pod Logs + +```bash +# OLake UI logs +kubectl logs -l app.kubernetes.io/name=olake-ui -f + +# Fusion logs +kubectl logs -l app.kubernetes.io/component=fusion -f + +# Temporal server logs +kubectl logs -l app.kubernetes.io/name=temporal-server -f +``` + +### Common Issues + +**Pods Stuck in Pending State:** +- Check if your cluster has sufficient resources +- Check node selectors or affinity rules +- Verify StorageClass is available and configured correctly + +**Pods Stuck in CrashLoopBackOff State:** +- If pod logs show error like `failed to ping database`, check the `Database Connection Issues` section below +- If pod restarts and goes into CrashLoopBackOff state, check for the resource requests and limits defined in Helm Values file +- If pod events show error like `failed to mount volume`, check if the nfs-server pod is up and running + +**Database Connection Issues:** +- Verify PostgreSQL pod is running: `kubectl get pods -l app.kubernetes.io/name=postgresql` +- In case, setup is done with External PostgreSQL Configuration, check the following: + - Check if the host is pointing to Writer instance and not a Reader of the database + - Check if password contains special characters like `@` or `#` etc. If yes, use a different password + - Check if `ssl_mode` is correctly set in the kubernetes secret for external database +- Check database connectivity from other pods +- Review database credentials in secrets + +## Migration Guides + +
    +Migrating to v0.0.12 + +When upgrading from a previous version, the `olake-signup-init` Job must be deleted before running `helm upgrade`. Kubernetes does not allow modifications to a Job's pod template, and the updated image references in this version will cause the upgrade to fail. + +```bash +kubectl delete job olake-signup-init -n olake +helm upgrade olake olake/olake --set fusion.enabled=true +``` +
    + +
    +Migrating to v0.0.7 (Standard Resources) + +Version 0.0.7 introduced a significant change to how ServiceAccount, RBAC, and Secret resources are managed. By default, `useStandardResources` is now set to `true`, which converts these from Helm Hooks to standard resources. This improved compatibility with ArgoCD and prevents race conditions during updates. + +**For New Installations:** +No action needed. The new default (`true`) is the recommended configuration. + +**For Existing Installations:** +Upgrading directly may cause `resource already exists` errors because Helm tries to adopt resources that were previously created by hooks. + +**Option 1: Maintain Legacy Behavior (Easiest)** +Set the flag to `false` in custom `values.yaml` to keep the old hook-based behavior: +```yaml +useStandardResources: false +``` + +**Option 2: Migrate to Standard Resources (Recommended)** +To adopt the new behavior, manually remove the hook annotations and label the resources for Helm adoption before upgrading: + +```bash +# 1. ServiceAccount +kubectl annotate serviceaccount olake-workers meta.helm.sh/release-name=olake meta.helm.sh/release-namespace=olake helm.sh/hook- helm.sh/hook-weight- helm.sh/hook-delete-policy- -n olake --overwrite +kubectl label serviceaccount olake-workers app.kubernetes.io/managed-by=Helm -n olake --overwrite + +# 2. Role +kubectl annotate role olake-workers meta.helm.sh/release-name=olake meta.helm.sh/release-namespace=olake helm.sh/hook- helm.sh/hook-weight- helm.sh/hook-delete-policy- -n olake --overwrite +kubectl label role olake-workers app.kubernetes.io/managed-by=Helm -n olake --overwrite + +# 3. RoleBinding +kubectl annotate rolebinding olake-workers meta.helm.sh/release-name=olake meta.helm.sh/release-namespace=olake helm.sh/hook- helm.sh/hook-weight- helm.sh/hook-delete-policy- -n olake --overwrite +kubectl label rolebinding olake-workers app.kubernetes.io/managed-by=Helm -n olake --overwrite + +# 4. Secret +kubectl annotate secret olake-workers-secret meta.helm.sh/release-name=olake meta.helm.sh/release-namespace=olake helm.sh/hook- helm.sh/hook-weight- helm.sh/hook-delete-policy- -n olake --overwrite +kubectl label secret olake-workers-secret app.kubernetes.io/managed-by=Helm -n olake --overwrite + +# 5. Perform the upgrade +helm upgrade olake olake/olake --set fusion.enabled=true +``` +
    + + + diff --git a/docs/fusion/install/olake-ui/index.mdx b/docs/fusion/install/olake-ui/index.mdx new file mode 100644 index 000000000..16a8a7e5c --- /dev/null +++ b/docs/fusion/install/olake-ui/index.mdx @@ -0,0 +1,310 @@ +--- +title: "OLake Fusion UI Installation Guide - Docker Compose Setup & Configuration" +description: Quickly deploy OLake Fusion UI with Docker Compose. Manage ingestion jobs, run table maintenance (compaction), configure encryption, customize data directory, and troubleshoot common deployment issues. +sidebar_label: Docker Compose +sidebar_position: 1 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import BrowserOnly from '@docusaurus/BrowserOnly'; + +# Docker Compose (OLake UI) + +OLake UI provides a complete Docker Compose stack for running the replication sync using different sources with orchestration. + +
    +Components & Architecture + +- **OLake UI**: Main web interface for job management and configuration +- **Temporal Worker**: Background worker for processing data replication jobs +- **PostgreSQL**: Primary database for storing job data, configurations, sync state, and Temporal visibility data +- **Temporal Server**: Workflow orchestration engine for managing job execution +- **Temporal UI**: Web interface for monitoring workflows and debugging +- **Signup Init**: One-time initialization service that creates the default admin user +- **Fusion**: Maintenance control plane and API server +- **Fusion Spark Optimizer**: Spark-on-Kubernetes execution layer for maintenance jobs (Kind cluster in Docker) + +
    + +![OLake platform Docker Compose architecture showing user, core services, containers, and external data sources.](/img/docs/fusion-architecture.webp) + +
    + +
    + +## Prerequisites + +The following requirements must be met before starting: +- [Docker](https://docs.docker.com/get-docker/) installed (Docker Desktop recommended) +- [Docker Compose](https://docs.docker.com/compose/) (included with Docker Desktop) +- Port 8000 available on the system +- System Requirements: + - Minimum: 8 vCPU, 16 GB RAM + - Recommended: 16 vCPU, 32 GB RAM + +## Quick Start + +### One-Command Setup + +The fastest way to get OLake UI running is with a single command: + +Setup Maintenance and Ingestion stack with following command: + +```bash +curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose-v1.yml | ENABLE_OPTIMIZATION="true" docker compose --profile fusion -f - up -d +``` + +The OLake UI server reads `ENABLE_OPTIMIZATION` from the environment. Set `ENABLE_OPTIMIZATION="true"` in the command above to access maintenance module; the `fusion` profile starts Fusion and related services. + +Spark maintenance executes on a Kind (Kubernetes in Docker) cluster (`fusion-cluster`) created by the Fusion profile. `fusion-db-init` downloads `config.yaml`, `kind-config.yaml`, and `spark-rbac.yaml` from the `olake-ui` GitHub repository over HTTPS at startup. Local overrides for those files are not supported at this time. + +This command will: +- Download the latest docker-compose.yml file +- Pull all required Docker images +- Start all services in the background +- Create a default admin user automatically + +### Access the Application + +- **OLake UI**: [http://localhost:8000](http://localhost:8000) + +### Login + +The default credentials are: +- **Username**: `admin` +- **Password**: `password` + +![Login page with fields for admin username and password](/img/docs/jobs/olake-job-0.webp) +
    + OLake UI Jobs +
    + +For instructions on setting up a table maintenance, see [Configure First Compaction](/docs/fusion/getting-started/configure-first-compaction/). + +## Service Configuration + +### Changing Admin Credentials + +The default admin user can be customized by editing the `docker-compose.yml` file before starting: + +```yaml +x-signup-defaults: + username: &defaultUsername "your-username" + password: &defaultPassword "your-secure-password" + email: &defaultEmail "your-email@example.com" +``` + +### Updating OLake UI Version + +To update OLake UI (Ingestion + Maintenance Stack) to the latest version, use the following commands based on your setup: + +```bash +docker ps -aq --filter name=fusion-cluster \ +| xargs -r docker rm -f; \ +curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose-v1.yml \ +| docker compose --profile fusion -f - down && \ +curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose-v1.yml \ +| ENABLE_OPTIMIZATION="true" docker compose --profile fusion -f - up -d +``` + +### Encryption Key Configuration + +OLake supports encryption of source and destination configurations stored in the database. Configure the encryption key in `docker-compose.yml`: + +- **Custom String**: Provide any string (OLake generates SHA-256 hash): + + ```yaml + x-encryption: + key: &encryptionKey "your-passphrase" + ``` + +- **AWS KMS**: Use a AWS KMS key ARN (recommended for production): + + ```yaml + x-encryption: + key: &encryptionKey "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012" + ``` + +- **Disable Encryption**: Use empty string: + + ```yaml + x-encryption: + key: &encryptionKey "" # No Encryption + ``` + +### Customizing Data Directory + +The default data directory can be changed by modifying the host persistence path: + +```yaml +x-app-defaults: + host_persistence_path: &hostPersistencePath /custom/path/to/olake-data + worker_config_volume_details: &workerConfigVolumeDetails + type: bind + source: *hostPersistencePath + target: /tmp/olake-config +``` + +This will create and use `/custom/path/to/olake-data` instead of the default `./olake-data` directory. + +### External PostgreSQL Configuration + +OLake supports using an **external PostgreSQL instance** instead of the built-in Postgres service included in the docker-compose file. This PostgreSQL stores all job data, configurations, sync state, and Temporal workflow data. + +Within the [compose file](https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose-v1.yml), an extension field named `x-db-envs` defines all database configuration values used by OLake and Temporal. These values can be modified to point to the external PostgreSQL instance. + +```yaml +x-db-envs: + DB_HOST: &DBHost postgresql + DB_PORT: &DBPort 5432 + DB_USER: &DBUser temporal + DB_PASSWORD: &DBPassword temporal + DB_SSLMODE: &DBSSLMode disable + OLAKE_DB_NAME: &olakeDBName postgres + TEMPORAL_DB_NAME: &temporalDBName temporal +``` + +If connection to external PostgreSQL instance is over TLS, the following variables need to be uncommented under `services.temporal.env` section in the compose file: + +```yaml + # for TLS enabled external postgres database + SQL_TLS: true + SQL_TLS_DISABLE_HOST_VERIFICATION: true + SQL_TLS_ENABLED: true + SQL_HOST_VERIFICATION: false +``` + +:::info **Optional** +A separate PostgreSQL container (`postgresql` service) is included by default. +When an external database is used, this service is not needed and can be stopped after all the services are started and healthy. +::: + +### Service Environment Variables + +Key environment variables that can be customized within `x-envs` section. These variables are automatically injected into the `olake-ui` and `olake-worker` containers: + +```yaml +x-envs: + shared: &sharedEnvs + CONTAINER_REGISTRY_BASE: ${CONTAINER_REGISTRY_BASE:-registry-1.docker.io} + OLAKE_SECRET_KEY: *encryptionKey + PERSISTENT_DIR: *hostPersistencePath + FOO: bar + KEY: value + NAME: example +``` + +### Data Persistence + +The Docker Compose setup creates the following data storage: + +#### OLake Config Directory +- `olake-data`: Local directory created in the current working directory + - Contains streams configurations, connection settings, and sync state + - Persists across container restarts and recreations + - Used by OLake UI and Temporal Worker services + +#### Docker Volumes +- `temporal-postgresql-data`: PostgreSQL database storage + - Contains workflow execution history, job metadata and sync state + - Used by the PostgreSQL service +- `temporal-elasticsearch-data` (Legacy only): Elasticsearch search index storage + - Used by the Elasticsearch service in the legacy configuration + +### Log Retention +OLake includes an automated log retention system that helps manage disk space by automatically cleaning up old log files. This prevents log files from accumulating indefinitely and consuming excessive storage space. + +#### What It Does +- Runs daily at midnight (00:00 server's local timezone) using a cron job +- Deletes entire log directories that are older than the configured retention period (defaults to 30 days) + +#### Configuration +Set the `LOG_RETENTION_PERIOD` environment variable in the docker-compose of [olake-ui](https://github.com/datazip-inc/olake-ui) to control how long logs are kept: + +```yml +# In docker-compose.yml +services: + temporal-worker: + environment: + LOG_RETENTION_PERIOD: "30" # Keep logs for 30 days +``` + +#### Monitoring Log Cleaner +The log cleanup process can be monitored through the logs of the `temporal-worker` service defined in the docker-compose configuration for [olake-ui](https://github.com/datazip-inc/olake-ui). + +When the log cleaner starts, it emits a log entry indicating the beginning of the process: +``` +Log cleaner started... +``` + +Every time the cron job runs, it emits the following log: +``` +Running log cleaner... +``` + +As each log directory is deleted, a log entry is generated showing the full path of the directory being removed: +``` +Deleting folder /path/to/olake/logs +``` + +### Temporal Retention Period + +OLake includes a configurable retention period for Temporal visibility data that controls how long job and sync history are stored and visible in the system. This helps manage database storage by automatically cleaning up old workflow execution history. + +#### Configuration +Set the `TEMPORAL_RETENTION_PERIOD` environment variable in the `temporal-worker` service within the docker-compose configuration for [olake-ui](https://github.com/datazip-inc/olake-ui) to control how long visibility data is retained: + +```yaml +# In docker-compose.yml +services: + temporal-worker: + environment: + TEMPORAL_RETENTION_PERIOD: "168h" #Keeps data visible for 7 days (by default) +``` + +**Configuration Constraints:** +- **Minimum value**: 1 day +- **Default value**: 7 days + +The retention period is specified in **hours**. Only job and sync history within the configured retention period will be visible in the OLake UI. + +## Troubleshooting + +### Common Issues + +#### Port Conflicts +If port binding errors occur: +1. Check what's using the ports: `lsof -i :8000` (on macOS/Linux) +2. Stop conflicting services or change ports in `docker-compose.yml` + +#### Database Connection Issues +- Ensure PostgreSQL container is healthy: `docker compose ps` +- Check PostgreSQL logs: `docker compose logs postgresql` + +#### Memory Issues +- Ensure Docker has at least 4GB RAM allocated +- Check Docker resource usage: `docker stats` + +#### Permission Issues +- On Linux, ensure the user is in the docker group +- Check file permissions for the `./olake-data/` directory (or custom path if modified) + +### Reset Everything + +The OLake stack can be completely reset with: + +```bash +# Removes containers and volumes +ENABLE_OPTIMIZATION="true" docker compose --profile fusion -f docker-compose-v1.yml down -v +# Removes the Kind cluster +docker ps -aq --filter name=fusion-cluster | xargs -r docker rm -f +# Fresh start +ENABLE_OPTIMIZATION="true" docker compose --profile fusion -f docker-compose-v1.yml up -d +``` + +**Warning**: This will delete all job data and configurations. + + + + diff --git a/docs/fusion/install/olake-ui/offline-environments-aws.mdx b/docs/fusion/install/olake-ui/offline-environments-aws.mdx new file mode 100644 index 000000000..e5bc17c7b --- /dev/null +++ b/docs/fusion/install/olake-ui/offline-environments-aws.mdx @@ -0,0 +1,227 @@ +--- +title: Offline Environments (AWS) +description: OLake UI installation in offline and air-gapped network environments on AWS +sidebar_position: 1 +--- + +# OLake UI for Offline Environments (AWS) + +OLake UI can be deployed in offline or air‑gapped AWS environments by using an Amazon ECR pull‑through cache to mirror required Docker images. This guide outlines how to configure the pull-through cache, pre‑pull connector images, and run OLake UI. + +ECR pull-through cache is an AWS service that automatically mirrors and caches Docker images from external registries like Docker Hub into your private ECR registry for offline access. + +## Prerequisites + +The following are required to begin: + +- An active **AWS account** with `Administrator Access` IAM permissions +- A **Docker Hub account** with permission to generate `Personal Access Token` +- **Docker installed** and configured on the machine where OLake UI is set up +- The **AWS CLI installed** and configured on the machine where OLake UI is set up + +## 1. Docker Hub Access Token + +To authenticate ECR Pull Through Cache with Docker Hub, a Personal Access Token (PAT) must be created. This token will be used by AWS to pull images. + +1. Log in to the Docker Hub account +2. Navigate to **Account Settings > Personal access tokens** +3. Provide a description for the token (e.g., "Access Token for ECR pull-through cache") +4. Set expiration date to **None** +5. Set the access permissions. For this use case, **Public Repo Read-only** access is sufficient +6. Click **Generate** + +
    + +![Docker personal access token setup instructions, including login command and token for CLI authentication](/img/docs/install/docker-hub-access-token.webp) + +
    + +:::warning Important +Copy the generated token and store it in a secure location. The token will not be visible again after the window is closed. +::: + +For more detailed instructions, refer to the [official Docker documentation on creating access tokens](https://docs.docker.com/docker-hub/access-tokens/). + +## 2. Store Docker Hub Credentials in AWS Secrets Manager + +Next, the Docker Hub credentials must be securely stored in AWS Secrets Manager. This allows ECR to authenticate with Docker Hub without exposing credentials in code or configuration files. + +1. Open the AWS Management Console and navigate to **Secrets Manager** +2. Click **Store a new secret** +3. For the **Secret type**, select **Other type of secret** +4. In the **Key/value pairs** section, create two key-value pairs: + - **Key**: `username`, **Value**: Your Docker Hub username + - **Key**: `accessToken`, **Value**: The Docker Hub Personal Access Token created in the previous step +5. For the **Secret name**, enter a descriptive name with the prefix `ecr-pullthroughcache/`. For example: `ecr-pullthroughcache/dockerhub-credentials` +6. Skip to the **Review** step and leave other values as default +7. Click **Store** to save the secret + +
    + +![AWS Secrets Manager page displaying DockerHub credentials with fields for username and access token, including secret metadata and permissions options](/img/docs/install/aws-secrets-manager.webp) + +
    + +## 3. Create the ECR Pull-Through Cache Rule + +Now, the pull-through cache rule can be created in ECR. This rule instructs ECR to cache images from Docker Hub whenever they are pulled through the private registry. + +1. In the AWS Management Console, navigate to **Elastic Container Registry (ECR)** +2. In the left-hand menu, under **Private registry** click on **Features and Settings** to expand, select **Pull through cache** +3. Click **Add rule** +4. For the **Upstream registry**, select **Docker Hub** (note that `registry-1.docker.io` is the official Docker Hub registry by default) +5. For **Authentication**, select **Use an existing AWS secret** and choose the secret created in Step 2 +6. For the **Cache repository prefix**, enter a prefix that will be used to create new repositories for the cached images (e.g., `dockerhub`) +7. For the **Upstream namespace**, choose **No Prefix** +8. Click **Create** + +
    + +![DockerHub ECR pull-through cache rule detail and error activity log are shown, with no recent error events found](/img/docs/install/ecr-pull-through-cache.webp) + +
    + +## 4. Configure IAM Permissions + +With the ECR Pull-through cache rule created, the necessary IAM permissions can be configured. An IAM role with the correct policy must be attached to the machine where OLake UI will be run. + +The policy should include the following permissions. Ensure the resource ARN is updated with the correct `region`, `account ID`, and the `ECR repository prefix` created in the previous section: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "ECRLogin", + "Effect": "Allow", + "Action": "ecr:GetAuthorizationToken", + "Resource": "*" + }, + { + "Sid": "PullFromDockerHubWithPrefix", + "Effect": "Allow", + "Action": [ + "ecr:CreatePullThroughCacheRule", + "ecr:CreateRepository", + "ecr:GetDownloadUrlForLayer", + "ecr:GetAuthorizationToken", + "ecr:BatchImportUpstreamImage", + "ecr:BatchGetImage", + "ecr:GetImageCopyStatus", + "ecr:InitiateLayerUpload", + "ecr:UploadLayerPart", + "ecr:CompleteLayerUpload", + "ecr:PutImage", + "ecr:ListImages", + "ecr:DescribeImages" + ], + "Resource": [ + "arn:aws:ecr:::repository/", + "arn:aws:ecr:::repository//*" + ] + } + ] +} +``` + +## 5. Configure VPC Endpoints for Offline Environments + +For a truly isolated environment, VPC endpoints need to be configured. This allows instances to communicate with AWS services without traversing the public internet. + +The following VPC endpoints need to be created: + +- `com.amazonaws..ecr.api` +- `com.amazonaws..ecr.dkr` +- `com.amazonaws..s3` (ECR uses S3 to store image layers) + +For detailed instructions on creating VPC endpoints, please refer to the [AWS documentation](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints.html). + +
    + +![AWS VPC Endpoints dashboard showing a filtered list of endpoint names, types, statuses, and service names.](/img/docs/install/vpc-endpoints.webp) + +
    + +## 6. Pre-pull Connector Images + +The OLake UI spins up separate Docker containers for different data sources (connectors). These connector images must also be pulled through the ECR pull-through cache before starting the main application stack. + +Run the following commands on the machine where Docker Compose will be run. These commands will pull the necessary connector images and ensure they are cached in the private ECR. Replace `` with the specific connector version required. Only **use stable release versions** (e.g., `v0.1.8`). + +```bash +# Docker login for AWS ECR repository +aws ecr get-login-password --region | docker login --username AWS --password-stdin .dkr.ecr..amazonaws.com + +# MySQL Connector +docker pull .dkr.ecr..amazonaws.com//olakego/source-mysql: + +# PostgreSQL Connector +docker pull .dkr.ecr..amazonaws.com//olakego/source-postgres: + +# MongoDB Connector +docker pull .dkr.ecr..amazonaws.com//olakego/source-mongodb: + +# Oracle DB Connector +docker pull .dkr.ecr..amazonaws.com//olakego/source-oracle: +``` + +:::info Note +This pre-pull step is a one-time action for each connector version. Once an image is pulled, it is cached within the private ECR. +::: + +## 7. Run the Application Stack + +With the pull-through cache configured and connector images pre-pulled, the main OLake application can be started. The OLake docker-compose-v1.yml is designed to use an environment variable to specify the container registry. This makes it easy to switch from Docker Hub to a private ECR. + +### Configure the Environment + +In the same directory where the OLake [docker-compose-v1.yml](https://raw.githubusercontent.com/datazip-inc/olake-ui/refs/heads/master/docker-compose-v1.yml) file is located, a new file named `.env` must be created. + +The `.env` file should contain the following line: + +```bash +CONTAINER_REGISTRY_BASE=".dkr.ecr..amazonaws.com/" + +# Example: CONTAINER_REGISTRY_BASE="111222333444.dkr.ecr.us-east-1.amazonaws.com/dockerhub" +``` + +Replace `` and `` with the appropriate AWS account ID and region. The `` must be replaced with the value created earlier. + +The docker-compose-v1.yml file for OLake is already configured to use this `CONTAINER_REGISTRY_BASE` variable for all service images. No modifications to the docker-compose-v1.yml file itself are necessary. + +### Start OLake UI + +With the environment configured, start the OLake UI stack: + +```bash +# Start the application stack +docker-compose -f docker-compose-v1.yml up -d +``` + +## Access the OLake UI + +The OLake UI will be available at: + +- **URL**: [http://localhost:8000](http://localhost:8000) +- **Username**: `admin` +- **Password**: `password` + +## Troubleshooting + +### Common Issues + +**ECR Authentication Failures:** + +- Ensure the IAM role has the correct ECR permissions + +**Image Pull Failures:** + +- Confirm the pull-through cache rule is correctly configured +- Verify the Docker Hub credentials in Secrets Manager +- Ensure VPC endpoints are properly set up for offline environments + +**Service Startup Issues:** + +- Check that all required images have been pre-pulled +- Verify the `.env` file contains the correct registry configuration +- Review Docker Compose logs: `docker-compose logs` diff --git a/docs/fusion/install/olake-ui/offline-environments-generic.mdx b/docs/fusion/install/olake-ui/offline-environments-generic.mdx new file mode 100644 index 000000000..2de9c1275 --- /dev/null +++ b/docs/fusion/install/olake-ui/offline-environments-generic.mdx @@ -0,0 +1,190 @@ +--- +title: Offline Environments (Generic) +description: Procedure for updating OLake Stack components and connector images in generic offline environments +sidebar_position: 2 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# OLake UI for Offline Environments (Generic) + +:::info +This guide currently covers the **update procedure** for existing offline installations. An **installation section** for generic offline environments will be added to this document soon. +::: + +This document outlines the procedure for **updating** the OLake Stack components and connector images in offline environments (air-gapped setups) hosted on cloud providers or on-premise infrastructure. + +## Prerequisites + +The following requirements must be satisfied prior to initiating the update process: + +- **Local Machine (or Laptop)**: A local machine (or Laptop) with active internet access and a functional Docker installation is required. +- **SSH Access**: SSH connectivity from the local machine to the Offline Node (On-prem/Cloud) must be established. +- **Offline Node Setup**: Docker and Docker Compose must be installed and operational on the Offline Node. +- **Architecture Compatibility**: The CPU architecture (ARM or x86) of the local machine must match that of the Offline Node. +- **Operational Stack**: An OLake Docker Compose stack is assumed to be currently running on the Offline Node (On-prem/Cloud). + +## 1. Image Acquisition (Local Machine) + +The required Docker images are retrieved on the internet-connected local machine. + +### Pulling Images + +The latest versions of the OLake core components and the specific versions of the required connectors are pulled from the public registry. + +```bash +# Pull Core Components +docker pull registry-1.docker.io/olakego/ui:latest +docker pull registry-1.docker.io/olakego/ui-worker:latest + +# Pull Connector Images +docker pull olakego/source-mysql: +docker pull olakego/source-postgres: +docker pull olakego/source-mongodb: +docker pull olakego/source-oracle: +docker pull olakego/source-kafka: +``` + +:::warning +Replace `` in the commands above with the specific tag number required for the update (e.g., `v0.3.4`). +::: + +
    + +Docker image pull + +
    + +### Creating the Archive + +Once the images are present locally, they are to be exported into separate tarballs. + +```bash +# Core Components +docker save -o olake-ui.tar registry-1.docker.io/olakego/ui:latest +docker save -o olake-ui-worker.tar registry-1.docker.io/olakego/ui-worker:latest + +# Connector Images +docker save -o source-mysql_.tar olakego/source-mysql: +docker save -o source-postgres_.tar olakego/source-postgres: +docker save -o source-mongodb_.tar olakego/source-mongodb: +docker save -o source-oracle_.tar olakego/source-oracle: +docker save -o source-kafka_.tar olakego/source-kafka: +``` + +
    + +Docker image save + +
    + +## 2. Transfer to Offline Node + +The generated archive files are transferred to the Offline Node using the Secure Copy Protocol (SCP). + +```bash +# Transfer Core Components +# Syntax: scp .tar @: +scp -i .pem olake-ui.tar user@offline-node-ip:/home/user/olake/ +scp -i .pem olake-ui-worker.tar user@offline-node-ip:/home/user/olake/ + +# Transfer Connector Images +scp -i .pem source-mysql_.tar user@offline-node-ip:/home/user/olake/ +scp -i .pem source-postgres_.tar user@offline-node-ip:/home/user/olake/ +scp -i .pem source-mongodb_.tar user@offline-node-ip:/home/user/olake/ +scp -i .pem source-oracle_.tar user@offline-node-ip:/home/user/olake/ +scp -i .pem source-kafka_.tar user@offline-node-ip:/home/user/olake/ + +# To transfer all generated tarballs at once +# Syntax: scp *.tar @: +scp -i .pem *.tar user@offline-node-ip:/home/user/olake/ +``` + +
    + +Docker image scp + +
    + +## 3. Deployment (Offline Node) + +The following operations are performed on the Offline Node to ingest the new images and apply the configuration updates. + +### Loading Images + +The Docker images are imported from the transferred archives into the Offline Node's local Docker registry. + +```bash +# Load Core Components +docker load -i olake-ui.tar +docker load -i olake-ui-worker.tar + +# Load Connector Images +docker load -i source-mysql_.tar +docker load -i source-postgres_.tar +docker load -i source-mongodb_.tar +docker load -i source-oracle_.tar +docker load -i source-kafka_.tar +``` + +
    + +Docker image load + +
    + +
    + +Docker image list + +
    + +### Applying Service Updates + +If the core components (`olake-ui` and `olake-worker`) were included in the update, the Docker Compose stack must be refreshed to utilize the new `latest` images. + + + + ```bash + curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose-v1.yml | docker compose -f - down && \ + curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose-v1.yml | docker compose -f - up -d + ``` + + + ```bash + curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose.yml | docker compose -f - down && \ + curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose.yml | docker compose -f - up -d + ``` + + + +:::note +If only connector images (`source-mysql`, `source-postgres`, etc.) were updated, the step above (starting and stopping the containers) is not required. The `olake-worker` service automatically scans the local registry and will display the newly loaded connector versions in the Source Create/Edit page immediately. +::: + +## Troubleshooting + +- **No space left on device**: Docker images can be significant in size. Ensure the Offline Node has sufficient disk space to hold both the compressed tarballs and the uncompressed images. +- **Connectors not appearing in UI**: If updated connectors do not appear in the dropdown list, perform a hard refresh of the browser page. You can verify the images were loaded correctly by running `docker image ls | grep 'olakego/source-'` on the Offline Node. +- **"archive/tar: invalid header"**: This error during `docker load` usually indicates a corrupted tarball. Verify the file size on the local and offline machines matches, and transfer the file again if necessary. diff --git a/docs/fusion/maintenance/catalogs.mdx b/docs/fusion/maintenance/catalogs.mdx new file mode 100644 index 000000000..dd62dd7a6 --- /dev/null +++ b/docs/fusion/maintenance/catalogs.mdx @@ -0,0 +1,80 @@ +--- +title: Iceberg Maintenance Catalogs +sidebar_label: Catalogs +--- + +# What are Catalogs? + +A **catalog** is a entity from where OLake Fusion fetches Iceberg tables. It stores reference to the latest metadata file which stores info such as table names, schemas, and file locations. OLake Fusion must know which catalog (and database) a table belongs to before maintenance can run on it. + +Catalogs are managed from the **Maintenance** section in the OLake UI. There are two types: **OLake Imported Catalogs** and **External Catalogs**. + +## OLake Imported Catalogs + +**OLake Imported Catalogs** are catalogs where data was ingested using OLake Go. Since the catalog connection details are already configured in OLake Go, they can be directly imported here without entering the credentials again. + +### How to add an OLake Imported Catalog + +1. In the OLake UI sidebar, open the **Maintenance** dropdown and go to the **Catalogs** page. +2. Click **New Catalog**. +![Catalogs page](/img/docs/iceberg-maintenance/catalogs/add-catalog.webp) + +3. Under **Import Catalog from destination**, open the dropdown and select the destination. OLake Fusion imports all catalog credentials from that destination. +![Catalog dropdown](/img/docs/iceberg-maintenance/catalogs/catalog-import-dropdown.webp) + + :::note JDBC Catalog — Catalog Name Must Match + For JDBC catalogs, the catalog name must match the one used during Ingestion. If no catalog name was provided during ingestion, the default is `olake_iceberg`. + ::: + +4. Click **Connect** to save and validate the catalog. +![Save Catalog](/img/docs/iceberg-maintenance/catalogs/connect-imported-catalog.webp) + +> **Note:** Duplicate catalog names are not allowed. If a catalog with the same name already exists, the new catalog will not be added. + +
    +On the Catalogs page, OLake Imported Catalogs are shown with **OLake** beside their name to distinguish them from External Catalogs. + +![OLake Imported Catalog with OLake label on Catalogs page](pathname:///img/docs/iceberg-maintenance/catalogs/olake-imported-catalogs.webp) + +
    + +## External Catalogs + +**External Catalogs** are catalogs added by entering connection details manually from the **Catalogs** page (without using **Import Catalog from destination**). + +Use an external catalog when: + +- Maintenance is needed on Iceberg tables created outside OLake. +- The same catalog is used by other systems and OLake Fusion should compact those tables without replicating data via Ingestion. + +### How to add an External Catalog + +1. In the OLake UI sidebar, open the **Maintenance** dropdown and go to the **Catalogs** page. +2. Click **New Catalog**. +3. Enter the catalog details manually (for example, catalog name, type, and connection settings). Do **not** use **Import Catalog from destination**. + +![Add catalog view](/img/docs/iceberg-maintenance/catalogs/add-catalog.webp) + +4. Click **Connect** to save and validate the catalog. + +![Catalog connected view](/img/docs/iceberg-maintenance/catalogs/connect-catalog.webp) + +Once a catalog is connected—whether imported or added as external—it appears in the **Select Catalog** dropdown on the **Tables** page. Selecting a catalog (and then a database) lists the tables available for maintenance configuration. + +## Compatibility to Iceberg Catalogs + +OLake Fusion supports multiple Iceberg catalog implementations, including REST catalog, Hive Metastore, and JDBC Catalog, letting you choose the one that best fits your environment. The table below shows the supported catalogs at a glance, with links to their setup guides. + +| | Catalog | +| ----------------------------------------------------------------------------------------- | ------------------- | +| AWS Glue logo | **AWS Glue** | +| | **REST (Generic)** | +| Nessie logo | **REST Nessie** | +| Polaris logo | **REST Polaris** | +| Unity Catalog logo | **REST Unity** | +| Lakekeeper logo | **REST Lakekeeper** | +| Amazon S3 logo | **S3 Tables** | +| JDBC logo | **JDBC** | +| Apache Hive logo | **Hive Metastore** | + +For a full walkthrough that includes adding a catalog and configuring the first table maintenance, see [Configure Your First Table Maintenance](/docs/fusion/getting-started/configure-first-compaction/). diff --git a/docs/iceberg-maintenance/metrics.mdx b/docs/fusion/maintenance/metrics.mdx similarity index 87% rename from docs/iceberg-maintenance/metrics.mdx rename to docs/fusion/maintenance/metrics.mdx index 1192bed90..b78c26d76 100644 --- a/docs/iceberg-maintenance/metrics.mdx +++ b/docs/fusion/maintenance/metrics.mdx @@ -5,12 +5,12 @@ sidebar_label: Metrics # Overview -The Metrics view helps you understand both the **current health of an Iceberg table** and the **impact of optimization runs**. +The Metrics view helps you understand both the **current health of an Iceberg table** and the **impact of compaction runs**. There are two types of metrics: - **Table Metrics** – always available for a table, showing its current state. -- **Run Metrics** – measurements for a single optimization run: input vs output data and delete files (counts and sizes) for that run. +- **Run Metrics** – measurements for a single compaction run: input vs output data and delete files (counts and sizes) for that run. ## Table Metrics @@ -41,7 +41,7 @@ These metrics describe the current layout and size of the Iceberg table: ## Run Metrics -Run Metrics describe what one optimization run did to the table: how many data and delete files went in, and how many data and delete files came out. Open them from the **Runs** page via the **View Metrics** action for the run you want to inspect. +Run Metrics describe what one compaction run did to the table: how many data and delete files went in, and how many data and delete files came out. Open them from the **Runs** page via the **View Metrics** action for the run you want to inspect. ![Run metrics view](pathname:///img/docs/iceberg-maintenance/metrics/view-run-metrics-button.webp) @@ -72,6 +72,6 @@ All **size** fields in **Table metrics** and **Run metrics** are shown in **byte ## Interpreting Metrics - A high **Delete Files Count** or very low **Average File Size** can indicate fragmentation and metadata overhead. -- After a successful optimization run, compare **Input** vs **Output**. +- After a successful compaction run, compare **Input** vs **Output**. - A smaller **Output delete size** (and/or fewer **Output** delete files) typically indicates the run reduced delete-file fragmentation. - Changes in **Input**/**Output** data file counts and data size show how much data was rewritten/compacted for that run’s scope. diff --git a/docs/iceberg-maintenance/runs-and-logs.mdx b/docs/fusion/maintenance/runs-and-logs.mdx similarity index 70% rename from docs/iceberg-maintenance/runs-and-logs.mdx rename to docs/fusion/maintenance/runs-and-logs.mdx index 2a3518880..5dd44f59b 100644 --- a/docs/iceberg-maintenance/runs-and-logs.mdx +++ b/docs/fusion/maintenance/runs-and-logs.mdx @@ -5,18 +5,18 @@ sidebar_label: Logs & Runs # What is a Run? -A **run** is a single execution of an optimization (Lite, Medium, or Full) on a specific Iceberg table. +A **run** is a single execution of a compaction (Lite, Medium, or Full) on a specific Iceberg table. -Each time you start an optimization for a table, OLake creates a new run entry. This entry tracks: +Each time you start a compaction for a table, OLake creates a new run entry. This entry tracks: -- The **table** being optimized -- The **optimization type** (Lite, Medium, or Full) +- The **table** being compacted +- The **compaction type** (Lite, Medium, or Full) - The **current status** of the run - The **start time and duration** - The **logs** for that execution - The **Metrics** of that execution -Runs help you understand what has been optimized, when it happened, and whether it completed successfully. +Runs help you understand what has been compacted, when it happened, and whether it completed successfully. ## Run Lifecycle @@ -24,11 +24,11 @@ Every run goes through a few simple states from start to finish: | State | Description | |----------|-----------------------------------------------------------------------------| -| Running | The optimization run is currently in progress for the table. | -| Success | The optimization run finished successfully without errors. | -| Failed | The optimization run stopped due to an error. Check the logs for details. | -| Cancelled| The optimization run was manually stopped by the user. | -| Skipped | The optimization run was skipped. +| Running | The compaction run is currently in progress for the table. | +| Success | The compaction run finished successfully without errors. | +| Failed | The compaction run stopped due to an error. Check the logs for details. | +| Cancelled| The compaction run was manually stopped by the user. | +| Skipped | The compaction run was skipped. ## Viewing Runs @@ -42,7 +42,7 @@ You can see all runs for a table from the **Tables** page: This opens the **Runs** view for that table. The Runs view typically includes: -- The **optimization type** (Lite, Medium, or Full) +- The **compaction type** (Lite, Medium, or Full) - The **current status** of each run - **Start time** and **duration** - A **Logs** column so you can inspect details for each run @@ -61,7 +61,7 @@ From the **Runs** view, you can open detailed logs for a specific run: This opens the **Logs** page for that run. The Logs page is split into: -- **Driver Logs** – high-level logs for the overall optimization run. +- **Driver Logs** – high-level logs for the overall compaction run. - **Sub Tasks Logs** – detailed logs for each sub-task that the run is broken into. The **Driver Logs** section gives you an overview of how the run progressed from start to finish. diff --git a/docs/fusion/release/maintenance/overview.mdx b/docs/fusion/release/maintenance/overview.mdx new file mode 100644 index 000000000..edc81694f --- /dev/null +++ b/docs/fusion/release/maintenance/overview.mdx @@ -0,0 +1,22 @@ +--- +title: "OLake Maintenance Release Overview & Updates | Features & Bug Fixes" +description: Explore OLake Maintenance release summaries, new features, bug fixes, and major changes. Get support and engage with the community for smooth updates. +sidebar_label: Start Here +sidebar_position: 1 +--- + +# Release Notes + +Welcome to the OLake Fusion release notes archive. Each release contains a changelog of new features, enhancements, and bug fixes. Use this landing page to: +- Quickly navigate to a specific version’s release notes +- Understand when major features and improvements were introduced +- Learn about backward-incompatible changes and deprecations + +**Why a Release Landing Page?** OLake Fusion evolves rapidly with new table-maintenance capabilities, tuning improvements, and stability enhancements. Since updates vary by version, including features, enhancements, and bug fixes, this page helps you identify when key maintenance capabilities were introduced, track deprecations, and access guidance for migrating between versions. + +### Navigate Releases + +Use the Release dropdown in the documentation sidebar to select any version’s detailed notes. Each version page includes: +- 🎯 What's New: Additions or enhancements introduced in this release across table-maintenance workflows. +- 🔧 Bug Fixes & Stability: Improvements and refinements to existing functionality in this release; reliability, performance, and compatibility fixes. + diff --git a/docs/fusion/release/maintenance/v0.1.0.mdx b/docs/fusion/release/maintenance/v0.1.0.mdx new file mode 100644 index 000000000..62ea0e21d --- /dev/null +++ b/docs/fusion/release/maintenance/v0.1.0.mdx @@ -0,0 +1,24 @@ +--- +title: "OLake Fusion (v0.1.0 - v0.1.3)" +--- + +# OLake Fusion (v0.1.0 - v0.1.3) +April 10, 2026 – July 02, 2026 + +## 🎯 What's New + +### Catalogs + +1. **Test catalog connection before creation -**
    Added a catalog connection test API that will validate catalog credentials/configuration before creating the catalog. + +## 🔧 Bug Fixes & Stability + +1. **Health score fix -**
    Restored missing `tableRuntime.setTableSummary(pendingInput)` updates in `TableRuntimeRefreshExecutor` after the cron changes. Health score calculation is now scoped to applicable Iceberg tables only. + +2. **Refresh table and Fusion startup failure on catalog error -**
    Fixed failures in refresh table and Fusion startup caused by catalog errors, and resolved an issue with the releaser of the latest tag. + +3. **Glue catalog authentication fix -**
    Fixed authentication failures when using AWS Glue as a catalog. + +4. **Fail reason now visible in driver logs -**
    When an optimizing process fails, the failure reason is now appended to the per-process driver log files served by `LogController`. + +5. **Recovered Iceberg tables stuck in planning state -**
    Tables left in `PLANNING` state with status code `500` are now detected on startup and moved to `PENDING` state `600`. \ No newline at end of file diff --git a/docs/getting-started/alerts-and-notifications.mdx b/docs/getting-started/alerts-and-notifications.mdx index 4a1682832..f9b1782a1 100644 --- a/docs/getting-started/alerts-and-notifications.mdx +++ b/docs/getting-started/alerts-and-notifications.mdx @@ -6,7 +6,7 @@ sidebar_label: Alerts and Notifications # Alerts and Notifications -OLake provides system-level alerting capabilities that notify you when jobs fail. You can configure webhook alerts to receive notifications in Slack, Microsoft Teams, or any other platform that supports webhooks. +OLake Go provides system-level alerting capabilities that notify you when jobs fail. You can configure webhook alerts to receive notifications in Slack, Microsoft Teams, or any other platform that supports webhooks. :::note Version Requirements To access the alerts and notifications feature, ensure you have the following minimum versions: @@ -18,12 +18,12 @@ To access the alerts and notifications feature, ensure you have the following mi ### Obtaining Webhook URLs -Before configuring alerts in OLake, you need to obtain a webhook URL from your preferred notification platform: +Before configuring alerts in OLake Go, you need to obtain a webhook URL from your preferred notification platform: - **Slack**: Follow the [Slack Incoming Webhooks guide](https://docs.slack.dev/messaging/sending-messages-using-incoming-webhooks/) to create a webhook URL for your Slack channel - **Microsoft Teams**: Follow the [Microsoft Teams Incoming Webhooks guide](https://learn.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook?tabs=newteams%2Cdotnet#create-an-incoming-webhook) to create a webhook URL for your Teams channel -### Setting Up Webhook URL in OLake +### Setting Up Webhook URL in OLake Go 1. Navigate to the **System Settings** page from the left sidebar in OLake UI 2. Click on the **Alerts and Notifications** tab @@ -40,11 +40,11 @@ Before configuring alerts in OLake, you need to obtain a webhook URL from your p Only one webhook URL can be configured per system. You cannot use multiple channels (e.g., both Slack and Teams) simultaneously. ::: -Once you configure the webhook URL, all jobs in your OLake instance will be automatically tracked. You will receive alerts for all job failures, including both newly created jobs and previously running jobs. +Once you configure the webhook URL, all jobs in your OLake Go instance will be automatically tracked. You will receive alerts for all job failures, including both newly created jobs and previously running jobs. ## Alert Message Format -When a job fails, OLake sends a POST request to your webhook URL with a JSON payload containing: +When a job fails, OLake Go sends a POST request to your webhook URL with a JSON payload containing:
    ![Example alert notification showing sync failure with job details and error message](../../static/img/docs/getting-started/alerts-and-notifications/alert-example.png) @@ -59,6 +59,6 @@ When a job fails, OLake sends a POST request to your webhook URL with a JSON pay - **Last Run Time**: The timestamp when the job was executed :::note System-Level Configuration -Alerts and notifications are configured at the system level, meaning all jobs will send alerts to the same webhook URL. This ensures you receive notifications for all job failures across your OLake instance. +Alerts and notifications are configured at the system level, meaning all jobs will send alerts to the same webhook URL. This ensures you receive notifications for all job failures across your OLake Go instance. ::: diff --git a/docs/getting-started/creating-first-pipeline.mdx b/docs/getting-started/creating-first-pipeline.mdx index 9dcbf8f7e..5e2793d1e 100644 --- a/docs/getting-started/creating-first-pipeline.mdx +++ b/docs/getting-started/creating-first-pipeline.mdx @@ -1,15 +1,15 @@ --- title: "Create Your First Data Pipeline | OLake Getting Started Guide" description: Step-by-step tutorial to create a data replication job in OLake, configuring source, destination, streams, and scheduling for seamless sync. -sidebar_label: Create Your First Job Pipeline +sidebar_label: Configure Your First Ingestion Job sidebar_position: 3 --- -# Get Started With First Job! +# Get Started With First Ingestion Job! -This guide is for end-users who want to replicate data between the various sources and destinations that OLake supports. Using the OLake UI, you can configure a **source**, set up a **destination**, and create a **job** to move data between them. +This guide is for end-users who want to replicate data between the various sources and destinations that OLake Go supports. Using the OLake UI, you can configure a **source**, set up a **destination**, and create a **job** to move data between them. -By the end of this tutorial, you’ll have a complete replication workflow running in OLake. +By the end of this tutorial, you’ll have a complete replication workflow running in OLake Go. ## Prerequisites @@ -17,12 +17,7 @@ Follow the [Quickstart Setup Guide](/docs/getting-started/quickstart) to ensure ### What is a Job? -A job in OLake is a pipeline that defines how data should be synchronized from a **source** (where your data comes from) to a **destination** (where your data goes). - -Sources and destinations can be: - -- **New** - configured during job creation. -- **Existing** - already set up and reused across multiple jobs. +A job in OLake Go is a pipeline that defines how data should be synchronized from a **source** (where your data comes from) to a **destination** (where your data goes). ### Two ways to create a Job @@ -32,8 +27,8 @@ Start from the **Jobs** page and set up everything in one flow. 1. Go to **Jobs** in the left menu and click **Create Job**. 1. Configure **job name & schedule** -1. Configure the **source**. -1. Configure the **destination**. +1. If the **source** is already configured select the source connector and existing source. Otherwise, configure a new **source**. +1. If the **destination** is already configured select the destination connector and existing destination. Otherwise, configure a new **destination**. 1. Configure streams and save. #### 2. Resource-first workflow: @@ -47,42 +42,30 @@ Set up your source and destination first, then link them in a job. 1. Configure streams and save. :::tip -The two methods achieve the same result. Choose **Job-first** if you want a guided setup in one go. -Choose **Resource-first** if your source and destination are already configured, or if you prefer to prepare them in advance. +- The two methods achieve the same result. Choose **Job-first** if you want a guided setup in one go. Choose **Resource-first** if your source and destination are already configured, or if you prefer to prepare them in advance. +- If you don't have a source or destination set up yet but still want to try OLake, head over to the [OLake Playground](https://olake.io/docs/getting-started/playground/) for a ready to use sandbox environment. :::
    ## Tutorial: Creating a Job -In this guide, we'll use the **Job-first workflow** to set up a job from configuring the source and destination to running it. If you prefer video, check out our [video walkthrough](#video-tutorial). +In this guide, we'll use the **Resource-first workflow** to set up a job from configuring the source and destination to running it. First things first, every job needs a source and a destination before it can run. For this demonstration, we'll use [**Postgres**](/docs/connectors/postgres) as the source and [**Apache Iceberg**](/iceberg/why-iceberg) with [**Glue Catalog**](/docs/writers/iceberg/catalog/glue/) as the destination. Let's get started! -### 1. Create a New Job +### 1. Create a New Source -Navigate to **Jobs** section and select **+ Create Job** button in the top right corner. This opens the Job creation wizard, starting with the **Configure Job Name & Schedule** step. +Navigate to **Sources** section and select **+ Create Source** button in the top right corner. -
    - ![OLake jobs dashboard with the Jobs tab, Create Job button, and Create your first Job button highlighted](../../static/img/docs/getting-started/create-your-first-job/job-create.webp) -
    +![Olake create source](../../static/img/docs/getting-started/create-your-first-job/create-source.webp) -### 2. Configure Job Name & Schedule +### 2. Configure Source -Give your job a descriptive name. For this guide, set the **Frequency** dropdown to **Every Day** and choose **12:00 AM** as the **Time**. - -
    - ![OLake Create Job page showing step 1, with job name, frequency dropdown (Every Day highlighted), and job start time settings](../../static/img/docs/getting-started/create-your-first-job/job-schedule.webp) -
    - -### 3. Configure Source - -Since we're following the **Job-first workflow**, select the **Set up a new source** option. - -For this guide, choose **Postgres** from the connector dropdown, and keep the **OLake version** set to the latest stable version. +For this guide, choose **Postgres** from the connector dropdown, and keep the **OLake Go version** set to the latest stable version.
    ![Job source @@ -96,7 +79,7 @@ Give your source a descriptive name, then fill in the required Postgres connecti creation](../../static/img/docs/getting-started/create-your-first-job/job-source-config.webp)
    -Once the test connection succeeds, OLake shows a success message and takes you to the destination configuration step. +Once the test connection succeeds, OLake Go shows a success message, and by clicking on destinations button it takes you to the destination configuration step. You can find the configuration and troubleshooting guides for all supported source connectors below. @@ -106,19 +89,29 @@ You can find the configuration and troubleshooting guides for all supported sour | Postgres | [Config](/docs/connectors/postgres#configuration) | | MongoDB | [Config](/docs/connectors/mongodb#configuration) | | Oracle | [Config](/docs/connectors/oracle#configuration) | +| MSSQL | [Config](/docs/connectors/mssql#configuration) | +| Kafka | [Config](/docs/connectors/kafka#configuration) | +| DB2 LUW | [Config](/docs/connectors/db2#configuration) | +| S3 | [Config](/docs/connectors/s3#configuration) | :::note If you plan to enable CDC (Change Data Capture), make sure a replication slot already exists on your Postgres database. You can learn how to check or create one in our [Replication Slot Guide](/docs/connectors/postgres/setup/generic). ::: +### 3. Create a New Destination + +Navigate to **Destination** section and select **+ Create Destination** button in the top right corner. + +![Olake create destination](../../static/img/docs/getting-started/create-your-first-job/create-dest.webp) + ### 4. Configure Destination Similarly, here we'll be using **Iceberg** with **AWS Glue Catalog** as the destination. -For this guide, select **Apache Iceberg** from the connector dropdown, and keep the **OLake version** set to the latest stable version. +For this guide, select **Apache Iceberg** from the connector dropdown, and keep the **OLake Go version** set to the latest stable version. -
    +
    ![Job destination creation](../../static/img/docs/getting-started/create-your-first-job/job-dest-connector.webp)
    @@ -127,17 +120,17 @@ Choose the catalog as **AWS Glue** from the Catalog Type dropdown.
    ![Job destination - catalog](../../static/img/docs/getting-started/create-your-first-job/configure_dest.png) + catalog](../../static/img/docs/getting-started/create-your-first-job/job-dest-catalog.webp)
    Give your destination a descriptive name, then fill in the required connection details in the Endpoint Config form.
    ![Job destination - config](../../static/img/docs/getting-started/create-your-first-job/dest_config.png) + config](../../static/img/docs/getting-started/create-your-first-job/job-dest-config.webp)
    -Once the test connection succeeds, OLake shows a success message and takes you to the streams configuration step. +Once the test connection succeeds, OLake Go shows a success message and by clicking on create job button it takes you to the job configuration step. You can find the configuration and troubleshooting guides for all supported destination connectors below. @@ -153,16 +146,32 @@ You can find the configuration and troubleshooting guides for all supported dest | Hive Catalog | [Config](/docs/writers/iceberg/catalog/hive#configuration) | | JDBC Catalog | [Config](/docs/writers/iceberg/catalog/jdbc#configuration) | | REST Catalog | [Config](/docs/writers/iceberg/catalog/rest?rest-catalog=generic#configuration) | - | Nessie Catalog | [Config](/docs/writers/iceberg/catalog/rest?rest-catalog=nessie#configuration) | - | LakeKeeper | [Config](/docs/writers/iceberg/catalog/rest?rest-catalog=lakekeeper#configuration) | - | S3 Tables | [Config](/docs/writers/iceberg/catalog/rest?rest-catalog=s3-tables#configuration) | - | Polaris | [Config](/docs/writers/iceberg/catalog/rest?rest-catalog=polaris#configuration) | - | Unity | [Config](/docs/writers/iceberg/catalog/rest?rest-catalog=unity#configuration) | + | Nessie Catalog | [Config](/docs/writers/iceberg/catalog/rest?rest-catalog=nessie#configuration-2) | + | LakeKeeper | [Config](/docs/writers/iceberg/catalog/rest?rest-catalog=lakekeeper#configuration-1) | + | S3 Tables | [Config](/docs/writers/iceberg/catalog/rest?rest-catalog=s3-tables#configuration-3) | + | Polaris | [Config](/docs/writers/iceberg/catalog/rest?rest-catalog=polaris#configuration-5) | + | Unity | [Config](/docs/writers/iceberg/catalog/rest?rest-catalog=unity#configuration-4) | + +### 5. Create a New Job + +Navigate to **Jobs** section and select **+ Create Job** button in the top right corner. + +![Olake create Job](../../static/img/docs/getting-started/create-your-first-job/create-job.webp) + +### 6. Configure Job + +Give your job a descriptive name. For this guide, set the **Frequency** dropdown to **Every Minute**. + +Next we have to select the source and destination that we created in the previous steps. First we need to select the **source connector** from the dropdown and then select the **source**. Similarly for the destination we have to select the **destination connector** from the dropdown and then select the **destination**. + +Once you have selected the source and destination, click on the **Next** button to continue. At this stage, the system validates both configurations. You can proceed to the Streams section only after both validations succeed and a success status is displayed. + +![Olake Job Configuration](../../static/img/docs/getting-started/create-your-first-job/select-source-dest.webp) -### 5. Configure Streams +### 7. Configure Streams -The **Streams** page is where you select which streams to replicate to the destination. -Here, you can choose your preferred [sync mode](/docs/understanding/terminologies/olake#2-sync-modes) and configure [partitioning](/docs/writers/parquet/partitioning) and [Destination Database](/docs/understanding/terminologies/olake#7-tablecolumn-normalization--destination-database-creation) as well as other stream-level settings here. +The **Streams** page is where you select which streams to replicate to the destination and configure stream-level properties for each selected stream. For more details, please check the [Stream Properties](/docs/understanding/terminologies/olake/#streams-properties). +Here, you can choose your preferred [sync mode](/docs/understanding/terminologies/olake#2-sync-modes) and configure [partitioning](/docs/writers/parquet/partitioning) and [Destination Database](/docs/understanding/terminologies/olake/#destination-database) as well as other stream-level settings here.
    ![OLake streams selection, employee_data and other tables checked, sync mode set to Full Refresh + CDC](../../static/img/docs/getting-started/create-your-first-job/job-streams.webp) @@ -171,7 +180,7 @@ Here, you can choose your preferred [sync mode](/docs/understanding/terminologie For this guide, we'll configure the following: - Replicate the `fivehundred` stream (name of the table). -- Use [**Full Refresh + CDC**](/docs/features/#2-sync-modes-supported) as the sync mode. +- Use [**Full Refresh + CDC**](/docs/understanding/terminologies/olake/#2-sync-modes) as the sync mode. - Enable **data Normalization**. - Modify Destination Database name (if required). - Replicate only data where `dropoff_datetime` >= `2010-01-01 00:00:00` (basically data from 2010 onward). @@ -180,7 +189,7 @@ For this guide, we'll configure the following: Let's start by selecting the `fivehundred` stream (or any stream from your source) by checking its checkbox to include it in the replication. Click the stream name to open the stream-level settings panel on the right side. -In the panel, set the **sync mode** to [**Full Refresh + CDC**](/docs/features/#2-sync-modes-supported), and enable **Normalization** by toggling the switch on. +In the panel, set the **sync mode** to [**Full Refresh + CDC**](/docs/understanding/terminologies/olake/#2-sync-modes), and enable **Normalization** by toggling the switch on.
    ![Job streams @@ -212,7 +221,7 @@ filter = "\"id-with#special!char\" = 1" ``` ::: -To edit the **Destination Database** name, select the edit icon beside the [Destination Database](/docs/understanding/terminologies/olake#7-tablecolumn-normalization--destination-database-creation) (Iceberg DB or S3 Folder) and make the changes. +To edit the **Destination Database** name, select the edit icon beside the [Destination Database](/docs/understanding/terminologies/olake/#destination-database) (Iceberg DB or S3 Folder) and make the changes.
    ![Job stream Destination @@ -232,8 +241,8 @@ The sync will start at the next scheduled time. You can also start it manually b ![OLake jobs dashboard with actions menu for sync, edit streams, pause, logs, settings, delete](../../static/img/docs/getting-started/create-your-first-job/job-sync-now.webp)
    -You can verify the sync status by checking the badge at the right end of the job row. Possible statuses include **Running**, **Failed**, and **Completed**. -You can also monitor the sync logs by selecting [**Job Logs and History**](/docs/getting-started/creating-first-pipeline#5-job-logs--history) from the job options menu. +You can verify the sync status by checking the last run status column. Possible statuses include **Running**, **Failed**, and **Completed**. +You can also monitor the sync logs by selecting [**Job Logs and History**](/docs/getting-started/job-level-properties/#8-job-logs--history) from the job options menu. - Job running: ![OLake jobs dashboard showing active job status as running for Postgres to Iceberg pipeline](../../static/img/docs/getting-started/create-your-first-job/job-running.webp) @@ -249,7 +258,7 @@ Yay! The sync is complete, and our data has been replicated to Iceberg exactly a
    -### 6. Manage Your Job +### 8. Manage Your Job Once your job is created, you can manage it from the **Jobs** page using the **Actions** menu **(⋮)** diff --git a/docs/getting-started/job-level-properties.mdx b/docs/getting-started/job-level-properties.mdx index c8f85b98b..6748f6ffd 100644 --- a/docs/getting-started/job-level-properties.mdx +++ b/docs/getting-started/job-level-properties.mdx @@ -6,7 +6,7 @@ sidebar_label: Job Level Features # Job Level Features -Once your job is created in OLake, you can manage and control it using various job-level features available through the **Actions** menu **(⋮)** on the Jobs page. These features allow you to trigger immediate syncs, modify stream configurations, pause or cancel running jobs, reset destination data, monitor sync history, and adjust job settings. This guide covers all the job-level operations you can perform to manage your data pipelines effectively. +Once your job is created in OLake Go, you can manage and control it using various job-level features available through the **Actions** menu **(⋮)** on the Jobs page. These features allow you to trigger immediate syncs, modify stream configurations, pause or cancel running jobs, reset destination data, monitor sync history, and adjust job settings. This guide covers all the job-level operations you can perform to manage your data pipelines effectively. ### 1. Max Parallel Threads for Discovery @@ -16,43 +16,81 @@ This feature allows you to set the maximum number of parallel threads used for d ![max discover threads](../../static/img/docs/getting-started/create-your-first-job/max-discover-thread.webp)
    -### 2. Sync Now +### 2. Job Configuration + +The job configuration property refers to the options that defines job’s name, schedule, and execution in the OLake Go’s system. \ +User has to start with job creation, which will be followed with source configuration, then destination configuration, checking and enabling relevant streams from the schema for sync, and then finally in job configuration, job name and frequency has to be set. + +- **Frequency Options:** + - Default options i.e every minute, hourly, daily, weekly + - Custom frequency: Specify a cron expression. + +OLake Go supports **Custom Frequency** using cron expressions in both 5-field and 7-field formats: + +- **5-field:** `minute, hour, day_of_month, month, day_of_week` +- **7-field:** `second, minute, hour, day_of_month, month, day_of_week, year` + +:::note +OLake Go does **not** support Quartz cron syntax. +::: + +**Guide to Cron Expression (5-field format):** + +| * | * | * | * | * | +|---|---|---|---|---| +| minute (0-59) | hour (0-23) | day of the month (1-31) | month (1-12) | day of the week (0-6) | + + +**Cron Examples (5-field format):** +- `* * * * *` = Every minute +- `0 * * * *` = Every hour +- `0 0 * * *` = Every day at 12:00 AM +- `0 0 * * FRI` = At midnight only on Fridays +- `0 0 1 * *` = At midnight on the 1st day of each month + +**Guide to Cron Expression (7-field format):** + +| * | * | * | * | * | * | * | +|---|---|---|---|---|---|---| +| second (0-59) | minute (0-59) | hour (0-23) | day of the month (1-31) | month (1-12) | day of the week (0-6) | year | + +**Cron Examples (7-field format):** +- `* * * * * * *` = Every second +- `0 * * * * * *` = Every minute +- `0 0 * * * * *` = Every hour +- `0 0 0 * * * *` = Every day (day_of_month) +- `0 0 0 1 * * *` = Every month (on day 1) +- `0 0 0 * * 5 *` = Every week (on Friday) +- `0 0 0 1 1 * *` = Every year (on January 1) + +
    + Olake Partition output +
    + +### 3. Sync Now Run the job immediately without waiting for the next scheduled time. -### 3. Edit Streams +### 4. Edit Streams -Use this option to modify which streams are included in your job and adjust their replication settings. -When you click **Edit Streams** you'll be redirected to the **Stream Configuration** page. +Use this option to modify which streams are included in your job. When you click **Edit Streams** you'll be redirected to the **Stream Configuration** page. Here you can: - **Add new streams** from your source. - **Change the sync mode** for selected streams. -- **Adjust partitioning** or **Normalization** for newly added streams. -- You can also navigate to **Source** and **Destination** settings using the stepper at the top-right of the page. +- **Adjust partitioning**, **data filter**, and **Normalization** for newly added streams.
    ![Edit streams](../../static/img/docs/getting-started/create-your-first-job/job-edit-streams-page.webp)
    -- By default, source and destination editing is locked click **Edit** to unlock them. - -
    - ![Edit streams - destination](../../static/img/docs/getting-started/create-your-first-job/job-edit-destination.webp) -
    - -:::note -You cannot directly change the **Normalization**, **data filter**, or **partition scheme** for existing streams. To update these: - -1. Unselect the stream. -1. Save the job. -1. Reopen **Edit Streams** and re-add the stream with the updated settings - ::: - -### 4. Pause Job +### 5. Pause Job Stops the job from running until resumed. Paused jobs appear under **Inactive Jobs**. Resume them anytime from the **Inactive Jobs** tab. @@ -61,7 +99,7 @@ Stops the job from running until resumed. Paused jobs appear under **Inactive Jo
    -### 5. Cancel Job +### 6. Cancel Job Stops a currently running sync safely. Available only while a sync is in "Running" status. @@ -99,7 +137,7 @@ Whether your sync resumes from where it stopped depends on if it generates a sta After cancellation completes, you can start the job again with **Sync Now** or wait for the next scheduled run. The next run resumes from the last saved state if a state file exists, or starts fresh if not. -### 6. Clear Destination +### 7. Clear Destination This feature enables users to erase the selected streams from a specific job or sync from the destination. It removes all data that was synced for that particular job, making it ideal when stream-level settings (such as normalization, partitioning, or filters) are misconfigured. Users can clear the destination, reconfigure their job settings, and initiate a fresh sync to rebuild a clean, correct snapshot without manual file or table deletion. @@ -107,7 +145,7 @@ This feature enables users to erase the selected streams from a specific job or It can be considered a reset mechanism for a job’s destination outputs. :::info Available from v0.3.0 -The Clear Destination feature is stable and available from OLake version **v0.3.0** onwards. +The Clear Destination feature is stable and available from OLake Go version **v0.3.0** onwards. ::: @@ -125,7 +163,7 @@ To trigger Clear Destination, navigate to the **Job Settings** page and click th Clear Destination is only available for active jobs. Inactive or paused jobs cannot trigger Clear Destination until they are resumed. ::: -#### How it works in OLake +#### How it works in OLake Go - Job‑level behavior: Clear Destination affects only the specific job it is triggered for. Multiple jobs can run Clear Destination simultaneously - Stream‑aware: Only the job's enabled streams are cleared other streams remain unaffected @@ -137,7 +175,7 @@ Clear Destination is only available for active jobs. Inactive or paused jobs can :::tip Best Practice for Kafka When using Clear Destination with Kafka source, the behavior depends on whether a consumer group ID is provided: -**Scenario 1: Consumer Group ID is blank (OLake-managed)** +**Scenario 1: Consumer Group ID is blank (OLake Go-managed)** - After Clear Destination is executed, when you sync again, the entire data (currently present in the partitions) will be backfilled. A full refresh will take place and all data will be captured from the beginning of the first offset present. @@ -155,35 +193,35 @@ When using Clear Destination with Kafka source, the behavior depends on whether - Example: Job "daily_orders" has enabled streams `orders` and `customers`. Clear Destination is initiated → both streams are cleared at the destination. The next run (scheduled or Sync Now) rebuilds both streams using the latest configuration. #### 2) Stream-level property changes in previously syncing streams -When stream settings that have already synced are changed, OLake clears only the affected streams so they can be rebuilt consistently. If a job is running, OLake auto‑cancels the current run after confirmation and starts Clear Destination for those streams. +When stream settings that have already synced are changed, OLake Go clears only the affected streams so they can be rebuilt consistently. If a job is running, OLake Go auto‑cancels the current run after confirmation and starts Clear Destination for those streams. - Normalization changes - - Example: Normalization is enabled so destination naming changes from raw to standardized table/column names. OLake clears the affected stream(s); the next run writes with normalized names. + - Example: Normalization is enabled so destination naming changes from raw to standardized table/column names. OLake Go clears the affected stream(s); the next run writes with normalized names. - Filter changes - - Example: A filter for the last 12 months (`order_date >= 2024-01-01`) is added. OLake clears that stream; the next run writes only filtered rows. + - Example: A filter for the last 12 months (`order_date >= 2024-01-01`) is added. OLake Go clears that stream; the next run writes only filtered rows. - Sync Mode changes - - Example: A stream is switched from Incremental to Full Refresh. OLake clears that stream so the next run writes a complete snapshot that matches the new mode. + - Example: A stream is switched from Incremental to Full Refresh. OLake Go clears that stream so the next run writes a complete snapshot that matches the new mode. - Primary Cursor value changes - - Example: The cursor is changed from `updated_at` to `id`. OLake drops the existing table and creates a new one as per new cursor. + - Example: The cursor is changed from `updated_at` to `id`. OLake Go drops the existing table and creates a new one as per new cursor. - Fallback Cursor Modification - - Adding, removing, or changing a Fallback Cursor value will lead to dropping of the existing table and creation of a new one, as OLake tracks both primary and fallback cursors. + - Adding, removing, or changing a Fallback Cursor value will lead to dropping of the existing table and creation of a new one, as OLake Go tracks both primary and fallback cursors. - Partition changes - - Example: Partitioning is changed from day to month to reduce file counts, or an existing partition is deleted. OLake clears the stream; the next run writes new partitions based on the updated configuration. + - Example: Partitioning is changed from day to month to reduce file counts, or an existing partition is deleted. OLake Go clears the stream; the next run writes new partitions based on the updated configuration. - Destination DB name change - Example: The destination DB is renamed from `sales_raw` to `sales_curated`. In this case, all the selected streams are dropped from the old destination database, and subsequent syncs write to the new database. - Append/Upsert mode change - - Example: The mode is switched from append to upsert to deduplicate by a primary key. OLake clears the stream; the next run reconstructs with upsert semantics. + - Example: The mode is switched from append to upsert to deduplicate by a primary key. OLake Go clears the stream; the next run reconstructs with upsert semantics. - Enabling New Streams or Re-enabling Disabled Streams - - Enabling new streams: When new streams are enabled and added to a job, OLake triggers Clear Destination for those new streams, but since there is no data in the destination yet, no data is deleted. - - Re-enabling disabled streams: If a stream is disabled and then re-enabled, OLake clears those streams from the destination and rebuilds them, as the source data state may have changed during the disabled period. If the re-enabled streams also have modified properties (such as normalization, partitioning, or filters), they are rebuilt with the updated configuration. + - Enabling new streams: When new streams are enabled and added to a job, OLake Go triggers Clear Destination for those new streams, but since there is no data in the destination yet, no data is deleted. + - Re-enabling disabled streams: If a stream is disabled and then re-enabled, OLake Go clears those streams from the destination and rebuilds them, as the source data state may have changed during the disabled period. If the re-enabled streams also have modified properties (such as normalization, partitioning, or filters), they are rebuilt with the updated configuration. Result: Only the modified streams are dropped and rebuilt at the next run; other enabled streams are unaffected. @@ -193,27 +231,27 @@ Result: Only the modified streams are dropped and rebuilt at the next run; other ::::: What happens next: -- OLake drops the affected streams from the destination so they can be rebuilt by the next run -- If the job is running, OLake auto‑cancels the current run after confirmation and initiates Clear Destination for the affected streams +- OLake Go drops the affected streams from the destination so they can be rebuilt by the next run +- If the job is running, OLake Go auto‑cancels the current run after confirmation and initiates Clear Destination for the affected streams - If the job is idle, the clear is queued; once it completes, the next run (scheduled or “Sync Now”) rebuilds with the new settings #### What appears in the UI - Job Settings page: A Clear Destination button opens a confirmation modal. If a run is active, cancellation is required before proceeding -- Edit Streams: Changing any previously syncing stream prompts a confirmation; if a run is active, OLake auto‑cancels it after confirmation and begins the clear for the affected streams +- Edit Streams: Changing any previously syncing stream prompts a confirmation; if a run is active, OLake Go auto‑cancels it after confirmation and begins the clear for the affected streams - Job Logs & History: A `Job Type` column indicates whether a run is a **Sync** or **Clear Destination** so progress and outcomes can be followed. This information is also visible on the Jobs home page to provide users with at-a-glance visibility of the job type. #### Step‑by‑step Workflow 1. Clear Destination is triggered. -2. OLake checks if the job is currently running. If running, cancellation of the job is required first. -3. OLake deletes destination data for the job’s enabled streams (or only affected streams). +2. OLake Go checks if the job is currently running. If running, cancellation of the job is required first. +3. OLake Go deletes destination data for the job’s enabled streams (or only affected streams). 4. Clear completes → The job becomes editable again; the next run rebuilds the destination with the corrected settings. -### 7. Job Logs & History +### 8. Job Logs & History This page lets you view and monitor a job's sync history and logs. You'll see a list of all current and past job runs. To view logs for a specific run, click **View Logs** in the Actions column. @@ -228,7 +266,7 @@ Once you click **View Logs**, you'll see the logs for the selected job run. ![Job logs](../../static/img/docs/getting-started/create-your-first-job/logs-page.webp)
    -### 8. Job settings +### 9. Job settings Here, you can edit the frequency, and other configuration settings. You can also pause or delete the job. diff --git a/docs/getting-started/olake-ui.mdx b/docs/getting-started/olake-ui.mdx index 0c3eb6c89..8e975daf5 100644 --- a/docs/getting-started/olake-ui.mdx +++ b/docs/getting-started/olake-ui.mdx @@ -24,20 +24,11 @@ The following requirements must be met before starting: ### One-Command Setup The fastest way to get OLake UI running is with a single command: - - - ```bash - curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose-v1.yml | docker compose -f - up -d - ``` - *This setup uses Postgres for both metadata and Temporal visibility.* - - - ```bash - curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose.yml | docker compose -f - up -d - ``` - *This setup uses Elasticsearch for Temporal visibility.* - - +```bash +curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose-v1.yml | docker compose -f - up -d +``` + +*This setup uses Postgres for both metadata and Temporal visibility.* This command will: - Download the latest docker-compose.yml file diff --git a/docs/getting-started/playground.mdx b/docs/getting-started/playground.mdx index a83f2f38d..2e85c823f 100644 --- a/docs/getting-started/playground.mdx +++ b/docs/getting-started/playground.mdx @@ -6,10 +6,11 @@ sidebar_position: 3 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; +import TOCTabLinker from '@site/src/components/TOCTabLinker'; -# OLake Playground +# OLake Go Playground -OLake Playground is a self-contained environment for exploring lakehouse architecture using [Apache Iceberg](/iceberg/why-iceberg). It comes preconfigured with all the required components, allowing you to experience the complete workflow without manual setup. +OLake Go Playground is a self-contained environment for exploring lakehouse architecture using [Apache Iceberg](/iceberg/why-iceberg). It comes preconfigured with all the required components, allowing you to experience the complete workflow without manual setup. + diff --git a/docs/install/olake-ui/offline-environments-aws.mdx b/docs/install/olake-ui/offline-environments-aws.mdx index e5bc17c7b..14325a929 100644 --- a/docs/install/olake-ui/offline-environments-aws.mdx +++ b/docs/install/olake-ui/offline-environments-aws.mdx @@ -171,23 +171,38 @@ This pre-pull step is a one-time action for each connector version. Once an imag ## 7. Run the Application Stack -With the pull-through cache configured and connector images pre-pulled, the main OLake application can be started. The OLake docker-compose-v1.yml is designed to use an environment variable to specify the container registry. This makes it easy to switch from Docker Hub to a private ECR. +With the pull-through cache configured and connector images pre-pulled, the main OLake Go application can be started. The OLake Go docker-compose-v1.yml is designed to use an environment variable to specify the container registry. This makes it easy to switch from Docker Hub to a private ECR or any self-managed repository for example Harbor, Nexus etc. ### Configure the Environment -In the same directory where the OLake [docker-compose-v1.yml](https://raw.githubusercontent.com/datazip-inc/olake-ui/refs/heads/master/docker-compose-v1.yml) file is located, a new file named `.env` must be created. +In the same directory where the OLake Go [docker-compose-v1.yml](https://github.com/datazip-inc/olake-ui/blob/master/docker-compose-v1.yml) file is located, create a new file named `.env`. -The `.env` file should contain the following line: +Before starting the stack, log in to your private registry from a terminal: ```bash -CONTAINER_REGISTRY_BASE=".dkr.ecr..amazonaws.com/" +docker login registry.your-company.com -u '' -p '' +``` + +Then add the following values to the `.env` file: + +```env +CONTAINER_REGISTRY_BASE="registry.your-company.com" -# Example: CONTAINER_REGISTRY_BASE="111222333444.dkr.ecr.us-east-1.amazonaws.com/dockerhub" +## For self-managed repositories +CONTAINER_REGISTRY_USERNAME="" +CONTAINER_REGISTRY_PASSWORD="" +CONTAINER_REGISTRY_INSECURE="" +CONTAINER_REGISTRY_TLS_SKIP_VERIFY="" +CONTAINER_REGISTRY_CA_CERT="" ``` -Replace `` and `` with the appropriate AWS account ID and region. The `` must be replaced with the value created earlier. +For AWS ECR specific example: + +```env +CONTAINER_REGISTRY_BASE="111222333444.dkr.ecr.us-east-1.amazonaws.com/dockerhub" +``` -The docker-compose-v1.yml file for OLake is already configured to use this `CONTAINER_REGISTRY_BASE` variable for all service images. No modifications to the docker-compose-v1.yml file itself are necessary. +The docker-compose-v1.yml file for OLake Go is already configured to use this `CONTAINER_REGISTRY_BASE` variable for all service images. No modifications to the docker-compose-v1.yml file itself are necessary. ### Start OLake UI diff --git a/docs/install/olake-ui/offline-environments-generic.mdx b/docs/install/olake-ui/offline-environments-generic.mdx index 2de9c1275..11446af0f 100644 --- a/docs/install/olake-ui/offline-environments-generic.mdx +++ b/docs/install/olake-ui/offline-environments-generic.mdx @@ -13,7 +13,7 @@ import TabItem from '@theme/TabItem'; This guide currently covers the **update procedure** for existing offline installations. An **installation section** for generic offline environments will be added to this document soon. ::: -This document outlines the procedure for **updating** the OLake Stack components and connector images in offline environments (air-gapped setups) hosted on cloud providers or on-premise infrastructure. +This document outlines the procedure for **updating** the OLake Go Stack components and connector images in offline environments (air-gapped setups) hosted on cloud providers or on-premise infrastructure. ## Prerequisites @@ -23,7 +23,7 @@ The following requirements must be satisfied prior to initiating the update proc - **SSH Access**: SSH connectivity from the local machine to the Offline Node (On-prem/Cloud) must be established. - **Offline Node Setup**: Docker and Docker Compose must be installed and operational on the Offline Node. - **Architecture Compatibility**: The CPU architecture (ARM or x86) of the local machine must match that of the Offline Node. -- **Operational Stack**: An OLake Docker Compose stack is assumed to be currently running on the Offline Node (On-prem/Cloud). +- **Operational Stack**: An OLake Go Docker Compose stack is assumed to be currently running on the Offline Node (On-prem/Cloud). ## 1. Image Acquisition (Local Machine) @@ -31,7 +31,7 @@ The required Docker images are retrieved on the internet-connected local machine ### Pulling Images -The latest versions of the OLake core components and the specific versions of the required connectors are pulled from the public registry. +The latest versions of the OLake Go core components and the specific versions of the required connectors are pulled from the public registry. ```bash # Pull Core Components diff --git a/docs/intro.mdx b/docs/intro.mdx index 13e220b7a..ea1a88681 100644 --- a/docs/intro.mdx +++ b/docs/intro.mdx @@ -1,7 +1,7 @@ --- title: "OLake Data Replication: Fastest Open Source Iceberg Lakehouse Tool" description: "Replicate databases 500X faster to Iceberg-based data lakes. OLake saves 90% cost, offers seamless real-time workflows, and integrates with major tools." -sidebar_label: Introduction +sidebar_label: Overview slug: / --- @@ -91,7 +91,7 @@ import Head from '@docusaurus/Head' -# Welcome to OLake +# Welcome to OLake Go

    @@ -99,10 +99,110 @@ import Head from '@docusaurus/Head'

    -

    Fastest open-source tool for replicating Databases to Apache Iceberg or Data Lakehouse. ⚡ Efficient, quick and scalable data ingestion for real-time analytics. Visit olake.io for the full documentation, and benchmarks

    +

    Fastest open-source tool for replicating Databases to Apache Iceberg or Data Lakehouse. ⚡ Efficient, quick and scalable data ingestion for real-time analytics.

    +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; -## Introduction to OLake +## What is OLake Go? + +OLake Go is a high-performance, open-source EL (Extract–Load) platform that bridges operational databases and open lakehouse storage - writing Apache Iceberg tables or Parquet files on Amazon S3 , enabling organizations to replicate data at scale with minimal overhead. Supporting Incremental Sync, Change Data Capture (CDC), and stateful, resumable syncs, OLake Go ensures your tables remain fresh, organized, and optimized for analytics. + +## Supported Sources + +OLake Go supports ingestion from the following sources: + +| Source | Full Refresh | Full Refresh + Incremental | Full Refresh + CDC | CDC Only | +| ----------------- | :----------: | :--------------: | -------------------------- | -------------------------- | +| **PostgreSQL** | ✅ | ✅ | ✅ | ✅ | +| **MySQL** | ✅ | ✅ | ✅ | ✅ | +| **MongoDB** | ✅ | ✅ | ✅ | ✅ | +| **Oracle Database** | ✅ | ✅ | — | — | +| **Apache Kafka** | — | — | — | ✅ | +| **DB2** | ✅ | ✅ | — | — | +| **MSSQL** | ✅ | ✅ | ✅ | ✅ | +| **S3** | ✅ | ✅ | — | — | + +## Destinations + +OLake Go writes data to open lakehouse storage: + +- **Parquet files** on object storage such as Amazon S3, MinIO, and Google Cloud Storage +- **Apache Iceberg** tables with support for multiple catalog integrations including: + - AWS Glue Data Catalog + - Apache Hive Metastore + - REST catalogs such as Nessie, Polaris and Unity Catalog + - JDBC catalogs + + To know more, read [OLake Go Catalog Integration](/docs/understanding/compatibility-catalogs). + +## Capabilities + +### 1. Parallelised Chunking + +Parallel chunking is a technique that splits large datasets or collections into smaller virtual chunks, allowing them to be read and processed simultaneously. It is used in sync modes such as Full Refresh, Full Refresh + CDC, and Full Refresh + Incremental. + +**What it does**: +- Splits big collections into manageable pieces without altering the underlying data. +- Each chunk can be processed parallely & independently. + +**Benefit**: +- Enables **parallel reads**, dramatically reducing the time needed to perform full snapshots or scans of large datasets. +- Improves ingestion speed, scalability, and overall system performance. + +### 2. Stateful, Resumable Syncs + +It ensures that data syncs resume automatically from the last checkpoint after interruptions. Applicable for Full Refresh + CDC and Full Refresh + Incremental & Strict CDC sync modes. + +**What it does**: +- Maintains **state** of in-progress syncs. +- Automatically **resumes** after crashes, network failures, or manual pauses. +- Eliminates the need to resync data from the beginning. Interrupted runs resume from the last checkpoint instead. + +**Benefit**: +- Reduces data duplication and processing time. +- Ensures **reliable**, **fault-tolerant** pipelines. +- Minimizes manual intervention for operational teams. + +### 3. Configurable Max Connections + +Each job can set its own maximum number of database connections to the source. This limit is **per job**, not shared across jobs—even when several jobs use the same source, each job’s setting applies only to that job. Helps prevent overload and ensures stable performance on the source system. + +### 4. Data Deduplication + +Data Deduplication ensures that only unique records are stored and processed in **Upsert** ingestion mode: saving space, reducing costs, and improving data quality. OLake Go automatically deduplicates data using the primary key from the source tables, guaranteeing that each primary key maps to a single row in the destination along with its corresponding olake_id. + +### 5. Hive Style Partitioning + +Partitioning is the process of dividing large datasets into smaller, more manageable segments based on specific column values (e.g., date, region, or category), improving query performance, scalability, and data organization. + +- [**Iceberg partitioning**](/docs/writers/iceberg/partitioning/) → Metadata-driven, no need for directory-based partitioning; enables efficient pruning and schema evolution. +- **S3-style partitioning** → Traditional folder-based layout (e.g., `year=2025/month=08/day=22/`) for compatibility with external tools. +- **Normalization** → Automatically expands **level-1 nested JSON fields** into top-level columns. + +**What it does**: + +- Converts nested JSON objects into **flat columns** for easier querying. +- Preserves all data while simplifying structure. + +**Benefit**: +- Makes **SQL queries simpler and faster**. +- Reduces the need for complex JSON parsing in queries. +- Improves readability and downstream analytics efficiency. + +### 6. Schema Evolution & Data Types Changes + +OLake Go automatically handles changes in your table's schema without breaking downstream jobs. Read More [Schema Evolution in OLake Go](/docs/features/schema) + +### 7. Dead Letter Queue Columns (Coming soon) + +The DLQ column handles values with data type changes not supported by Iceberg / Parquet destinations type promotions, safely storing them without loss. This prevents sync failures and ensures downstream models remain stable. By isolating incompatible values, it allows users to continue syncing data seamlessly while addressing type mismatches at their convenience, improving reliability and reducing manual intervention. + +### 8. Two Phase Commit + +OLake Go uses this mechanism for the Iceberg destination during full refresh, incremental, and CDC (MongoDB, PostgreSQL, MySQL, MSSQL), so per-chunk commit status is tracked via destination state making sure the syncs avoid inconsistencies and duplicate writes. + + \ No newline at end of file diff --git a/docs/release/28-04-2025.mdx b/docs/release/ingestion/28-04-2025.mdx similarity index 100% rename from docs/release/28-04-2025.mdx rename to docs/release/ingestion/28-04-2025.mdx diff --git a/docs/release/overview.mdx b/docs/release/ingestion/overview.mdx similarity index 51% rename from docs/release/overview.mdx rename to docs/release/ingestion/overview.mdx index d2938a7cf..db833e8fe 100644 --- a/docs/release/overview.mdx +++ b/docs/release/ingestion/overview.mdx @@ -1,23 +1,23 @@ --- -title: "OLake Release Overview & Updates | Features & Bug Fixes" -description: Explore OLake release summaries, new features, bug fixes, and major changes. Get support and engage with the community for smooth updates. +title: "OLake Ingestion Release Overview & Updates | Features & Bug Fixes" +description: Explore OLake Ingestion release summaries, new features, bug fixes, and major changes. Get support and engage with the community for smooth updates. sidebar_label: Start Here sidebar_position: 1 --- # Release Notes -Welcome to the OLake release notes archive. Each release contains a changelog of new features, enhancements, and bug fixes. Use this landing page to: +Welcome to the OLake Go release notes archive. Each release contains a changelog of new features, enhancements, and bug fixes. Use this landing page to: - Quickly navigate to a specific version’s release notes - Understand when major features were introduced - Learn about backward-incompatible changes and deprecations -**Why a Release Landing Page?** OLake evolves rapidly with new connectors, performance improvements, and stability enhancements. Since not all features are available in every version, this page helps you identify when key features were introduced, track deprecations, and access guidance for migrating between versions. +**Why a Release Landing Page?** OLake Go evolves rapidly with new connectors, performance improvements, and stability enhancements. Since not all features are available in every version, this page helps you identify when key features were introduced, track deprecations, and access guidance for migrating between versions. ### Navigate Releases Use the Release dropdown in the documentation sidebar to select any version’s detailed notes. Each version page includes: -- 🎯 What's New: Additions or enhancements introduced in this release across Sources, Destinations, Catalogs, or core Platform features; typically new connectors, capabilities, flags, or behavior that expand OLake’s functionality. +- 🎯 What's New: Additions or enhancements introduced in this release across Sources, Destinations, Catalogs, or core Platform features; typically new connectors, capabilities, flags, or behavior that expand OLake Go’s functionality. - 🔧 Bug Fixes & Stability: Improvements and refinements to existing functionality in this release; reliability, performance, and compatibility fixes (e.g., retries, state handling, type conversions, dependency updates). diff --git a/docs/release/v0.1.0-v0.1.1.mdx b/docs/release/ingestion/v0.1.0-v0.1.1.mdx similarity index 92% rename from docs/release/v0.1.0-v0.1.1.mdx rename to docs/release/ingestion/v0.1.0-v0.1.1.mdx index a429e2cad..f61ecc71c 100644 --- a/docs/release/v0.1.0-v0.1.1.mdx +++ b/docs/release/ingestion/v0.1.0-v0.1.1.mdx @@ -1,9 +1,8 @@ --- -title: "OLake (v0.1.0 - v0.1.1)" -description: "OLake v0.1.0-v0.1.1 adds MongoDB, Postgres, MySQL sources, Iceberg and Parquet writers, CDC sync mode, schema discovery improvements, and key bug fixes" +title: "OLake Go (v0.1.0 - v0.1.1)" --- -# OLake (v0.1.0 – v0.1.1) +# OLake Go (v0.1.0 – v0.1.1) June 13 – June 18, 2025 ## 🎯 What's New diff --git a/docs/release/v0.1.2-v0.1.5.mdx b/docs/release/ingestion/v0.1.2-v0.1.5.mdx similarity index 85% rename from docs/release/v0.1.2-v0.1.5.mdx rename to docs/release/ingestion/v0.1.2-v0.1.5.mdx index f4eddadb9..f15445064 100644 --- a/docs/release/v0.1.2-v0.1.5.mdx +++ b/docs/release/ingestion/v0.1.2-v0.1.5.mdx @@ -1,9 +1,8 @@ --- -title: "OLake (v0.1.2 - v0.1.5)" -description: "OLake v0.1.2-v0.1.5 introduces Oracle source connector, Unity Catalog support, telemetry via Segment IO & Mixpanel, config decryption, and Iceberg deduplication fix" +title: "OLake Go (v0.1.2 - v0.1.5)" --- -# OLake (v0.1.2 – v0.1.5) +# OLake Go (v0.1.2 – v0.1.5) June 26 – July 01, 2025 ## 🎯 What's New diff --git a/docs/release/v0.1.6-v0.1.8.mdx b/docs/release/ingestion/v0.1.6-v0.1.8.mdx similarity index 92% rename from docs/release/v0.1.6-v0.1.8.mdx rename to docs/release/ingestion/v0.1.6-v0.1.8.mdx index 74ce815db..28101f6e7 100644 --- a/docs/release/v0.1.6-v0.1.8.mdx +++ b/docs/release/ingestion/v0.1.6-v0.1.8.mdx @@ -1,9 +1,8 @@ --- -title: "OLake (v0.1.6 - v0.1.8)" -description: "OLake v0.1.6-v0.1.8 adds incremental MongoDB/Oracle sync, Oracle filter and chunking, MySQL binlog permission checks, and fixes Postgres CDC and discovery issues" +title: "OLake Go (v0.1.6 - v0.1.8)" --- -# OLake (v0.1.6 – v0.1.8) +# OLake Go (v0.1.6 – v0.1.8) July 17 – July 30, 2025 ## 🎯 What's New diff --git a/docs/release/v0.1.9-v0.1.11.mdx b/docs/release/ingestion/v0.1.9-v0.1.11.mdx similarity index 90% rename from docs/release/v0.1.9-v0.1.11.mdx rename to docs/release/ingestion/v0.1.9-v0.1.11.mdx index 6c73fc202..b2c2cebb0 100644 --- a/docs/release/v0.1.9-v0.1.11.mdx +++ b/docs/release/ingestion/v0.1.9-v0.1.11.mdx @@ -1,9 +1,8 @@ --- -title: "OLake (v0.1.9 - v0.1.11)" -description: "OLake v0.1.9-v0.1.11 introduces MongoDB multi-cursor, incremental MySQL/Postgres sync, batch size consistency, relational DB normalization, and bug fixes" +title: "OLake Go (v0.1.9 - v0.1.11)" --- -# OLake (v0.1.9 – v0.1.11) +# OLake Go (v0.1.9 – v0.1.11) August 15 – August 27, 2025 ## 🎯 What's New diff --git a/docs/release/v0.2.0-v0.2.1.mdx b/docs/release/ingestion/v0.2.0-v0.2.1.mdx similarity index 91% rename from docs/release/v0.2.0-v0.2.1.mdx rename to docs/release/ingestion/v0.2.0-v0.2.1.mdx index 00da67f85..6c91a059a 100644 --- a/docs/release/v0.2.0-v0.2.1.mdx +++ b/docs/release/ingestion/v0.2.0-v0.2.1.mdx @@ -1,9 +1,8 @@ --- -title: "OLake (v0.2.0 - v0.2.1)" -description: "OLake v0.2.0-v0.2.1 introduces namespace normalization, spec commands, Java writer refactor, AWS IRSA fixes, and gRPC dependency resolutions for stability" +title: "OLake Go (v0.2.0 - v0.2.1)" --- -# OLake (v0.2.0 – v0.2.1) +# OLake Go (v0.2.0 – v0.2.1) August 15 – August 27, 2025 ## 🎯 What's New diff --git a/docs/release/v0.2.10.mdx b/docs/release/ingestion/v0.2.10.mdx similarity index 90% rename from docs/release/v0.2.10.mdx rename to docs/release/ingestion/v0.2.10.mdx index 9c86829f8..e479d717b 100644 --- a/docs/release/v0.2.10.mdx +++ b/docs/release/ingestion/v0.2.10.mdx @@ -1,9 +1,8 @@ --- -title: "OLake (v0.2.10 - v0.3.4)" -description: "OLake v0.2.10-v0.3.4 introduces Kafka source connector for streaming data ingestion and replication to Apache Iceberg lakehouses." +title: "OLake Go (v0.2.10 - v0.3.4)" --- -# OLake (v0.2.10 - v0.3.4) +# OLake Go (v0.2.10 - v0.3.4) October 31 – November 28, 2025 ## 🎯 What's New diff --git a/docs/release/v0.2.2-v0.2.4.mdx b/docs/release/ingestion/v0.2.2-v0.2.4.mdx similarity index 83% rename from docs/release/v0.2.2-v0.2.4.mdx rename to docs/release/ingestion/v0.2.2-v0.2.4.mdx index 6c69a0a29..92c33c578 100644 --- a/docs/release/v0.2.2-v0.2.4.mdx +++ b/docs/release/ingestion/v0.2.2-v0.2.4.mdx @@ -1,9 +1,8 @@ --- -title: "OLake (v0.2.2 - v0.2.4)" -description: "OLake v0.2.2-v0.2.4 updates include custom DB preservation for new streams, MySQL empty table sync fix, and MongoDB _id type fallback improvements" +title: "OLake Go (v0.2.2 - v0.2.4)" --- -# OLake (v0.2.2 – v0.2.4) +# OLake Go (v0.2.2 – v0.2.4) September 16 – September 19, 2025 ## 🎯 What's New diff --git a/docs/release/v0.2.5-v0.2.7.mdx b/docs/release/ingestion/v0.2.5-v0.2.7.mdx similarity index 94% rename from docs/release/v0.2.5-v0.2.7.mdx rename to docs/release/ingestion/v0.2.5-v0.2.7.mdx index 73268f07f..e6c5d7ddf 100644 --- a/docs/release/v0.2.5-v0.2.7.mdx +++ b/docs/release/ingestion/v0.2.5-v0.2.7.mdx @@ -1,9 +1,8 @@ --- -title: "OLake (v0.2.5 - v0.2.7)" -description: "OLake v0.2.5-v0.2.7 introduces PgOutput plugin for faster Postgres CDC, MongoDB IAM authentication, and enhanced normalization with improved performance." +title: "OLake Go (v0.2.5 - v0.2.7)" --- -# OLake (v0.2.5 - v0.2.7) +# OLake Go (v0.2.5 - v0.2.7) September 20 – October 11, 2025 ## 🎯 What's New diff --git a/docs/release/v0.2.8.mdx b/docs/release/ingestion/v0.2.8.mdx similarity index 95% rename from docs/release/v0.2.8.mdx rename to docs/release/ingestion/v0.2.8.mdx index fece2892d..bc70fd063 100644 --- a/docs/release/v0.2.8.mdx +++ b/docs/release/ingestion/v0.2.8.mdx @@ -1,9 +1,8 @@ --- -title: "OLake (v0.2.8 - v0.2.9)" -description: "OLake v0.2.8-v0.2.9 adds sync progress tracking for Oracle, fixes MongoDB sharded clusters, improves MySQL CDC, and enhances Postgres normalization." +title: "OLake Go (v0.2.8 - v0.2.9)" --- -# OLake (v0.2.8 - v0.2.9) +# OLake Go (v0.2.8 - v0.2.9) October 11 – October 30, 2025 ## 🎯 What's New diff --git a/docs/release/v0.3.14.mdx b/docs/release/ingestion/v0.3.14.mdx similarity index 98% rename from docs/release/v0.3.14.mdx rename to docs/release/ingestion/v0.3.14.mdx index 9b051fe70..256a39a1c 100644 --- a/docs/release/v0.3.14.mdx +++ b/docs/release/ingestion/v0.3.14.mdx @@ -1,4 +1,8 @@ -# OLake (v0.3.14 - v0.3.16) +--- +title: "OLake Go (v0.3.14 - v0.3.16)" +--- + +# OLake Go (v0.3.14 - v0.3.16) January 13, 2026 – February 09, 2026 ## 🎯 What's New diff --git a/docs/release/v0.3.17.mdx b/docs/release/ingestion/v0.3.17.mdx similarity index 96% rename from docs/release/v0.3.17.mdx rename to docs/release/ingestion/v0.3.17.mdx index fb3f83a97..7ff82e728 100644 --- a/docs/release/v0.3.17.mdx +++ b/docs/release/ingestion/v0.3.17.mdx @@ -1,4 +1,8 @@ -# OLake (v0.3.17 - v0.3.18) +--- +title: "OLake Go (v0.3.17 - v0.3.18)" +--- + +# OLake Go (v0.3.17 - v0.3.18) February 10, 2026 – February 20, 2026 ## 🎯 What's New diff --git a/docs/release/v0.3.5.mdx b/docs/release/ingestion/v0.3.5.mdx similarity index 96% rename from docs/release/v0.3.5.mdx rename to docs/release/ingestion/v0.3.5.mdx index 1ba3aa343..445d82c97 100644 --- a/docs/release/v0.3.5.mdx +++ b/docs/release/ingestion/v0.3.5.mdx @@ -1,9 +1,8 @@ --- -title: "OLake (v0.3.5 - v0.3.8)" -description: "OLake v0.3.5-v0.3.8 release notes" +title: "OLake Go (v0.3.5 - v0.3.8)" --- -# OLake (v0.3.5 - v0.3.8) +# OLake Go (v0.3.5 - v0.3.8) November 29 – December 28, 2025 ## 🎯 What's New diff --git a/docs/release/v0.3.9-v0.3.11.mdx b/docs/release/ingestion/v0.3.9-v0.3.11.mdx similarity index 97% rename from docs/release/v0.3.9-v0.3.11.mdx rename to docs/release/ingestion/v0.3.9-v0.3.11.mdx index bb363a466..069c7065d 100644 --- a/docs/release/v0.3.9-v0.3.11.mdx +++ b/docs/release/ingestion/v0.3.9-v0.3.11.mdx @@ -1,4 +1,8 @@ -# OLake (v0.3.9 - v0.3.13) +--- +title: "OLake Go (v0.3.9 - v0.3.13)" +--- + +# OLake Go (v0.3.9 - v0.3.13) December 29, 2025 – January 12, 2026 ## 🎯 What's New diff --git a/docs/release/v0.4.0.mdx b/docs/release/ingestion/v0.4.0.mdx similarity index 93% rename from docs/release/v0.4.0.mdx rename to docs/release/ingestion/v0.4.0.mdx index 762b0b88b..2603846c0 100644 --- a/docs/release/v0.4.0.mdx +++ b/docs/release/ingestion/v0.4.0.mdx @@ -1,4 +1,8 @@ -# OLake (v0.4.0 - v0.4.2) +--- +title: "OLake Go (v0.4.0 - v0.4.2)" +--- + +# OLake Go (v0.4.0 - v0.4.2) February 21, 2026 – March 5, 2026 ## 🎯 What's New diff --git a/docs/release/v0.5.0.mdx b/docs/release/ingestion/v0.5.0.mdx similarity index 96% rename from docs/release/v0.5.0.mdx rename to docs/release/ingestion/v0.5.0.mdx index 059cb0819..3efa8d1b2 100644 --- a/docs/release/v0.5.0.mdx +++ b/docs/release/ingestion/v0.5.0.mdx @@ -1,4 +1,8 @@ -# OLake (v0.5.0 - v0.5.2) +--- +title: "OLake Go (v0.5.0 - v0.5.2)" +--- + +# OLake Go (v0.5.0 - v0.5.2) March 6, 2026 – March 20, 2026 ## 🎯 What's New diff --git a/docs/release/ingestion/v0.6.0.mdx b/docs/release/ingestion/v0.6.0.mdx new file mode 100644 index 000000000..e65aaee2d --- /dev/null +++ b/docs/release/ingestion/v0.6.0.mdx @@ -0,0 +1,45 @@ +--- +title: "OLake Go (v0.6.0 - v0.6.5)" +--- + +# OLake Go (v0.6.0 - v0.6.5) +March 21, 2026 – April 20, 2026 + +## 🎯 What's New + +### Sources + +1. **Filters for CDC and incremental syncs -**
    Added filtering support for CDC and incremental syncs so users can selectively sync records based on configured conditions. + +2. **Postgres strict SSL verification support -**
    Added `verify-ca` and `verify-full` SSL support for Postgres using PEM certificate content input. Also removed support for passing `sslrootcert`, `sslcert`, and `sslkey` as file paths; these fields now expect the actual PEM content instead. + +3. **Skip ProduceSchema overhead during sync -**
    Optimized sync performance by skipping heavy schema inference (like MongoDB collection scans or Kafka message decoding) during sync. Drivers now rely on the pre-discovered schema from `streams.json` via a new `SyncContext`, and `ProduceSchema` is only executed for user-selected streams rather than all available source tables/collections. + +4. **Partitioning support for non-normalized streams -**
    Partition transforms now work correctly when `normalization=false` for both legacy and Arrow Iceberg writers. Previously, non-normalized streams silently wrote all data to a null partition due to missing column values during pre-shaping and case mismatches in column name lookups. + +## 🔧 Bug Fixes & Stability + +1. **Datatype utility unit tests -**
    Added unit test coverage for `utils/typeutils/datatype.go` across type detection, comparison, timestamp precision, and SQL type mapping, and fixed invalid `reflect.Value` handling to map to `types.Null`. + +2. **Oracle incremental cursor timezone fix -**
    Fixed incremental cursor handling for Oracle `TIMESTAMP` and `DATE` columns by stripping the session timezone offset before saving cursor values, so TZ-naive columns don’t shift and re-read already synced records. + +3. **MongoDB date out of range fix -**
    Fixed a fatal crash (`json: error calling MarshalJSON`) during MongoDB syncs with `normalization: false` caused by dates exceeding Go's [0, 9999] year limit. Added bounds checking to clamp years `< 1` to 1970 and years `> 9999` to 9999, ensuring dates safely map to valid JSON timestamps. + +4. **Clear destination support for GCP -**
    Fixed clear destination for GCP buckets by handling the unsupported bulk delete operation and added retry backoff. + +5. **Increase Java writer startup timeout -**
    Extended the Java writer process startup timeout from 30 seconds to 600 seconds (10 minutes). + +6. **MongoDB nested dates out of range fix -**
    Fixed fatal crashes (`json: error calling MarshalJSON`) when unnormalized MongoDB documents contained nested `DateTime` values with years outside Go’s supported [0, 9999] range by configuring the MongoDB client to decode BSON `DateTime` into safe, clamped `time.Time` values at decode-time. + +7. **PostgreSQL backfill chunking for multi-level partitions -**
    Fixed a bug where multi-level partitioned PostgreSQL tables were silently skipped during backfill. The chunk discovery query now properly finds all leaf partitions (using `pg_partition_tree` for PG 12+ and a recursive `pg_inherits` CTE for older versions) instead of only looking one level deep, and stale statistics without an `ANALYZE` run now return a clear error rather than silently skipping the table. + +8. **Column selection integration tests -**
    Added integration tests for column selection to verify that only specified columns are synced. Tests cover both inclusion and exclusion logic, as well as validating that columns removed from the source during a backfill stop being ingested while the remaining columns continue processing correctly. + +9. **Go Security workflow vulnerability and Go version update -**
    Updated the Go Security workflow to make `govulncheck` respect defined exemptions (failing only on unexempted vulnerabilities) and upgraded the Go version to `v1.25.9` (which includes recent security fixes). + +10. **Kafka integration test -**
    Added an integration test for the Kafka driver to validate end-to-end sync behavior, and fixed parsing and comparison bugs that could cause incorrect data transformation and validation mismatches. + +11. **MSSQL boolean filter literal fix -**
    MSSQL does not support `TRUE`/`FALSE` as SQL literals, so filter SQL generation now converts boolean values to `1`/`0` instead. This fix is applied consistently across both the structured `filter_config` and legacy string filter paths. + +12. **Kafka sync: skip unparseable last partition message -**
    When the last message in a partition was neither JSON nor Avro, the sync had no way to parse it and would stall indefinitely, waiting until the context was cancelled. Fixed by skipping such messages — the offset is still marked in Kafka to advance past it, but the message is not written to the destination, allowing the sync to continue cleanly. + diff --git a/docs/release/ingestion/v0.7.0.mdx b/docs/release/ingestion/v0.7.0.mdx new file mode 100644 index 000000000..ee7439469 --- /dev/null +++ b/docs/release/ingestion/v0.7.0.mdx @@ -0,0 +1,65 @@ +--- +title: "OLake Go (v0.7.0 - v0.7.8)" +--- + +# OLake Go (v0.7.0 - v0.7.8) +April 21, 2026 – June 29, 2026 + +## 🎯 What's New + +### Sources + +1. **MySQL chunking optimisation -**
    Replaced repeated database lookups during chunk discovery with mathematical range splitting — arithmetic progression for numeric primary keys and Unicode-encoded range splitting for string keys — significantly reducing chunk generation time for large tables while ensuring correct collation-aware ordering. + +2. **SSH tunnel support for DB2 and MSSQL -**
    Added SSH tunnel configuration for the DB2 and MSSQL drivers. DB2 uses a local TCP proxy on `localhost:0` forwarded through the SSH client (`since go_ibm_db` has no Go-level dial hook), while MSSQL routes connections via `go-mssqldb's` `Connector.Dialer` and `HostDialer` interfaces with remote-side DNS resolution. + +3. **Schema filtering for PostgreSQL discovery -**
    Added an optional schemas config field to restrict the discover operation to user-specified PostgreSQL schemas. When omitted, existing behaviour is preserved and all non-system schemas are discovered. + +4. **MSSQL read replica support -**
    Added optional `jdbc_url_params` to the MSSQL source so you can target Always On read replicas (for example with read-intent), and updated CDC to use replica-safe paths that avoid primary-only agent/msdb and capture-instance management on secondaries. + +5. **MongoDB delete pre-image capture -**
    Added support to capture the full document on delete events using `fullDocumentBeforeChange: "whenAvailable"` for MongoDB 6.0+ clusters with pre-images enabled, falling back to `_id`-only `documentKey` when pre-images are unavailable to preserve existing behaviour. + +6. **Optimized chunking strategies for MSSQL -**
    Adds faster and more efficient chunk planning for MSSQL full-load syncs. Uses page-level metadata to split tables without scanning them (SQL Server 2012+, requires `VIEW DATABASE STATE`; not supported on Azure SQL DB/MI). Falls back to statistical sampling when the primary strategy is unavailable. + +7. **Capture instance management on read replicas -**
    Adds an optional `primary_config` (host, port, username, password) to the MSSQL driver so CDC capture instance operations run against the Always On primary while ingestion continues from the read-only secondary. Database, SSL, and JDBC params are inherited from the main connection, and the existing `ssh_config` tunnel is reused for both connections. + +8. **Kafka consumer migration to Franz-Go -**
    Migrates the Kafka consumer from Segment `Kafka-Go` to `Franz-Go` to improve consumer group handling and to exit cleanly on rebalance behavior to avoid duplicate data writes in destination. + +9. **Bulk read path for DB2 backfill and incremental sync -**
    Adds a fast bulk-read path using a patched `go_ibm_db` driver that fetches 200 rows per ODBC call via block fetch, replacing the previous one-row-at-a-time `database/sql` scan. Column types are resolved once at setup, and a producer-consumer pipeline overlaps I/O and CPU, significantly reducing bottlenecks on large tables. + +### Destinations + +1. **Skip equality deletes for CDC inserts post-backfill -**
    Equality deletes are now skipped for CDC inserts once the backfill→CDC overlap window is complete, reducing unnecessary write overhead. A new `dedup_inserts` flag on the Iceberg `olake_2pc` table property tracks this — Java sets it to `true` on backfill commit, and Go clears it to `false` after the first successful CDC commit. This applies to both the Arrow and legacy gRPC writers. + +2. **2PC integration tests -**
    Added integration tests for two-phase commit (2PC) to validate end-to-end behavior and improve reliability of 2PC flows. + +3. **Configurable destination column naming strategy -**
    Adds a `use_source_column_names` flag in CLI or the **Source Naming Convention** toggle in the UI to control how destination columns are named. By default, it is disabled and column names continue to be normalized (e.g. My-Column → my_column). When enabled, original source column names are preserved as is (e.g. My-Column → My-Column). + +## 🔧 Bug Fixes & Stability + +1. **Upgrade pgx/v5 to v5.9.2 for security fixes -**
    Upgraded `github.com/jackc/pgx/v5` from `v5.7.3` to `v5.9.2` to remediate two security vulnerabilities: a critical memory-safety flaw (`CVE-2026-33816`) that could allow memory corruption and a low-severity SQL injection advisory (`GHSA-j88v-2chj-qfwx`). No existing functionality is affected by this upgrade. + +2. **Oracle chunk boundary query optimisation -**
    Replaced N+1 sequential database round trips in `splitViaTableIteration` with a single `NTILE`-based query to fetch all chunk boundaries at once, with a fallback to the original loop when table stats are unavailable. + +3. **Iceberg positional delete file fix for CDC upserts -**
    Compaction was failing when multiple changes for the same `_olake_id` arrived in a single batch, caused by a positional delete file referencing multiple data files. Fixed by creating one positional delete file per data file reference. + +4. **PostgreSQL primary key discovery fix via pg_catalog -**
    `information_schema.key_column_usage` incorrectly included foreign key columns as primary keys, causing wrong `_olake_id` hashes, missed equality deletes, and duplicate rows in Iceberg on CDC upserts. Replaced with a `pg_catalog`-based query that returns only true primary keys and works correctly for read-only roles on managed databases like RDS, Supabase, and Render. + +5. **MySQL CDC charset corruption fix for non-UTF8 columns -**
    ENUM and string columns using non-UTF8 charsets (`utf16`, `ucs2`, `latin1`) were silently corrupted during CDC due to blind `[]byte` → `string` casts. Fixed by adding collation-aware decoding using `TableMapEvent.CollationMap()` and `EnumSetCollationMap()`. + +6. **MongoDB primary key pinning for deterministic deduplication -**
    Previously, all indexed fields were treated as primary keys, so updates to non-unique indexed fields changed the `_olake_id` and broke Iceberg equality deletes, creating duplicate rows. The primary key is now pinned strictly to MongoDB’s guaranteed-unique `_id`, ensuring stable hashes and correct deduplicated upserts. + +7. **DB2 driver download fix in integration tests -**
    DB2 integration tests now reuse the already-installed `clidriver` by copying it into the workspace, so Docker containers find it locally instead of repeatedly hitting the flaky IBM CDN download path. + +8. **Upgrade `golang.org/x/crypto` to v0.52.0 for SSH security fixes -**
    Upgraded `golang.org/x/crypto` from `v0.50.0` to `v0.52.0` across all Go modules to patch five SSH-related vulnerabilities reported by `govulncheck` (GO-2026-5013, GO-2026-5017, GO-2026-5018, GO-2026-5019, GO-2026-5020), bringing the workspace onto the latest secure SSH implementation. + +9. **Graceful shutdown via SIGINT/SIGTERM-aware root context -**
    Wired SIGINT/SIGTERM into the Cobra root context using `signal.NotifyContext`, so CDC, backfill, and destination writers now respect `ctx.Done()` and shut down cleanly on pod eviction, `docker stop`, or Ctrl-C instead of being killed mid-read. + +10. **Fixed edge cases in `ReformatValue` and `ReformatBool` -**
    Corrected two bugs in value reformatting logic, added unit test coverage for `reformat.go`. + +11. **Fixed TOAST column values being nulled on update events -**
    Unchanged TOAST columns in PostgreSQL update events were incorrectly emitted as `null` when `pgoutput` omitted the column data for unchanged values. For `REPLICA IDENTITY FULL` tables, the fix now preserves the existing value from the old tuple, preventing data loss on updates. + +12. **Fixed false LSN mismatch error during Postgres CDC 2PC recovery -**
    A crash after slot acknowledgement but before `state.json` was written caused a non-retryable lsn mismatch error on the next run. Fixed by skipping `validateGlobalState` when `confirmed_flush_lsn` already matches the metadata-committed LSN. + +13. **Bumped golang.org/x/net to v0.55.0 -**
    Upgraded indirect dependency across all driver modules to remediate 6 HIGH severity CVEs flagged by the `trivy-go` security scanner. + diff --git a/docs/release/ingestion/v0.8.0.mdx b/docs/release/ingestion/v0.8.0.mdx new file mode 100644 index 000000000..e1ab52ac0 --- /dev/null +++ b/docs/release/ingestion/v0.8.0.mdx @@ -0,0 +1,30 @@ +--- +title: "OLake Go (v0.8.0 - v0.8.2)" +--- + +# OLake Go (v0.8.0 - v0.8.2) +June 29, 2026 – July 11, 2026 + +## 🎯 What's New + +### Sources + +1. **2PC support for Kafka -**
    Adds two-phase commit support for Kafka by persisting consumer group ID, partition, and offset information in destination metadata, enabling reliable recovery so Kafka consumption can resume correctly from the last committed state. + +### Destinations + +1. **Single JVM per process for Iceberg writes -**
    Previously, every Iceberg writer spawned its own JVM per stream/chunk, causing excessive memory usage and OOM risk on large concurrent syncs. Now a single shared JVM handles all streams, with `ThreadSession` based isolation preserving per-stream state and cutting memory overhead. + +## 🔧 Bug Fixes & Stability + +1. **Refactored MySQL chunking helpers for testability -**
    Split the MySQL `GetOrSplitChunks` flow into smaller helper functions for chunk sizing, column selection, bound selection, and split strategy routing, enabling independent unit tests without changing existing chunking behavior. + +2. **Placeholder for unavailable TOAST values in Postgres CDC -**
    Added an `olake_unavailable_value` placeholder to mark unchanged TOAST columns whose values aren't available in PostgreSQL logical replication events. + +3. **Fixed discover failure for schema/table names containing "." -**
    Tables with a `.` in their schema or name (e.g. `user1.test_table`) broke discover, since namespace and name were joined into one string and later re-split on `.`, aborting discovery entirely. Fixed by carrying a `types.StreamID{Namespace, Name}` struct through the driver contract across all 8 drivers, avoiding the round-trip. + +4. **Fixed nil pointer panic in `DropStreams` -**
    `destination.DropStreams` panicked when a writer returned no shutdown callback, breaking clear-destination and any path relying on `DropStreams`. Fixed by guarding against a nil shutdown callback before invoking it. + +5. **Fixed CDC insert failure on GCS via S3-compat -**
    CDC syncs to Iceberg on GCS failed at commit because deleting an unwritten equality-delete file returned 404 on GCS, crashing the sync. Fixed by wrapping the S3 client so a delete on a missing key is treated as success, restoring AWS-like delete behavior only when a custom `s3_endpoint` is configured. + +6. **Fixed partition spec column casing mismatch in Iceberg -**
    Table creation failed when `use_source_column_names` was disabled, since destination schema columns were normalized to lowercase (e.g. `ID` → `id`) but the partition spec payload still carried the original source column name, causing the Java writer to fail building the `PartitionSpec` against the lowercased schema. Fixed by using the resolved destination column name when populating the partition spec. \ No newline at end of file diff --git a/docs/release/ingestion/v0.9.0.mdx b/docs/release/ingestion/v0.9.0.mdx new file mode 100644 index 000000000..ea84d444c --- /dev/null +++ b/docs/release/ingestion/v0.9.0.mdx @@ -0,0 +1,32 @@ +--- +title: "OLake Go (v0.9.0)" +--- + +# OLake Go (v0.9.0) +July 12, 2026 – July 24, 2026 + +## 🎯 What's New + +### Platform Features + +1. **OLake Go version in startup logs -**
    Added OLake Go version logging during application startup so the running version is immediately visible in the logs. This makes it easier to verify deployments, troubleshoot issues, and confirm the exact OLake Go version running in an environment. + +2. **Faster integration test startup -**
    Updated the OLake Go Docker image to use a shared custom base image across all drivers, reducing the time required to spawn fresh driver containers during integration tests. This speeds up test execution and improves the development workflow. + +3. **Simplified local development and testing -**
    Added new `make` targets to simplify local development and testing. Contributors can now start test databases, build drivers, and run integration, two-phase commit, and unit tests using the same workflow as CI, making it easier to set up a development environment and ensure consistent test execution. + +4. **Enhanced sync statistics -**
    Expanded `stats.json` to report both the number of records and the total source bytes processed during a sync, providing better visibility into data movement and sync performance. Metrics are now accurately tracked across retries to prevent overcounting, and CPU usage is included to help monitor sync performance. + +### Catalogs + +1. **Google BigLake catalog support -**
    Upgraded Apache Iceberg to v1.10.2, adding support for the Google BigLake catalog. Now we can configure BigLake catalogs directly from OLake Go, including the required Google authentication and catalog settings. + +### Destinations + +1. **Rolling file writes for the S3 destination -**
    Added rolling file writes for the S3 destination, allowing large partitions to be written as multiple size-bounded files instead of a single output file. This reduces memory usage during syncs and produces more manageable output files for large datasets. + +## 🔧 Bug Fixes & Stability + +1. **Preserved record order during concurrent filtering -**
    Fixed an issue where records could be processed out of order during concurrent filtering, potentially resulting in out-of-order CDC events. Filtering now preserves the original record order while retaining the performance benefits of concurrent evaluation. + +2. **Improved integration test reliability -**
    Fixed an issue where stale Iceberg writer processes could prevent subsequent integration tests from starting, causing port conflicts between test runs. Test environments now clean up leaked processes correctly, ensuring reliable sequential test execution. \ No newline at end of file diff --git a/docs/release/v0.6.0.mdx b/docs/release/v0.6.0.mdx deleted file mode 100644 index 09e832fec..000000000 --- a/docs/release/v0.6.0.mdx +++ /dev/null @@ -1,17 +0,0 @@ -# OLake (v0.6.0) -March 21, 2026 – March 24, 2026 - -## 🎯 What's New - -### Sources - -1. **Filters for CDC and incremental syncs -**
    Added filtering support for CDC and incremental syncs so users can selectively sync records based on configured conditions. - -2. **Postgres strict SSL verification support -**
    Added `verify-ca` and `verify-full` SSL support for Postgres using PEM certificate content input. Also removed support for passing `sslrootcert`, `sslcert`, and `sslkey` as file paths; these fields now expect the actual PEM content instead. - -## 🔧 Bug Fixes & Stability - -1. **Datatype utility unit tests -**
    Added unit test coverage for `utils/typeutils/datatype.go` across type detection, comparison, timestamp precision, and SQL type mapping, and fixed invalid `reflect.Value` handling to map to `types.Null`. - -2. **Oracle incremental cursor timezone fix -**
    Fixed incremental cursor handling for Oracle `TIMESTAMP` and `DATE` columns by stripping the session timezone offset before saving cursor values, so TZ-naive columns don’t shift and re-read already synced records. - diff --git a/docs/shared/config/GlueIcebergWriterCLIConfigDetails.mdx b/docs/shared/config/GlueIcebergWriterCLIConfigDetails.mdx index b3a2560f8..8e6fb25fd 100644 --- a/docs/shared/config/GlueIcebergWriterCLIConfigDetails.mdx +++ b/docs/shared/config/GlueIcebergWriterCLIConfigDetails.mdx @@ -5,11 +5,12 @@ | **aws_region** | `` | AWS region containing the S3 bucket and Glue Data Catalog resources. | | **aws_access_key**| `XXX` | AWS access key with sufficient permissions for S3 and Glue. Optional if using IAM role attached to running instance/pod. | | **aws_secret_key**| `XXX` | AWS secret key with sufficient permissions for S3 and Glue. Optional if using IAM role attached to running instance/pod. | -| **Catalog Name** | `olake_iceberg` | Enter a name for your catalog. | -| **S3 Endpoint** | `http://` | Endpoint for the S3 service. | -| **Custom Glue Endpoint Configuration** | `true`/`false` | Enable custom Glue endpoint configuration. | -| **Glue Catalog ID** | `123456789012` | AWS account ID used as the Glue Data Catalog identifier. | -| **Glue Access Key** | `XXX` | Access key for authenticating Glue catalog requests. Required when Glue credentials differ from S3 credentials. | -| **Glue Secret Key** | `XXX` | Secret key for authenticating Glue catalog requests. Required when Glue credentials differ from S3 credentials. | -| **Glue Endpoint** | `https://` | Custom endpoint URL for AWS Glue or a Glue-compatible catalog service. | -| **Glue Region** | `` | Region for the Glue catalog, if different from the S3 region. Falls back to AWS Region if not set. | \ No newline at end of file +| **catalog_name** | `olake_iceberg` | Enter a name for your catalog. | +| **s3_endpoint** | `http://` | Endpoint for the S3 service. | +| **glue_additional_config** | `true`/`false` | Enable custom Glue endpoint configuration. | +| **glue_catalog_id** | `123456789012` | AWS account ID used as the Glue Data Catalog identifier. | +| **glue_access_key** | `XXX` | Access key for authenticating Glue catalog requests. Required when Glue credentials differ from S3 credentials. | +| **glue_secret_key** | `XXX` | Secret key for authenticating Glue catalog requests. Required when Glue credentials differ from S3 credentials. | +| **glue_endpoint** | `https://` | Custom endpoint URL for AWS Glue or a Glue-compatible catalog service. | +| **glue_region** | `` | Region for the Glue catalog, if different from the S3 region. Falls back to AWS Region if not set. | +| **arrow_writes** | `false` | Writes data and delete files using Apache Arrow based writer and registers them in Iceberg. | diff --git a/docs/shared/config/GlueIcebergWriterConfig.mdx b/docs/shared/config/GlueIcebergWriterConfig.mdx index 0a829ed21..2d6da9b05 100644 --- a/docs/shared/config/GlueIcebergWriterConfig.mdx +++ b/docs/shared/config/GlueIcebergWriterConfig.mdx @@ -12,7 +12,8 @@ description: "Configure AWS Glue Data Catalog for OLake Iceberg writer. Setup Gl "iceberg_s3_path": "s3:///", "aws_region": "", "aws_access_key": "XXX", - "aws_secret_key": "XXX" + "aws_secret_key": "XXX", + "arrow_writes": false } } ``` diff --git a/docs/shared/config/GlueIcebergWriterUIConfigDetails.mdx b/docs/shared/config/GlueIcebergWriterUIConfigDetails.mdx index ac384830c..5eabdbacb 100644 --- a/docs/shared/config/GlueIcebergWriterUIConfigDetails.mdx +++ b/docs/shared/config/GlueIcebergWriterUIConfigDetails.mdx @@ -1,7 +1,7 @@ | Parameter | Sample Value | Description | |-------------------|------------------------------------------------|---------------------------------------------------------------------------------------------------------| -| **Iceberg S3 Path (Warehouse)** | `s3:///` | S3 bucket path where Iceberg table data and metadata files will be stored. | +| **S3 Path (Warehouse)** | `s3:///` | S3 bucket path where Iceberg table data and metadata files will be stored. | | **AWS Region** | `` | AWS region containing the S3 bucket and Glue Data Catalog resources. | | **AWS Access Key**| `XXX` | AWS access key ID for authentication. Optional if using IAM roles or instance profiles. | | **AWS Secret Key**| `XXX` | AWS secret access key for authentication. Optional if using IAM roles or instance profiles. | @@ -12,4 +12,5 @@ | **Glue Access Key** | `XXX` | Access key for authenticating Glue catalog requests. Required when Glue credentials differ from S3 credentials. | | **Glue Secret Key** | `XXX` | Secret key for authenticating Glue catalog requests. Required when Glue credentials differ from S3 credentials. | | **Glue Endpoint** | `https://` | Custom endpoint URL for AWS Glue or a Glue-compatible catalog service. | -| **Glue Region** | `` | Region for the Glue catalog, if different from the S3 region. Falls back to AWS Region if not set. | \ No newline at end of file +| **Glue Region** | `` | Region for the Glue catalog, if different from the S3 region. Falls back to AWS Region if not set. | +| **Enable Arrow Writes** | `false` | Writes data and delete files using Apache Arrow based writer and registers them in Iceberg. | diff --git a/docs/shared/config/HiveIcebergWriterCLIConfigDetails.mdx b/docs/shared/config/HiveIcebergWriterCLIConfigDetails.mdx index da37cf838..f89d9620e 100644 --- a/docs/shared/config/HiveIcebergWriterCLIConfigDetails.mdx +++ b/docs/shared/config/HiveIcebergWriterCLIConfigDetails.mdx @@ -6,8 +6,9 @@ | **aws_secret_key** | `XXX` | AWS secret key with sufficient permissions for S3. Optional if using IAM role attached to running instance/pod. | | **s3_endpoint** | `http://S3_ENDPOINT` | Specifies the endpoint URL for the S3 service. This may be used when connecting to an S3-compatible storage service like MinIO running on localhost. | | **hive_uri** | `thrift://:9083` or `thrift://METASTORE_IP:9083` | Defines the URI of the Hive Metastore service that the writer will connect to for catalog interactions. `METASTORE_IP` will be provided by GCP's Hive dataproc metastore or `thrift://localhost:9083` or `thrift://host.docker.internal:9083` if you are using local setup using docker compose | -| **catalog name** | `olake_iceberg` | Enter a name for your catalog. | +| **catalog_name** | `olake_iceberg` | Enter a name for your catalog. | | **s3_use_ssl** | `false` | Indicates whether SSL is enabled for S3 connections. "false" means that SSL is disabled for these communications. | | **s3_path_style** | `true` | Determines if path-style access is used for S3. "true" means that the writer will use path-style addressing instead of the default virtual-hosted style. | | **hive_clients** | `5` | Specifies the number of Hive clients allocated for managing interactions with the Hive Metastore. | | **hive_sasl_enabled** | `false` | Indicates whether SASL authentication is enabled for the Hive connection. "false" means that SASL is disabled. | +| **arrow_writes** | `false` | Writes data and delete files using Apache Arrow based writer and registers them in Iceberg. | \ No newline at end of file diff --git a/docs/shared/config/HiveIcebergWriterConfig.mdx b/docs/shared/config/HiveIcebergWriterConfig.mdx index defbda663..afa4efb06 100644 --- a/docs/shared/config/HiveIcebergWriterConfig.mdx +++ b/docs/shared/config/HiveIcebergWriterConfig.mdx @@ -18,7 +18,8 @@ description: "Configure Apache Hive Metastore catalog for OLake Iceberg writer. "s3_use_ssl": false, "s3_path_style": true, "hive_clients": 5, - "hive_sasl_enabled": false + "hive_sasl_enabled": false, + "arrow_writes": false } } ``` diff --git a/docs/shared/config/HiveIcebergWriterUIConfigDetails.mdx b/docs/shared/config/HiveIcebergWriterUIConfigDetails.mdx index 67c8b4a20..8170678c4 100644 --- a/docs/shared/config/HiveIcebergWriterUIConfigDetails.mdx +++ b/docs/shared/config/HiveIcebergWriterUIConfigDetails.mdx @@ -11,3 +11,4 @@ | **Use Path Style for S3** | `true` | Determines if path-style access is used for S3. "true" means that the writer will use path-style addressing instead of the default virtual-hosted style. | | **Hive Clients** | `5` | Specifies the number of Hive clients allocated for managing interactions with the Hive Metastore. | | **Enable SASL for Hive** | `false` | Indicates whether SASL authentication is enabled for the Hive connection. "false" means that SASL is disabled. | +| **Enable Arrow Writes** | `false` | Writes data and delete files using Apache Arrow based writer and registers them in Iceberg. | \ No newline at end of file diff --git a/docs/shared/config/LocalParquetConfig.mdx b/docs/shared/config/LocalParquetConfig.mdx index 10d716dcd..17cea2ce8 100644 --- a/docs/shared/config/LocalParquetConfig.mdx +++ b/docs/shared/config/LocalParquetConfig.mdx @@ -14,5 +14,6 @@ |---------------------------|---------------|---------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **type** | string | `"PARQUET"` | Specifies the output file format. Currently, only the Parquet format is supported. | | **writer.local_path** | string | `"./mnt/config"` | The directory in user's local machine where Parquet files will be stored. | +| **writer.max_file_size_mb** | number | `512` | *(Optional)* Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to `512` when unset. Accepts a positive number (integers e.g. `512`, `128` or fractional e.g. `0.1` for `100KB`). | > **Note:** This configuration enables the Parquet local writer. For more details, check out the [README section](https://github.com/datazip-inc/olake/blob/master/README.md). diff --git a/docs/shared/config/MinioJDBCIcebergWriterUIConfigLocalDetailsOLakeCLI.mdx b/docs/shared/config/MinioJDBCIcebergWriterUIConfigLocalDetailsOLakeCLI.mdx index d620698fb..fd733a8d6 100644 --- a/docs/shared/config/MinioJDBCIcebergWriterUIConfigLocalDetailsOLakeCLI.mdx +++ b/docs/shared/config/MinioJDBCIcebergWriterUIConfigLocalDetailsOLakeCLI.mdx @@ -2,7 +2,7 @@ | Parameter | Sample Value | Description | |---------------------|------------------------------------------------|---------------------------------------------------------------------------------------------------------| | **jdbc_url** | `jdbc:postgresql://DB_URL:5432/iceberg` | JDBC connection string for the catalog database. Replace `DB_URL` with your database host or use `host.docker.internal` for local Docker containers. | -| **Catalog Name** | `olake_iceberg` | Enter a name for your catalog. | +| **catalog_name** | `olake_iceberg` | Enter a name for your catalog. | | **jdbc_username** | `iceberg` | Database username for JDBC catalog authentication. | | **jdbc_password** | `password` | Database password for JDBC catalog authentication. | | **iceberg_s3_path** | `s3://warehouse` | S3-compatible storage path for Iceberg table data and metadata files. Use standard `s3://` protocol or `s3a://` in case you are using Minio. | @@ -12,3 +12,4 @@ | **aws_access_key** | `admin` | S3 access key ID for authentication. Use MinIO credentials for local development. | | **aws_region** | `us-east-1` | AWS region identifier for S3 bucket location. Required even for non-AWS S3-compatible services. | | **aws_secret_key** | `password` | S3 secret access key for authentication. Use MinIO credentials for local development. | +| **arrow_writes** | `false` | Writes data and delete files using Apache Arrow based writer and registers them in Iceberg. | \ No newline at end of file diff --git a/docs/shared/config/MinioJDBCIcebergWriterUIConfigLocalDetailsOLakeUI.mdx b/docs/shared/config/MinioJDBCIcebergWriterUIConfigLocalDetailsOLakeUI.mdx index 6c4b69aed..b354c3dff 100644 --- a/docs/shared/config/MinioJDBCIcebergWriterUIConfigLocalDetailsOLakeUI.mdx +++ b/docs/shared/config/MinioJDBCIcebergWriterUIConfigLocalDetailsOLakeUI.mdx @@ -11,4 +11,5 @@ | **Use Path Style for S3** | `true` | Use path-style S3 addressing (`endpoint/bucket/key`) instead of virtual-hosted style. Required for MinIO and some S3-compatible services. | | **AWS Access Key** | `admin` | S3 access key ID for authentication. Use MinIO credentials for local development. **Optional** if using IAM role attached to running instance/pod. | | **AWS Region** | `us-east-1` | AWS region identifier for S3 bucket location. Required even for non-AWS S3-compatible services. | -| **AWS Secret Key** | `password` | S3 secret access key for authentication. Use MinIO credentials for local development. **Optional** if using IAM role attached to running instance/pod. | \ No newline at end of file +| **AWS Secret Key** | `password` | S3 secret access key for authentication. Use MinIO credentials for local development. **Optional** if using IAM role attached to running instance/pod. | +| **Enable Arrow Writes** | `false` | Writes data and delete files using Apache Arrow based writer and registers them in Iceberg. | \ No newline at end of file diff --git a/docs/shared/config/MySQLToIcebergDatatypes.mdx b/docs/shared/config/MySQLToIcebergDatatypes.mdx index b2738699b..d044f4be8 100644 --- a/docs/shared/config/MySQLToIcebergDatatypes.mdx +++ b/docs/shared/config/MySQLToIcebergDatatypes.mdx @@ -10,6 +10,10 @@
    +:::info INT UNSIGNED mapping change +From OLake Go version `0.4.1` and above, `INT UNSIGNED` maps to `bigint` in the destination. In versions earlier than `0.4.1`, it maps to `int`. +::: + :::info timestamptz timezone -OLake always ingests timestamp data in UTC format, independent of the source timezone. +OLake Go always ingests timestamp data in UTC format, independent of the source timezone. ::: \ No newline at end of file diff --git a/docs/shared/config/OracleToIcebergDatatypes.mdx b/docs/shared/config/OracleToIcebergDatatypes.mdx index 3c307c1da..b3c100a32 100644 --- a/docs/shared/config/OracleToIcebergDatatypes.mdx +++ b/docs/shared/config/OracleToIcebergDatatypes.mdx @@ -12,5 +12,5 @@
    :::info timestamptz timezone -OLake always ingests timestamp data in UTC format, independent of the source timezone. +OLake Go always ingests timestamp data in UTC format, independent of the source timezone. ::: \ No newline at end of file diff --git a/docs/shared/config/PostgresSourceConfig.mdx b/docs/shared/config/PostgresSourceConfig.mdx index 71d051355..e55b129a2 100644 --- a/docs/shared/config/PostgresSourceConfig.mdx +++ b/docs/shared/config/PostgresSourceConfig.mdx @@ -12,6 +12,7 @@ description: "Configure PostgreSQL source connection for OLake. Setup logical re "password": "password", "jdbc_url_params": {"connectTimeout":"20"}, "retry_count": 3, + "schemas": ["public", "analytics"], "ssl": { "mode": "disable" }, diff --git a/docs/shared/config/PostgresToIcebergDatatypes.mdx b/docs/shared/config/PostgresToIcebergDatatypes.mdx index 4c59411e9..d3d997c5b 100644 --- a/docs/shared/config/PostgresToIcebergDatatypes.mdx +++ b/docs/shared/config/PostgresToIcebergDatatypes.mdx @@ -13,5 +13,5 @@ :::info timestamptz timezone -OLake always ingests timestamp data in UTC format, independent of the source timezone. +OLake Go always ingests timestamp data in UTC format, independent of the source timezone. ::: \ No newline at end of file diff --git a/docs/shared/config/RESTIcebergWriterConfig.mdx b/docs/shared/config/RESTIcebergWriterConfig.mdx index 63b7add0f..c64be661f 100644 --- a/docs/shared/config/RESTIcebergWriterConfig.mdx +++ b/docs/shared/config/RESTIcebergWriterConfig.mdx @@ -14,7 +14,8 @@ description: "Configure REST catalog for OLake Iceberg writer. Setup Polaris, Ne "s3_endpoint": "http://:9090", "aws_region": "", "aws_access_key": "", - "aws_secret_key": "" + "aws_secret_key": "", + "arrow_writes": false } } ``` \ No newline at end of file diff --git a/docs/shared/config/RESTIcebergWriterConfigDetails.mdx b/docs/shared/config/RESTIcebergWriterConfigDetails.mdx index 545d44e73..5241dfe8d 100644 --- a/docs/shared/config/RESTIcebergWriterConfigDetails.mdx +++ b/docs/shared/config/RESTIcebergWriterConfigDetails.mdx @@ -4,15 +4,16 @@ |----------------------|-------------------------------|----------------------------------------------------------------------------------------------------------------| | **catalog_type** | `rest` | Defines the catalog type used by the writer. "rest" means the writer interacts with a RESTful catalog service. | | **rest_catalog_url** | `http://:8181` | Specifies the endpoint URL for the REST catalog service that the writer will connect to. | -| **Catalog Name** | `olake_iceberg` | Enter a name for your catalog. | +| **catalog_name** | `olake_iceberg` | Enter a name for your catalog. | | **iceberg_s3_path** | `s3://` | Determines the S3 path or storage location for Iceberg data. | | **s3_endpoint** | `http://:9000` | Endpoint for the S3 service (Minio in this case). | | **aws_region** | `` | Specifies the AWS region associated with the S3 bucket where the data is stored. | | **aws_access_key** | `` | AWS access key (Optional). | | **aws_secret_key** | `` | AWS secret key (Optional). | +| **arrow_writes** | `false` | Writes data and delete files using Apache Arrow based writer and registers them in Iceberg. | :::note Catalog Name Supported for v0.3.5 and above -For the **catalog name**, OLake only supports lowercase letters and underscores. Spaces and special characters are not supported. +For the **catalog name**, OLake Go only supports lowercase letters and underscores. Spaces and special characters are not supported. :::
    diff --git a/docs/shared/config/RESTIcebergWriterUIConfigDetails.mdx b/docs/shared/config/RESTIcebergWriterUIConfigDetails.mdx index f07b768e8..a57544c36 100644 --- a/docs/shared/config/RESTIcebergWriterUIConfigDetails.mdx +++ b/docs/shared/config/RESTIcebergWriterUIConfigDetails.mdx @@ -4,14 +4,15 @@ |----------------------|-------------------------------|-----------------------------------------------------------------------------------------------------------------------| | **REST Catalog URL** | `http://:8181` | Specifies the endpoint URL for the REST catalog service that the writer will connect to. | | **Catalog Name** | `olake_iceberg` | Enter a name for your catalog. | -| **Iceberg S3 Path** | `s3://` | Determines the S3 path or storage location for Iceberg data. "warehouse" represents the designated storage directory. | +| **S3 Path** | `s3://` | Determines the S3 path or storage location for Iceberg data. "warehouse" represents the designated storage directory. | | **S3 Endpoint** | `http://:9000` | Endpoint for the S3 service. | | **AWS Region** | `` | Specifies the AWS region associated with the S3 bucket where the data is stored. | | **AWS Access Key** | `` | AWS access key (Optional). | | **AWS Secret Key** | `` | AWS secret key (Optional). | +| **Enable Arrow Writes** | `false` | Writes data and delete files using Apache Arrow based writer and registers them in Iceberg. | :::note Catalog Name Supported for v0.3.5 and above -For the **catalog name**, OLake only supports lowercase letters and underscores. Spaces and special characters are not supported. +For the **catalog name**, OLake Go only supports lowercase letters and underscores. Spaces and special characters are not supported. :::
    diff --git a/docs/shared/config/S3ConfigDetails.mdx b/docs/shared/config/S3ConfigDetails.mdx index e56bf6df5..5873b75e7 100644 --- a/docs/shared/config/S3ConfigDetails.mdx +++ b/docs/shared/config/S3ConfigDetails.mdx @@ -7,3 +7,4 @@ | **s3\_secret\_key** | The AWS/GCS HMAC secret key used for S3 authentication. This key should be kept secure. | string | A valid AWS/GCS HMAC secret key | | **s3\_endpoint** | *(Optional)* The **custom endpoint** for S3-compatible services. **Required** for **GCS** using HMAC keys. | string | `"https://storage.googleapis.com"` | | **s3\_path** | *(Optional)* The specific path (or prefix) within the S3 bucket where data files will be written. This is typically a folder path that starts with a `/` (e.g. `"/data"`). | string | A valid path string | +| **max\_file\_size\_mb** | *(Optional)* Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to `512` when unset. | number | A positive number (integers e.g. `512`, `128` or fractional e.g. `0.1` for `100KB`) | diff --git a/docs/shared/config/S3ConfigUIDetails.mdx b/docs/shared/config/S3ConfigUIDetails.mdx index 8d7631a08..352e13780 100644 --- a/docs/shared/config/S3ConfigUIDetails.mdx +++ b/docs/shared/config/S3ConfigUIDetails.mdx @@ -6,3 +6,4 @@ | **S3 Secret Key** | The AWS/MinIO/GCS HMAC secret key used for S3 authentication. This key should be kept secure. | string | A valid AWS/GCS HMAC secret key | | **S3 Path** | The specific path (or prefix) within the S3 bucket where data files will be written. This is typically a folder path that starts with a `/` (e.g. `"/data"`). | string | A valid path string | | **S3 Endpoint** | *(Optional)* Custom S3-compatible endpoint. **Required** when using **GCS** HMAC keys or **MinIO** S3. | string | `"https://storage.googleapis.com"`, `"https://:9000"` | +| **Max File Size (MB)** | *(Optional)* Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to `512` when unset. | number | A positive number (integers e.g. `512`, `128` or fractional e.g. `0.1` for `100KB`) | diff --git a/docs/shared/config/S3ToIcebergDatatypes.mdx b/docs/shared/config/S3ToIcebergDatatypes.mdx index c66e2a5f4..db2b0aca5 100644 --- a/docs/shared/config/S3ToIcebergDatatypes.mdx +++ b/docs/shared/config/S3ToIcebergDatatypes.mdx @@ -30,5 +30,5 @@ ::: :::info timestamptz timezone -OLake always ingests timestamp data in UTC format, independent of the source timezone. +OLake Go always ingests timestamp data in UTC format, independent of the source timezone. ::: diff --git a/docs/understanding/compatibility-catalogs.mdx b/docs/understanding/compatibility-catalogs.mdx index f0f518218..415a7d5e0 100644 --- a/docs/understanding/compatibility-catalogs.mdx +++ b/docs/understanding/compatibility-catalogs.mdx @@ -6,7 +6,7 @@ sidebar_label: Compatibility to Iceberg Catalogs # Compatibility to Iceberg Catalogs -OLake supports multiple Iceberg catalog implementations, including [REST catalog](/docs/writers/iceberg/catalog/rest/), [Hive Metastore](/docs/writers/iceberg/catalog/hive/), and [JDBC Catalog](/docs/writers/iceberg/catalog/jdbc/), letting you choose the one that best fits your environment. The table below shows the supported catalogs at a glance, with links to their setup guides. +OLake Go supports multiple Iceberg catalog implementations, including [REST catalog](/docs/writers/iceberg/catalog/rest/), [Hive Metastore](/docs/writers/iceberg/catalog/hive/), and [JDBC Catalog](/docs/writers/iceberg/catalog/jdbc/), letting you choose the one that best fits your environment. The table below shows the supported catalogs at a glance, with links to their setup guides. | | Catalog | Link | | ----------------------------------------------------------------------------------------- | ------------------- | ------------------------------------------------------------------ | @@ -16,6 +16,6 @@ OLake supports multiple Iceberg catalog implementations, including [REST catalog | Polaris logo | **REST Polaris** | [Link](/docs/writers/iceberg/catalog/rest?rest-catalog=polaris) | | Unity Catalog logo | **REST Unity** | [Link](/docs/writers/iceberg/catalog/rest?rest-catalog=unity) | | Lakekeeper logo | **REST Lakekeeper** | [Link](/docs/writers/iceberg/catalog/rest?rest-catalog=lakekeeper) | -| Amazon S3 logo | **REST S3** | [Link](/docs/writers/iceberg/catalog/rest?rest-catalog=s3-tables) | +| Amazon S3 logo | **S3 Tables** | [Link](/docs/writers/iceberg/catalog/rest?rest-catalog=s3-tables) | | JDBC logo | **JDBC** | [Link](/docs/writers/iceberg/catalog/jdbc) | | Apache Hive logo | **Hive Metastore** | [Link](/docs/writers/iceberg/catalog/hive) | diff --git a/docs/understanding/compatibility-engines.mdx b/docs/understanding/compatibility-engines.mdx index 8eca809b4..361afe624 100644 --- a/docs/understanding/compatibility-engines.mdx +++ b/docs/understanding/compatibility-engines.mdx @@ -4,42 +4,44 @@ description: "Explore OLake support for querying Iceberg tables using Athena, Sp sidebar_label: Compatibility with Query Engines --- -# Compatibility with Query Engines - -You can query OLake Iceberg tables from multiple engines. The table below shows catalog compatibility at a glance, with links to setup guides. - -| Query Tool | AWS Glue | Hive Metastore | JDBC | REST | Docs | -|-----------------------|--------|--------|--------|--------|-------------------------------------------------------------------------------------------| -| Amazon Athena | ✅ | ❌ | ❌ | ❌ | [AWS Docs](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html) | -| Apache Spark (v3.3+) | ✅ | ✅ | ✅ | ✅ | [Spark Docs](https://iceberg.apache.org/docs/latest/spark-configuration/) | -| Apache Flink (v1.18+) | ✅ | ✅ | ✅ | ✅ | [Flink Docs](https://iceberg.apache.org/docs/latest/flink-configuration/) | -| Trino (v475 +) | ✅ | ✅ | ✅ | ✅ | [Trino Docs](https://trino.io/docs/current/object-storage/metastores.html) | -| Starburst Enterprise | ✅ | ✅ | ✅ | ✅ | [Starburst Docs](https://docs.starburst.io/latest/object-storage/metastores.html) | -| [Presto](/blog/building-open-data-lakehouse-with-olake-presto) (v0.288 +) | ✅ | ✅ | ✅ | ✅ | [Presto Guide](https://ibm.github.io/presto-iceberg-lab/lab-1/) | -| Apache Hive (v4.0) | ✅ | ✅ | ❌ | ✅ | [Hive Docs](https://iceberg.apache.org/docs/latest/hive/) | -| Apache Impala (v4.4) | ❌ | ✅ | ❌ | ❌ | [Impala Docs](https://impala.apache.org/docs/build/html/topics/impala_iceberg.html) | -| Dremio (v25/26) | ✅ | ✅ | ❌ | ✅ | [Dremio Docs](https://docs.dremio.com/current/release-notes/version-260-release/) | -| DuckDB (v1.2.1) | ❌ | ❌ | ❌ | ✅ | [DuckDB Docs](https://duckdb.org/docs/stable/extensions/iceberg/overview.html) | -| ClickHouse (v24.3 +) | ❌ | ❌ | ❌ | ✅ | [ClickHouse Docs](https://clickhouse.com/docs/engines/table-engines/integrations/iceberg) | -| StarRocks (v3.2 +) | ❌ | ✅ | ❌ | ✅ | [StarRocks Docs](https://docs.starrocks.io/docs/quick_start/iceberg/) | -| Apache Doris (v2.1 +) | ✅ | ✅ | ❌ | ✅ | [Doris Docs](https://doris.apache.org/docs/dev/lakehouse/best-practices/doris-iceberg) | -| BigQuery (BigLake) | ❌ | ❌ | ❌ | ❌ | [BigQuery Docs](https://cloud.google.com/bigquery/docs/iceberg-external-tables) | -| Snowflake (GA) | ❌ | ❌ | ❌ | ✅ | [Snowflake Docs](https://docs.snowflake.com/en/user-guide/tables-iceberg) | -| Databricks (Unity) | ❌ | ❌ | ❌ | ✅ | [Databricks Docs](https://docs.databricks.com/external-access/iceberg) | - - +# OLake Go Compatibility with Query Engines + +You can query OLake Go Iceberg tables from multiple engines. The table below shows catalog compatibility at a glance, with links to setup guides. + +| Query Tool | AWS Glue | Hive Metastore | JDBC | REST Generic | Lakekeeper | Unity | Polaris | S3 Tables | Nessie | Docs | +|-----------------------|--------|--------|--------|--------|------------|-------|---------|-----------|--------|-------------------------------------------------------------------------------------------| +| Amazon Athena | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | [AWS Docs](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html) | +| Apache Spark (v3.3+) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | [Spark Docs](https://iceberg.apache.org/docs/latest/spark-configuration/) | +| Apache Flink (v1.18+) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | [Flink Docs](https://iceberg.apache.org/docs/latest/flink-configuration/) | +| Trino (v475 +) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | [Trino Docs](https://trino.io/docs/current/object-storage/metastores.html) | +| Starburst Enterprise | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | [Starburst Docs](https://docs.starburst.io/latest/object-storage/metastores.html) | +| [Presto](/blog/building-open-data-lakehouse-with-olake-presto) (v0.288 +) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | [Presto Guide](https://ibm.github.io/presto-iceberg-lab/lab-1/) | +| Apache Hive (v4.2.0) | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | [Hive Docs](https://iceberg.apache.org/docs/latest/hive/) | +| Apache Impala (v5.0) | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ | [Impala Docs](https://impala.apache.org/docs/build/html/topics/impala_iceberg.html) | +| Dremio (v25/26) | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | [Dremio Docs](https://docs.dremio.com/dremio-cloud/bring-data/connect/catalogs/) | +| DuckDB (v1.2.1) | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | [DuckDB Docs](https://duckdb.org/docs/stable/extensions/iceberg/overview.html) | +| ClickHouse (v25.8 +) | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | [ClickHouse Docs](https://clickhouse.com/docs/engines/table-engines/integrations/iceberg) | +| StarRocks (v4.0 +) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | [StarRocks Docs](https://docs.starrocks.io/docs/quick_start/iceberg/) | +| Apache Doris (v3.1 +) | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | [Doris Docs](https://doris.apache.org/docs/dev/lakehouse/best-practices/doris-iceberg) | +| BigQuery (BigLake) | ✅ | ❌ | ❌ | ❌ | ❌ | ⚠️ | ❌ | ❌ | ❌ | [BigQuery Docs](https://docs.cloud.google.com/lakehouse/docs/about-cross-cloud-lakehouse) | +| Snowflake (GA) | ⚠️ | ❌ | ❌ | ⚠️ | ⚠️ | ⚠️ | ⚠️ | ⚠️ | ⚠️ | [Snowflake Docs](https://docs.snowflake.com/en/user-guide/tables-iceberg) | +| Databricks (Spark) | ✅ | ✅ | ✅ | ✅ | ✅ | ⚠️ | ✅ | ✅ | ✅ | [Databricks Docs](https://docs.databricks.com/external-access/iceberg) | +| AWS Redshift | ✅ | ❌ | ❌ | ❌ | ❌ | ⚠️ | ✅ | ✅ | ❌ | [Redshift Docs](https://docs.aws.amazon.com/redshift/latest/dg/querying-iceberg.html) | +| Azure Synapse Analytics | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | [Synapse Docs](https://learn.microsoft.com/en-us/azure/data-factory/format-iceberg) | + +:::info Symbol Guide +- ✅ — Query Engine connects to this catalog and reliably reads what OLake Go writes +- ⚠️ — Query engine connects to this catalog but can only read **append-only** OLake Go writes +- ❌ — Query engine doesn't support this catalog type +::: ### Important Notes - -- **Amazon Athena** → Supports Iceberg v2 tables **only when registered in Glue**. + - **Presto** → REST catalog supported from **v0.288+**; Glue requires the AWS SDK fat-jar. -- **Apache Hive v4.0** → Glue requires adding the AWS bundle. -- **Impala** → Works with Hive Metastore; Glue/REST only via Hive federation. -- **Dremio** → Supports Polaris/REST/Nessie natively; JDBC not supported. -- **DuckDB** → Supports REST catalogs (Nessie, Tabular); Glue/Hive not yet supported. -- **ClickHouse** → Iceberg tables are **read-only**; REST support stable from v24.12+. -- **BigQuery** → Reads Iceberg manifests directly, without a catalog. -- **Snowflake** → Can read external REST catalogs, but they are **read-only**. -- **Databricks Unity Catalog** → REST endpoint allows federation with Glue, Hive, and Snowflake catalogs. +- **Apache Hive v4.2.0** → Glue requires adding the AWS bundle. +- **Impala** → Reliable support for Hive Metastore; limited REST catalog support (read only). +- **ClickHouse** → Full read support; experimental Iceberg write support introduced in v25.7+ +- **Snowflake** → Can read external REST catalogs, but tables are read-only for external catalog workflows. +- **Azure Synapse Analytics** → Iceberg support via Spark pools requires manual runtime configuration. diff --git a/docs/understanding/terminologies/general.mdx b/docs/understanding/terminologies/general.mdx index 082135b26..3742462ad 100644 --- a/docs/understanding/terminologies/general.mdx +++ b/docs/understanding/terminologies/general.mdx @@ -1,67 +1,209 @@ --- title: "Key Data Engineering Terminologies Explained- A Glossary" description: "Understand key terms like data lake, lakehouse, CDC, schema evolution, and concurrency in this OLake glossary for data engineering and replication." -sidebar_label: General Terminologies +sidebar_label: Terminologies --- -# General Terminologies +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import OlakeTerminologies from '@site/docs/understanding/terminologies/olake.mdx'; +import TOCTabLinker from '@site/src/components/TOCTabLinker'; + + + + + + +## OLake Go Terminologies + +## 1. Source + +A Source is the system from which OLake Go reads data. This could be a database (MongoDB, Postgres, MySQL, Oracle), basically a data service. When you create a source in OLake Go, you are telling the platform where the data should come from. + +| Concept | Description | +|--------|--------------| +| **Active source** | Linked to at least one job; OLake Go reads from this source and sends data to a destination. | +| **Inactive source** | Created in OLake Go but not assigned to any job yet; no data is read until a job uses it. | + + +## 2. Destination + +A Destination is the system where OLake Go writes data after it has been extracted from a source. In OLake Go, destinations define where your data will be stored and in what format. Currently, OLake supports two types of destinations: Amazon S3 and Apache Iceberg. + +| Concept | Description | +|--------|--------------| +| **Active destination** | Assigned to at least one job; OLake Go is delivering data into this destination. | +| **Inactive destination** | Created in OLake Go but not linked to any job yet; no data until a job uses it. | + + +## 3. Jobs + +A Job is the pipeline or process in OLake Go that moves data from a source to a destination. A job defines what data is moved, how it is moved (full refresh, incremental, CDC), and where it is delivered. Jobs are the central element of OLake, as they connect sources and destinations. + +| Concept | Description | +|--------|--------------| +| **Active job** | Running or scheduled; OLake Go transfers data per the job configuration. Newly created jobs appear here while active. | +| **Inactive job** | Paused; no transfer until resumed. Configuration and state are kept. | +| **Saved job** | Saved configuration used to start new runs; when scheduled or run, it appears under Active Jobs. | +| **Failed job** | Execution error (for example network, schema, or permissions); needs investigation. | + +:::note Restrictions for inactive jobs +When a job is inactive, certain job-level features are unavailable: + +- **Sync Now** — cannot trigger immediate syncs +- **Edit Streams** — stream configuration cannot be modified +- **Clear Destination** — cannot clear destination data + +To use these features, resume the job to move it back to active status. +::: + + +## 4. OLake Go-Generated Columns + +When OLake Go replicates data from source (like PostgreSQL, MySQL, Oracle or MongoDB) to destination formats (Apache Iceberg, Parquet), it automatically adds several metadata columns to track the lifecycle and processing history of each record. These columns help users understand how and when each row was captured and written to the destination. + +OLake Go adds these metadata columns to every destination table: `_op_type`, `_olake_id`, `_olake_timestamp`, and `_cdc_timestamp`. + +### 1. Operation Types (`_op_type`): + - **Read/Snapshot (`r`) -** + - Appears during initial full table loads (snapshots) from the source database. + - **Example**: When you first sync a table i.e. full refresh, all existing rows get `_op_type = 'r'`. + - **Create/Insert (`c`) -** + - Generated during CDC, when new records are inserted into the source database. + - **Example**: After initial sync, inserting a new record creates a row with `_op_type = 'c'`. + - **Update (`u`) -** + - Created when existing records are modified in the source database. + - **Example**: Updating a column value from the table generates a new row with `_op_type = 'u'`. + - **Delete (`d`) -** + - Generated when records are deleted from the source database. + - When a record is deleted, primary key column and OLake-generated columns retain their values for tracking, while all other columns are set to either `None` or they appear blank. + - **Example**: Deleting a record from the table creates a tombstone row with `_op_type = 'd'`. + +### 2. Timestamps: + - `_olake_timestamp` - + - Captures the exact time when OLake Go processed and wrote the record to the destination. + - Useful for tracking when data was ingested into the data lake and for debugging sync latency and understanding processing order. + - `_cdc_timestamp` - + - Reflects the actual time when the change occurred in the source database. + - Present only when using Update Method (during Source Configuration) as `CDC`. + - Whenever a full refresh is performed, snapshot records `(_op_type = 'r')` will have the `_cdc_timestamp` set to the epoch time **(1970-01-01)** indicating that it is not a CDC record. Even in (Full Refresh+ CDC) mode, the very first sync run starts with a full refresh — in this case too, you may see **(1970-01-01)** timestamps. + - For CDC records `(_op_type = 'c', 'u', 'd')`, it provides the precise timestamp of when the insert, update, or delete happened in the source. + +### 3. Record Identification (`_olake_id`): + OLake Go assigns a unique, deterministic identifier to every record as it is processed. The identifier is designed primarily for deduplication and internal ordering/tracking. + + #### How it is generated: + - **Single primary key present in the source table -** + - `_olake_id` equals the record’s primary key value and duplicates on the key will be detected. + - **Example:** If the primary key is `id` with value 123, then `_olake_id` will be 123. + - **Composite primary key present in the source table -** + - `_olake_id` is a stable hash of all primary key columns for the record. Hence ensures that duplicates on the key will be detected. + - **Example:** If the composite primary key is `(order_id, product_id)` with values (456, 789), then `_olake_id` will be a hash of (456, 789). + - **With no primary key in the source table -** + - `_olake_id` equals the hash of all columns of the record. But in this situation, deduplication cannot be guaranteed as two different records could have the same hash value. + - **Example:** If a record has columns `(name, age, city)` with values (Alice, 30, NY), then `_olake_id` will be hash of (Alice, 30, NY). + +### 4. CDC Metadata Columns + +OLake Go also writes **driver-specific CDC ordering metadata columns**. These fields capture the exact log position and ordering information from each source system’s native CDC mechanism and is supported for **MongoDB**, **Postgres**, **MySQL** and **MSSQL** drivers. + +These columns are especially useful in scenarios where multiple changes happen within the same transaction (and thus share a single `_cdc_timestamp`), because they provide a stable, log-based key (for example, binlog position or LSN) that downstream systems can use to order events exactly as they occurred in the source. + +#### MongoDB + +- **Column:** `_cdc_resume_token` + - `_cdc_resume_token` stores the MongoDB Change Streams resume token that identifies the exact position in the oplog where this change event was captured. + - **Example:** `_cdc_resume_token:"82698C3243000000022B0429296E1404"` + +#### MySQL + +- **Columns:** `_cdc_binlog_file_name`, `_cdc_binlog_file_pos` + - Together, these columns identify the exact location in MySQL's binary log where the change event was read. `_cdc_binlog_file_name` specifies which binlog file, and `_cdc_binlog_file_pos` indicates the byte position within that file. + - **Example:** `_cdc_binlog_file_name = "mysql-bin.000003"`, `_cdc_binlog_file_pos = 1027` + +#### Postgres + +- **Column:** `_cdc_lsn` + - `_cdc_lsn` stores the Write-Ahead Log (WAL) Log Sequence Number that represents the precise position in Postgres's transaction log where this change was recorded. LSN values are monotonically increasing and can be used to order events in the exact sequence they were applied by Postgres. + - **Example:** `_cdc_lsn = "16/B374D848"` + +#### MSSQL + +- **Columns:** `_cdc_start_lsn`, `_cdc_seqval` + - `_cdc_start_lsn` indicates the starting Log Sequence Number for the change row in SQL Server's CDC change table, while `_cdc_seqval` is a sequence value that orders multiple changes that share the same LSN. Together, they provide a stable ordering key that matches SQL Server's internal CDC ordering. + - **Example:** `_cdc_start_lsn = "000000bb000055d00003"`, `_cdc_seqval = "000000bb000055d00002"` + +:::note Backward compatibility +This feature is available starting from **v0.3.16**.
    For **jobs created before** this feature was released, these columns are always present for Parquet-on-S3 destinations, but for **Iceberg** they appear only when Normalization was turned **on** and will not be present for older jobs where Iceberg Normalization was **off**. +::: + +     + + +## General Terminologies ### 1. Data Lake: -A Data Lake is a central storage system that holds large volumes of raw or processed data in open formats such as Parquet, CSV, or JSON. In OLake, data is written to object stores like Amazon S3 in Parquet format, making it easy to use these storage systems as data lakes for analytics and processing. +A Data Lake is a central storage system that holds large volumes of raw or processed data in open formats such as Parquet, CSV, or JSON. In OLake Go, data is written to object stores like Amazon S3 in Parquet format, making it easy to use these storage systems as data lakes for analytics and processing. ### 2. Data Lakehouse: -A Data Lakehouse is a modern architecture that combines the scalability and flexibility of data lakes with the reliability and performance features of data warehouses, such as transactions, schema enforcement, and query optimization. In OLake, this is achieved through Apache Iceberg, which adds ACID transactions and schema evolution on top of data lakes, enabling a true lakehouse environment. +A Data Lakehouse is a modern architecture that combines the scalability and flexibility of data lakes with the reliability and performance features of data warehouses, such as transactions, schema enforcement, and query optimization. In OLake Go, this is achieved through Apache Iceberg, which adds ACID transactions and schema evolution on top of data lakes, enabling a true lakehouse environment. ### 3. Data Warehouse: Data Warehouse is a system where cleaned and structured data from many sources is stored for analysis and reporting. ### 4. Snapshot: -A Snapshot is a one-time, point-in-time capture of the entire dataset from the source. In OLake, an initial snapshot is taken when a table or collection is first onboarded. +A Snapshot is a one-time, point-in-time capture of the entire dataset from the source. In OLake Go, an initial snapshot is taken when a table or collection is first onboarded. ### 5. Polymorphic (or Heterogeneous) Data: Polymorphic (or Heterogeneous) Data means data in the same dataset that doesn’t always follow the exact same structure. This is common in NoSQL systems like MongoDB, where one record might have extra fields that another record doesn’t. For instance, in an e-commerce database, one product document might include a “color” field while another doesn’t, and tools need to handle both without errors. ### 6. Chunking: -Chunking (or Parallel Chunk-Based Loading) is the process of splitting a large dataset, such as a table or collection, into smaller segments called chunks. In OLake, all the chunks are first created, and then processed in parallel during full refresh operations. This approach makes it possible to load large collections more efficiently, significantly reducing the overall time needed for high-volume data transfers. +Chunking (or Parallel Chunk-Based Loading) is the process of splitting a large dataset, such as a table or collection, into smaller segments called chunks. In OLake Go, all the chunks are first created, and then processed in parallel during full refresh operations. This approach makes it possible to load large collections more efficiently, significantly reducing the overall time needed for high-volume data transfers. ### 7. Thread Count/Concurrency: -Thread Count (or Concurrency) is the number of parallel processes or threads used to read, transform, and write data simultaneously. Increasing concurrency often speeds up ingestion and improves throughput, but it can also put additional load on the source system. In OLake, this is managed using max_threads in the CLI or Max Threads in the UI, which defines how many chunks or streams are processed in parallel. +Thread Count (or Concurrency) is the number of parallel processes or threads used to read, transform, and write data simultaneously. Increasing concurrency often speeds up ingestion and improves throughput, but it can also put additional load on the source system. In OLake Go, this is managed using max_threads in the CLI or Max Threads in the UI, which defines how many chunks or streams are processed in parallel. ### 8. Writer: -A Writer in OLake is the component responsible for formatting and writing data from the source to the chosen destination, such as Parquet files or Apache Iceberg tables. By writing directly from the source to the destination, OLake removes the need for an intermediate data queue, which reduces latency and simplifies the overall pipeline. +A Writer in OLake Go is the component responsible for formatting and writing data from the source to the chosen destination, such as Parquet files or Apache Iceberg tables. By writing directly from the source to the destination, OLake Go removes the need for an intermediate data queue, which reduces latency and simplifies the overall pipeline. ### 9. Parquet: -Parquet is a columnar storage file format designed for efficient data compression and encoding, making it highly optimized for analytical workloads. In OLake, data is stored in Parquet format so that downstream systems like Spark or Trino, can query it efficiently, improving both performance and storage utilisation. +Parquet is a columnar storage file format designed for efficient data compression and encoding, making it highly optimized for analytical workloads. In OLake Go, data is stored in Parquet format so that downstream systems like Spark or Trino, can query it efficiently, improving both performance and storage utilisation. ### 10. Apache Iceberg: -Apache Iceberg is an open table format built for large-scale analytics that provides features like ACID transactions, schema evolution, partitioning, and time travel. In OLake, the Iceberg Writer outputs data as Iceberg-compatible Parquet files, allowing users to build a true Lakehouse architecture with reliable transaction semantics and efficient query performance. +Apache Iceberg is an open table format built for large-scale analytics that provides features like ACID transactions, schema evolution, partitioning, and time travel. In OLake Go, the Iceberg Writer outputs data as Iceberg-compatible Parquet files, allowing users to build a true Lakehouse architecture with reliable transaction semantics and efficient query performance. ### 11. Equality Deletes (Iceberg): -Equality Deletes in Apache Iceberg are a way to handle row-level deletions by matching specific record attributes, such as primary keys, to identify which rows should be removed. In OLake, equality deletes are used to implement upserts, ensuring that changes captured through CDC are correctly applied so that updated or deleted rows are accurately reflected in the Iceberg table. +Equality Deletes in Apache Iceberg are a way to handle row-level deletions by matching specific record attributes, such as primary keys, to identify which rows should be removed. In OLake Go, equality deletes are used to implement upserts, ensuring that changes captured through CDC are correctly applied so that updated or deleted rows are accurately reflected in the Iceberg table. ### 12. Real Time Replication: -Real-Time Replication is the process of continuously capturing and applying database changes with minimal delay, ensuring the destination stays closely in sync with the source. Change Data Capture (CDC) in OLake captures and applies database changes from the source to the destination. By default, CDC runs as a job — it captures recent changes, replicates them, and then stops. However, when combined with orchestration tools (such as Airflow), these jobs can be scheduled at frequent intervals, effectively enabling continuous or near real-time replication to keep destinations closely in sync with sources. +Real-Time Replication is the process of continuously capturing and applying database changes with minimal delay, ensuring the destination stays closely in sync with the source. Change Data Capture (CDC) in OLake Go captures and applies database changes from the source to the destination. By default, CDC runs as a job — it captures recent changes, replicates them, and then stops. However, when combined with orchestration tools (such as Airflow), these jobs can be scheduled at frequent intervals, effectively enabling continuous or near real-time replication to keep destinations closely in sync with sources. ### 13. State Management: -State Management is the practice of tracking metadata about processed data such as the last CDC offset or the last completed chunk so that jobs can resume or continue without losing or duplicating records. In OLake, this is handled through a state file that stores checkpoint offsets or timestamps, allowing interrupted processes to restart from the exact point of failure and ensuring data consistency. +State Management is the practice of tracking metadata about processed data such as the last CDC offset or the last completed chunk so that jobs can resume or continue without losing or duplicating records. In OLake Go, this is handled through a state file that stores checkpoint offsets or timestamps, allowing interrupted processes to restart from the exact point of failure and ensuring data consistency. ### 14. Catalog: A catalog in Apache Iceberg is the metadata and namespace service that manages Iceberg tables. It acts as the central registry where tables are created, organized, and discovered . It stores table metadata locations (not the data itself) and provides a namespace structure (like databases & schemas in SQL). ### 15. Flattening (JSON Flattening): -JSON Flattening converts nested JSON fields into top-level columns for simpler and faster querying. OLake currently supports Level-1 JSON flattening through its ‘Normalization’ feature, converting top-level nested fields into separate columns for easier querying. +JSON Flattening converts nested JSON fields into top-level columns for simpler and faster querying. OLake Go currently supports Level-1 JSON flattening through its ‘Normalization’ feature, converting top-level nested fields into separate columns for easier querying. ### 16. Performance Benchmarks: -Performance Benchmarks are structured tests that measure how efficiently a system processes data under defined conditions, such as dataset size, concurrency, or hardware resources. In OLake, benchmarks focus on metrics like throughput (rows per second) and total load times for large datasets, often demonstrating faster ingestion and replication compared to traditional ETL or CDC tools. +Performance Benchmarks are structured tests that measure how efficiently a system processes data under defined conditions, such as dataset size, concurrency, or hardware resources. In OLake Go, benchmarks focus on metrics like throughput (rows per second) and total load times for large datasets, often demonstrating faster ingestion and replication compared to traditional ETL or CDC tools. ### 17. Monitoring & Alerting: -Monitoring & Alerting refers to the practices and tools used to track system activity, capture metrics, and generate notifications when issues or anomalies occur. In OLake, monitoring provides real-time visibility into all sync modes, while alerting notifies users about events like schema changes in the source or job failures, ensuring problems are quickly identified and addressed. +Monitoring & Alerting refers to the practices and tools used to track system activity, capture metrics, and generate notifications when issues or anomalies occur. In OLake Go, monitoring provides real-time visibility into all sync modes, while alerting notifies users about events like schema changes in the source or job failures, ensuring problems are quickly identified and addressed. ### 18. BYOC (Bring Your Own Cloud): -BYOC (Bring Your Own Cloud) is the approach of running software within the user’s chosen cloud or infrastructure instead of being tied to a vendor-managed environment. In OLake, this means the platform is cloud-agnostic, supporting deployments across AWS, GCP, Azure, or even on-premises setups, giving users flexibility without vendor lock-in. +BYOC (Bring Your Own Cloud) is the approach of running software within the user’s chosen cloud or infrastructure instead of being tied to a vendor-managed environment. In OLake Go, this means the platform is cloud-agnostic, supporting deployments across AWS, GCP, Azure, or even on-premises setups, giving users flexibility without vendor lock-in. ### 19. Query Engines (Trino, Spark, Flink, Snowflake): -Trino, Spark, Flink, Snowflake are popular data processing and query engines that can work directly with open data formats like Parquet and Iceberg. In OLake, writing data in these open formats ensures seamless compatibility, allowing these engines to query and process the data without requiring proprietary connectors or vendor lock-in. +Trino, Spark, Flink, Snowflake are popular data processing and query engines that can work directly with open data formats like Parquet and Iceberg. In OLake Go, writing data in these open formats ensures seamless compatibility, allowing these engines to query and process the data without requiring proprietary connectors or vendor lock-in. ### 20. gRPC: gRPC is a communication framework created by Google that lets different services talk to each other quickly and efficiently. It's often used in systems where data needs to move in real time, like event-driven applications or data pipelines. In data engineering, gRPC helps microservices share data safely and at high speed, supports streaming of large datasets (like logs or analytics data), and is also used to connect with machine learning models for fast predictions. @@ -127,10 +269,13 @@ CDC is a mode where only changes in the source database—inserts, updates, and Full Refresh is the process of reloading an entire table or collection from the source into the destination. This ensures the destination is a complete, up-to-date copy of the source, but it can be time-consuming and resource-intensive for large datasets. Full refresh is typically used for the first load of a dataset or when incremental tracking is not possible. ### 41. Incremental: -Incremental replication is a method of loading only the new or updated records from a source system since the last successful run, using a tracking column such as a timestamp or an incrementing ID (cursor key). In OLake, incremental replication works through a cursor-based approach. Each run tracks the highest cursor_key value (for example, last_updated_at or an increasing primary key) that was processed previously. On the next run, OLake fetches only the rows where cursor_key is greater than the saved checkpoint and appends those records to the destination. This avoids reloading the full dataset and makes ongoing synchronization more efficient. +Incremental replication is a method of loading only the new or updated records from a source system since the last successful run, using a tracking column such as a timestamp or an incrementing ID (cursor key). In OLake Go, incremental replication works through a cursor-based approach. Each run tracks the highest cursor_key value (for example, last_updated_at or an increasing primary key) that was processed previously. On the next run, OLake Go fetches only the rows where cursor_key is greater than the saved checkpoint and appends those records to the destination. This avoids reloading the full dataset and makes ongoing synchronization more efficient. ### 42. Position Deletes (Iceberg): Position deletes in Apache Iceberg identify which rows to remove by file path and row position within that file, rather than by matching column values. They are used when the table has no unique key or when deletes are applied at write time. During compaction, position deletes are merged into the data files so that deleted rows are physically removed and storage can be reclaimed. ### 43. Small Files: -Small files are data files that are much smaller than the ideal size for efficient querying and storage (for example, well below the target or block size). Having many small files can hurt query performance (more metadata and I/O) and increase storage overhead. Iceberg maintenance (compaction or optimization) combines small files into larger ones so that reads and scans are more efficient. \ No newline at end of file +Small files are data files that are much smaller than the ideal size for efficient querying and storage (for example, well below the target or block size). Having many small files can hurt query performance (more metadata and I/O) and increase storage overhead. Iceberg maintenance (compaction or optimization) combines small files into larger ones so that reads and scans are more efficient. + + +     \ No newline at end of file diff --git a/docs/understanding/terminologies/olake.mdx b/docs/understanding/terminologies/olake.mdx index 7e67eaa6f..f304045eb 100644 --- a/docs/understanding/terminologies/olake.mdx +++ b/docs/understanding/terminologies/olake.mdx @@ -6,13 +6,13 @@ sidebar_position: 2 ## Streams -A Stream in OLake represents a unit of data (such as a table or collection) discovered from a source. This panel lets you choose which streams to sync, how the data is synced (i.e. you can choose from different sync modes), what schemas they use, and also allows the user to do partitioning on data before loading it into the destination. +A Stream in OLake Go represents a unit of data (such as a table or collection) discovered from a source. This panel lets you choose which streams to sync, how the data is synced (i.e. you can choose from different sync modes), what schemas they use, and also allows the user to do partitioning on data before loading it into the destination. ### Streams Properties ### 1. Normalization -Normalization in OLake is the transformation step that does Level-1 flattening of data in nested JSON format, mapping fields to proper columns, thus making data ready to be written into Iceberg/Parquet format tables. +Normalization in OLake Go is the transformation step that does Level-1 flattening of data in nested JSON format, mapping fields to proper columns, thus making data ready to be written into Iceberg/Parquet format tables. - Detects schema evolution (adds, drops, type promotions) and writes according to Iceberg v2 spec. - Flattens nested structures so records become query-friendly. - Focuses on mapping source types to Iceberg/Parquet types. @@ -60,9 +60,9 @@ Normalization must be enabled at the schema configuration step per table or stre ### 2. Sync Modes -Sync modes in Olake define the strategy used to replicate data from a source system to a destination. Each mode represents a different approach to data synchronization, with specific behaviours, guarantees, and performance characteristics. +Sync modes in OLake Go define the strategy used to replicate data from a source system to a destination. Each mode represents a different approach to data synchronization, with specific behaviours, guarantees, and performance characteristics. -OLake supports 4 distinct sync modes: +OLake Go supports 4 distinct sync modes: 1. **Full Refresh:** Entire table is re-copied from source to destination in parallel chunks. Useful as a main sync mode or for initial loads. @@ -74,7 +74,7 @@ OLake supports 4 distinct sync modes: A delta-sync strategy that only processes new or changed records since the last sync. Requires primary (mandatory) and secondary cursor (optional) fields for change detection. Similar to CDC sync, an initial full-refresh takes place in this as well. :::info Cursor fields are columns in the source table used to track the last synced records. \ - Olake allows setting up to two cursor fields: + OLake Go allows setting up to two cursor fields: - **Primary Cursor:** This is the mandatory cursor field through which the changes in the records are captured and compared. - **Secondary Cursor:** In case primary cursor's value is null, then the value of secondary cursor is considered if provided. ::: @@ -94,24 +94,24 @@ OLake supports 4 distinct sync modes: ### 3. Data Filter -The data filter feature allows selective ingestion from source databases by applying SQL-style `WHERE` clauses or BSON-based conditions during ingestion. +The data filter feature allows selective ingestion from source systems by applying filtering conditions before writing to the destination, so only the required subset of data is replicated. - Ensures only selected data enters the pipeline, saving on transfer, storage, and processing. - Supports combining up to two conditions with logical operators (AND/OR). - Operators: `>`, `<`, `=`, `!=`, `>=`, `<=` -- Values can be `numbers`, `quoted strings/timestamps/ids (eg.created_at > \"2025-08-21 17:38:35.017\")`, or `null`. +- Values can be `numbers`, `quoted strings/timestamps/ids (eg.created_at > 2025-08-21 17:38:35.017)`, or `null`. -**Adoption of filter in drivers:** -- **Postgres:** During chunk processing, filters are applied alongside chunk conditions, ensuring only matching records are ingested—even with CTID-based chunking. -- **MySQL:** During chunk processing, filters are applied within each chunk so only relevant rows are returned, even with limit-offset chunking. -- **MongoDB:** During chunk processing, filters are enforced in the aggregation pipeline’s $match stage to ensure only compliant documents are processed. -- **Oracle:** Similar to Postgres and MySQL, filters are applied within each chunk’s scan, guaranteeing only records satisfying conditions are ingested. -- **DB2:** Similar to Postgres and MySQL, filters are applied within each chunk’s scan, guaranteeing only records satisfying conditions are ingested. +:::note +Data filter is supported **only when Normalisation is enabled** for the job. +::: + +:::info CDC/Incremental Filter Behavior + - From **OLake Go connector v0.6.0** and **OLake UI v0.4.1** onward, data filter is now available for **CDC and Incremental sync** as well, and the filter configured during **Full Refresh** will be applied during subsequent **CDC and Incremental** syncs. + - Data filtering for CDC and Incremental is available **only for jobs created on OLake Go connector v0.6.0 or later**. For jobs created on earlier versions, even if a data filter is configured for Full Refresh, it will not be applied during CDC and Incremental, even if the OLake Go version is upgraded; new job must be created to use data filtering for CDC and Incremental. + - If you update an existing job’s filter after it has been created and scheduled (for example, changing conditions a few days later), OLake Go will automatically perform **Clear Destination**, and the next sync will run as a **Full Refresh** that applies the new filter conditions. + - If a job was originally created without any filter and you later add a filter, OLake Go will again perform **Clear Destination**, and the next sync will be a **Full Refresh** that uses the newly added filter. +::: - :::note - If using DB2 as source, then filter for timestamp should be in the format of `2025-01-01 10:15:30.123456` - ::: -
    Olake Partition output After streams discovery is complete, OLake presents all available columns for each selected table. Schema management in OLake allows you to control which columns from your source tables are synced to the destination, enabling you to evolve your schema by selecting or deselecting columns as needed. +Schema is the structure of the tables (or streams) that OLake creates when it scans and discovers the source data. The ability to adjust a schema (add/remove columns, change types) without rewriting the entire table is called `Schema Evolution`. For more information, refer to [Schema Evolution Feature](/docs/features/schema/).
    After streams discovery is complete, OLake Go presents all available columns for each selected table. Schema management in OLake Go allows you to control which columns from your source tables are synced to the destination, enabling you to evolve your schema by selecting or deselecting columns as needed.
    + +- **Source Naming Convention:** + - By default, this toggle is **disabled**, ensuring that column names are normalised to follow [destination naming conventions](/docs/understanding/terminologies/olake/#6-tablecolumn-normalization--destination-database-creation) before being written to the destination. + + ![Disabled Source Naming Convention](/img/docs/terminologies/source_naming_off.webp) + + - When enabled, column names are written to the destination exactly as they appear in the source. + + ![Enabled Source Naming Convention](/img/docs/terminologies/source_naming_on.webp) + :::note Version Compatibility -Column selection and automatic sync of new columns are available in **OLake v0.4.0** and **OLake UI v0.3.1** or later.
    Jobs created before these versions must be run on OLake v0.4.0+ and OLake UI v0.3.1+ to use these features; newly created jobs have them enabled by default. +- Column selection and automatic sync of new columns are available in **OLake Go v0.4.0** and **OLake UI v0.3.1** or later.
    Jobs created before these versions must be run on OLake Go v0.4.0+ and OLake UI v0.3.1+ to use these features; newly created jobs have them enabled by default. +- Source Naming Convention is available in **OLake Go v0.7.7** and **OLake UI v0.4.7** or later. ::: --- @@ -200,40 +212,7 @@ Unlike traditional systems like Hive, Iceberg's approach uses "hidden partitioni --- -### 6. Job Configuration - -The job configuration property refers to the options that defines job’s name, schedule, and execution in the Olake’s system. \ -User has to start with job creation, which will be followed with source configuration, then destination configuration, checking and enabling relevant streams from the schema for sync, and then finally in job configuration, job name and frequency has to be set. - -- **Frequency Options:** - - Default options i.e every minute, hourly, daily, weekly - - Custom frequency: Specify a cron expression. - -**Guide to Cron Expression:** - -| * | * | * | * | * | -|---|---|---|---|---| -| minute (0-59) | hour (0-23) | day of the month (1-31) | month (1-12) | day of the week (0-6) | - - -**Cron Examples:** -- `* * * * *` = Every minute -- `0 * * * *` = Every hour -- `0 0 * * *` = Every day at 12:00 AM -- `0 0 * * FRI` = At midnight only on Fridays -- `0 0 1 * *` = At midnight on the 1st day of each month - -
    - Olake Partition output -
    - ---- - -### 7. Table/Column Normalization & Destination Database Creation +### 6. Table/Column Normalization & Destination Database Creation #### Table/Column Normalization @@ -291,7 +270,7 @@ Once the sync is complete, the streams will be available in the Iceberg database --- -### 8. Modes to Ingest Data (Upsert vs Append) +### 7. Modes to Ingest Data (Upsert vs Append) These modes control how records are written to the destination for CDC and Incremental syncs. These modes can be configured in the Streams panel after discovery completes. @@ -340,179 +319,51 @@ These modes control how records are written to the destination for CDC and Incre --- -## OLake-Generated Columns - -When OLake replicates data from source (like PostgreSQL, MySQL, Oracle or MongoDB) to destination formats (Apache Iceberg, Parquet), it automatically adds several metadata columns to track the lifecycle and processing history of each record. These columns help users understand how and when each row was captured and written to the destination. - -OLake adds these metadata columns to every destination table: `_op_type`, `_olake_id`, `_olake_timestamp`, and `_cdc_timestamp`. - -### 1. Operation Types (`_op_type`): - - **Read/Snapshot (`r`) -** - - Appears during initial full table loads (snapshots) from the source database. - - **Example**: When you first sync a table i.e. full refresh, all existing rows get `_op_type = 'r'`. - - **Create/Insert (`c`) -** - - Generated during CDC, when new records are inserted into the source database. - - **Example**: After initial sync, inserting a new record creates a row with `_op_type = 'c'`. - - **Update (`u`) -** - - Created when existing records are modified in the source database. - - **Example**: Updating a column value from the table generates a new row with `_op_type = 'u'`. - - **Delete (`d`) -** - - Generated when records are deleted from the source database. - - When a record is deleted, primary key column and OLake-generated columns retain their values for tracking, while all other columns are set to either `None` or they appear blank. - - **Example**: Deleting a record from the table creates a tombstone row with `_op_type = 'd'`. - -### 2. Timestamps: - - `_olake_timestamp` - - - Captures the exact time when OLake processed and wrote the record to the destination. - - Useful for tracking when data was ingested into the data lake and for debugging sync latency and understanding processing order. - - `_cdc_timestamp` - - - Reflects the actual time when the change occurred in the source database. - - Present only when using Update Method (during Source Configuration) as `CDC`. - - Whenever a full refresh is performed, snapshot records `(_op_type = 'r')` will have the `_cdc_timestamp` set to the epoch time **(1970-01-01)** indicating that it is not a CDC record. Even in (Full Refresh+ CDC) mode, the very first sync run starts with a full refresh — in this case too, you may see **(1970-01-01)** timestamps. - - For CDC records `(_op_type = 'c', 'u', 'd')`, it provides the precise timestamp of when the insert, update, or delete happened in the source. - -### 3. Record Identification (`_olake_id`): - OLake assigns a unique, deterministic identifier to every record as it is processed. The identifier is designed primarily for deduplication and internal ordering/tracking. - - #### How it is generated: - - **Single primary key present in the source table -** - - `_olake_id` equals the record’s primary key value and duplicates on the key will be detected. - - **Example:** If the primary key is `id` with value 123, then `_olake_id` will be 123. - - **Composite primary key present in the source table -** - - `_olake_id` is a stable hash of all primary key columns for the record. Hence ensures that duplicates on the key will be detected. - - **Example:** If the composite primary key is `(order_id, product_id)` with values (456, 789), then `_olake_id` will be a hash of (456, 789). - - **With no primary key in the source table -** - - `_olake_id` equals the hash of all columns of the record. But in this situation, deduplication cannot be guaranteed as two different records could have the same hash value. - - **Example:** If a record has columns `(name, age, city)` with values (Alice, 30, NY), then `_olake_id` will be hash of (Alice, 30, NY). - -### 4. CDC Metadata Columns - -OLake also writes **driver-specific CDC ordering metadata columns**. These fields capture the exact log position and ordering information from each source system’s native CDC mechanism and is supported for **MongoDB**, **Postgres**, **MySQL** and **MSSQL** drivers. - -These columns are especially useful in scenarios where multiple changes happen within the same transaction (and thus share a single `_cdc_timestamp`), because they provide a stable, log-based key (for example, binlog position or LSN) that downstream systems can use to order events exactly as they occurred in the source. - -#### MongoDB +### 8. Bulk Configure -- **Column:** `_cdc_resume_token` - - `_cdc_resume_token` stores the MongoDB Change Streams resume token that identifies the exact position in the oplog where this change event was captured. - - **Example:** `_cdc_resume_token:"82698C3243000000022B0429296E1404"` +Bulk Configure lets users apply the same configuration to multiple selected streams at once. When working with sources containing multiple streams, users often need to apply the same configuration across several streams. Bulk Configuration significantly reduces setup time and improves usability. -#### MySQL +#### Supported Bulk Configurations -- **Columns:** `_cdc_binlog_file_name`, `_cdc_binlog_file_pos` - - Together, these columns identify the exact location in MySQL's binary log where the change event was read. `_cdc_binlog_file_name` specifies which binlog file, and `_cdc_binlog_file_pos` indicates the byte position within that file. - - **Example:** `_cdc_binlog_file_name = "mysql-bin.000003"`, `_cdc_binlog_file_pos = 1027` +When multiple streams are selected using checkboxes, users can trigger a bulk configuration action to apply the following properties for all selected streams at once: -#### Postgres +- **Sync Mode** +- **Cursor Fields:** Shown only if Sync mode is *Full Refresh + Incremental*. Only columns that are common across all selected streams are available for cursor field selection. +- **Data Filter (Column and Values):** Only columns that are common across all selected streams and have similar data types are available for data filter configuration. +- **Partition Regex Value** -- **Column:** `_cdc_lsn` - - `_cdc_lsn` stores the Write-Ahead Log (WAL) Log Sequence Number that represents the precise position in Postgres's transaction log where this change was recorded. LSN values are monotonically increasing and can be used to order events in the exact sequence they were applied by Postgres. - - **Example:** `_cdc_lsn = "16/B374D848"` - -#### MSSQL - -- **Columns:** `_cdc_start_lsn`, `_cdc_seqval` - - `_cdc_start_lsn` indicates the starting Log Sequence Number for the change row in SQL Server's CDC change table, while `_cdc_seqval` is a sequence value that orders multiple changes that share the same LSN. Together, they provide a stable ordering key that matches SQL Server's internal CDC ordering. - - **Example:** `_cdc_start_lsn = "000000bb000055d00003"`, `_cdc_seqval = "000000bb000055d00002"` - -:::note Backward compatibility -This feature is available starting from **v0.3.16**.
    For **jobs created before** this feature was released, these columns are always present for Parquet-on-S3 destinations, but for **Iceberg** they appear only when Normalization was turned **on** and will not be present for older jobs where Iceberg Normalization was **off**. -::: - - - - -## Source - -A Source is the system from which OLake reads data. This could be a database (MongoDB, Postgres, MySQL, Oracle), basically a data service. When you create a source in OLake, you are telling the platform where the data should come from. +::::info Important Considerations +- **Overwriting Existing Configurations:** Existing configurations are overwritten only for the modified properties. If a user uses the bulk configure option to change only a few properties of a previously configured stream, properties configured outside the bulk configure feature remain unchanged. +- **Warning on Reconfiguration:** If a few streams are already configured and the user configures them again via Bulk Configuration, a warning modal will appear to confirm the action. Changing an already configured stream property will trigger a clear destination run. +:::: -| Concept | Description | -|--------|-------------|-----------------| -| **Active source** | Linked to at least one job; OLake reads from this source and sends data to a destination. | -| **Inactive source** | Created in OLake but not assigned to any job yet; no data is read until a job uses it. | +#### Step 1: Initiate Bulk Configuration -### Source Properties +From the Streams page, the **Bulk Configure** button can be used to apply the same configuration across multiple streams at once. -#### 1. SSH Configuration +![Bulk Configuration Button](/img/docs/terminologies/bulk-configure-button.webp) -OLake can connect to a database through an SSH tunnel instead of connecting directly.
    -Currently, SSH tunneling is supported for Postgres and MySQL.
    +#### Step 2: Select Streams -The following parameters can be configured to establish connection via SSH tunnel: +After clicking **Bulk Configure**, a modal opens where the target streams can be selected for bulk configuration. Once stream selection is complete, click **Configure Streams** to continue. - - - -
    - ![SSH configuration options dropdown showing No Tunnel, SSH Key Authentication, and SSH Password Authentication selections.](/img/docs/terminologies/ssh_config.webp) -
    - - | Field | Description | Example Value | - |------------------------------------------------------|---------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------| - | SSH Config `required` | Describes how user want to connect to the SSH tunnel. | `No Tunnel`
    `SSH Key Authentication`
    `SSH Password Authentication` | - | Host `required` | Host address of the SSH tunnel | `my-host` | - | Port `required` | Port number of the SSH tunnel | `22` | - | Username `required` | Username for the SSH tunnel | `my-user` | - | Private Key ^ | Private key used for SSH authentication. | 
    -----BEGIN OPENSSH PRIVATE KEY-----
    my-private-key
    -----END OPENSSH PRIVATE KEY-----
    | - | Passphrase | Passphrase to decrypt the private key. Leave blank if the key is not encrypted. | `my-passphrase` | - | Password ^ | Password for SSH authentication. | `my-password` | +![Bulk Configuration Streams Selection](/img/docs/terminologies/bulk-configure-select-streams.webp) +#### Step 3: Select Configuration -     - - ```json - "ssh_config": { - "host": "my-tunnel-host", - "port": 22, - "username": "my-tunnel-user", - "password": "tunnel-password", - "private_key": "-----BEGIN OPENSSH PRIVATE KEY-----\nmy-private-key\n-----END OPENSSH PRIVATE KEY-----", - "passphrase": "my-passphrase" - } - ``` - - | Field | Description | Type | Example Value | - |------------------------------------------------------|---------------------------------------------------------------------------------|---------|--------------------------------------------------------------------------------------------| - | host `required` | Host address of the SSH tunnel | String | `"my-host"` | - | port `required` | Port number of the SSH tunnel | Integer | `22` | - | username `required` | Username for the SSH tunnel | String | `"my-user"` | - | private_key ^ | Private key used for SSH authentication. | String | `"-----BEGIN OPENSSH PRIVATE KEY-----\nmy-private-key\n-----END OPENSSH PRIVATE KEY-----"` | - | passphrase | Passphrase to decrypt the private key. Leave blank if the key is not encrypted. | String | `"my-passphrase"` | - | password ^ | Password for SSH authentication. | String | `"my-password"` | +The next window displays the selected streams for bulk configuration. Configure the properties to be applied across those streams. The properties that are configured are marked with a warning sign, and only those marked properties are applied. +![Bulk Configuration on the streams](/img/docs/terminologies/configure-bulk-streams.webp) - **^ Only one among `private_key` or `password` is required** - - `passphrase` is required only if you use an encrypted `private_key`. - - When defining `private_key` in the `source.json` file, use `\n` to represent newlines within the single-line string. - -     +#### Step 4: Review and Apply Configuration -## Destination +In the final window, the selected configuration can be reviewed before submission. After confirmation, clicking **Apply Changes** applies the configuration to the selected streams. -A Destination is the system where OLake writes data after it has been extracted from a source. In OLake, destinations define where your data will be stored and in what format. Currently, OLake supports two types of destinations: Amazon S3 and Apache Iceberg. +![Review and Apply Bulk Configuration](/img/docs/terminologies/review-bulk-config.webp) -| Concept | Description | -|--------|-------------|-----------------| -| **Active destination** | Assigned to at least one job; OLake is delivering data into this destination. | -| **Inactive destination** | Created in OLake but not linked to any job yet; no data until a job uses it. | -## Jobs -A Job is the pipeline or process in OLake that moves data from a source to a destination. A job defines what data is moved, how it is moved (full refresh, incremental, CDC), and where it is delivered. Jobs are the central element of OLake, as they connect sources and destinations. -| Concept | Description | -|--------|-------------|-----------------| -| **Active job** | Running or scheduled; OLake transfers data per the job configuration. Newly created jobs appear here while active. | -| **Inactive job** | Paused; no transfer until resumed. Configuration and state are kept. | -| **Saved job** | Saved configuration used to start new runs; when scheduled or run, it appears under Active Jobs. | -| **Failed job** | Execution error (for example network, schema, or permissions); needs investigation. | -:::note Restrictions for inactive jobs -When a job is inactive, certain job-level features are unavailable: -- **Sync Now** — cannot trigger immediate syncs -- **Edit Streams** — stream configuration cannot be modified -- **Clear Destination** — cannot clear destination data -To use these features, resume the job to move it back to active status. -::: diff --git a/docs/writers/iceberg/azure.mdx b/docs/writers/iceberg/azure.mdx index 8d51e95d2..1d39b15dc 100644 --- a/docs/writers/iceberg/azure.mdx +++ b/docs/writers/iceberg/azure.mdx @@ -5,24 +5,24 @@ sidebar_label: Iceberg On Azure sidebar_position: 2 --- -## Iceberg Lakehouse on Azure: A Step-by-Step Guide with OLake and Lakekeeper -Running Iceberg on Azure usually means gluing together storage, catalogs, and data-loading scripts yourself. OLake does that for you: it streams database changes, writes Iceberg-compatible Parquet files directly to ADLS Gen2, and updates the catalog automatically. +## Iceberg Lakehouse on Azure: A Step-by-Step Guide with OLake Go and Lakekeeper +Running Iceberg on Azure usually means gluing together storage, catalogs, and data-loading scripts yourself. OLake Go does that for you: it streams database changes, writes Iceberg-compatible Parquet files directly to ADLS Gen2, and updates the catalog automatically. The result is a query-ready Iceberg table on Azure with almost zero manual setup. -In this guide, we will demonstrate how OLake makes it easy to build a production-ready data pipeline on Azure. We will use OLake to orchestrate a best-in-class open-source stack: +In this guide, we will demonstrate how OLake Go makes it easy to build a production-ready data pipeline on Azure. We will use OLake Go to orchestrate a best-in-class open-source stack: * **Lakekeeper:** As our open-source Iceberg REST Catalog to manage table metadata. * **Apache Iceberg:** As the open table format for reliability and performance. * **Azure Data Lake Storage (ADLS) Gen2:** As the scalable storage layer for our data. -By following these steps, you will see firsthand how OLake can help you set up a fully functional pipeline, writing data into a query-ready Iceberg table on your Azure subscription with minimal friction. +By following these steps, you will see firsthand how OLake Go can help you set up a fully functional pipeline, writing data into a query-ready Iceberg table on your Azure subscription with minimal friction. ## Architecture Overview Before we begin, it's important to understand how these components interact: -* **OLake (The Writer):** Connects to your source database, reads data, formats it into Parquet files, and writes those files directly to Azure ADLS Gen2. -* **Lakekeeper (The Catalog):** Manages the metadata for our Iceberg tables. OLake communicates with Lakekeeper's REST API to commit new data files and update table versions. It does **not** store the data itself. +* **OLake Go (The Writer):** Connects to your source database, reads data, formats it into Parquet files, and writes those files directly to Azure ADLS Gen2. +* **Lakekeeper (The Catalog):** Manages the metadata for our Iceberg tables. OLake Go communicates with Lakekeeper's REST API to commit new data files and update table versions. It does **not** store the data itself. * **Azure ADLS Gen2 (The Storage):** This is the durable, scalable storage layer in Azure where all the actual Iceberg data (Parquet files) will reside. ## Prerequisites @@ -76,7 +76,7 @@ Now, let's get the Lakekeeper REST Catalog running using its official Docker Com This will start the Lakekeeper REST catalog service and its required PostgreSQL backend. Lakekeeper's API will now be available on your local machine at `http://localhost:8181`. ### Step 3: Add the Azure Warehouse to Lakekeeper -Before Olake can use Lakekeeper, we must configure Lakekeeper to be aware of our ADLS Gen2 storage. +Before OLake Go can use Lakekeeper, we must configure Lakekeeper to be aware of our ADLS Gen2 storage. 1. **Access the Lakekeeper UI:** Open your browser and navigate to `http://localhost:8181`. 2. **Add a New Warehouse:** Find the section for adding a new storage profile or warehouse. @@ -94,11 +94,11 @@ Before Olake can use Lakekeeper, we must configure Lakekeeper to be aware of our Lakekeeper Warehouse Configuration -### Step 4: Set Up the OLake Environment -Now, let's get the Olake UI running using its official Docker Compose setup. +### Step 4: Set Up the OLake Go Environment +Now, let's get the OLake UI running using its official Docker Compose setup. -1. **Get the OLake Docker Compose file:** Follow the instructions at the [OLake Getting Started](/docs/getting-started/quickstart) to get the `docker-compose.yml` file. -2. **Start OLake:** Follow the steps from the docs and remember to modify the directory in `docker-compose.yml` where OLake's persistent data and configuration will be stored, once done, run the following command: +1. **Get the OLake Go Docker Compose file:** Follow the instructions at the [OLake Go Getting Started](/docs/getting-started/quickstart) to get the `docker-compose.yml` file. +2. **Start OLake Go:** Follow the steps from the docs and remember to modify the directory in `docker-compose.yml` where OLake Go's persistent data and configuration will be stored, once done, run the following command: ```sh docker compose up -d @@ -106,10 +106,10 @@ Now, let's get the Olake UI running using its official Docker Compose setup. This will start the OLake UI along with Temporal and PostgreSQL services. OLake’s UI will now be available on your local machine at http://localhost:8000. -### Step 5: Configure the Iceberg Destination in the Olake UI -With all services running, we will now connect Olake to Lakekeeper and Azure. +### Step 5: Configure the Iceberg Destination in the OLake UI +With all services running, we will now connect OLake Go to Lakekeeper and Azure. -1. **Log in to Olake UI:** Open your browser and navigate to http://localhost:8000. +1. **Log in to OLake UI:** Open your browser and navigate to http://localhost:8000. 2. **Navigate to Destinations:** Go to the **Destinations** page and click **"Create Destination"**, then select **Apache Iceberg**. 3. **Fill in the Endpoint config:** * **Catalog Type**: `REST Catalog` @@ -119,7 +119,7 @@ With all services running, we will now connect Olake to Lakekeeper and Azure. * **S3 Endpoint**: `https://olakehouse.dfs.core.windows.net` * **AWS Access Key**: Leave empty. * **AWS Secret Key**: Leave empty. -4. **Save and Test** the destination to ensure Olake can communicate with both Lakekeeper and Azure. +4. **Save and Test** the destination to ensure OLake Go can communicate with both Lakekeeper and Azure. ![Screenshot of OLake Iceberg destination config with REST Catalog, Iceberg S3 path, and Azure Data Lake endpoint.](/img/docs/adls/azure_dest.png) diff --git a/docs/writers/iceberg/catalog/glue.mdx b/docs/writers/iceberg/catalog/glue.mdx index 5d00c2a5c..d6de1d966 100644 --- a/docs/writers/iceberg/catalog/glue.mdx +++ b/docs/writers/iceberg/catalog/glue.mdx @@ -13,17 +13,17 @@ import GlueIcebergWriterCLIConfigDetails from '@site/docs/shared/config/GlueIceb # AWS Glue Catalog Write Guide -OLake integrates with AWS Glue Catalog to provide full support for **Apache Iceberg tables**. +OLake Go integrates with AWS Glue Catalog to provide full support for **Apache Iceberg tables**. This setup ensures that: - **Data** is stored in Amazon S3 (Parquet + metadata files) - **Metadata** is managed in AWS Glue Catalog (schemas, partitions, table properties) -- **OLake** seamlessly writes into Iceberg tables through Glue APIs +- **OLake Go** seamlessly writes into Iceberg tables through Glue APIs --- ## Prerequisites -Before configuring OLake with AWS Glue Catalog, ensure the following are set up: +Before configuring OLake Go with AWS Glue Catalog, ensure the following are set up: #### 1. Amazon S3 Bucket @@ -50,6 +50,7 @@ Here is a sample IAM policy example: "glue:CreateTable", "glue:CreateDatabase", "glue:GetTable", + "glue:GetTables", "glue:GetDatabase", "glue:GetDatabases", "glue:SearchTables", @@ -111,9 +112,9 @@ Here is a sample IAM policy example: For AWS Glue Catalog, the catalog name is fixed to `olake_iceberg` and cannot be customized. ::: -**Click `Next ->`** to test the connection and verify that OLake can validate both Glue Catalog and S3 access. +**Click `Next ->`** to test the connection and verify that OLake Go can validate both Glue Catalog and S3 access. -After you have successfully set up the destination: [Configure your streams](/docs/getting-started/creating-first-pipeline#4-configure-streams) +After you have successfully set up the destination: [Configure your streams](/docs/getting-started/creating-first-pipeline/#7-configure-streams) @@ -138,7 +139,7 @@ After you have successfully set up the destination: [Run the Discover command](/ :::tip **Connection Testing** -OLake will automatically test: +OLake Go will automatically test: - AWS credentials validity - S3 bucket access permissions - Glue Catalog connectivity @@ -150,7 +151,7 @@ OLake will automatically test: ## Querying Data :::info **Query Your Data with AWS Athena** -Once OLake has written data to Iceberg tables in AWS Glue Catalog, you can query the data using AWS Athena: +Once OLake Go has written data to Iceberg tables in AWS Glue Catalog, you can query the data using AWS Athena: ```sql SELECT * FROM "ICEBERG_DATABASE_NAME"."TABLE_NAME" LIMIT 10; @@ -159,7 +160,7 @@ SELECT * FROM "ICEBERG_DATABASE_NAME"."TABLE_NAME" LIMIT 10; ## Troubleshooting -The OLake Iceberg Writer with AWS Glue Catalog stops immediately upon encountering errors to ensure data integrity. Below are common issues and their fixes: +The OLake Go Iceberg Writer with AWS Glue Catalog stops immediately upon encountering errors to ensure data integrity. Below are common issues and their fixes: - AccessDeniedException: User is not authorized to perform action - Cause: IAM role or user lacks required permissions for AWS Glue or S3 operations. @@ -176,6 +177,7 @@ The OLake Iceberg Writer with AWS Glue Catalog stops immediately upon encounteri "glue:GetDatabase", "glue:CreateTable", "glue:GetTable", + "glue:GetTables", "glue:UpdateTable", "glue:GetPartitions" ], @@ -216,7 +218,7 @@ The OLake Iceberg Writer with AWS Glue Catalog stops immediately upon encounteri - Database does not exist in Glue Catalog - Cause: Specified database name doesn't exist in AWS Glue Data Catalog. - - Fix: OLake will automatically create the database if you have `glue:CreateDatabase` permissions. Verify permissions or create manually: + - Fix: OLake Go will automatically create the database if you have `glue:CreateDatabase` permissions. Verify permissions or create manually: ```bash aws glue create-database --database-input Name=iceberg_db --region us-east-1 ``` @@ -253,4 +255,4 @@ The OLake Iceberg Writer with AWS Glue Catalog stops immediately upon encounteri ```bash aws glue delete-table --database-name iceberg_db --name table_name --region us-east-1 ``` - - Or use OLake's schema evolution capabilities for compatible changes + - Or use OLake Go's schema evolution capabilities for compatible changes diff --git a/docs/writers/iceberg/catalog/hive.mdx b/docs/writers/iceberg/catalog/hive.mdx index 67503c753..f6eec592b 100644 --- a/docs/writers/iceberg/catalog/hive.mdx +++ b/docs/writers/iceberg/catalog/hive.mdx @@ -12,22 +12,23 @@ import HiveIcebergWriterConfigDetails from '../../../shared/config/HiveIcebergWr import HiveIcebergWriterUIConfigDetails from '../../../shared/config/HiveIcebergWriterUIConfigDetails.mdx'; import HiveIcebergWriterCLIConfigDetails from '../../../shared/config/HiveIcebergWriterCLIConfigDetails.mdx'; import CatalogQuery from '../../../shared/CatalogQuery.mdx'; +import BrowserOnly from '@docusaurus/BrowserOnly'; # Hive Catalog Write Guide -OLake integrates with **Hive Catalog** to provide full support for **Apache Iceberg tables**. +OLake Go integrates with **Hive Catalog** to provide full support for **Apache Iceberg tables**. With this setup: - **Data** is stored in object storage (S3, GCS, MinIO, or any S3-compatible system). - **Metadata** is managed by Hive Metastore. -- **OLake** seamlessly writes into Iceberg tables using Hive Metastore + Object storage. +- **OLake Go** seamlessly writes into Iceberg tables using Hive Metastore + Object storage. --- ## Prerequisites -Before configuring OLake with Hive Catalog, ensure the following: +Before configuring OLake Go with Hive Catalog, ensure the following: #### 1. Hive Metastore A Hive Metastore service will serve as the Iceberg metadata catalog. This can be: @@ -55,10 +56,10 @@ A bucket for storing **Iceberg data files (Parquet + metadata).** :::note Catalog Name Supported for v0.3.5 and above -For the **catalog name**, OLake only supports lowercase letters and underscores. Spaces and special characters are not supported. +For the **catalog name**, OLake Go only supports lowercase letters and underscores. Spaces and special characters are not supported. ::: -After you have successfully set up the destination: [Configure your streams](/docs/getting-started/creating-first-pipeline#4-configure-streams) +After you have successfully set up the destination: [Configure your streams](/docs/getting-started/creating-first-pipeline/#7-configure-streams) @@ -69,7 +70,7 @@ After you have successfully set up the destination: [Configure your streams](/do :::note Catalog Name Supported for v0.3.5 and above -For the **catalog name**, OLake only supports lowercase letters and underscores. Spaces and special characters are not supported. +For the **catalog name**, OLake Go only supports lowercase letters and underscores. Spaces and special characters are not supported. ::: After you have successfully set up the destination: [Run the Discover command](/docs/install/docker-cli#discover-command ) @@ -80,13 +81,56 @@ After you have successfully set up the destination: [Run the Discover command](/ --- - + +{() => { + const React = require('react'); + + const gcpHeadings = new Set([ + 'gcp-dataproc-metastore', + 'step-by-step-setup', + 'gcp-hive-olake-go-destination-config', + 'notes' + ]); + + const localHeadings = new Set([ + 'docker-compose-setup', + 'starting-the-environment', + 'destination-configuration-local-hive--minio' + ]); + + React.useEffect(() => { + const patch = () => { + const tocLinks = document.querySelectorAll('.table-of-contents__link'); + + tocLinks.forEach(link => { + const href = link.getAttribute('href') || ''; + const hash = href.split('#')[1]; + if (!hash) return; + + if (gcpHeadings.has(hash)) { + link.setAttribute('href', `?setup-type=gcp-dataproc#${hash}`); + } else if (localHeadings.has(hash)) { + link.setAttribute('href', `?setup-type=local-docker#${hash}`); + } + }); + }; + + patch(); + const t = setTimeout(patch, 500); + return () => clearTimeout(t); + }, []); + + return null; +}} + + + ### GCP Dataproc Metastore {#gcp-dataproc-metastore} -OLake supports using Google Cloud Dataproc Metastore (Hive) as the Iceberg catalog and Google Cloud Storage (GCS) as the data lake destination. This allows you to leverage GCP-native services for scalable, managed metadata and storage. +OLake Go supports using Google Cloud Dataproc Metastore (Hive) as the Iceberg catalog and Google Cloud Storage (GCS) as the data lake destination. This allows you to leverage GCP-native services for scalable, managed metadata and storage. **Dataproc Metastore** ![Metastore service configuration panel showing service properties and Hive warehouse GCS bucket path](/img/docs/iceberg/hive-gcp-dataproc.webp) @@ -105,9 +149,9 @@ OLake supports using Google Cloud Dataproc Metastore (Hive) as the Iceberg catal - Click Submit. Creation may take 20–30 minutes. 3. **Expose the Metastore endpoint** to the network where OLake will run (ensure network connectivity and firewall rules allow access to the Thrift port). 4. **Create or choose a GCS bucket** for Iceberg data. -5. **Deploy OLake** in the same network (or with access to the Metastore endpoint). +5. **Deploy OLake Go** in the same network (or with access to the Metastore endpoint). -### GCP-Hive OLake Destination Config +### GCP-Hive OLake Go Destination Config @@ -120,6 +164,7 @@ OLake supports using Google Cloud Dataproc Metastore (Hive) as the Iceberg catal | **Hive URI** | `thrift://:9083` | Dataproc Metastore Thrift endpoint. | | **Hive Clients** | `10` | Number of concurrent Hive clients. | | **Enable SASL for Hive** | `false` | Leave disabled unless your metastore requires SASL. | +| **Enable Arrow Writes** | `false` | Writes data and delete files using Apache Arrow based writer and registers them in Iceberg. | @@ -134,7 +179,8 @@ OLake supports using Google Cloud Dataproc Metastore (Hive) as the Iceberg catal "hive_clients": 10, "hive_sasl_enabled": false, "iceberg_s3_path": "gs:///hive-warehouse", - "aws_region": "us-central1" + "aws_region": "us-central1", + "arrow_writes": false } } ``` @@ -151,7 +197,7 @@ OLake supports using Google Cloud Dataproc Metastore (Hive) as the Iceberg catal - The `hive_uri` must use the Thrift protocol and point to your Dataproc Metastore endpoint. - The `iceberg_s3_path` can use the `gs://` prefix for GCS buckets. -- Ensure OLake has network access to the Metastore and permissions to write to the GCS bucket. +- Ensure OLake Go has network access to the Metastore and permissions to write to the GCS bucket. - Data written will be in Iceberg format, queryable via compatible engines (e.g., Spark, Trino) configured with the same Hive Metastore and GCS bucket. @@ -428,7 +474,7 @@ No Spark image is required for this setup. ### Destination Configuration (Local Hive + MinIO) -Assuming OLake is running locally in another Docker Compose stack, configure the destination as follows. +Assuming OLake Go is running locally in another Docker Compose stack, configure the destination as follows. @@ -480,7 +526,7 @@ Assuming OLake is running locally in another Docker Compose stack, configure the ## Troubleshooting -The OLake Hive Catalog connector stops immediately upon encountering errors to ensure data accuracy. Below are common issues and their fixes: +The OLake Go Hive Catalog connector stops immediately upon encountering errors to ensure data accuracy. Below are common issues and their fixes: - Hive Metastore JAR Dependencies Missing - Cause: Required JAR files not available in Hive Metastore classpath for S3 and PostgreSQL connectivity. diff --git a/docs/writers/iceberg/catalog/jdbc.mdx b/docs/writers/iceberg/catalog/jdbc.mdx index 491b2bd31..5b831fb4b 100644 --- a/docs/writers/iceberg/catalog/jdbc.mdx +++ b/docs/writers/iceberg/catalog/jdbc.mdx @@ -8,18 +8,18 @@ sidebar_position: 3 # JDBC/SQL Catalog Write Guide -OLake integrates with **JDBC/SQL catalogs** (such as PostgreSQL, MySQL, etc.) to provide full support for **Apache Iceberg tables**. +OLake Go integrates with **JDBC/SQL catalogs** (such as PostgreSQL, MySQL, etc.) to provide full support for **Apache Iceberg tables**. With this setup: - **Data** is stored in object storage (S3, MinIO, or any S3-compatible system). - **Metadata** is managed in a relational database (via JDBC). -- **OLake** seamlessly writes into Iceberg tables using JDBC + object storage. +- **OLake Go** seamlessly writes into Iceberg tables using JDBC + object storage. --- ## Prerequisites -Before configuring OLake with JDBC Catalog, ensure the following: +Before configuring OLake Go with JDBC Catalog, ensure the following: #### 1. Relational Database @@ -27,7 +27,7 @@ Before configuring OLake with JDBC Catalog, ensure the following: #### Required Database Permissions -The JDBC catalog user must have sufficient privileges to manage Iceberg metadata tables. OLake requires the following database permissions: +The JDBC catalog user must have sufficient privileges to manage Iceberg metadata tables. OLake Go requires the following database permissions: > **CREATE TABLE** - Creates Iceberg catalog metadata tables (`iceberg_tables`, `iceberg_namespace_properties`, etc.) on first connection > @@ -72,12 +72,12 @@ After setting up the source, configure your **destination with JDBC Catalog**. :::note Catalog Name Supported for v0.3.5 and above -For the **catalog name**, OLake only supports lowercase letters and underscores. Spaces and special characters are not supported. +For the **catalog name**, OLake Go only supports lowercase letters and underscores. Spaces and special characters are not supported. ::: -**Click `Next →`** to test the connection. OLake will verify JDBC + object storage connectivity. +**Click `Next →`** to test the connection. OLake Go will verify JDBC + object storage connectivity. -After you have successfully set up the destination: [Configure your streams](/docs/getting-started/creating-first-pipeline#4-configure-streams) +After you have successfully set up the destination: [Configure your streams](/docs/getting-started/creating-first-pipeline/#7-configure-streams) @@ -100,7 +100,8 @@ Create a `destination.json` with the following configuration: "s3_path_style": true, "aws_access_key": "admin", "aws_region": "us-east-1", - "aws_secret_key": "password" + "aws_secret_key": "password", + "arrow_writes": false } } ``` @@ -108,7 +109,7 @@ Create a `destination.json` with the following configuration: :::note Catalog Name Supported for v0.3.5 and above -For the **catalog name**, OLake only supports lowercase letters and underscores. Spaces and special characters are not supported. +For the **catalog name**, OLake Go only supports lowercase letters and underscores. Spaces and special characters are not supported. ::: Run the sync command using this `destination.json`. @@ -119,7 +120,7 @@ After you have successfully set up the destination: [Run the Discover command](/ :::tip Connection Testing -OLake automatically validates: +OLake Go automatically validates: * JDBC connectivity & authentication * Object storage access (S3/MinIO) @@ -130,7 +131,7 @@ OLake automatically validates: #### Local Development Setup -Here's an example `docker-compose.yml` for OLake with **PostgreSQL JDBC Catalog + MinIO**: +Here's an example `docker-compose.yml` for OLake Go with **PostgreSQL JDBC Catalog + MinIO**: ```yml version: "3.9" @@ -208,7 +209,7 @@ volumes: ## Troubleshooting -The OLake JDBC Catalog connector stops immediately upon encountering errors to ensure data accuracy. Below are common issues and their fixes: +The OLake Go JDBC Catalog connector stops immediately upon encountering errors to ensure data accuracy. Below are common issues and their fixes: - Connection Refused to Host:Port - Cause: JDBC database not accessible or network connectivity issues. diff --git a/docs/writers/iceberg/catalog/rest.mdx b/docs/writers/iceberg/catalog/rest.mdx index 985719129..1ab9d3da9 100644 --- a/docs/writers/iceberg/catalog/rest.mdx +++ b/docs/writers/iceberg/catalog/rest.mdx @@ -5,12 +5,29 @@ sidebar_label: 2. REST sidebar_position: 2 --- +import TOCTabLinker from '@site/src/components/TOCTabLinker'; + # REST Catalog The REST catalog is a standardized API designed to simplify the management of Apache Iceberg tables across diverse engines and programming languages. By providing a unified client interface, it eliminates the need for separate catalog integrations for engines like Spark, Flink, Trino, or languages like Java. Built on an OpenAPI specification, the REST catalog offers a modern, flexible alternative to the Hive Metastore's Thrift interface, tailored specifically for Iceberg's architecture. + + + + {/* REST - Generic */} @@ -42,7 +59,7 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al

    - After you have successfully set up the destination: [Configure your streams](/docs/getting-started/creating-first-pipeline#4-configure-streams ) + After you have successfully set up the destination: [Configure your streams](/docs/getting-started/creating-first-pipeline/#7-configure-streams) @@ -170,7 +187,7 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al docker-compose up -d ``` :::note - All services involved in the sync OLake, REST Catalog Service, MinIO, and Postgres must run in the **same Docker network**. + All services involved in the sync OLake Go, REST Catalog Service, MinIO, and Postgres must run in the **same Docker network**. ::: --- @@ -232,18 +249,19 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al |----------------------|-------------------------|-----------------------------------------------------------------------------| | **REST Catalog URI** | `http://:8181/catalog` | Endpoint for the Lakekeeper REST catalog service. | | **Catalog Name** | `olake_iceberg` | Enter a name for your catalog. | - | **Iceberg S3 Path** | `` | The name of the warehouse you created in Lakekeeper | + | **S3 Path** | `` | The name of the warehouse you created in Lakekeeper | | **S3 Endpoint** | `http://:9000` | Endpoint for the S3-compatible service (e.g., MinIO, AWS S3). | | **AWS Region** | `` | Region of the S3 bucket. | | **AWS Access Key** | `` | Access key for S3 authentication. | | **AWS Secret Key** | `` | Secret key for S3 authentication. | - + | **Enable Arrow Writes** | `false` | Writes data and delete files using Apache Arrow based writer and registers them in Iceberg. | + :::note Catalog Name Supported for v0.3.5 and above For the **catalog name**, OLake only supports lowercase letters and underscores. Spaces and special characters are not supported. :::
    - After you have successfully set up the destination: [Configure your streams](/docs/getting-started/creating-first-pipeline#4-configure-streams ) + After you have successfully set up the destination: [Configure your streams](/docs/getting-started/creating-first-pipeline/#7-configure-streams)     @@ -260,7 +278,8 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al "s3_endpoint": "http://:9000", "aws_access_key": "", "aws_secret_key": "", - "aws_region": "" + "aws_region": "", + "arrow_writes": false } } ``` @@ -271,13 +290,14 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al |----------------------|-------------------------|-----------------------------------------------------------------------------| | **catalog_type** | `rest` | Defines the catalog type used by the writer. "rest" means the writer interacts with a RESTful catalog service. | | **rest_catalog_url** | `http://:8181/catalog` | Endpoint for the Lakekeeper REST catalog service. | - | **Catalog Name** | `olake_iceberg` | Enter a name for your catalog. | + | **catalog_name** | `olake_iceberg` | Enter a name for your catalog. | | **iceberg_s3_path** | `` | The name of the warehouse you created in Lakekeeper | | **s3_endpoint** | `http://:9000` | Endpoint for the MinIO S3-compatible service. | | **aws_access_key** | `` | MinIO access key for authentication (Optional). | | **aws_secret_key** | `` | MinIO secret key for authentication (Optional). | | **aws_region** | `` | Specifies the AWS region associated with the S3 bucket. | - + | **arrow_writes** | `false` | Writes data and delete files using Apache Arrow based writer and registers them in Iceberg. | + :::note Catalog Name Supported for v0.3.5 and above For the **catalog name**, OLake only supports lowercase letters and underscores. Spaces and special characters are not supported. ::: @@ -507,15 +527,16 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al | Parameter | Sample Value | Description | |----------------------|-----------------------------------------|----------------------------------------------------------------------------------------------------------------| - | **catalog_type** | `rest` | Defines the catalog type used by the writer. "rest" means the writer interacts with a RESTful catalog service. | - | **rest_catalog_url** | `http://:19120/iceberg/` | Specifies the endpoint URL for the Nessie REST catalog service.
    **Linux:** Use machine's IP (e.g., `http://192.168.1.100:19120/iceberg/`)
    **macOS:** Use `http://host.docker.internal:19120/iceberg/` | + | **Catalog Type** | `rest` | Defines the catalog type used by the writer. "rest" means the writer interacts with a RESTful catalog service. | + | **REST Catalog URI** | `http://:19120/iceberg/` | Specifies the endpoint URL for the Nessie REST catalog service.
    **Linux:** Use machine's IP (e.g., `http://192.168.1.100:19120/iceberg/`)
    **macOS:** Use `http://host.docker.internal:19120/iceberg/` | | **Catalog Name** | `olake_iceberg` | Enter a name for your catalog. | - | **iceberg_s3_path** | `s3://` | Determines the S3 path or storage location for Iceberg data in the warehouse bucket.
    Example: `s3://warehouse` for the local testing | - | **s3_endpoint** | `http://:9000/` | Endpoint for the MinIO S3-compatible service.
    **Linux:** Use machine's IP (e.g., `http://192.168.1.100:9000/`)
    **macOS:** Use `http://host.docker.internal:9000/` | - | **aws_access_key** | `` | MinIO access key for authentication (Optional).
    Default from docker-compose: `minio` | - | **aws_secret_key** | `` | MinIO secret key for authentication (Optional).
    Default from docker-compose: `minio123` | - | **aws_region** | `` | Specifies the AWS region associated with the S3 bucket where the data is stored.
    Example: `us-east-1` | - + | **S3 Path** | `s3://` | Determines the S3 path or storage location for Iceberg data in the warehouse bucket.
    Example: `s3://warehouse` for the local testing | + | **S3 Endpoint** | `http://:9000/` | Endpoint for the MinIO S3-compatible service.
    **Linux:** Use machine's IP (e.g., `http://192.168.1.100:9000/`)
    **macOS:** Use `http://host.docker.internal:9000/` | + | **AWS Access Key** | `` | MinIO access key for authentication (Optional).
    Default from docker-compose: `minio` | + | **AWS Secret Key** | `` | MinIO secret key for authentication (Optional).
    Default from docker-compose: `minio123` | + | **AWS Region** | `` | Specifies the AWS region associated with the S3 bucket where the data is stored.
    Example: `us-east-1` | + | **Enable Arrow Writes** | `false` | Writes data and delete files using Apache Arrow based writer and registers them in Iceberg. | + :::note Catalog Name Supported for v0.3.5 and above For the **catalog name**, OLake only supports lowercase letters and underscores. Spaces and special characters are not supported. ::: @@ -535,10 +556,10 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al | **REST Signing Region** | `us-east-1` | Region for AWS Signature V4 signing. | | **REST Enable Signature V4** | `true` | Enable AWS Signature V4 signing (boolean). | | **Disable Identifier Tables** | `false` | Needed to set `true` for Databricks Unity Catalog as it doesn't support identifier fields | - +
    - After you have successfully set up the destination: [Configure your streams](/docs/getting-started/creating-first-pipeline#4-configure-streams ) + After you have successfully set up the destination: [Configure your streams](/docs/getting-started/creating-first-pipeline/#7-configure-streams)     @@ -556,7 +577,8 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al "s3_endpoint": "http://:9000/", "aws_access_key": "", "aws_secret_key": "", - "aws_region": "" + "aws_region": "", + "arrow_writes": false } } ``` @@ -566,13 +588,14 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al |----------------------|-----------------------------------------|----------------------------------------------------------------------------------------------------------------| | **catalog_type** | `rest` | Defines the catalog type used by the writer. "rest" means the writer interacts with a RESTful catalog service. | | **rest_catalog_url** | `http://:19120/iceberg/` | Specifies the endpoint URL for the Nessie REST catalog service.
    **Linux:** Use machine's IP (e.g., `http://192.168.1.100:19120/iceberg/`)
    **macOS:** Use `http://host.docker.internal:19120/iceberg/` | - | **Catalog Name** | `olake_iceberg` | Enter a name for your catalog. | + | **catalog_name** | `olake_iceberg` | Enter a name for your catalog. | | **iceberg_s3_path** | `s3://` | Determines the S3 path or storage location for Iceberg data in the warehouse bucket.
    Example: `s3://warehouse` for the local testing | | **s3_endpoint** | `http://:9000/` | Endpoint for the MinIO S3-compatible service.
    **Linux:** Use machine's IP (e.g., `http://192.168.1.100:9000/`)
    **macOS:** Use `http://host.docker.internal:9000/` | | **aws_access_key** | `` | MinIO access key for authentication (Optional).
    Default from docker-compose: `minio` | | **aws_secret_key** | `` | MinIO secret key for authentication (Optional).
    Default from docker-compose: `minio123` | | **aws_region** | `` | Specifies the AWS region associated with the S3 bucket where the data is stored.
    Example: `us-east-1` | - + | **arrow_writes** | `false` | Writes data and delete files using Apache Arrow based writer and registers them in Iceberg. | + :::note Catalog Name Supported for v0.3.5 and above For the **catalog name**, OLake only supports lowercase letters and underscores. Spaces and special characters are not supported. ::: @@ -773,14 +796,14 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al |------------------|--------------------------------------------|-------------------------------------------------------------------------------------------------| | **REST Catalog URI** | `https://s3tables..amazonaws.com/iceberg` | Specifies the endpoint URL for the S3 Tables REST catalog service. | | **Catalog Name** | `olake_iceberg` | Enter a name for your catalog. | - | **Iceberg S3 Path** | `arn:aws:s3tables:::bucket/` | Determines the S3 Tables ARN for the bucket and namespace. | + | **S3 Path** | `arn:aws:s3tables:::bucket/` | Determines the S3 Tables ARN for the bucket and namespace. | | **AWS Region** | `` | Specifies the AWS region for the S3 Tables service. | | **AWS Access Key** | `` | AWS access key for authentication. | | **AWS Secret Key** | `` | AWS secret key for authentication. | | **REST Signing Name** | `s3tables` | Service name for AWS Signature V4. | | **REST Signing Region** | `` | Region for AWS Signature V4 signing. | | **REST Enable Signature V4** | `true` | Enable AWS Signature V4 signing. | - + | **Enable Arrow Writes** | `false` | Writes data and delete files using Apache Arrow based writer and registers them in Iceberg. | :::note Catalog Name Supported for v0.3.5 and above For the **catalog name**, OLake only supports lowercase letters and underscores. Spaces and special characters are not supported. ::: @@ -798,7 +821,7 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al
    - After you have successfully set up the destination: [Configure your streams](/docs/getting-started/creating-first-pipeline#4-configure-streams ) + After you have successfully set up the destination: [Configure your streams](/docs/getting-started/creating-first-pipeline/#7-configure-streams) @@ -819,7 +842,8 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al "aws_region": "", "rest_signing_name": "s3tables", "rest_signing_region": "", - "rest_signing_v_4": true + "rest_signing_v_4": true, + "arrow_writes": false } } ``` @@ -829,7 +853,7 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al |------------------|--------------------------------------------|----------------------------------------------------------------------------------------------------------------| | **catalog_type** | `rest` | Defines the catalog type used by the writer. "rest" means the writer interacts with a RESTful catalog service. | | **rest_catalog_url** | `https://s3tables..amazonaws.com/iceberg` | Specifies the endpoint URL for the S3 Tables REST catalog service. | - | **Catalog Name** | `olake_iceberg` | Enter a name for your catalog. | + | **catalog_name** | `olake_iceberg` | Enter a name for your catalog. | | **iceberg_s3_path** | `arn:aws:s3tables:::bucket/` | Determines the S3 Tables ARN for the bucket and namespace. | | **aws_access_key** | `` | AWS access key for authentication. | | **aws_secret_key** | `` | AWS secret key for authentication. | @@ -837,7 +861,8 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al | **rest_signing_name** | `s3tables` | Service name for AWS Signature V4. | | **rest_signing_region** | `` | Region for AWS Signature V4 signing. | | **rest_signing_v_4** | `true` | Enable AWS Signature V4 signing (boolean). | - + | **arrow_writes** | `false` | Writes data and delete files using Apache Arrow based writer and registers them in Iceberg. | + :::note Catalog Name Supported for v0.3.5 and above For the **catalog name**, OLake only supports lowercase letters and underscores. Spaces and special characters are not supported. ::: @@ -896,7 +921,7 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al OLake supports Databricks Unity Catalog as a REST catalog destination using Token-based or OAuth2 authentication via the Iceberg REST Catalog API. - ## ⚠️ Important Limitations + ## Important Limitations - **Append Only**: Unity Catalog supports append operations only (no equality delete-based updates). - **Managed Tables Only**: Iceberg REST writes are supported only for Unity Catalog managed Iceberg tables. @@ -986,6 +1011,7 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al | **Catalog Name** | `olake_iceberg` | Name of the Unity Catalog | | **S3 Path** | `workspace` | Name of the catalog in Unity Catalog (e.g., "workspace", "main"). This appears as `iceberg_s3_path` in JSON config. | | **Disable Identifier Field For Tables** | `true` | Required because Unity Catalog does not support equality deletes | + | **Enable Arrow Writes** | `false` | Writes data and delete files using Apache Arrow based writer and registers them in Iceberg. | :::note Catalog Name Supported for v0.3.5 and above For the catalog name, OLake only supports lowercase letters and underscores. Spaces and special characters are not supported. @@ -1029,7 +1055,8 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al "rest_catalog_url": "https:///api/2.1/unity-catalog/iceberg-rest", "iceberg_s3_path": "", "token": "", - "no_identifier_fields": true + "no_identifier_fields": true, + "arrow_writes": false } } ``` @@ -1047,6 +1074,7 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al - `NAMESPACE` -> Namespace name inside catalog (e.g., "default") - `DATABRICK_USER_PERSONAL_ACCESS_TOKEN` -> Go to Settings > Developer > Create Personal Access Token - `no_identifier_fields` -> Set to `true` (Required for environments that don't support equality delete-based updates, such as Databricks Unity managed Iceberg tables) + - `arrow_writes` -> Set to `false` ### OAuth2 Authentication (Alternative - Currently Having Issues) @@ -1066,7 +1094,8 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al "oauth2_uri": "", "credential": ":", "scope": "", - "no_identifier_fields": true + "no_identifier_fields": true, + "arrow_writes": false } } ``` @@ -1144,20 +1173,20 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al | **Catalog Type** | `rest` | Defines the catalog type used by the writer. | | **REST Catalog URL** | `http://:8181/api/catalog ` | Endpoint URL for the Polaris REST catalog service. | | **Catalog Name** | `olake_iceberg` | Enter a name for your catalog. | - | **Iceberg S3 Path** | `` | Name of the Polaris catalog. | + | **S3 Path** | `` | Name of the Polaris catalog. | | **REST Auth Type** | `oauth2` | Type of authentication (e.g., "oauth2"). | | **REST Auth URI** | `http://:8181/api/catalog/v1/oauth/tokens` | OAuth2 server URI for authentication. | | **REST Credential** | `:` | OAuth2 client ID and secret, formatted as client_id:client_secret. | | **REST Scope** | `PRINCIPAL_ROLE:ALL` | OAuth2 scopes (space-separated). | | **AWS Region** | `` | AWS region associated with the S3 bucket. | - + | **Enable Arrow Writes** | `false` | Writes data and delete files using Apache Arrow based writer and registers them in Iceberg. | :::note Catalog Name Supported for v0.3.5 and above For the **catalog name**, OLake only supports lowercase letters and underscores. Spaces and special characters are not supported. :::
    - After the destination has been successfully configured: [Configure the streams](/docs/getting-started/creating-first-pipeline#4-configure-streams ) + After the destination has been successfully configured: [Configure the streams](http://localhost:3000/docs/getting-started/creating-first-pipeline/#7-configure-streams) @@ -1176,7 +1205,8 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al "oauth2_uri": "http://:8181/api/catalog/v1/oauth/tokens", "credential": ":", "scope": "PRINCIPAL_ROLE:ALL", - "aws_region": "" + "aws_region": "", + "arrow_writes": false } } ``` @@ -1186,14 +1216,15 @@ Built on an OpenAPI specification, the REST catalog offers a modern, flexible al |----------------------|-----------------------------------------------------------|----------------------------------------------------------------------------------------------------------------| | **catalog_type** | `rest` | Defines the catalog type used by the writer. "rest" means the writer interacts with a RESTful catalog service. | | **rest_catalog_url** | `http://:8181/api/catalog` | Specifies the endpoint URL for the Polaris REST catalog service that the writer will connect to. | - | **Catalog Name** | `olake_iceberg` | Enter a name for your catalog. | + | **catalog_name** | `olake_iceberg` | Enter a name for your catalog. | | **iceberg_s3_path** | `` | Name of the Polaris catalog. | | **rest_auth_type** | `oauth2` | Authentication type for Polaris REST catalog service. | | **oauth2_uri** | `http://:8181/api/catalog/v1/oauth/tokens` | OAuth2 server URI for Polaris authentication. | | **credential** | `:` | OAuth2 client ID and secret, formatted as client_id:client_secret. | | **scope** | `PRINCIPAL_ROLE:ALL` | OAuth2 scopes for Polaris authentication. | | **aws_region** | `` | Specifies the AWS region associated with the S3 bucket where the data is stored. | - + | **arrow_writes** | `false` | Writes data and delete files using Apache Arrow based writer and registers them in Iceberg. | + :::note Catalog Name Supported for v0.3.5 and above For the **catalog name**, OLake only supports lowercase letters and underscores. Spaces and special characters are not supported. ::: @@ -1408,4 +1439,108 @@ networks: - If TLS is enabled, check that certificates are trusted and correctly configured + {/* REST - BigLake */} + + **BigLake** is Google Cloud's fully managed lakehouse solution that provides an Apache Iceberg REST catalog endpoint. It enables seamless interoperability across query engines like Apache Spark, BigQuery, and other compatible tools, with built-in support for Google Cloud authentication and fine-grained access control. + + :::note BigLake Catalog Supported for v0.9.0 and above + BigLake catalog support is available in OLake Go source version **0.9.0** and above. + ::: + + ## Prerequisites + + **Required services**: + - **Google Cloud Project** – with billing enabled and the BigLake API activated. + - **Cloud Storage Bucket** – for storing Iceberg table data and metadata. + - **BigLake Catalog** – created in Google Cloud. + - **Service Account** – with appropriate BigLake and Cloud Storage permissions. + + **Required permissions**: + - **Service Account Roles**: + - `BigLake Admin` (`roles/biglake.admin`) – for administrative tasks and catalog management. + - `BigLake Editor` (`roles/biglake.editor`) – for writing table data. + - `BigLake Viewer` (`roles/biglake.viewer`) – for reading table data. + - `Storage Object User` (`roles/storage.objectUser`) – on all associated Cloud Storage buckets. + + For detailed setup instructions, prerequisites, and IAM configuration, refer to the [Google Cloud BigLake REST Catalog documentation](https://docs.cloud.google.com/lakehouse/docs/set-up-lakehouse-iceberg-rest-catalog). + + --- + + ## Configuration + + Configure the following fields to connect OLake Go to your BigLake REST catalog. + + ### Authentication + + BigLake requires **Google OAuth** authentication using a service account. This is the only supported authentication method. + + + + +
    + +
    + ![REST endpoint config form with required fields for Iceberg database connection.](/img/docs/iceberg/catalog/rest/rest_biglake_ui.webp) +
    + + **BigLake Configuration Parameters:** + | Parameter | Sample Value | Description | + |------------------|--------------------------------------------|-------------------------------------------------------------------------------------------------| + | **REST Catalog URI**
    `required` | `https://biglake.googleapis.com/iceberg/v1/restcatalog` | Endpoint URL for the Google Cloud BigLake REST catalog service. | + | **S3 Path**
    `required` | **Single-bucket catalog:**
    `gs://` | BigLake catalog path to use, OLake Go uses this to look up the storage configuration (e.g. Cloud Storage buckets) already defined in BigLake.

    **Single-bucket catalog:** This configuration restricts your catalog to a single bucket and locks the catalog name to the bucket name. | + | **REST Auth Type**
    `required` | `org.apache.iceberg.gcp.auth.GoogleAuthManager` | Iceberg authentication manager for Google Cloud OAuth. BigLake supports only this auth type. | + | **GCP Service Account JSON**
    `required` | `{ "type": "service_account", ... }` | JSON content of the GCP service account key file. OLake Go uses this to authorize REST catalog and Cloud Storage access in your Google Cloud project. | + | **Catalog Name** | `olake_iceberg` | Name of the Iceberg catalog OLake Go registers tables under. Defaults to `olake_iceberg` if left empty. | + | **GCP Auth Scopes** | `https://www.googleapis.com/auth/cloud-platform` | Defines which OAuth scopes Google grants on the access token, comma-separated if requesting more than one. | + | **Enable Arrow Writes** | `false/true` | Writes data and delete files using Apache Arrow based writer and registers them in Iceberg. | + + :::note Catalog Name Supported for v0.3.5 and above + For the **catalog name**, OLake Go only supports lowercase letters and underscores. Spaces and special characters are not supported. + ::: + + Click **`Create ->`** to test the connection and verify that OLake Go can validate both the BigLake REST catalog endpoint and authentication. + +     + + + + To connect to Iceberg using BigLake as the catalog, create `destination.json` with the following configuration: + + ```json title="destination.json" +{ + "type": "ICEBERG", + "writer": { + "catalog_type": "rest", + "rest_catalog_url": "https://biglake.googleapis.com/iceberg/v1/restcatalog", + "catalog_name": "olake_iceberg", + "iceberg_s3_path": "gs://", + "rest_auth_type": "org.apache.iceberg.gcp.auth.GoogleAuthManager", + "gcp_service_account_json": "{ "type": "service_account", ... }", + "arrow_writes": false + } + } +``` + + **BigLake Configuration Parameters** + | Parameter | Sample Value | Description | + |------------------|--------------------------------------------|----------------------------------------------------------------------------------------------------------------| + | **catalog_type**
    `required` | `rest` | Defines the catalog type used by the writer. "rest" means the writer interacts with a RESTful catalog service. | + | **rest_catalog_url**
    `required`| `https://biglake.googleapis.com/iceberg/v1/restcatalog` | Endpoint URL for the Google Cloud BigLake REST catalog service. | + | **iceberg_s3_path**
    `required`| **Single-bucket catalog:**
    `gs://`
    | BigLake catalog path to use, OLake Go uses this to look up the storage configuration (e.g. Cloud Storage buckets) already defined in BigLake.

    **Single-bucket catalog:** Use the format `gs://BUCKET_NAME`. This configuration restricts your catalog to a single bucket and locks the catalog name to the bucket name. | + | **rest_auth_type**
    `required`| `org.apache.iceberg.gcp.auth.GoogleAuthManager` | Iceberg authentication manager for Google Cloud OAuth. BigLake supports only this auth type. | + | **gcp_service_account_json**
    `required`| `{"type": "service_account", ...}` | JSON content of the GCP service account key file. OLake Go uses this to authorize REST catalog and Cloud Storage access in your Google Cloud project. | + | **catalog_name** | `olake_iceberg` | Name of the Iceberg catalog OLake Go registers tables under. Defaults to `olake_iceberg` if left empty. | + | **gcp_auth_scopes** | `https://www.googleapis.com/auth/cloud-platform` | Defines which OAuth scopes Google grants on the access token, comma-separated if requesting more than one. | + | **arrow_writes** | `false/true` | Writes data and delete files using Apache Arrow based writer and registers them in Iceberg. | + + :::note Catalog Name Supported for v0.3.5 and above + For the **catalog name**, OLake Go only supports lowercase letters and underscores. Spaces and special characters are not supported. + ::: + +     + +     + +     + diff --git a/docs/writers/iceberg/gcp.mdx b/docs/writers/iceberg/gcp.mdx index f60608a71..7adb9ecf0 100644 --- a/docs/writers/iceberg/gcp.mdx +++ b/docs/writers/iceberg/gcp.mdx @@ -7,7 +7,7 @@ sidebar_position: 3 # Google Cloud Storage (GCS) Writer Overview -Currently OLake supports writing data to Google Cloud Storage (GCS), using [`Hive Catalog`](/docs/writers/iceberg/catalog/hive#gcp-dataproc-metastore) that supports Iceberg. This allows you to leverage GCS as a scalable and durable storage solution for your data lakehouse architecture. +Currently OLake Go supports writing data to Google Cloud Storage (GCS), using [`Hive Catalog`](/docs/writers/iceberg/catalog/hive#gcp-dataproc-metastore) that supports Iceberg. This allows you to leverage GCS as a scalable and durable storage solution for your data lakehouse architecture. More details on how to configure and use the GCS writer will be available soon. In the meantime, you can refer to the [Iceberg writer documentation](/docs/writers/iceberg/catalog/rest?rest-catalog=generic) for general guidance on Iceberg integration, as the principles are similar across different storage backends. diff --git a/docs/writers/iceberg/partitioning.mdx b/docs/writers/iceberg/partitioning.mdx index 4fedfb36d..8fd007f88 100644 --- a/docs/writers/iceberg/partitioning.mdx +++ b/docs/writers/iceberg/partitioning.mdx @@ -81,7 +81,7 @@ table { Partitioning groups rows that share common values at **write-time**, so that queries filtering on those values read only the relevant files. The result is **fewer data files scanned, less I/O, and faster queries**. -OLake allows seamless writing into **partitioned Iceberg tables**, supporting all built-in transformations in **Apache Iceberg™**. +OLake Go allows seamless writing into **partitioned Iceberg tables**, supporting all built-in transformations in **Apache Iceberg™**. ### How Iceberg Handles Partitions? @@ -99,7 +99,7 @@ When you run a query like `WHERE date > …`, Iceberg consults the metadata to s Learn more about [Iceberg Partitioning](https://iceberg.apache.org/docs/1.7.2/partitioning/). -### How to Add a Partition in OLake? +### How to Add a Partition in OLake Go? When adding a partition, you need to configure two components: @@ -139,12 +139,11 @@ Check all the available transforms [below](#supported-transformations). -1. Before adding partitioning, make sure you have [configured your destination](/docs/getting-started/creating-first-pipeline#3-configure-destination). +1. Before adding partitioning, make sure you have [configured your destination](/docs/getting-started/creating-first-pipeline/#4-configure-destination). 2. Then select your table. -3. Keep **Normalization** enabled. -4. Select **Partitioning** in the right tab. -5. Add your **partition field** along with the **transform**. -6. Then we can move forward to [Schedule a Job](/docs/getting-started/creating-first-pipeline#5-schedule-job). +3. Select **Partitioning** in the right tab. +4. Add your **partition field** along with the **transform**. +5. Then we can move forward to [Schedule a Job](/docs/getting-started/creating-first-pipeline/#6-configure-job).
    @@ -167,7 +166,6 @@ Check all the available transforms [below](#supported-transformations). { "stream_name": "my_stream", "partition_regex": "/{created_at, year}", - "normalization": true } ] } diff --git a/docs/writers/iceberg/troubleshooting-local.mdx b/docs/writers/iceberg/troubleshooting-local.mdx index b380f2201..591649767 100644 --- a/docs/writers/iceberg/troubleshooting-local.mdx +++ b/docs/writers/iceberg/troubleshooting-local.mdx @@ -7,6 +7,15 @@ sidebar_position: 8 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; +import TOCTabLinker from '@site/src/components/TOCTabLinker'; + + @@ -69,12 +78,17 @@ If the version is older, update Java to a supported version. Reload the shell co - Verify Docker and Docker Compose are installed and running. - Use `docker compose ps` to check service statuses. ---- - -### **For catalog-specific issues, refer to the corresponding documentation.** - ---- - +### 7. For catalog-specific issues, refer to the corresponding documentation + +- [AWS Glue](/docs/writers/iceberg/catalog/glue/#troubleshooting) +- [REST Generic](/docs/writers/iceberg/catalog/rest/?rest-catalog=generic#troubleshooting) +- [REST Lakekeeper](/docs/writers/iceberg/catalog/rest/?rest-catalog=lakekeeper#troubleshooting-1) +- [REST Nessie](/docs/writers/iceberg/catalog/rest/?rest-catalog=nessie#troubleshooting-2) +- [REST S3 Tables](/docs/writers/iceberg/catalog/rest/?rest-catalog=s3-tables#troubleshooting-3) +- [REST Unity](/docs/writers/iceberg/catalog/rest/?rest-catalog=unity#troubleshooting-4) +- [REST Apache Polaris](/docs/writers/iceberg/catalog/rest/?rest-catalog=polaris#troubleshooting-5) +- [JDBC](/docs/writers/iceberg/catalog/jdbc/#troubleshooting) +- [Hive](/docs/writers/iceberg/catalog/hive/#troubleshooting) diff --git a/docs/writers/parquet/config.mdx b/docs/writers/parquet/config.mdx index d668817e5..811de3fbc 100644 --- a/docs/writers/parquet/config.mdx +++ b/docs/writers/parquet/config.mdx @@ -7,7 +7,7 @@ sidebar_position: 2 # S3 Parquet -OLake supports writing in parquet format directly to S3. Before proceeding with the S3 Parquet destination, we recommend reviewing the [Getting Started](/docs/getting-started/quickstart) and [Installation](/docs/install/olake-ui) sections. +OLake Go supports writing in parquet format directly to S3. Before proceeding with the S3 Parquet destination, we recommend reviewing the [Getting Started](/docs/getting-started/quickstart) and [Installation](/docs/install/olake-ui) sections. ## Prerequisites - Before setting up the destination, make sure you have successfully set up the [`source`](/docs/getting-started/creating-first-pipeline#2-configure-source). @@ -31,17 +31,17 @@ Steps to get started:- 3. Select `AWS S3` as the Destination type from Connector drop down. 4. Fill in the required connection details in the form. 5. Click on `Create ->`. -6. OLake will test the destination connection and display the results. If the connection is successful, you will see a success message. If there are any issues, OLake will provide error messages to help you troubleshoot. +6. OLake Go will test the destination connection and display the results. If the connection is successful, you will see a success message. If there are any issues, OLake Go will provide error messages to help you troubleshoot. -This will create a S3 destination in OLake, now you can use this destination in your [Jobs Pipeline](../../jobs/overview) to sync data from any [Source](../../connectors/overview) to [AWS S3](../../writers/parquet/s3). +This will create a S3 destination in OLake Go, now you can use this destination in your **Jobs Pipeline** to sync data from any **Source** to **AWS S3**. ![OLake S3 destination config fields and reference guide.](/img/docs/s3/s3-destination.webp) ### Using AWS Amazon S3 Credentials {#s3-cli-configuration-definition} -OLake supports direct syncing of data from source to AWS S3 using Amazon S3 credentials.\ +OLake Go supports direct syncing of data from source to AWS S3 using Amazon S3 credentials.\ For this, refer to [IAM permission](/docs/writers/parquet/permission) needed for Amazon-powered AWS S3. User needs to provide, - AWS S3 bucket path @@ -57,7 +57,7 @@ If using AWS IAM Role with the required permissions, the AWS Access Key and Secr ### Using GCS-compatible S3 Credentials -OLake supports writing data to Google Cloud Storage (GCS) in parquet format. +OLake Go supports writing data to Google Cloud Storage (GCS) in parquet format. Google Cloud Storage provides an S3-compatible interface, allowing you to use S3-compatible tools and libraries to interact with GCS buckets and objects. This interoperability enables seamless integration and ingestion of data into GCS by supporting the Amazon S3 API, which means you can use existing S3 tools and workflows with minimal changes such as updating the endpoint to `https://storage.googleapis.com` and authenticating via HMAC keys (discussed in next section). This compatibility simplifies migrations, data transfers, and tool usage across platforms. For role based permissions, refer to [GCP IAM Permission](/docs/writers/parquet/permission#2-gcs-google-cloud-storage-iam-policy). @@ -66,7 +66,7 @@ For role based permissions, refer to [GCP IAM Permission](/docs/writers/parquet/ - HMAC keys will act as the access key and secret key for S3 writer. - In Google Cloud Console, go to storage, then settings, then select Interoperability. -- Copy the request endpoint and provide it as the S3 endpoint in OLake. +- Copy the request endpoint and provide it as the S3 endpoint in OLake Go. - Create HMAC keys for the service account, which will have an access key and corresponding secret key. - Use those HMAC keys as S3 access key and secret key. @@ -80,7 +80,7 @@ Refer - https://cloud.google.com/storage/docs/authentication/hmackeys ### Using Minio S3 Credentials -OLake supports S3-compatible MinIO as well in its S3 destination configuration. +OLake Go supports S3-compatible MinIO as well in its S3 destination configuration. User can create a MinIO service account, create bucket in it, and provide MinIO access key, secret key with bucket URL in the config. \ In S3 endpoint, URL through MinIO bucket is accessible has to be provided. @@ -141,7 +141,7 @@ Please change the `s3_endpoint` in the current config below and provide the actu :::info 1. The generated `.parquet` files use SNAPPY compression ([Read more](https://en.wikipedia.org/wiki/Snappy_(compression))). Note that SNAPPY is no longer supported by S3 Select when performing queries. -2. OLake creates a test folder named `olake_writer_test` containing a single text file (`.txt`) with the content: +2. OLake Go creates a test folder named `olake_writer_test` containing a single text file (`.txt`) with the content: ```text S3 write test ``` diff --git a/docs/writers/parquet/partitioning.mdx b/docs/writers/parquet/partitioning.mdx index 1d1feee79..5f7be96c5 100644 --- a/docs/writers/parquet/partitioning.mdx +++ b/docs/writers/parquet/partitioning.mdx @@ -9,7 +9,7 @@ import HighPartitionWarning from '@site/docs/shared/HighPartitionWarning.mdx'; # Data Partitioning -OLake supports data partitioning when writing to S3. You can define a `partition_regex` during `schema changes` in OLake UI or in `streams.json` in OLake CLI. This regex determines how data is partitioned into folders within your S3 bucket. Partitioning is defined on a per-stream (table or collection) basis. +OLake Go supports data partitioning when writing to S3. You can define a `partition_regex` during `schema changes` in OLake UI or in `streams.json` in OLake CLI. This regex determines how data is partitioned into folders within your S3 bucket. Partitioning is defined on a per-stream (table or collection) basis. @@ -142,7 +142,7 @@ The partition regex supports several patterns to handle both column values and t ### Supported Timestamp Formats for Partitioning -OLake attempts to parse a wide range of timestamp formats when generating partition folders. The following formats are currently supported: +OLake Go attempts to parse a wide range of timestamp formats when generating partition folders. The following formats are currently supported: ```text "2006-01-02", diff --git a/docs/writers/parquet/troubleshoot.mdx b/docs/writers/parquet/troubleshoot.mdx index 607a18ff3..dee6dc203 100644 --- a/docs/writers/parquet/troubleshoot.mdx +++ b/docs/writers/parquet/troubleshoot.mdx @@ -30,7 +30,7 @@ Symptom: failed to write test file to S3: AccessDenied: Access Denied OR failed ``` **Cause:** -- The AWS credentials supplied to Olake do **not** have the minimum set of S3 actions or the bucket ARN is incorrect. +- The AWS credentials supplied to OLake Go do **not** have the minimum set of S3 actions or the bucket ARN is incorrect. **Resolution:** - Make sure the IAM user / role has the JSON policy shown in the [IAM Permissions](/docs/writers/parquet/permission) page attached. diff --git a/docusaurus.config.js b/docusaurus.config.js index 186e4f710..26a68dda1 100644 --- a/docusaurus.config.js +++ b/docusaurus.config.js @@ -1,4 +1,47 @@ const imageFetchPriorityRehypePlugin = require('./src/plugins/image-fetchpriority-rehype-plugin') +const fs = require('fs') +const path = require('path') + +/** + * Latest OLake Go release, derived at build time from the release notes in + * docs/release/ingestion. Those notes are the page the landing-page bulletin + * card links to, so deriving from them keeps the label and the link in sync + * automatically instead of hardcoding a version that goes stale. + * + * A note's title may cover a range ("OLake Go (v0.8.0 - v0.8.2)"), so the + * displayed version is the LAST version mentioned in the newest note's title, + * falling back to the filename. + */ +function getLatestOlakeRelease() { + const dir = path.join(__dirname, 'docs/release/ingestion') + const toParts = (v) => v.split('.').map(Number) + const files = fs + .readdirSync(dir) + .map((f) => f.match(/^v(\d+\.\d+\.\d+)\.mdx$/)) + .filter(Boolean) + .map((m) => ({ file: `v${m[1]}`, version: m[1] })) + .sort((a, b) => { + const [A, B] = [toParts(a.version), toParts(b.version)] + return A[0] - B[0] || A[1] - B[1] || A[2] - B[2] + }) + + if (!files.length) return { version: '', label: 'OLake', docPath: '/docs/release/ingestion' } + + const newest = files[files.length - 1] + let version = newest.version + try { + const title = fs + .readFileSync(path.join(dir, `${newest.file}.mdx`), 'utf8') + .match(/^title:\s*"?([^"\n]+)"?/m) + const mentioned = title && title[1].match(/v(\d+\.\d+\.\d+)/g) + if (mentioned && mentioned.length) version = mentioned[mentioned.length - 1].slice(1) + } catch { + /* fall back to the filename version */ + } + return { version, label: `OLake v${version}`, docPath: `/docs/release/ingestion/${newest.file}` } +} + +const latestOlakeRelease = getLatestOlakeRelease() // This runs in Node.js - Don't use client-side code here (browser APIs, JSX...) /** @type {import('@docusaurus/types').Config} */ @@ -8,6 +51,13 @@ const config = { 'Fastest open-source tool for replicating Databases to Data Lake in Open Table Formats like Apache Iceberg. Efficient, quick and scalable data ingestion for real-time analytics. Supporting Postgres, MongoDB, MySQL, Oracle and Kafka with 5-500x faster than alternatives.', favicon: 'img/logo/olake-blue.svg', + // Exposed to components via useDocusaurusContext().siteConfig.customFields + customFields: { + latestOlakeVersion: latestOlakeRelease.version, + latestOlakeReleaseLabel: latestOlakeRelease.label, + latestOlakeReleasePath: latestOlakeRelease.docPath + }, + // Set the production url of your site here url: 'https://olake.io', // Set the // pathname under which your site is served @@ -38,7 +88,11 @@ const config = { }, // Client modules for handling client-side functionality - clientModules: [require.resolve('./src/clientModules/hashScroll.ts')], + clientModules: [ + require.resolve('./src/clientModules/hashScroll.ts'), + require.resolve('./src/clientModules/deferredGtag.ts'), + require.resolve('./src/clientModules/deferredReo.ts') + ], presets: [ [ @@ -52,6 +106,10 @@ const config = { }, blog: false, + // GA is loaded by src/clientModules/deferredGtag.ts instead of the preset, so the + // 166KB script stays off the critical path. Re-enabling this would double-load it. + gtag: undefined, + sitemap: { lastmod: 'date', changefreq: 'weekly', @@ -92,20 +150,81 @@ const config = { } ], + // Site-wide, non-per-page tags only. og:*/twitter:card/twitter:title/twitter:description/ + // twitter:image are owned per-page by src/theme/DocItem, BlogPostPage, BlogListPage and + // src/pages/* — do not duplicate them here. + headTags: [ + { + tagName: 'meta', + attributes: { name: 'msvalidate.01', content: 'C36AD97FE1CEDCD4041338A807D6BC4C' } + }, + { + tagName: 'meta', + attributes: { name: 'twitter:site', content: '@_olake' } + }, + { + tagName: 'meta', + attributes: { + name: 'robots', + content: 'follow, index, max-snippet:-1, max-video-preview:-1, max-image-preview:large' + } + }, + // Critical resource preloads for mobile performance + { + tagName: 'link', + attributes: { + rel: 'preload', + href: '/img/logo/olake-blue-with-text.svg', + as: 'image', + type: 'image/svg+xml', + fetchpriority: 'high' + } + }, + // No preload for /img/site/hero-section.svg — it belonged to the v1 homepage and is + // no longer rendered by any routed page, so preloading it cost 22KB at high priority + // on every page for nothing. It is still referenced by JSON-LD in src/data/landing/seo.ts. + // Preconnect to Google Fonts + { + tagName: 'link', + attributes: { + rel: 'preconnect', + href: 'https://fonts.googleapis.com' + } + }, + { + tagName: 'link', + attributes: { + rel: 'preconnect', + href: 'https://fonts.gstatic.com', + crossorigin: 'anonymous' + } + }, + // DNS prefetch for HubSpot forms + { + tagName: 'link', + attributes: { + rel: 'dns-prefetch', + href: 'https://js.hsforms.net' + } + }, + // OpenSearch + { + tagName: 'link', + attributes: { + rel: 'search', + type: 'application/opensearchdescription+xml', + title: 'OLake Documentation', + href: '/opensearch.xml' + } + } + ], + themeConfig: /** @type {import('@docusaurus/preset-classic').ThemeConfig} */ ({ // Replace with your project's social card image: 'img/logo/olake-blue-with-text.webp', - // announcementBar: { - // id: 'monthly-events-2025', - // content: 'Monthly events are hereView upcoming webinars. Check out! 🎉', - // backgroundColor: '#193ae6', - // textColor: 'white', - // isCloseable: true, - // }, - docs: { sidebar: { autoCollapseCategories: true, @@ -122,8 +241,25 @@ const config = { src: 'img/logo/olake-blue-with-text.svg' }, items: [ - { to: '/docs', label: 'Docs', position: 'left' }, - { to: '/ai-lake', label: 'Pricing', position: 'left' }, + { + type: 'dropdown', + position: 'left', + label: 'Docs', + activeBasePath: '/docs', + items: [ + { + label: 'OLake Go', + to: '/docs', + activeBaseRegex: '^/docs(?!/fusion)' + }, + { + label: 'OLake Fusion', + to: '/docs/fusion/getting-started/overview', + activeBaseRegex: '^/docs/fusion' + } + ] + }, + { to: '/contact', label: 'Pricing', position: 'left' }, { to: '/blog', label: 'Blogs', position: 'left' }, { @@ -180,7 +316,7 @@ const config = { }, { - href: 'https://join.slack.com/t/getolake/shared_invite/zt-2uyphqf69-KQxih9Gwd4GCQRD_XFcuyw', + href: 'https://olake.io/slack', position: 'right', className: 'header-slack-link' }, @@ -191,254 +327,13 @@ const config = { }, { label: 'Talk to us', - href: '/#olake-form-product', + href: '/contact', position: 'right', className: 'dev-portal-signup dev-portal-link' } ] }, - metadata: [ - // { name: 'robots', content: 'noindex, nofollow' }, - { name: 'OLake', content: 'ETL tool, ELT tool, open source' }, - { name: 'twitter:card', content: 'summary_large_image' }, - { name: 'twitter:site', content: '@olake.io' }, - { name: 'msvalidate.01', content: 'C36AD97FE1CEDCD4041338A807D6BC4C' } - ], - headTags: [ - // Critical resource preloads for mobile performance - { - tagName: 'link', - attributes: { - rel: 'preload', - href: '/img/logo/olake-blue-with-text.svg', - as: 'image', - type: 'image/svg+xml', - fetchpriority: 'high' - } - }, - { - tagName: 'link', - attributes: { - rel: 'preload', - href: '/img/site/hero-section.svg', - as: 'image', - type: 'image/svg+xml', - fetchpriority: 'high' - } - }, - // Font optimization - preconnect to Google Fonts - { - tagName: 'link', - attributes: { - rel: 'preconnect', - href: 'https://fonts.googleapis.com' - } - }, - { - tagName: 'link', - attributes: { - rel: 'preconnect', - href: 'https://fonts.gstatic.com', - crossorigin: 'anonymous' - } - }, - // Minimal font optimization - only DNS prefetch for performance - { - tagName: 'link', - attributes: { - rel: 'dns-prefetch', - href: 'https://fonts.googleapis.com' - } - }, - // DNS prefetch for external resources - { - tagName: 'link', - attributes: { - rel: 'dns-prefetch', - href: 'https://js.hsforms.net' - } - }, - { - tagName: 'link', - attributes: { - rel: 'dns-prefetch', - href: 'https://www.google-analytics.com' - } - }, - { - tagName: 'link', - attributes: { - rel: 'dns-prefetch', - href: 'https://www.googletagmanager.com' - } - }, - // Preconnect to critical domains - { - tagName: 'link', - attributes: { - rel: 'preconnect', - href: 'https://olake.io', - crossorigin: 'anonymous' - } - }, - // Canonical URL - Removed hardcoded canonical tag - // Docusaurus automatically generates proper canonical URLs for each page - // OpenSearch meta tags - { - tagName: 'link', - attributes: { - rel: 'search', - type: 'application/opensearchdescription+xml', - title: 'OLake Documentation', - href: '/opensearch.xml' - } - }, - // Enhanced Open Graph Meta Tags - { - tagName: 'meta', - attributes: { - property: 'og:type', - content: 'website' - } - }, - { - tagName: 'meta', - attributes: { - property: 'og:title', - content: 'OLake - The Open Lakehouse Platform' - } - }, - { - tagName: 'meta', - attributes: { - property: 'og:description', - content: - 'Fastest way to replicate MongoDB data in Apache Iceberg. Open-source data lakehouse platform for modern data engineering.' - } - }, - { - tagName: 'meta', - attributes: { - property: 'og:image', - content: 'https://olake.io/img/logo/olake-blue.webp' - } - }, - { - tagName: 'meta', - attributes: { - property: 'og:site_name', - content: 'OLake' - } - }, - { - tagName: 'meta', - attributes: { - property: 'og:locale', - content: 'en_US' - } - }, - // Enhanced Open Graph Meta Tags - { - tagName: 'meta', - attributes: { - property: 'og:image:type', - content: 'image/webp' - } - }, - { - tagName: 'meta', - attributes: { - property: 'og:image:width', - content: '1200' - } - }, - { - tagName: 'meta', - attributes: { - property: 'og:image:height', - content: '630' - } - }, - // Enhanced Twitter Meta Tags - { - tagName: 'meta', - attributes: { - name: 'twitter:creator', - content: '@_olake' - } - }, - { - tagName: 'meta', - attributes: { - name: 'twitter:title', - content: 'OLake - The Open Lakehouse Platform' - } - }, - { - tagName: 'meta', - attributes: { - name: 'twitter:description', - content: - 'OLake is the fastest data replication platform, built to stream operational databases into Apache Iceberg in real time with full CDC, incremental sync, and zero-lag reliability.' - } - }, - { - tagName: 'meta', - attributes: { - name: 'twitter:image', - content: 'https://olake.io/img/logo/olake-blue.webp' - } - }, - { - tagName: 'meta', - attributes: { - name: 'twitter:image:alt', - content: 'OLake - The Open Lakehouse Platform' - } - }, - // Enhanced Twitter Meta Tags - { - tagName: 'meta', - attributes: { - name: 'twitter:label1', - content: 'Written by' - } - }, - { - tagName: 'meta', - attributes: { - name: 'twitter:data1', - content: 'OLake Team' - } - }, - // Enhanced Bot Directives - { - tagName: 'meta', - attributes: { - name: 'robots', - content: 'follow, index, max-snippet:-1, max-video-preview:-1, max-image-preview:large' - } - }, - // Bing Webmaster Verification - { - tagName: 'meta', - attributes: { - name: 'msvalidate.01', - content: 'C36AD97FE1CEDCD4041338A807D6BC4C' - } - }, - // Enhanced Favicon Support - { - tagName: 'link', - attributes: { - rel: 'icon', - type: 'image/svg+xml', - href: '/img/logo/olake-blue.svg' - } - } - ], - colorMode: { defaultMode: 'light', // dark or light disableSwitch: false, @@ -621,31 +516,152 @@ const config = { return undefined }, redirects: [ - // British → American spelling (`optimization` URLs) + // Features page replaced by intro (intro.mdx has slug: / so it lives at /docs/) { - to: '/docs/iceberg-maintenance/optimization/overview', - from: '/docs/iceberg-maintenance/optimisation/overview' + to: '/docs/', + from: '/docs/features' }, + // Fusion release notes moved to fusion/release/maintenance { - to: '/docs/iceberg-maintenance/optimization/configuration', - from: '/docs/iceberg-maintenance/optimisation/configuration' + to: '/docs/fusion/release/maintenance/overview', + from: '/docs/release/maintenance/overview' + }, + { + to: '/docs/fusion/release/maintenance/v0.1.0', + from: '/docs/release/maintenance/v0.1.0' + }, + // Fusion maintenance pages moved from iceberg-maintenance to fusion/maintenance + { + to: '/docs/fusion/maintenance/catalogs', + from: '/docs/iceberg-maintenance/catalogs' + }, + { + to: '/docs/fusion/maintenance/metrics', + from: '/docs/iceberg-maintenance/metrics' + }, + { + to: '/docs/fusion/maintenance/runs-and-logs', + from: '/docs/iceberg-maintenance/runs-and-logs' + }, + // Fusion overview moved from iceberg-maintenance to fusion/getting-started + { + to: '/docs/fusion/getting-started/overview', + from: [ + '/docs/iceberg-maintenance/overview', + '/docs/iceberg-maintenance/compaction/overview', + '/docs/iceberg-maintenance/optimisation/overview', + '/docs/iceberg-maintenance/optimization/overview' + ] + }, + { + to: '/docs/fusion/compaction/configuration', + from: [ + '/docs/iceberg-maintenance/compaction/configuration', + '/docs/iceberg-maintenance/optimisation/configuration', + '/docs/iceberg-maintenance/optimization/configuration' + ] }, { - to: '/docs/getting-started/configure-first-optimization', - from: '/docs/getting-started/configure-first-optimisation' + to: '/docs/fusion/getting-started/configure-first-compaction', + from: [ + '/docs/getting-started/configure-first-compaction', + '/docs/getting-started/configure-first-optimisation', + '/docs/getting-started/configure-first-optimization' + ] }, { to: '/docs/benchmarks/ingestion', from: '/docs/benchmarks' }, { - to: '/docs/benchmarks/optimization', - from: '/docs/benchmarks/optimisation' + to: '/docs/fusion/getting-started/compaction', + from: [ + '/docs/benchmarks/compaction', + '/docs/benchmarks/optimisation', + '/docs/benchmarks/optimization' + ] }, { to: '/docs/install/kubernetes', from: '/docs/install/kubernetes-ingestion' }, + { + to: '/docs/fusion/install/kubernetes-compaction', + from: [ + '/docs/install/kubernetes-compaction', + '/docs/install/kubernetes-optimisation', + '/docs/install/kubernetes-optimization' + ] + }, + // Legacy release-note URLs -> ingestion release notes + { + to: '/docs/release/ingestion/overview', + from: '/docs/release/overview' + }, + { + to: '/docs/release/ingestion/v0.6.0', + from: '/docs/release/v0.6.0' + }, + { + to: '/docs/release/ingestion/v0.5.0', + from: '/docs/release/v0.5.0' + }, + { + to: '/docs/release/ingestion/v0.4.0', + from: '/docs/release/v0.4.0' + }, + { + to: '/docs/release/ingestion/v0.3.17', + from: '/docs/release/v0.3.17' + }, + { + to: '/docs/release/ingestion/v0.3.14', + from: '/docs/release/v0.3.14' + }, + { + to: '/docs/release/ingestion/v0.3.9-v0.3.11', + from: '/docs/release/v0.3.9-v0.3.11' + }, + { + to: '/docs/release/ingestion/v0.3.5', + from: '/docs/release/v0.3.5' + }, + { + to: '/docs/release/ingestion/v0.2.10', + from: '/docs/release/v0.2.10' + }, + { + to: '/docs/release/ingestion/v0.2.8', + from: '/docs/release/v0.2.8' + }, + { + to: '/docs/release/ingestion/v0.2.5-v0.2.7', + from: '/docs/release/v0.2.5-v0.2.7' + }, + { + to: '/docs/release/ingestion/v0.2.2-v0.2.4', + from: '/docs/release/v0.2.2-v0.2.4' + }, + { + to: '/docs/release/ingestion/v0.2.0-v0.2.1', + from: '/docs/release/v0.2.0-v0.2.1' + }, + { + to: '/docs/release/ingestion/v0.1.9-v0.1.11', + from: '/docs/release/v0.1.9-v0.1.11' + }, + { + to: '/docs/release/ingestion/v0.1.6-v0.1.8', + from: '/docs/release/v0.1.6-v0.1.8' + }, + { + to: '/docs/release/ingestion/v0.1.2-v0.1.5', + from: '/docs/release/v0.1.2-v0.1.5' + }, + { + to: '/docs/release/ingestion/v0.1.0-v0.1.1', + from: '/docs/release/v0.1.0-v0.1.1' + }, { to: '/docs/benchmarks/ingestion?tab=mongodb', @@ -963,7 +979,7 @@ const config = { from: '/docs/connectors/mongodb/state' }, { - to: '/docs/release/overview', + to: '/docs/release/ingestion/overview', from: '/docs/release-notes' }, { diff --git a/sidebars.js b/sidebars.js index 55cf87da0..994d32d2a 100644 --- a/sidebars.js +++ b/sidebars.js @@ -24,316 +24,113 @@ const docSidebar = { // module.exports = { docSidebar: [ // OVERVIEW - sectionHeader("Overview"), - 'intro', + sectionHeader("GET STARTED"), + { type: 'doc', id: 'intro', label: 'Overview' }, { type: 'category', label: 'Benchmarks', items: [ - { - type: 'doc', - id: 'benchmarks/ingestion', - label: 'Ingestion Benchmarks', - }, - { - type: 'doc', - id: 'benchmarks/optimization', - label: 'Optimization Benchmarks', - }, - { - type: 'doc', - id: 'dmsvsolake', - label: 'AWS DMS vs OLake', - }, + { type: 'doc', id: 'benchmarks/ingestion', label: 'Ingestion Benchmarks' }, + { type: 'doc', id: 'dmsvsolake', label: 'AWS DMS vs OLake Go' }, ], }, + { type: 'doc', id: 'getting-started/quickstart', label: 'Quickstart' }, + { type: 'doc', id: 'getting-started/creating-first-pipeline', label: 'Configure Your Pipeline' }, + { type: 'doc', id: 'getting-started/playground', label: 'Playground' }, + + // INSTALL & CONFIGURE + sectionHeader("INSTALL & CONFIGURE"), { type: 'category', - label: 'Install', + label: 'Docker Compose (UI)', items: [ - { - type: 'category', - label: 'Docker (UI)', - items: [ - { - type: 'doc', - id: 'install/olake-ui/index', - label: 'Docker Compose', - }, - { - type: 'doc', - id: 'install/olake-ui/offline-environments-aws', - label: 'Offline Environments (AWS)', - }, - { - type: 'doc', - id: 'install/olake-ui/offline-environments-generic', - label: 'Offline Environments (Generic)', - }, - ], - }, - { - type: 'doc', - id: 'install/docker-cli', - label: 'Docker Compose (CLI)', - }, - { - type: 'doc', - id: 'install/kubernetes', - label: 'Kubernetes Installation', - }, + { type: 'doc', id: 'install/olake-ui/index', label: 'Docker Compose' }, + { type: 'doc', id: 'install/olake-ui/offline-environments-aws', label: 'Offline Environments (AWS)' }, + { type: 'doc', id: 'install/olake-ui/offline-environments-generic', label: 'Offline Environments (Generic)' }, ], }, + { type: 'doc', id: 'install/docker-cli', label: 'Docker CLI' }, + { type: 'doc', id: 'install/kubernetes', label: 'Kubernetes/Helm' }, - // SERVICES - sectionHeader("SERVICES"), + // MOVE AND MANAGE DATA + sectionHeader("MOVE AND MANAGE DATA"), { type: 'category', - label: 'Ingestion', + label: 'Sources', + items: [ + { type: 'doc', id: 'connectors/postgres/index', label: 'PostgreSQL' }, + { type: 'doc', id: 'connectors/mongodb/index', label: 'MongoDB' }, + { type: 'doc', id: 'connectors/mysql/index', label: 'MySQL' }, + { type: 'doc', id: 'connectors/oracle/index', label: 'Oracle' }, + { type: 'doc', id: 'connectors/kafka/index', label: 'Kafka' }, + { type: 'doc', id: 'connectors/db2/index', label: 'DB2 LUW' }, + { type: 'doc', id: 'connectors/s3/index', label: 'S3' }, + { type: 'doc', id: 'connectors/mssql/index', label: 'MSSQL' }, + ], + }, + { + type: 'category', + label: 'Destinations', items: [ - { - type: 'doc', - id: 'features/index', - label: 'Overview', - }, - - { - type: 'category', - label: 'Quickstart', - items: [ - { - type: 'doc', - id: 'getting-started/quickstart', - label: 'Getting Started', - }, - { - type: 'doc', - id: 'getting-started/creating-first-pipeline', - label: 'Create Your First Job Pipeline', - }, - { - type: 'doc', - id: 'getting-started/playground', - label: 'Playground', - }, - ], - }, - { - type: 'category', - label: 'Sources', - items: [ - { - type: 'doc', - id: 'connectors/postgres/index', - label: 'PostgreSQL', - }, - { - type: 'doc', - id: 'connectors/mongodb/index', - label: 'MongoDB', - }, - { - type: 'doc', - id: 'connectors/mysql/index', - label: 'MySQL', - }, - { - type: 'doc', - id: 'connectors/oracle/index', - label: 'Oracle', - }, - { - type: 'doc', - id: 'connectors/kafka/index', - label: 'Kafka', - }, - { - type: 'doc', - id: 'connectors/db2/index', - label: 'DB2 LUW', - }, - { - type: 'doc', - id: 'connectors/s3/index', - label: 'S3', - }, - { - type: 'doc', - id: 'connectors/mssql/index', - label: 'MSSQL', - }, - ], - }, { type: 'category', - label: 'Destinations', + label: 'Apache Iceberg', items: [ { type: 'category', - label: 'Apache Iceberg', + label: 'Catalogs', items: [ - { - type: 'category', - label: 'Catalogs', - items: [ - { - type: 'doc', - id: 'writers/iceberg/catalog/glue', - label: 'AWS Glue', - }, - { - type: 'doc', - id: 'writers/iceberg/catalog/rest', - label: 'REST Catalog', - }, - { - type: 'doc', - id: 'writers/iceberg/catalog/jdbc', - label: 'JDBC Catalog', - }, - { - type: 'doc', - id: 'writers/iceberg/catalog/hive', - label: 'Hive Metastore', - }, - ], - }, - { - type: 'doc', - id: 'writers/iceberg/partitioning', - label: 'Data Partitioning', - }, - { - type: 'doc', - id: 'writers/iceberg/azure', - label: 'Iceberg On Azure', - }, - { - type: 'doc', - id: 'writers/iceberg/gcp', - label: 'Iceberg on Google Cloud', - }, - { - type: 'doc', - id: 'writers/iceberg/troubleshooting-local', - label: 'Troubleshooting & Local Testing', - }, - ], - }, - { - type: 'category', - label: 'Parquet Writer', - items: [ - { - type: 'doc', - id: 'writers/parquet/config', - label: 'Configuration', - }, - { - type: 'doc', - id: 'writers/parquet/permission', - label: 'IAM Permissions', - }, - { - type: 'doc', - id: 'writers/parquet/partitioning', - label: 'Partitioning', - }, - { - type: 'doc', - id: 'writers/parquet/troubleshoot', - label: 'Troubleshooting', - }, + { type: 'doc', id: 'writers/iceberg/catalog/glue', label: 'AWS Glue' }, + { type: 'doc', id: 'writers/iceberg/catalog/rest', label: 'REST Catalog' }, + { type: 'doc', id: 'writers/iceberg/catalog/jdbc', label: 'JDBC Catalog' }, + { type: 'doc', id: 'writers/iceberg/catalog/hive', label: 'Hive Metastore' }, ], }, + { type: 'doc', id: 'writers/iceberg/partitioning', label: 'Data Partitioning' }, + { type: 'doc', id: 'writers/iceberg/azure', label: 'Iceberg On Azure' }, + { type: 'doc', id: 'writers/iceberg/gcp', label: 'Iceberg on Google Cloud' }, + { type: 'doc', id: 'writers/iceberg/troubleshooting-local', label: 'Troubleshooting & Local Testing' }, ], }, { type: 'category', - label: 'Features', + label: 'Parquet Writer', items: [ - { - type: 'doc', - id: 'getting-started/job-level-properties', - label: 'Job level features', - }, - { - type: 'doc', - id: 'getting-started/alerts-and-notifications', - label: 'Alerts & Notifications', - }, + { type: 'doc', id: 'writers/parquet/config', label: 'Configuration' }, + { type: 'doc', id: 'writers/parquet/permission', label: 'IAM Permissions' }, + { type: 'doc', id: 'writers/parquet/partitioning', label: 'Partitioning' }, + { type: 'doc', id: 'writers/parquet/troubleshoot', label: 'Troubleshooting' }, ], }, - { - type: 'doc', - id: 'understanding/terminologies/olake', - label: 'Properties', - }, ], }, - { type: 'category', - label: 'Iceberg Maintenance', + label: 'Properties', items: [ - { - type: 'doc', - id: 'iceberg-maintenance/overview', - label: 'Overview', - }, - { - type: 'category', - label: 'Quickstart', - items: [ - { - type: 'doc', - id: 'getting-started/configure-first-optimization', - label: 'Configure Your First Optimization', - }, - ], - }, - { - type: 'doc', - id: 'iceberg-maintenance/catalogs', - label: 'Catalogs', - }, - { - type: 'category', - label: 'Optimization', - items: [ - { - type: 'doc', - id: 'iceberg-maintenance/optimization/overview', - label: 'Types of Optimization', - }, - { - type: 'doc', - id: 'iceberg-maintenance/optimization/configuration', - label: 'Configuration', - }, - ], - }, - { - type: 'doc', - id: 'iceberg-maintenance/runs-and-logs', - label: 'Logs & Runs', - }, - { - type: 'doc', - id: 'iceberg-maintenance/metrics', - label: 'Metrics', - }, + { type: 'doc', id: 'getting-started/job-level-properties', label: 'Job-level Properties' }, + { type: 'doc', id: 'understanding/terminologies/olake', label: 'Stream-level Properties' }, + { type: 'doc', id: 'getting-started/alerts-and-notifications', label: 'Alerts & Notifications' }, ], }, - // UNDERSTANDING OLAKE - sectionHeader("UNDERSTANDING OLAKE"), - 'understanding/terminologies/general', - 'core/architecture', - 'understanding/compatibility-catalogs', - 'understanding/compatibility-engines', - 'core/use-cases', - + // CORE CONCEPTS + sectionHeader("CORE CONCEPTS"), + { type: 'doc', id: 'core/architecture', label: 'Core Architecture' }, + { type: 'doc', id: 'understanding/compatibility-catalogs', label: 'Catalog Compatibility' }, + { type: 'doc', id: 'understanding/compatibility-engines', label: 'Query Engine Compatibility' }, + { type: 'doc', id: 'understanding/terminologies/general', label: 'Terminologies' }, + { type: 'doc', id: 'features/schema', label: 'Schema Evolution' }, + { type: 'doc', id: 'core/use-cases', label: 'Use Cases' }, + + // API REFERENCE + sectionHeader("API DOCUMENTATION"), + { + type: 'doc', + label: 'OLake UI API', + id: 'api/olake-ui-api', + }, + // Community sectionHeader("COMMUNITY"), 'community/contributing', @@ -356,37 +153,98 @@ const docSidebar = { // RELEASE NOTES sectionHeader("RELEASE NOTES"), + 'release/ingestion/overview', { type: 'category', - label: 'Ingestion', + label: 'Versions', items: [ - 'release/overview', - 'release/v0.6.0', - 'release/v0.5.0', - 'release/v0.4.0', - 'release/v0.3.17', - 'release/v0.3.14', - 'release/v0.3.9-v0.3.11', - 'release/v0.3.5', - 'release/v0.2.10', - 'release/v0.2.8', - 'release/v0.2.5-v0.2.7', - 'release/v0.2.2-v0.2.4', - 'release/v0.2.0-v0.2.1', - 'release/v0.1.9-v0.1.11', - 'release/v0.1.6-v0.1.8', - 'release/v0.1.2-v0.1.5', - 'release/v0.1.0-v0.1.1', + 'release/ingestion/v0.9.0', + 'release/ingestion/v0.8.0', + 'release/ingestion/v0.7.0', + 'release/ingestion/v0.6.0', + 'release/ingestion/v0.5.0', + 'release/ingestion/v0.4.0', + 'release/ingestion/v0.3.17', + 'release/ingestion/v0.3.14', + 'release/ingestion/v0.3.9-v0.3.11', + 'release/ingestion/v0.3.5', + 'release/ingestion/v0.2.10', + 'release/ingestion/v0.2.8', + 'release/ingestion/v0.2.5-v0.2.7', + 'release/ingestion/v0.2.2-v0.2.4', + 'release/ingestion/v0.2.0-v0.2.1', + 'release/ingestion/v0.1.9-v0.1.11', + 'release/ingestion/v0.1.6-v0.1.8', + 'release/ingestion/v0.1.2-v0.1.5', + 'release/ingestion/v0.1.0-v0.1.1', ], }, + ], +}; + +// ─── FUSION SIDEBAR ──────────────────────────────────────────────────────────── +const fusionDocSidebar = { + fusionDocSidebar: [ + + // GET STARTED + sectionHeader("GET STARTED"), + { type: 'doc', id: 'fusion/getting-started/overview', label: 'Overview' }, + { type: 'doc', id: 'fusion/getting-started/compaction', label: 'Benchmarks' }, + { type: 'doc', id: 'fusion/getting-started/quickstart', label: 'Quickstart' }, + { type: 'doc', id: 'fusion/getting-started/configure-first-compaction', label: 'Configure Your First Compaction' }, + + // INSTALL & CONFIGURE + sectionHeader("INSTALL & CONFIGURE"), + { + type: 'category', + label: 'Docker Compose (UI)', + items: [ + { type: 'doc', id: 'fusion/install/olake-ui/index', label: 'Docker Compose' }, + { type: 'doc', id: 'fusion/install/olake-ui/offline-environments-aws', label: 'Offline Environments (AWS)' }, + { type: 'doc', id: 'fusion/install/olake-ui/offline-environments-generic', label: 'Offline Environments (Generic)' }, + ], + }, + { type: 'doc', id: 'fusion/install/kubernetes-compaction', label: 'Kubernetes/Helm' }, + + // MAINTENANCE + sectionHeader("MAINTENANCE"), + { type: 'doc', id: 'fusion/maintenance/catalogs', label: 'Catalogs' }, + { + type: 'category', + label: 'Compaction', + items: [ + { type: 'doc', id: 'fusion/compaction/types-of-compaction', label: 'Types of Compaction' }, + { type: 'doc', id: 'fusion/compaction/configuration', label: 'Configuration' }, + ], + }, + { type: 'doc', id: 'fusion/maintenance/runs-and-logs', label: 'Runs & Logs' }, + { type: 'doc', id: 'fusion/maintenance/metrics', label: 'Metrics' }, + + // CORE CONCEPTS + sectionHeader("CORE CONCEPTS"), + { type: 'doc', id: 'fusion/core/architecture', label: 'Architecture' }, + { type: 'doc', id: 'fusion/core/compatibility/query-engines', label: 'Query Engines' }, + { type: 'doc', id: 'fusion/core/terminologies', label: 'Terminologies' }, + { type: 'doc', id: 'fusion/core/use-cases', label: 'Use Cases' }, + + // COMMUNITY + sectionHeader("COMMUNITY"), + { type: 'doc', id: 'fusion/community/contributing', label: 'Contributing' }, + { type: 'doc', id: 'fusion/community/issues-and-prs', label: 'How to Raise a PR' }, + { type: 'doc', id: 'fusion/community/code-of-conduct', label: 'Code of Conduct' }, + { type: 'doc', id: 'fusion/community/channels', label: 'Channels' }, + + // RELEASE NOTES + sectionHeader("RELEASE NOTES"), + 'fusion/release/maintenance/overview', { type: 'category', - label: 'Optimization', + label: 'Versions', items: [ - 'release/overview' + 'fusion/release/maintenance/v0.1.0', ], }, ], }; -export default docSidebar; \ No newline at end of file +export default { ...docSidebar, ...fusionDocSidebar }; \ No newline at end of file diff --git a/src/clientModules/deferredGtag.ts b/src/clientModules/deferredGtag.ts new file mode 100644 index 000000000..0be32e2db --- /dev/null +++ b/src/clientModules/deferredGtag.ts @@ -0,0 +1,81 @@ +// Loads GA after the browser goes idle instead of during page load, keeping 166KB of +// third-party JS off the critical path. dataLayer is initialised immediately, so hits +// fired before the script arrives are queued and flushed on load — nothing is lost. + +import ExecutionEnvironment from '@docusaurus/ExecutionEnvironment' + +const GA_ID = 'G-GTNTGHDNZW' +const IDLE_TIMEOUT = 3000 + +// `gtag` is already declared on Window in global.d.ts; only dataLayer is missing. +declare global { + interface Window { + dataLayer: unknown[] + } +} + +type RouteLocation = { pathname: string; search: string; hash: string } + +let scriptRequested = false + +function initDataLayer(): void { + if (typeof window.gtag === 'function') return + window.dataLayer = window.dataLayer || [] + // gtag.js expects the raw `arguments` object here, not an array — this is the shape + // Google's own snippet pushes, and the one verified to produce hits. + window.gtag = function gtag() { + // eslint-disable-next-line prefer-rest-params + window.dataLayer.push(arguments) + } + window.gtag('js', new Date()) + window.gtag('config', GA_ID, { anonymize_ip: true }) +} + +function loadGtagScript(): void { + if (scriptRequested) return + scriptRequested = true + const script = document.createElement('script') + script.async = true + script.src = `https://www.googletagmanager.com/gtag/js?id=${GA_ID}` + document.head.appendChild(script) +} + +function scheduleLoad(): void { + const schedule = () => { + if (typeof window.requestIdleCallback === 'function') { + window.requestIdleCallback(loadGtagScript, { timeout: IDLE_TIMEOUT }) + } else { + window.setTimeout(loadGtagScript, 1500) + } + } + + if (document.readyState === 'complete') schedule() + else window.addEventListener('load', schedule, { once: true }) + + // Anyone who engages before idle fires gets the script straight away. + const opts: AddEventListenerOptions = { once: true, passive: true } + ;['pointerdown', 'keydown', 'touchstart'].forEach((evt) => + window.addEventListener(evt, loadGtagScript, opts) + ) +} + +if (ExecutionEnvironment.canUseDOM) { + initDataLayer() + scheduleLoad() +} + +// Docusaurus is a SPA, so only the first page view is automatic; the rest are sent here. +export function onRouteDidUpdate({ + location, + previousLocation +}: { + location: RouteLocation + previousLocation: RouteLocation | null +}): void { + if (!previousLocation || previousLocation.pathname === location.pathname) return + window.gtag?.('event', 'page_view', { + page_title: document.title, + page_location: window.location.href, + page_path: location.pathname + location.search + location.hash + }) +} diff --git a/src/clientModules/deferredReo.ts b/src/clientModules/deferredReo.ts new file mode 100644 index 000000000..04ed27ba0 --- /dev/null +++ b/src/clientModules/deferredReo.ts @@ -0,0 +1,53 @@ +// Loads Reo.dev (reo.js) after the browser goes idle instead of during page load, the same +// way deferredGtag.ts handles GA — keeps this third-party script off the critical path so +// it doesn't push LCP/FCP back the way the un-deferred GA snippet did. + +import ExecutionEnvironment from '@docusaurus/ExecutionEnvironment' + +const REO_CLIENT_ID = 'fffabce108fa8ea' +const IDLE_TIMEOUT = 3000 + +declare global { + interface Window { + Reo?: { + init: (config: { clientID: string; enableThirdPartyTracking?: boolean }) => void + } + } +} + +let scriptRequested = false + +function loadReoScript(): void { + if (scriptRequested) return + scriptRequested = true + const script = document.createElement('script') + script.src = `https://static.reo.dev/${REO_CLIENT_ID}/reo.js` + script.defer = true + script.onload = () => { + window.Reo?.init({ clientID: REO_CLIENT_ID, enableThirdPartyTracking: true }) + } + document.head.appendChild(script) +} + +function scheduleLoad(): void { + const schedule = () => { + if (typeof window.requestIdleCallback === 'function') { + window.requestIdleCallback(loadReoScript, { timeout: IDLE_TIMEOUT }) + } else { + window.setTimeout(loadReoScript, 1500) + } + } + + if (document.readyState === 'complete') schedule() + else window.addEventListener('load', schedule, { once: true }) + + // Anyone who engages before idle fires gets the script straight away. + const opts: AddEventListenerOptions = { once: true, passive: true } + ;['pointerdown', 'keydown', 'touchstart'].forEach((evt) => + window.addEventListener(evt, loadReoScript, opts) + ) +} + +if (ExecutionEnvironment.canUseDOM) { + scheduleLoad() +} diff --git a/src/clientModules/hashScroll.ts b/src/clientModules/hashScroll.ts index 50ab33f59..27293a7c5 100644 --- a/src/clientModules/hashScroll.ts +++ b/src/clientModules/hashScroll.ts @@ -112,8 +112,26 @@ export function onClientEntry(): void { } } -export function onRouteDidUpdate({ location }: { location: { hash: string } }): void { - if (location.hash) { - handleSpaNavigation() +export function onRouteDidUpdate({ + location, + previousLocation, +}: { + location: { pathname: string; search: string; hash: string } + previousLocation?: { pathname: string; search: string; hash: string } | null +}): void { + if (!location.hash) return + + // Match Docusaurus core: query-string-only updates (e.g. ) should not + // re-run anchor correction — it would keep jumping back to the hash while the user + // interacts with UI further down the page (same pathname + same hash, search changed). + if ( + previousLocation && + location.pathname === previousLocation.pathname && + location.hash === previousLocation.hash && + location.search !== previousLocation.search + ) { + return } + + handleSpaNavigation() } diff --git a/src/components/CollapsibleTip.tsx b/src/components/CollapsibleTip.tsx new file mode 100644 index 000000000..16a727855 --- /dev/null +++ b/src/components/CollapsibleTip.tsx @@ -0,0 +1,36 @@ +import React, {type ReactNode} from 'react'; +import {Details} from '@docusaurus/theme-common/Details'; +import Translate from '@docusaurus/Translate'; +import IconTip from '@theme/Admonition/Icon/Tip'; +import clsx from 'clsx'; + +type Props = { + summary: string; + children: ReactNode; +}; + +/** Collapsible block with the same green styling and header as a :::tip admonition. */ +export default function CollapsibleTip({summary, children}: Props) { + return ( +
    + + + + + + tip + + +
    + ); +} diff --git a/src/components/FusionBlogCTA.tsx b/src/components/FusionBlogCTA.tsx new file mode 100644 index 000000000..536835c3e --- /dev/null +++ b/src/components/FusionBlogCTA.tsx @@ -0,0 +1,53 @@ +import React from 'react'; +import { FaExternalLinkAlt, FaGithub } from 'react-icons/fa'; + +const FusionBlogCTA = () => { + return ( +
    +

    + OLake Fusion +

    +

    + Open-source lakehouse maintenance for Apache Iceberg tables. 50% cheaper (2x faster) compaction than Vanilla Spark. +

    + + + +
    + Contact us at hello@olake.io +
    +
    + ); +}; + +export default FusionBlogCTA; diff --git a/src/components/TOCTabLinker.tsx b/src/components/TOCTabLinker.tsx new file mode 100644 index 000000000..a4997c8c7 --- /dev/null +++ b/src/components/TOCTabLinker.tsx @@ -0,0 +1,80 @@ +import { useEffect, useRef } from 'react'; + +/** + * Patches Docusaurus TOC sidebar links so clicking them preserves the active + * tab query param instead of stripping it from the URL. + * + * Background: Docusaurus generates TOC links as bare `#hash` anchors at build + * time with no awareness of Tabs `queryString` state. This component fixes + * that at runtime using a MutationObserver so the patch also covers + * late-mounted TOC nodes (e.g. mobile TOC appearing on first resize). + * + * @param headingMap Maps each tab's query-param value → the heading IDs it owns. + * @param queryParam The `queryString` prop value on your . Default: 'config-type'. + * + * @example + * // Two tabs (standard connector pages) + * + */ + +interface Props { + /** Maps each tab query-param value to the heading IDs belonging to that tab. */ + headingMap: Record; + /** The query param name used on the component. */ + queryParam?: string; +} + +export default function TOCTabLinker({ + headingMap, + queryParam = 'config-type', +}: Props): null { + // Build a reverse lookup: headingId → tabValue. + // Stored in refs so the effect closure always reads the latest values + // without needing to re-run observer setup on every render. + const reverseMapRef = useRef>(new Map()); + const queryParamRef = useRef(queryParam); + + reverseMapRef.current = new Map( + Object.entries(headingMap).flatMap(([tabValue, headings]) => + headings.map((h) => [h, tabValue]) + ) + ); + queryParamRef.current = queryParam; + + useEffect(() => { + const patch = () => { + document + .querySelectorAll('.table-of-contents__link') + .forEach((link) => { + const href = link.getAttribute('href') ?? ''; + // Skip links already patched (no longer a bare #hash). + if (!href.startsWith('#')) return; + const hash = href.slice(1); + const tabValue = reverseMapRef.current.get(hash); + if (tabValue !== undefined) { + link.setAttribute( + 'href', + `?${queryParamRef.current}=${tabValue}#${hash}` + ); + } + }); + }; + + // Patch immediately — TOC is already in the DOM when useEffect fires. + patch(); + + // Re-patch when new TOC nodes are added (e.g. mobile TOC mounts lazily). + const observer = new MutationObserver(patch); + observer.observe(document.body, { childList: true, subtree: true }); + + return () => observer.disconnect(); + }, []); // Refs are stable; no stale closures, no deps needed. + + return null; +} diff --git a/src/components/TestimonialCard.tsx b/src/components/TestimonialCard.tsx index 2168ba950..48fde1538 100644 --- a/src/components/TestimonialCard.tsx +++ b/src/components/TestimonialCard.tsx @@ -1,43 +1,87 @@ import React from 'react'; -interface TestimonialCardProps { - quote: string; +interface Author { name: string; title: string; - imageSrc: string; + imageSrc?: string; imageAlt?: string; + linkedinUrl?: string; +} + +interface TestimonialCardProps { + quote: string; + // Either pass a single author via flat props... + name?: string; + title?: string; + imageSrc?: string; + imageAlt?: string; + linkedinUrl?: string; + // ...or pass multiple via the authors array. + authors?: Author[]; +} + +const getInitials = (name: string) => + name + .split(' ') + .map((n) => n[0]) + .slice(0, 2) + .join('') + .toUpperCase(); + +function AuthorBadge({ author }: { author: Author }) { + return ( +
    + {author.imageSrc ? ( + {author.imageAlt + ) : ( +
    + {getInitials(author.name)} +
    + )} +
    + {author.linkedinUrl ? ( + + {author.name} + + ) : ( +

    {author.name}

    + )} +

    {author.title}

    +
    +
    + ); } const TestimonialCard: React.FC = ({ quote, + authors, name, title, imageSrc, - imageAlt + imageAlt, + linkedinUrl, }) => { + const list: Author[] = authors ?? (name && title ? [{ name, title, imageSrc, imageAlt, linkedinUrl }] : []); + return (
    -

    - "{quote}" -

    -
    - {imageAlt -
    -

    - {name} -

    -

    - {title} -

    -
    +

    "{quote}"

    +
    + {list.map((author, i) => ( + + ))}
    ); }; -export default TestimonialCard; - +export default TestimonialCard; \ No newline at end of file diff --git a/src/components/customers/CustomerCard.tsx b/src/components/customers/CustomerCard.tsx index e54deb9ea..87f7387e0 100644 --- a/src/components/customers/CustomerCard.tsx +++ b/src/components/customers/CustomerCard.tsx @@ -54,7 +54,7 @@ const CustomerCard: React.FC = ({ /> {/* Company Logo/Name Overlay */}
    - + {companyName}
    diff --git a/src/components/landing/Navbar/SiteNavbar.css b/src/components/landing/Navbar/SiteNavbar.css new file mode 100644 index 000000000..3852cfa76 --- /dev/null +++ b/src/components/landing/Navbar/SiteNavbar.css @@ -0,0 +1,304 @@ +/* Shared landing-page navbar — values taken verbatim from the v2 homepage + design's NAV block. + * + * NOTE ON SPECIFICITY: the generated page stylesheets contain + * `.olakehome-page a { color: #193AE6 }` (0,1,1), which outranks a bare + * `.olake-nav-cta` (0,1,0) and repainted the CTA's white text blue-on-blue + * (invisible) and the Pricing link blue. Every rule that sets a link colour + * is therefore qualified with `.olake-nav` to reach (0,2,0) and win. */ + +.olake-nav { + /* sticky + z-index come from the design; without them the bar scrolls away */ + position: sticky; + top: 0; + z-index: 50; + width: 100%; + background: rgba(240, 242, 250, 0.82); + backdrop-filter: blur(14px); + border-bottom: 1px solid rgba(0, 0, 0, 0.06); + font-family: 'Space Grotesk', sans-serif; +} + +.olake-nav-inner { + display: flex; + align-items: center; + justify-content: space-between; + padding: 20px 48px; + max-width: 1360px; + margin: 0 auto; +} + +.olake-nav-left { + display: flex; + align-items: center; + gap: 44px; +} + +.olake-nav .olake-nav-logo { + display: flex; + align-items: center; +} + +.olake-nav-logo img { + height: 32px; + width: auto; +} + +.olake-nav-links { + display: flex; + align-items: center; + gap: 30px; + font-size: 14px; + font-weight: 500; + color: #424865; +} + +.olake-nav-item { + position: relative; + padding: 8px 0; +} + +.olake-nav-trigger { + display: flex; + align-items: center; + gap: 5px; + cursor: pointer; + color: #121212; +} + +.olake-nav-caret { + font-size: 9px; + margin-top: 2px; +} + +.olake-nav-panel { + position: absolute; + top: 36px; + left: -16px; + background: #e7ecff; + border: 1px solid rgba(0, 0, 0, 0.09); + border-radius: 12px; + padding: 8px; + min-width: 190px; + display: flex; + flex-direction: column; + visibility: hidden; + opacity: 0; + transition: opacity 0.12s ease; + z-index: 60; +} + +/* Invisible bridge across the gap between the trigger and the panel. + * + * The panel is offset to `top: 36px` but the trigger's box ends a few pixels + * above that, leaving a strip where the pointer is over neither element — + * which fires mouseleave on .olake-nav-item and closes the menu while you are + * still moving toward it. + * + * This works because the panel is a DOM child of .olake-nav-item, so the + * pointer never actually leaves that subtree (mouseleave does not fire when + * moving onto a descendant, even one positioned outside the parent's box). + * The pseudo-element inherits `visibility: hidden` while closed, so it never + * blocks clicks on whatever sits underneath. */ +.olake-nav-panel::before { + content: ''; + position: absolute; + top: -14px; + left: 0; + right: 0; + height: 14px; +} + +.olake-nav-panel.is-open { + visibility: visible; + opacity: 1; +} + +.olake-nav .olake-nav-panel-link { + padding: 10px 14px; + border-radius: 8px; + color: #0a0f23; + font-weight: 500; + text-decoration: none; + white-space: nowrap; +} + +.olake-nav .olake-nav-panel-link:hover { + background: rgba(0, 0, 0, 0.05); + color: #0a0f23; + text-decoration: none; +} + +.olake-nav .olake-nav-plain { + color: #121212; + text-decoration: none; +} + +.olake-nav .olake-nav-plain:hover { + color: #000; + text-decoration: none; +} + +.olake-nav-right { + display: flex; + align-items: center; + gap: 18px; +} + +.olake-nav .olake-nav-stars { + display: flex; + align-items: center; + gap: 8px; + font-family: 'JetBrains Mono', monospace; + font-size: 12px; + color: #424865; + text-decoration: none; + border: 1px solid rgba(0, 0, 0, 0.1); + border-radius: 8px; + padding: 7px 12px; + white-space: nowrap; +} + +.olake-nav .olake-nav-stars:hover { + color: #424865; + text-decoration: none; +} + +.olake-nav-stars img { + height: 28px; + width: 28px; + border-radius: 5px; +} + +.olake-nav-star-mark { + color: #3fa872; +} + +.olake-nav .olake-nav-slack { + display: flex; + align-items: center; +} + +.olake-nav-slack img { + height: 30px; + width: 30px; +} + +.olake-nav .olake-nav-cta { + background: #193ae6; + color: #fff; + padding: 9px 20px; + border-radius: 8px; + font-weight: 600; + font-size: 14px; + text-decoration: none; + white-space: nowrap; + box-shadow: + 0 4px 0 #5762da, + 0 5px 8px rgba(255, 255, 255, 0.18), + inset 0 1px 0 rgba(0, 0, 0, 0.32); + transition: transform 0.06s ease, box-shadow 0.06s ease; +} + +.olake-nav .olake-nav-cta:hover { + color: #fff; + text-decoration: none; + transform: translateY(4px); + box-shadow: 0 0 0 #5762da, inset 0 1px 0 rgba(0, 0, 0, 0.2); +} + +/* --- mobile --------------------------------------------------------------- + These routes hide Docusaurus' navbar (and therefore its mobile sidebar), so + this carries its own menu. 1279px matches the navbar-breakpoint plugin. */ + +.olake-nav-burger { + display: none; + flex-direction: column; + gap: 5px; + cursor: pointer; + padding: 6px; + background: none; + border: none; +} + +.olake-nav-burger span { + width: 22px; + height: 2px; + background: #121212; + border-radius: 2px; +} + +.olake-nav-mobile { + display: flex; + flex-direction: column; + gap: 2px; + padding: 8px 20px 20px; + background: #f0f2fa; + border-top: 1px solid rgba(0, 0, 0, 0.06); +} + +.olake-nav-mobile-group { + display: flex; + flex-direction: column; + padding-bottom: 6px; +} + +.olake-nav-mobile-heading { + font-family: 'JetBrains Mono', monospace; + font-size: 11px; + letter-spacing: 0.08em; + color: #7b84a4; + padding: 12px 6px 4px; +} + +.olake-nav .olake-nav-mobile-link { + padding: 11px 6px; + color: #0a0f23; + font-weight: 600; + font-size: 15px; + text-decoration: none; + border-bottom: 1px solid rgba(0, 0, 0, 0.06); +} + +.olake-nav .olake-nav-mobile-link:hover { + color: #193ae6; + text-decoration: none; +} + +.olake-nav .olake-nav-mobile-cta { + margin-top: 14px; + background: #193ae6; + color: #fff; + padding: 12px 20px; + border-radius: 8px; + font-weight: 600; + font-size: 14px; + text-align: center; + text-decoration: none; +} + +.olake-nav .olake-nav-mobile-cta:hover { + color: #fff; + text-decoration: none; +} + +@media (max-width: 1279px) { + .olake-nav-inner { + padding: 12px 20px; + } + + .olake-nav-links, + .olake-nav-stars { + display: none; + } + + .olake-nav-burger { + display: flex; + } +} + +@media (min-width: 1280px) { + .olake-nav-mobile { + display: none; + } +} diff --git a/src/components/landing/Navbar/SiteNavbar.tsx b/src/components/landing/Navbar/SiteNavbar.tsx new file mode 100644 index 000000000..0f3e01175 --- /dev/null +++ b/src/components/landing/Navbar/SiteNavbar.tsx @@ -0,0 +1,143 @@ +import React, { useState, type ReactNode } from 'react' +import Link from '@docusaurus/Link' +import { cn } from '@site/src/lib/utils' +import useGetReleases from '@site/src/hooks/useGetReleases' +import { + NAV_DROPDOWNS, + PRICING_LINK, + GITHUB_REPO_URL, + SLACK_URL, + CTA, + type NavDropdown +} from './navData' + +/** Formats 4231 -> "4.2k", matching the design's star pill. */ +function formatStars(n: number): string { + if (!n) return '—' + return n >= 1000 ? `${(n / 1000).toFixed(1)}k` : String(n) +} + +function Dropdown({ dropdown }: { dropdown: NavDropdown }) { + const [open, setOpen] = useState(false) + return ( +
    setOpen(true)} + onMouseLeave={() => setOpen(false)} + > +
    + {dropdown.label} + ▾ +
    + {/* Always rendered so the links are crawlable and work without JS; only visibility toggles. */} +
    + {dropdown.items.map((item) => ( + + {item.label} + + ))} +
    +
    + ) +} + +interface SiteNavbarProps { + /** Optional extra content at the far right, after the CTA. */ + trailing?: ReactNode +} + +/** + * The navbar shared by the three landing pages, following the v2 homepage + * design. Those routes hide Docusaurus' own navbar (see custom.css), so this + * carries its own mobile menu — without it there would be no navigation + * below the 1279px breakpoint. + */ +export default function SiteNavbar({ trailing }: SiteNavbarProps) { + const { stargazersCount } = useGetReleases() + const stars = formatStars(stargazersCount) + const [mobileOpen, setMobileOpen] = useState(false) + + return ( +
    +
    +
    + + OLake + +
    + {NAV_DROPDOWNS.map((d) => ( + + ))} + + {PRICING_LINK.label} + +
    +
    + +
    + + + + ★ {stars} + + + + {/* the design uses the Slack artwork here, not an icon font */} + Slack + + + {CTA.label} + + {trailing} + +
    +
    + + {mobileOpen && ( +
    + {NAV_DROPDOWNS.map((d) => ( +
    +
    {d.label}
    + {d.items.map((item) => ( + setMobileOpen(false)} + > + {item.label} + + ))} +
    + ))} + setMobileOpen(false)} + > + {PRICING_LINK.label} + + setMobileOpen(false)}> + {CTA.label} + +
    + )} +
    + ) +} diff --git a/src/components/landing/Navbar/navData.ts b/src/components/landing/Navbar/navData.ts new file mode 100644 index 000000000..6daebd09b --- /dev/null +++ b/src/components/landing/Navbar/navData.ts @@ -0,0 +1,81 @@ +export interface NavLink { + label: string + href: string +} + +export interface NavDropdown { + label: string + items: NavLink[] +} + +/** + * Information architecture for the shared site-wide navbar. + * + * Styling/structure follow the v2 homepage design, but the design's nav only + * covers Products / Docs / Resources / Pricing. Everything that existed in + * docusaurus.config.js is preserved here so no destination is lost: + * + * config item -> where it lives now + * -------------------------------------------------------------- + * Docs (dropdown) -> Docs (unchanged) + * Blogs -> Resources > Blog + * Customer Stories -> Resources > Customer Stories + * Community > Webinars & Events -> Resources > Webinars & Events + * Community > OLake Community -> Community > OLake Community + * Community > Top Contributors -> Community > Top Contributors + * Community > Contributor's Prog. -> Community > Contributor's Program + * Community > GSoC -> Community > GSoC + * Iceberg (dropdown) -> Iceberg (kept top-level, as requested) + * Pricing -> Pricing + * + * NOTE: `themeConfig.navbar.items` is not read for these — the design's + * dropdown panels can't be expressed as plain Docusaurus `dropdown` items. + * That config must still stay non-empty: theme-common disables the mobile + * burger entirely when `navbar.items.length === 0`. + */ +export const NAV_DROPDOWNS: NavDropdown[] = [ + { + label: 'Products', + items: [ + { label: 'OLake Go', href: '/olake-go' }, + { label: 'OLake Fusion', href: '/olake-fusion' } + ] + }, + { + label: 'Docs', + items: [ + { label: 'OLake Go', href: '/docs' }, + { label: 'OLake Fusion', href: '/docs/fusion/getting-started/overview' } + ] + }, + { + label: 'Resources', + items: [ + { label: 'Blog', href: '/blog' }, + { label: 'Customer Stories', href: '/customer-stories' }, + { label: 'Webinars & Events', href: '/webinar' } + ] + }, + { + label: 'Community', + items: [ + { label: 'OLake Community', href: '/community' }, + { label: 'Top Contributors', href: '/community/contributors' }, + { label: 'Contributor Program', href: '/community/contributor-program' }, + { label: 'GSoC', href: '/community/gsoc' } + ] + }, + { + label: 'Iceberg', + items: [ + { label: 'Iceberg Blogs', href: '/iceberg' }, + { label: 'Query Engine', href: '/iceberg/query-engine' } + ] + } +] + +export const PRICING_LINK: NavLink = { label: 'Pricing', href: '/contact' } + +export const GITHUB_REPO_URL = 'https://github.com/datazip-inc/olake' +export const SLACK_URL = '/slack' +export const CTA = { label: "Try, it's free!", href: '/docs/getting-started/quickstart/' } diff --git a/src/components/landing/pages/cssToObj.ts b/src/components/landing/pages/cssToObj.ts new file mode 100644 index 000000000..b7311e940 --- /dev/null +++ b/src/components/landing/pages/cssToObj.ts @@ -0,0 +1,54 @@ +import type { CSSProperties } from 'react' + +/** + * Parses a raw CSS declaration string into a React style object. + * + * The designs build some styles as strings in their logic block (e.g. + * `icebergStyle`), and JSX `style` needs an object — this converts at the + * consumption point so the generated markup can stay a literal copy. + */ +export function cssToObj(css: string | CSSProperties | undefined | null): CSSProperties { + if (!css) return {} + if (typeof css !== 'string') return css + + const out: Record = {} + let depth = 0 + let quote: string | null = null + let cur = '' + const decls: string[] = [] + + for (const ch of css) { + if (quote) { + cur += ch + if (ch === quote) quote = null + continue + } + if (ch === '"' || ch === "'") { + quote = ch + cur += ch + continue + } + if (ch === '(') depth++ + if (ch === ')') depth-- + if (ch === ';' && depth === 0) { + if (cur.trim()) decls.push(cur.trim()) + cur = '' + continue + } + cur += ch + } + if (cur.trim()) decls.push(cur.trim()) + + for (const d of decls) { + const i = d.indexOf(':') + if (i < 0) continue + const name = d.slice(0, i).trim() + const value = d.slice(i + 1).trim() + if (!name) continue + const key = name.startsWith('--') + ? name + : name.replace(/-([a-z])/g, (_, c: string) => c.toUpperCase()) + out[key] = value + } + return out as CSSProperties +} diff --git a/src/components/landing/pages/olake-fusion.css b/src/components/landing/pages/olake-fusion.css new file mode 100644 index 000000000..e42aa3b19 --- /dev/null +++ b/src/components/landing/pages/olake-fusion.css @@ -0,0 +1,64 @@ +html:has(.olakefusion-page), body:has(.olakefusion-page){ height: auto; min-height: 100%; } +html:has(.olakefusion-page){ scroll-snap-type: y mandatory; } +body:has(.olakefusion-page){ margin: 0; background: #F5F6FA; } +.olakefusion-page *{ box-sizing: border-box; } +.olakefusion-page a{ color: #3D4FF0; text-decoration: none; } +.olakefusion-page a:hover{ color: #2A38C4; } +.olakefusion-page h1, .olakefusion-page h2, .olakefusion-page h3, .olakefusion-page h4, .olakefusion-page h5, .olakefusion-page h6{ margin: 0; font-weight: inherit; } +@keyframes floatBox{ 0%, 100% { transform: translateY(0); } 50% { transform: translateY(-8px); } } +@keyframes travelDot{ 0% { left: 0%; opacity: 0; } 10% { opacity: 1; } 90% { opacity: 1; } 100% { left: 100%; opacity: 0; } } +@keyframes pulseRow{ 0%, 100% { background: #171E44; } 50% { background: #2A3568; } } +@keyframes logoSpin{ 0%, 100% { transform: rotate(0deg) scale(1); } 50% { transform: rotate(8deg) scale(1.08); } } +@keyframes fadeSlideIn{ from { opacity: 0; transform: translateX(14px); } to { opacity: 1; transform: translateX(0); } } +@keyframes wordFloat{ 0%, 100% { transform: translateY(0) rotate(var(--r, 0deg)); } 50% { transform: translateY(-5px) rotate(var(--r, 0deg)); } } +@keyframes feedIn{ 0% { transform: translateX(0) scale(1); opacity:1; } 65% { opacity:1; } 100% { transform: translateX(120px) scale(0.3); opacity:0; } } +@keyframes compactAppear{ 0%, 15% { opacity:0; transform:scale(0.5); } 35%, 85% { opacity:1; transform:scale(1); } 100% { opacity:0; transform:scale(1.1); } } +@keyframes fusionPulseRing{ 0%, 100% { box-shadow: 0 0 0 0 rgba(61,79,240,0.35); } 50% { box-shadow: 0 0 0 12px rgba(61,79,240,0); } } +@keyframes lightUp1{ 0%, 8% { background:#E4E6F0; } 16%, 88% { background:#A9D8F0; } 96%, 100% { background:#E4E6F0; } } +@keyframes lightUp2{ 0%, 32% { background:#E4E6F0; } 40%, 88% { background:#A9D8F0; } 96%, 100% { background:#E4E6F0; } } +@keyframes lightUp3{ 0%, 56% { background:#E4E6F0; } 64%, 88% { background:#A9D8F0; } 96%, 100% { background:#E4E6F0; } } +.olakefusion-page .nav-burger{ display: none; } +@media (max-width: 900px){ +.olakefusion-page .scroll-pipeline{ display: none !important; } +html:has(.olakefusion-page){ scroll-snap-type: none; } +.olakefusion-page .nav-links, .olakefusion-page .nav-star{ display: none !important; } +.olakefusion-page .nav-burger{ display: flex !important; } +.olakefusion-page .nav-inner{ padding: 14px 20px !important; } +.olakefusion-page .hero-wrap{ padding: 24px 20px 8px !important; } +.olakefusion-page .hero-title{ font-size: 32px !important; } +.olakefusion-page .hero-sub{ width: auto !important; max-width: 100% !important; height: auto !important; font-size: 16px !important; } +.olakefusion-page .hero-btns{ flex-wrap: wrap !important; gap: 12px !important; } +.olakefusion-page .problem-grid{ grid-template-columns: repeat(2, 1fr) !important; } +.olakefusion-page .problem-wrap{ padding: 32px 20px 56px !important; } +.olakefusion-page .problem-title{ font-size: 26px !important; } +.olakefusion-page .features-wrap{ padding: 48px 20px 20px !important; } +.olakefusion-page .features-grid{ grid-template-columns: 1fr !important; gap: 28px !important; } +.olakefusion-page .features-card{ padding: 24px !important; } +.olakefusion-page .sec-title{ font-size: 28px !important; } +.olakefusion-page .bench-wrap{ padding: 60px 16px 30px !important; } +.olakefusion-page .cta-wrap{ padding: 0 16px !important; } +.olakefusion-page .cta-inner{ padding: 28px 24px !important; } +.olakefusion-page .cta-title{ width: auto !important; max-width: 100% !important; height: auto !important; font-size: 24px !important; } +.olakefusion-page .faq-wrap{ padding: 48px 20px 90px !important; } +.olakefusion-page .faq-title{ font-size: 26px !important; } +.olakefusion-page .footer-wrap{ padding: 48px 24px 0 !important; } +.olakefusion-page .footer-hero{ font-size: 28px !important; } +.olakefusion-page .footer-watermark{ font-size: 120px !important; } +} +@media (max-width: 560px){ +.olakefusion-page .hero-title{ font-size: 27px !important; } +.olakefusion-page .problem-grid{ grid-template-columns: 1fr !important; } +.olakefusion-page .arch-inner{ transform: scale(0.46); transform-origin: top center; } +.olakefusion-page .arch-wrap{ height: 118px !important; overflow: hidden; padding: 6px 12px !important; margin: 8px auto 8px !important; } +.olakefusion-page .bench-row{ padding-left: 16px !important; padding-right: 16px !important; } +.olakefusion-page .bench-cell-metric{ font-size: 13px !important; } +.olakefusion-page .faq-q{ font-size: 16px !important; } +.olakefusion-page .footer-watermark{ font-size: 84px !important; } +} + +/* style-hover rules */ +.olakefusion-page .olakefusion-h0:hover{background:#F4F2EC !important;} +.olakefusion-page .olakefusion-h1:hover{color:#1B1E2B !important;} +.olakefusion-page .olakefusion-h2:hover{box-shadow: 0 2px 0 #23309E !important; transform: translateY(1px) !important;} +.olakefusion-page .olakefusion-h3:hover{box-shadow: 0 2px 0 #10173A !important; transform: translateY(1px) !important;} +.olakefusion-page .olakefusion-h4:hover{border-color:#3D4FF0 !important; color:#3D4FF0 !important;} diff --git a/src/components/landing/pages/olake-go.css b/src/components/landing/pages/olake-go.css new file mode 100644 index 000000000..3b55f9cc6 --- /dev/null +++ b/src/components/landing/pages/olake-go.css @@ -0,0 +1,63 @@ +html:has(.olakego-page), body:has(.olakego-page){ height: auto; min-height: 100%; } +html:has(.olakego-page){ scroll-snap-type: y proximity; } +body:has(.olakego-page){ margin: 0; background: #F5F6FA; font-family: 'Space Grotesk', sans-serif; } +.olakego-page *{ box-sizing: border-box; } +.olakego-page a{ color: #3D4FF0; text-decoration: none; } +.olakego-page a:hover{ color: #2A38C4; } +@keyframes floatBox{ 0%, 100% { transform: translateY(0); } 50% { transform: translateY(-8px); } } +@keyframes travelDot{ 0% { left: 0%; opacity: 0; } 10% { opacity: 1; } 90% { opacity: 1; } 100% { left: 100%; opacity: 0; } } +@keyframes pulseRow{ 0%, 100% { background: #171E44; } 50% { background: #2A3568; } } +@keyframes logoSpin{ 0%, 100% { transform: rotate(0deg) scale(1); } 50% { transform: rotate(8deg) scale(1.08); } } +@keyframes fadeSlideIn{ from { opacity: 0; transform: translateX(14px); } to { opacity: 1; transform: translateX(0); } } +@keyframes wordFloat{ 0%, 100% { transform: translateY(0) rotate(var(--r, 0deg)); } 50% { transform: translateY(-5px) rotate(var(--r, 0deg)); } } +@keyframes feedIn{ 0% { transform: translateX(0) scale(1); opacity:1; } 65% { opacity:1; } 100% { transform: translateX(120px) scale(0.3); opacity:0; } } +@keyframes compactAppear{ 0%, 15% { opacity:0; transform:scale(0.5); } 35%, 85% { opacity:1; transform:scale(1); } 100% { opacity:0; transform:scale(1.1); } } +@keyframes fusionPulseRing{ 0%, 100% { box-shadow: 0 0 0 0 rgba(61,79,240,0.35); } 50% { box-shadow: 0 0 0 12px rgba(61,79,240,0); } } +@keyframes lightUp1{ 0%, 8% { background:#E4E6F0; } 16%, 88% { background:#A9D8F0; } 96%, 100% { background:#E4E6F0; } } +@keyframes lightUp2{ 0%, 32% { background:#E4E6F0; } 40%, 88% { background:#A9D8F0; } 96%, 100% { background:#E4E6F0; } } +@keyframes lightUp3{ 0%, 56% { background:#E4E6F0; } 64%, 88% { background:#A9D8F0; } 96%, 100% { background:#E4E6F0; } } +.olakego-page .nav-burger{ display: none; } +@media (max-width: 900px){ +.olakego-page .scroll-pipeline{ display: none !important; } +html:has(.olakego-page){ scroll-snap-type: none; } +.olakego-page .nav-links, .olakego-page .nav-star{ display: none !important; } +.olakego-page .nav-burger{ display: flex !important; } +.olakego-page .nav-inner{ padding: 14px 20px !important; } +.olakego-page .hero-wrap{ padding: 24px 20px 8px !important; } +.olakego-page .hero-title{ font-size: 32px !important; } +.olakego-page .hero-sub{ width: auto !important; max-width: 100% !important; height: auto !important; font-size: 16px !important; } +.olakego-page .hero-btns{ flex-wrap: wrap !important; gap: 12px !important; } +.olakego-page .problem-grid{ grid-template-columns: repeat(2, 1fr) !important; } +.olakego-page .problem-wrap{ padding: 32px 20px 56px !important; } +.olakego-page .problem-title{ font-size: 26px !important; } +.olakego-page .features-wrap{ padding: 48px 20px 20px !important; } +.olakego-page .features-grid{ grid-template-columns: 1fr !important; gap: 28px !important; } +.olakego-page .features-card{ padding: 24px !important; } +.olakego-page .sec-title{ font-size: 28px !important; } +.olakego-page .bench-wrap{ padding: 60px 16px 30px !important; } +.olakego-page .bench-stats{ grid-template-columns: 1fr !important; } +.olakego-page .cta-wrap{ padding: 0 16px !important; } +.olakego-page .cta-inner{ padding: 28px 24px !important; } +.olakego-page .cta-title{ width: auto !important; max-width: 100% !important; height: auto !important; font-size: 24px !important; } +.olakego-page .faq-wrap{ padding: 48px 20px 90px !important; } +.olakego-page .faq-title{ font-size: 26px !important; } +.olakego-page .footer-wrap{ padding: 48px 24px 0 !important; } +.olakego-page .footer-hero{ font-size: 28px !important; } +.olakego-page .footer-watermark{ font-size: 120px !important; } +} +@media (max-width: 560px){ +.olakego-page .hero-title{ font-size: 27px !important; } +.olakego-page .problem-grid{ grid-template-columns: 1fr !important; } +.olakego-page .arch-inner{ transform: scale(0.46); transform-origin: top center; } +.olakego-page .arch-wrap{ height: auto !important; overflow: visible; padding: 6px 12px !important; margin: 8px auto 8px !important; } +.olakego-page .bench-row{ padding-left: 16px !important; padding-right: 16px !important; } +.olakego-page .bench-cell-metric{ font-size: 13px !important; } +.olakego-page .faq-q{ font-size: 16px !important; } +.olakego-page .footer-watermark{ font-size: 84px !important; } +} + +/* style-hover rules */ +.olakego-page .olakego-h0:hover{background:#F4F2EC !important;} +.olakego-page .olakego-h1:hover{color:#1B1E2B !important;} +.olakego-page .olakego-h2:hover{box-shadow: 0 2px 0 #23309E !important; transform: translateY(1px) !important;} +.olakego-page .olakego-h3:hover{box-shadow: 0 2px 0 #10173A !important; transform: translateY(1px) !important;} diff --git a/src/components/landing/pages/olake-home.css b/src/components/landing/pages/olake-home.css new file mode 100644 index 000000000..84216c16c --- /dev/null +++ b/src/components/landing/pages/olake-home.css @@ -0,0 +1,53 @@ +html:has(.olakehome-page), body:has(.olakehome-page){ height: auto; min-height: 100%; scroll-behavior: smooth; } +body:has(.olakehome-page){ margin: 0; background: #f0f2fa; } +.olakehome-page *{ box-sizing: border-box; } +.olakehome-page a{ color: #193AE6; text-decoration: none; } +.olakehome-page a:hover{ color: #1430b8; } +.olakehome-page ::selection{ background: #193AE6; color: #000000; } +@keyframes gridPan{ from { background-position: 0 0; } to { background-position: 0 -80px; } } +@keyframes blink{ 0%, 49% { opacity: 1; } 50%, 100% { opacity: 0; } } +/* Seamless marquee: the track holds N identical copies and scrolls left by exactly + one copy's width. Each group's trailing padding matches its gap so the track is + exactly N copies wide; `--marquee-copies` comes from React's actual group count. */ +.olakehome-page .olakehome-marquee{ + display: flex; + width: max-content; + will-change: transform; + animation: tickerScroll 26s linear infinite; + -webkit-animation: tickerScroll 26s linear infinite; +} +.olakehome-page .olakehome-marquee-group{ + --marquee-gap: 28px; + display: flex; + gap: var(--marquee-gap); + padding-right: var(--marquee-gap); + flex-shrink: 0; +} +@keyframes tickerScroll{ + from { transform: translateX(0); } + to { transform: translateX(calc(-100% / var(--marquee-copies, 4))); } +} +@media (prefers-reduced-motion: reduce){ + .olakehome-page .olakehome-marquee{ animation: none; } +} +@keyframes glowPulse{ 0%, 100% { opacity: 0.5; } 50% { opacity: 1; } } +@keyframes floatUp{ from { opacity: 0; transform: translateY(16px); } to { opacity: 1; transform: translateY(0); } } +@keyframes barGrow{ from { transform: scaleY(0.15); } to { transform: scaleY(1); } } +@keyframes kenburns{ 0% { transform: scale(1) translate(0,0); } 100% { transform: scale(1.14) translate(-2%,-2%); } } +@keyframes chipBlink{ + 0%, 100% { background:rgba(25,58,230,0.12); border-color:rgba(120,140,255,0.4); color:#AEB9FF; box-shadow:0 0 0 0 rgba(25,58,230,0); } + 50% { background:rgba(25,58,230,0.32); border-color:#6E7CFF; color:#ffffff; box-shadow:0 0 16px 1px rgba(79,91,255,0.5); } + } +@keyframes hubGlow{ 0%, 100% { box-shadow:0 0 0 10px rgba(25, 58, 230, 0.06), 0 0 40px -10px rgba(25, 58, 230, 0.4); } 50% { box-shadow:0 0 0 16px rgba(25, 58, 230, 0.12), 0 0 80px -4px rgba(25, 58, 230, 0.85); } } +@keyframes nodeBlink{ 0%, 80%, 100% { box-shadow:0 8px 22px -10px rgba(0,0,0,0.18); transform:scale(1); } 10% { box-shadow:0 0 0 4px rgba(25,58,230,0.22), 0 12px 28px -8px rgba(25,58,230,0.55); transform:scale(1.07); } } + +/* style-hover rules */ +.olakehome-page .olakehome-h0:hover{background:rgba(0, 0, 0, 0.05) !important;} +.olakehome-page .olakehome-h1:hover{color:#000000 !important;} +.olakehome-page .olakehome-h2:hover{transform:translateY(4px) !important; box-shadow:0 0 0 #5762da, inset 0 1px 0 rgba(0, 0, 0, 0.2) !important;} +.olakehome-page .olakehome-h3:hover{transform:translateY(6px) !important; box-shadow:0 0 0 #5762da, inset 0 1px 0 rgba(0, 0, 0, 0.2) !important;} +.olakehome-page .olakehome-h4:hover{transform:translateY(6px) !important; box-shadow:0 0 0 rgba(0, 0, 0, 0.1), inset 0 1px 0 rgba(0, 0, 0, 0.08) !important;} +.olakehome-page .olakehome-h5:hover{transform:translateY(-4px) !important; box-shadow:0 32px 60px -26px rgba(25, 58, 230, 0.55), inset 0 1px 0 rgba(255, 255, 255, 0.22) !important;} +.olakehome-page .olakehome-h6:hover{opacity:0.72 !important;} +.olakehome-page .olakehome-h7:hover{transform:translateY(-6px) !important; box-shadow:0 34px 60px -28px rgba(0,0,0,0.55) !important;} +.olakehome-page .olakehome-h8:hover{opacity:0.65 !important;} diff --git a/src/components/landing/pages/overrides.css b/src/components/landing/pages/overrides.css new file mode 100644 index 000000000..67c25d85d --- /dev/null +++ b/src/components/landing/pages/overrides.css @@ -0,0 +1,98 @@ +/* ========================================================================= + Hand-maintained overrides for the three literal-ported landing pages. + + IMPORTANT: olake-go.css / olake-fusion.css / olake-home.css are GENERATED + from the design templates and are overwritten on every regeneration. Put + any deliberate deviation from the designs HERE instead, so it survives. + ========================================================================= */ + +/* ------------------------------------------------------------------------- + / (v2 homepage) — USER STORIES cards. + + The design draws these as plain
    s. They are converted to links to + each customer's story (see postprocess_pages.js), so they need a hover + affordance — the design has none because they weren't clickable. + ------------------------------------------------------------------------- */ +.olakehome-page a[aria-label='Read the customer story'] { + text-decoration: none; + transition: transform 0.2s ease, box-shadow 0.2s ease; +} + +.olakehome-page a[aria-label='Read the customer story']:hover { + transform: translateY(-4px); + box-shadow: 0 32px 60px -26px rgba(0, 0, 0, 0.6); +} + +/* ------------------------------------------------------------------------- + Shared footer hover states. + + All three landing pages render one canonical footer, so its hover styles + live here rather than in the per-page generated stylesheets (whose hover + classes are page-scoped, e.g. `.olakego-h0`). The Go design's social boxes + are unlinked
    s with no hover at all; these match the Fusion design, + which implements the same boxes as real links. + ------------------------------------------------------------------------- */ +.olake-design-page .olake-footer-social { + transition: border-color 0.15s ease, color 0.15s ease; +} + +.olake-design-page .olake-footer-social:hover { + border-color: #3d4ff0 !important; + color: #3d4ff0 !important; +} + +.olake-design-page .olake-footer-link { + transition: color 0.15s ease; +} + +.olake-design-page .olake-footer-link:hover { + color: #3d4ff0 !important; +} + +/* ------------------------------------------------------------------------- + /olake-go — benchmark table horizontal-overflow "dead zone". + + The design sizes the table's inner wrapper to `min-width: 860px`, while + `.bench-wrap` is `max-width:1080px` with 64px side padding. So the table + only fits without scrolling at viewports >= 988px (1080 - 128 = 952, and + viewport - 128 >= 860). The design's own breakpoint drops that padding to + 16px at <= 900px, which makes it fit again (900 - 32 = 868 >= 860). + + That leaves ~901-987px where the table is wider than its container and + clips its last column. This closes that gap by applying the narrower side + padding across the dead zone too (987 - 32 = 955 >= 860). + + Only the horizontal padding is touched, so the design's vertical rhythm + above the 900px breakpoint is preserved exactly. + ------------------------------------------------------------------------- */ +@media (max-width: 987px) and (min-width: 901px) { + .olakego-page .bench-wrap { + padding-left: 16px !important; + padding-right: 16px !important; + } +} + +/* ------------------------------------------------------------------------- + / (v2 homepage) — footer responsive rules. + + The homepage uses the OLake Go footer verbatim (all three landing pages + share it), but the v2 design ships no media queries at all, so the Go + design's footer breakpoints have to be carried over by hand. + ------------------------------------------------------------------------- */ +@media (max-width: 900px) { + .olakehome-page .footer-wrap { + padding: 48px 24px 0 !important; + } + .olakehome-page .footer-hero { + font-size: 28px !important; + } + .olakehome-page .footer-watermark { + font-size: 120px !important; + } +} + +@media (max-width: 560px) { + .olakehome-page .footer-watermark { + font-size: 84px !important; + } +} diff --git a/src/components/landing/pages/useFusionLogic.ts b/src/components/landing/pages/useFusionLogic.ts new file mode 100644 index 000000000..446b3637b --- /dev/null +++ b/src/components/landing/pages/useFusionLogic.ts @@ -0,0 +1,384 @@ +// @ts-nocheck +import { useState, useRef, useEffect, useMemo } from 'react' + +export function useFusionLogic(props = {}) { + const [state, setStateRaw] = useState({ + faqs: [ + { + q: 'What is OLake Fusion?', + a: 'OLake Fusion is a self-hosted open-source software that keeps your lakehouse tables efficient, compact, and query-ready as your data continuously grows. As Iceberg tables evolve through ingestion, updates, and deletes, they accumulate small files, delete files, and excess metadata, all of which degrade query performance and increase storage overhead over time. Fusion manages that for you automatically.' + }, + { + q: 'When do I actually need table maintenance?', + a: "Six situations call for it: frequent data ingestion or updates, accumulation of small files, presence of delete files, high partition cardinality, degrading query performance, and growing table size over time. If your tables are under continuous CDC pressure, you're in all six." + }, + { + q: 'Can I use OLake Fusion together with OLake Go?', + a: "Yes, that's the intended path. Iceberg Maintenance ships as a maintenance module inside the OLake UI from v0.4.0, so Go handles ingestion and Fusion handles maintenance from the same place. New users can start with a combined Ingestion + Maintenance setup, and existing OLake Go users just upgrade the UI (Docker or Helm/Kubernetes) to unlock the module, no separate tool to adopt. Our own compaction benchmark ran exactly this way: OLake Go ingesting the TPC-H lineitem table while Fusion compacted it." + }, + { + q: 'Can I use OLake Fusion on its own, without OLake Go?', + a: "Yes, Fusion maintains Apache Iceberg tables through your Iceberg catalog, so it operates on the tables themselves rather than on OLake's ingestion pipeline." + }, + { + q: 'How much faster is Fusion than Apache Spark?', + a: 'On a CDC-like TPC-H workload (300 GB, ~1.8 billion rows in lineitem), Fusion compacted in 27 minutes 2 seconds versus Spark rewrite_data_files at 55 minutes 47 seconds; 2.06× faster on identical infrastructure.' + }, + { + q: 'Does it cost less than running Spark compaction?', + a: 'Yes. On the same $2.36/hour infrastructure, the benchmark job cost $1.06 with Fusion and $2.19 with Spark, roughly half, because the job finishes in half the time.' + }, + { + q: 'Do I have to pause ingestion or queries while compaction runs?', + a: 'No. In the benchmark, compaction ran for two hours while CDC-style updates (~200,000 rows every 2 minutes) and repeated TPC-H Query 6 executions continued in parallel, specifically to measure how concurrent compaction affects query latency and stability.' + }, + { + q: 'How often does compaction run?', + a: "Fusion uses tiered triggers rather than one blunt job: Lite every 20 minutes and Medium every 40 minutes in the benchmark, plus Full as a periodic deep-clean for much larger datasets, terabyte-scale tables where long-term small-file buildup is higher. At the benchmark's sub-100 GB destination size, Full wasn't needed." + }, + { + q: 'How do I deploy it, and what does it cost to license?', + a: 'Fusion is open-source and is deployed on Docker or Kubernetes. You pay only for the compute and storage you provision.' + }, + { + q: 'How much configuration does it need?', + a: 'One parameter: target-size (512 MB in the benchmark). For comparison, the Spark rewrite_data_files job in the same test needed seven: strategy, target/max/min file size, concurrent rewrites, partial-progress, and delete-file threshold.' + } + ], + openFaq: -1, + activeFeature: 0, + progress: 0, + resourcesOpen: false, + productOpen: false, + contributorsOpen: false, + paused: false, + benchmarksInfoOpen: false, + mobileMenuOpen: false + }) + const stateRef = useRef(state) + stateRef.current = state + const setState = (u) => + setStateRaw((s) => { + const p = typeof u === 'function' ? u(s) : u + return p ? { ...s, ...p } : s + }) + const self = useRef({}).current + + const toggleFaq = (i) => { + setState((s) => ({ openFaq: s.openFaq === i ? -1 : i })) + } + + const tick = () => { + if (!self._started) return + setState((s) => { + if (s.paused) return null + const next = s.progress + 1 + if (next >= 100) { + return { progress: 0, activeFeature: (s.activeFeature + 1) % 4 } + } + return { progress: next } + }) + } + + const selectFeature = (i) => { + setState({ activeFeature: i, progress: 100, paused: true }) + } + + const toggleBenchmarkInfo = () => { + setState((s) => ({ benchmarksInfoOpen: !s.benchmarksInfoOpen })) + } + + const toggleMobileMenu = () => { + setState((s) => ({ mobileMenuOpen: !s.mobileMenuOpen })) + } + + const openResources = () => { + setState({ resourcesOpen: true }) + } + + const closeResources = () => { + setState({ resourcesOpen: false, contributorsOpen: false }) + } + + const openProduct = () => { + setState({ productOpen: true }) + } + + const closeProduct = () => { + setState({ productOpen: false }) + } + + const openContributors = () => { + setState({ contributorsOpen: true }) + } + + const closeContributors = () => { + setState({ contributorsOpen: false }) + } + + const wordCloud = (words, emphasize, matchIndex, bigWords) => { + const sizes = [34, 22, 27, 24, 31, 21, 25] + const weights = [700, 500, 600, 500, 700, 500, 600] + const colors = ['#10173A', '#8890C4', '#10173A', '#3D4FF0', '#10173A', '#8890C4', '#10173A'] + const rots = [-3, 2, -1, 3, -2, 1, 0] + const emph = (emphasize || []).map((w) => w.toLowerCase().replace(/[^a-z']/g, '')) + const big = (bigWords || []).map((w) => w.toLowerCase().replace(/[^a-z']/g, '')) + const mi = matchIndex || {} + return words.map((text, i) => { + const norm = text.toLowerCase().replace(/[^a-z']/g, '') + const isEmph = emph.includes(norm) + const isBig = big.includes(norm) + const idx = mi[i] !== undefined ? mi[i] : i + return { + text, + size: isEmph || isBig ? 33 : sizes[idx % sizes.length], + weight: isEmph ? 700 : weights[idx % weights.length], + color: isEmph ? '#10173A' : colors[idx % colors.length], + opacity: isEmph ? 1 : 0.75 + (0.25 * ((idx * 37) % 10)) / 10, + rot: isEmph ? 0 : rots[idx % rots.length], + delay: (i % 5) * 0.3 + } + }) + } + + const renderVals = () => { + const posX = props.icebergPosX ?? 50 + const posY = props.icebergPosY ?? 50 + const zoom = props.icebergZoom ?? 100 + const op = (props.icebergOpacity ?? 32) / 100 + const icebergStyle = `position:absolute; inset:0; width:100%; height:100%; object-fit:cover; object-position:${posX}% ${posY}%; transform:scale(${zoom / 100}); opacity:${op}; pointer-events:none;` + return { + icebergStyle, + problemCards: [ + { + words: wordCloud( + ['Small', 'files', 'PILE UP', 'faster', 'than', 'you', 'notice'], + ['Small', 'files', 'PILE UP'] + ) + }, + { + words: wordCloud( + ['Queries', 'get', 'SLOWER', 'every', 'day'], + ['Queries', 'get', 'SLOWER'] + ) + }, + { + words: wordCloud( + ['Problems', "aren't", 'visible', 'until', "they're", 'expensive'], + ['Problems', 'expensive'], + { 4: 2 } + ) + }, + { + words: wordCloud( + ['Compaction', 'becomes', 'a', 'debugging', 'issue'], + ['Compaction'], + null, + ['debugging'] + ) + } + ], + features: [0, 1, 2, 3].map((i) => { + const list = [ + { + title: 'TIERED COMPACTION', + body: 'Trigger-based tiers instead of one blunt job, so light cleanup runs constantly and deep rewrites only run when they\u2019re actually needed.', + tag: 'tiered' + }, + { + title: 'REDUCED DECAY', + body: 'Small files, delete files, and excess metadata pile up as tables evolve. Fusion resolves them periodically so query performance never degrades.', + tag: 'decay' + }, + { + title: 'EASY CONFIGURATION', + body: 'One target-size parameter replaces the seven Spark rewrite_data_files needs. Same result, far less to manage.', + tag: 'config' + }, + { + title: 'SELF-HOSTED', + body: 'Open-source and deployable on Docker or Kubernetes so you pay only for the compute and storage you provision.', + tag: 'hosted' + } + ][i] + const active = stateRef.current.activeFeature === i + return { + ...list, + active, + titleColor: active ? '#fff' : '#BFC7F2', + progress: active ? stateRef.current.progress : 0, + isTiered: list.tag === 'tiered', + isDecay: list.tag === 'decay', + isConfig: list.tag === 'config', + isHosted: list.tag === 'hosted', + onSelect: () => selectFeature(i) + } + }), + benchmarkRows: [ + { + metric: 'Total compaction time', + spark: '55m 47s', + fusion: '27m 02s', + delta: '2.06X Faster' + }, + { + metric: 'Compaction cost / job', + spark: '$2.19', + fusion: '$1.06', + delta: '52% Less Cost' + }, + { metric: 'Config parameters', spark: '10+', fusion: '1', delta: '10X Simpler' } + ], + faqs: stateRef.current.faqs.map((f, i) => ({ + ...f, + open: stateRef.current.openFaq === i, + sign: stateRef.current.openFaq === i ? '\u2191' : '\u2193', + onToggle: () => toggleFaq(i) + })), + toggleFaq: (i) => toggleFaq(i), + selectFeature: (i) => selectFeature(i), + benchmarksInfoOpen: stateRef.current.benchmarksInfoOpen, + benchmarkInfoArrow: stateRef.current.benchmarksInfoOpen ? 'rotate(180deg)' : 'rotate(0deg)', + toggleBenchmarkInfo: () => toggleBenchmarkInfo(), + mobileMenuOpen: stateRef.current.mobileMenuOpen, + toggleMobileMenu: () => toggleMobileMenu(), + resourcesOpen: stateRef.current.resourcesOpen, + productOpen: stateRef.current.productOpen, + openProduct: () => openProduct(), + closeProduct: () => closeProduct(), + contributorsOpen: stateRef.current.contributorsOpen, + openResources: () => openResources(), + closeResources: () => closeResources(), + openContributors: () => openContributors(), + closeContributors: () => closeContributors() + } + } + + useEffect(() => { + self._tick = setInterval(() => tick(), 100) + self._started = false + + self._measureOrigin = () => { + const node = document.getElementById('opt-node') + if (node) { + const r = node.getBoundingClientRect() + self._originDocX = r.left + r.width / 2 + self._originDocY = r.top + window.scrollY + r.height / 2 + } + } + + self._base = { x: 0, y: 0 } + self._dodge = { x: 0, y: 0 } + self._dodgeTarget = { x: 0, y: 0 } + + self._render = () => { + const x = self._base.x + self._dodge.x + const y = self._base.y + self._dodge.y + const els = [ + ['scroll-packet', 0], + ['trail-1', 4], + ['trail-2', 8], + ['trail-3', 12] + ] + for (const [id, lag] of els) { + const el = document.getElementById(id) + if (el) el.style.transform = 'translate(' + x + 'px,' + (y - lag) + 'px)' + } + } + + self._onScroll = () => { + if (self._raf) return + self._raf = requestAnimationFrame(() => { + self._raf = null + const doc = document.documentElement + const max = doc.scrollHeight - window.innerHeight || 1 + const p = Math.min(1, Math.max(0, window.scrollY / max)) + const vh = window.innerHeight + const containerLeft = window.innerWidth - 26 - 70 + // gutter path target + const gutterX = 12 + Math.sin(p * Math.PI * 6) * 16 + const gutterY = 40 + p * (vh - 120) + // origin (emerging from the Optimized Iceberg Tables node) + const originX = (self._originDocX ?? containerLeft) - containerLeft - 17 + const originY = (self._originDocY ?? 60) - window.scrollY - 17 + // blend from origin to gutter over first 12% of scroll + let b = Math.min(1, p / 0.12) + b = b * b * (3 - 2 * b) + const x = originX + (gutterX - originX) * b + const y = originY + (gutterY - originY) * b + self._base.x = x + self._base.y = y + self._render() + }) + } + window.addEventListener('scroll', self._onScroll, { passive: true }) + self._hoisted_resize_0 = () => { + self._measureOrigin() + self._onScroll() + } + window.addEventListener('resize', self._hoisted_resize_0, { passive: true }) + self._measureOrigin() + self._onScroll() + + // Playful dodge: the packet flees the cursor so it can never be clicked + self._onMouseMove = (e) => { + const el = document.getElementById('scroll-packet') + if (!el || getComputedStyle(el.parentElement).display === 'none') return + const r = el.getBoundingClientRect() + const cx = r.left + r.width / 2 + const cy = r.top + r.height / 2 + const dx = cx - e.clientX + const dy = cy - e.clientY + const dist = Math.hypot(dx, dy) || 1 + const R = 130 + if (dist < R) { + const push = (R - dist) * 1.4 + self._dodgeTarget.x = Math.max(-150, Math.min(150, (dx / dist) * push)) + self._dodgeTarget.y = Math.max(-200, Math.min(200, (dy / dist) * push)) + } + } + window.addEventListener('mousemove', self._onMouseMove, { passive: true }) + + // Single smoothing loop: ease applied dodge toward target, drift target home + self._dodgeTick = () => { + self._dodgeTarget.x *= 0.92 + self._dodgeTarget.y *= 0.92 + self._dodge.x += (self._dodgeTarget.x - self._dodge.x) * 0.18 + self._dodge.y += (self._dodgeTarget.y - self._dodge.y) * 0.18 + if (Math.abs(self._dodge.x) < 0.3) self._dodge.x = 0 + if (Math.abs(self._dodge.y) < 0.3) self._dodge.y = 0 + self._render() + self._decayRaf = requestAnimationFrame(self._dodgeTick) + } + self._dodgeTick() + + const el = document.getElementById('features') + if (el && 'IntersectionObserver' in window) { + self._observer = new IntersectionObserver( + (entries) => { + if (entries[0].isIntersecting && !self._started) { + self._started = true + setState({ progress: 0 }) + } + }, + { threshold: 0.4 } + ) + self._observer.observe(el) + } else { + self._started = true + } + + return () => { + clearInterval(self._tick) + if (self._observer) self._observer.disconnect() + if (self._onScroll) window.removeEventListener('scroll', self._onScroll) + if (self._onMouseMove) window.removeEventListener('mousemove', self._onMouseMove) + if (self._decayRaf) cancelAnimationFrame(self._decayRaf) + + window.removeEventListener('resize', self._hoisted_resize_0) + } + // eslint-disable-next-line react-hooks/exhaustive-deps + }, []) + + return renderVals() +} diff --git a/src/components/landing/pages/useGoLogic.ts b/src/components/landing/pages/useGoLogic.ts new file mode 100644 index 000000000..8e298c55d --- /dev/null +++ b/src/components/landing/pages/useGoLogic.ts @@ -0,0 +1,473 @@ +// @ts-nocheck +import { useState, useRef, useEffect, useMemo } from 'react' +import { + CONNECTORS, + CONNECTOR_BENCHMARKS, + CONNECTOR_CDC_BENCHMARKS, + CONNECTOR_METRIC_LABELS, + TOOLS +} from '@site/src/data/benchmarkData' + +// Row order of the benchmark table, top to bottom. +const BENCHMARK_METRICS = ['rowsSynced', 'elapsedTime', 'speed', 'comparison', 'cost'] + +// The table renders a fixed 4 competitor columns. Kafka is benchmarked against +// Apache Flink rather than Debezium, which is why the set is per-connector. +const competitorKeys = (bench) => + bench.rowsSynced.flink !== undefined + ? ['airbyte', 'fivetran', 'estuary', 'flink'] + : ['airbyte', 'fivetran', 'debezium', 'estuary'] + +export function useGoLogic(props = {}) { + const [state, setStateRaw] = useState({ + faqs: [ + { + q: 'How to get started?', + a: 'Check the Quickstart Guide. With a single Docker command you can spin up OLake and access the UI.' + }, + { + q: 'Is OLake really open source?', + a: 'Yes. OLake is fully open source under the Apache 2.0 license. You can explore the GitHub repository (already starred by 1K+ developers) and use it freely without hidden costs.' + }, + { + q: 'Is there any enterprise plan?', + a: "We're actively working on providing enterprise support, from professional assistance and pilot programs to helping teams scale OLake in production. You can reach out at hello@olake.io to learn more." + }, + { + q: 'How can I contribute?', + a: 'Join our Slack community, review the Contribution Guide, and explore "Good First Issues" on GitHub. Contributors can get their pull requests merged and be part of building the fastest open-source Iceberg-native ingestion tool.' + }, + { + q: 'Why should I use OLake?', + a: 'OLake makes data replication into Apache Iceberg seamless, faster, and cost-efficient. It handles real-time CDC, schema and partition evolution, full and incremental syncs, and compaction, all without vendor lock-in, so your Iceberg tables stay open, scalable, and ready for analytics.' + }, + { + q: 'What data platforms and tools does OLake integrate with?', + a: 'As of now, we integrate with Apache Iceberg as a destination. You can query it from most big data platforms like Snowflake, Databricks, Redshift and BigQuery.' + } + ], + openFaq: -1, + activeFeature: 0, + progress: 0, + resourcesOpen: false, + productOpen: false, + contributorsOpen: false, + paused: false, + benchmarksInfoOpen: false, + activeSource: 0, + benchmarkMode: 'full_load', + mobileMenuOpen: false + }) + const stateRef = useRef(state) + stateRef.current = state + const setState = (u) => + setStateRaw((s) => { + const p = typeof u === 'function' ? u(s) : u + return p ? { ...s, ...p } : s + }) + const self = useRef({}).current + + const toggleFaq = (i) => { + setState((s) => ({ openFaq: s.openFaq === i ? -1 : i })) + } + + const selectSource = (i) => { + setState({ activeSource: i }) + } + + const buildBenchmarks = () => { + const G = '#2E9E44', + GB = '#EAF6E9' + const connector = CONNECTORS[stateRef.current.activeSource] || CONNECTORS[0] + const dataset = + stateRef.current.benchmarkMode === 'cdc' ? CONNECTOR_CDC_BENCHMARKS : CONNECTOR_BENCHMARKS + const bench = dataset[connector.id] + const comps = competitorKeys(bench) + const cols = [ + { + name: 'Metrics', + sub: '', + hasSub: false, + olake: false, + color: '#10173A', + bg: 'transparent', + align: 'flex-start' + }, + { + name: 'OLake Go', + sub: TOOLS.olake.description, + hasSub: true, + olake: true, + color: '#10173A', + bg: GB, + align: 'center' + }, + ...comps.map((key) => ({ + name: TOOLS[key].name, + sub: '', + hasSub: false, + olake: false, + color: '#10173A', + bg: 'transparent', + align: 'center' + })) + ] + const cell = (text, type) => ({ + text, + color: type === 'olake' ? G : type === 'cmp' ? '#3D4FF0' : '#4A5170', + weight: type === 'olake' || type === 'cmp' ? 700 : 500, + bg: type === 'olake' ? GB : 'transparent' + }) + const table = BENCHMARK_METRICS.map((metric) => { + const row = bench[metric] + const isCmp = metric === 'comparison' + return { + label: CONNECTOR_METRIC_LABELS[metric], + sub: + metric === 'cost' + ? 'OLake is OSS and self-hosted — only pay for your infrastructure.' + : '', + hasSub: metric === 'cost', + // OLake is the baseline for the comparison row, so it shows a dash, not a multiplier. + cells: [ + cell(isCmp ? '–' : row.olake, 'olake'), + ...comps.map((key) => cell(row[key] ?? '-', isCmp ? 'cmp' : 'normal')) + ] + } + }) + return { cols, table, comingSoon: !bench.hasData, sourceName: connector.name } + } + + const selectBenchmarkMode = (mode) => { + setState({ benchmarkMode: mode }) + } + + const tick = () => { + if (!self._started) return + setState((s) => { + if (s.paused) return null + const next = s.progress + 1 + if (next >= 100) { + return { progress: 0, activeFeature: (s.activeFeature + 1) % 4 } + } + return { progress: next } + }) + } + + const selectFeature = (i) => { + setState({ activeFeature: i, progress: 100, paused: true }) + } + + const toggleBenchmarkInfo = () => { + setState((s) => ({ benchmarksInfoOpen: !s.benchmarksInfoOpen })) + } + + const toggleMobileMenu = () => { + setState((s) => ({ mobileMenuOpen: !s.mobileMenuOpen })) + } + + const openResources = () => { + setState({ resourcesOpen: true }) + } + + const closeResources = () => { + setState({ resourcesOpen: false, contributorsOpen: false }) + } + + const openProduct = () => { + setState({ productOpen: true }) + } + + const closeProduct = () => { + setState({ productOpen: false }) + } + + const openContributors = () => { + setState({ contributorsOpen: true }) + } + + const closeContributors = () => { + setState({ contributorsOpen: false }) + } + + const sentence = (words, emphasize, base) => { + const emph = (emphasize || []).map((w) => w.toLowerCase().replace(/[^a-z']/g, '')) + return words.map((text) => { + const norm = text.toLowerCase().replace(/[^a-z']/g, '') + const isEmph = emph.includes(norm) + return { + text, + size: isEmph ? base + 7 : base, + weight: isEmph ? 700 : 500, + color: isEmph ? '#3D4FF0' : '#10173A', + opacity: isEmph ? 1 : 0.7 + } + }) + } + + const renderVals = () => { + const posX = props.icebergPosX ?? 50 + const posY = props.icebergPosY ?? 50 + const zoom = props.icebergZoom ?? 100 + const op = (props.icebergOpacity ?? 32) / 100 + const icebergStyle = `position:absolute; inset:0; width:100%; height:100%; object-fit:cover; object-position:${posX}% ${posY}%; transform:scale(${zoom / 100}); opacity:${op}; pointer-events:none;` + const bm = buildBenchmarks() + return { + icebergStyle, + problemSentences: [ + { + top: 27, + left: 42, + rot: -3, + delay: 0, + words: sentence( + ['Slow', 'syncs', 'BLOCK', 'your', 'analytics'], + ['Slow', 'syncs', 'BLOCK'], + 28 + ) + }, + { + top: 43, + left: 57, + rot: 2, + delay: 0.6, + words: sentence(['Legacy', 'ETL', 'costs', 'PILE UP', 'fast'], ['costs', 'PILE UP'], 24) + }, + { + top: 64, + left: 46, + rot: -2, + delay: 1.2, + words: sentence( + ['CDC', 'pipelines', 'BREAK', 'silently', 'in', 'production'], + ['CDC', 'BREAK'], + 25 + ) + }, + { + top: 85, + left: 55, + rot: 3, + delay: 0.3, + words: sentence(['Schema', 'drift', 'stalls', 'ingestion'], ['Schema', 'drift'], 23) + } + ], + features: [0, 1, 2, 3].map((i) => { + const list = [ + { + title: 'FULL, INCREMENTAL & CDC SYNCS', + body: 'Run full loads, incremental pulls, or real-time change data capture, whatever each table needs, all from a single tool.', + tag: 'tiered' + }, + { + title: 'SCHEMA & PARTITION EVOLUTION', + body: 'Source schemas change and partitions grow. OLake Go evolves your Iceberg tables automatically so pipelines never break.', + tag: 'decay' + }, + { + title: 'PARALLELISED CHUNKING', + body: 'Large collections are split into virtual chunks read in parallel, dramatically cutting the time for full snapshots of big datasets.', + tag: 'chunk', + stats: ['Parallel reads', 'Virtual chunks', 'Faster snapshots'] + }, + { + title: 'STATEFUL, RESUMABLE SYNCS', + body: 'Syncs checkpoint their progress and resume automatically after crashes, network failures, or pauses, never from scratch.', + tag: 'resume', + stats: ['Checkpointed', 'Auto-resume', 'Fault-tolerant'] + } + ][i] + const active = stateRef.current.activeFeature === i + const builtIn = ['tiered', 'decay', 'config', 'hosted'] + return { + ...list, + active, + titleColor: active ? '#fff' : '#BFC7F2', + progress: active ? stateRef.current.progress : 0, + isTiered: list.tag === 'tiered', + isDecay: list.tag === 'decay', + isConfig: list.tag === 'config', + isHosted: list.tag === 'hosted', + isChunk: list.tag === 'chunk', + isResume: list.tag === 'resume', + isGeneric: !builtIn.includes(list.tag), + hasStats: Array.isArray(list.stats), + onSelect: () => selectFeature(i) + } + }), + benchSources: (() => { + const act = stateRef.current.activeSource + return CONNECTORS.map((connector, i) => ({ + name: connector.name, + onSelect: () => selectSource(i), + color: act === i ? '#3D4FF0' : '#5B6484', + bg: act === i ? '#E7EAFE' : 'transparent', + weight: act === i ? 700 : 500 + })) + })(), + benchModes: [ + { key: 'full_load', label: 'Full Load' }, + { key: 'cdc', label: 'CDC' } + ].map((mode) => { + const active = stateRef.current.benchmarkMode === mode.key + return { + ...mode, + active, + color: active ? '#3D4FF0' : '#5B6484', + weight: active ? 700 : 500, + onSelect: () => selectBenchmarkMode(mode.key) + } + }), + benchModeIndicatorLeft: + stateRef.current.benchmarkMode === 'full_load' ? '4px' : 'calc(50% + 0px)', + benchCols: bm.cols, + benchTable: bm.table, + benchComingSoon: bm.comingSoon, + benchHasData: !bm.comingSoon, + benchSourceName: bm.sourceName, + faqs: stateRef.current.faqs.map((f, i) => ({ + ...f, + open: stateRef.current.openFaq === i, + sign: stateRef.current.openFaq === i ? '\u2191' : '\u2193', + onToggle: () => toggleFaq(i) + })), + toggleFaq: (i) => toggleFaq(i), + selectFeature: (i) => selectFeature(i), + benchmarksInfoOpen: stateRef.current.benchmarksInfoOpen, + benchmarkInfoArrow: stateRef.current.benchmarksInfoOpen ? 'rotate(180deg)' : 'rotate(0deg)', + toggleBenchmarkInfo: () => toggleBenchmarkInfo(), + mobileMenuOpen: stateRef.current.mobileMenuOpen, + toggleMobileMenu: () => toggleMobileMenu(), + resourcesOpen: stateRef.current.resourcesOpen, + productOpen: stateRef.current.productOpen, + openProduct: () => openProduct(), + closeProduct: () => closeProduct(), + contributorsOpen: stateRef.current.contributorsOpen, + openResources: () => openResources(), + closeResources: () => closeResources(), + openContributors: () => openContributors(), + closeContributors: () => closeContributors() + } + } + + useEffect(() => { + self._tick = setInterval(() => tick(), 100) + self._started = false + + self._drawLinks = () => { + const svg = document.getElementById('arch-links') + const inner = document.querySelector('.arch-inner') + const node = document.getElementById('olake-node') + if (!svg || !inner || !node) return + const srcs = Array.from(document.querySelectorAll('.src-box')) + const dests = Array.from(document.querySelectorAll('.dest-box')) + const c = inner.getBoundingClientRect() + const nb = node.getBoundingClientRect() + // Skip when the diagram has stacked/wrapped (mobile) + if (!srcs.length || !dests.length || srcs[0].getBoundingClientRect().right > nb.left + 4) { + svg.innerHTML = '' + return + } + svg.setAttribute('viewBox', '0 0 ' + c.width + ' ' + c.height) + // Extend endpoints a few px INTO the boxes so lines visibly touch (svg sits behind boxes) + const nodeL = { x: nb.left - c.left + 10, y: nb.top - c.top + nb.height / 2 } + const nodeR = { x: nb.right - c.left - 10, y: nb.top - c.top + nb.height / 2 } + let defs = + '' + + '' + + '' + + '' + let paths = '', + dots = '' + srcs.forEach((s, i) => { + const r = s.getBoundingClientRect() + const p = { x: r.right - c.left - 8, y: r.top - c.top + r.height / 2 } + const dx = Math.max(28, (nodeL.x - p.x) * 0.5) + const d = + 'M' + + p.x + + ',' + + p.y + + ' C' + + (p.x + dx) + + ',' + + p.y + + ' ' + + (nodeL.x - dx) + + ',' + + nodeL.y + + ' ' + + nodeL.x + + ',' + + nodeL.y + paths += '' + dots += + '' + }) + dests.forEach((s, i) => { + const r = s.getBoundingClientRect() + const p = { x: r.left - c.left + 8, y: r.top - c.top + r.height / 2 } + const dx = Math.max(28, (p.x - nodeR.x) * 0.5) + const d = + 'M' + + nodeR.x + + ',' + + nodeR.y + + ' C' + + (nodeR.x + dx) + + ',' + + nodeR.y + + ' ' + + (p.x - dx) + + ',' + + p.y + + ' ' + + p.x + + ',' + + p.y + paths += '' + dots += + '' + }) + svg.innerHTML = defs + paths + dots + } + self._drawRaf = requestAnimationFrame(() => self._drawLinks()) + setTimeout(() => self._drawLinks(), 400) + self._onArchResize = () => self._drawLinks() + window.addEventListener('resize', self._onArchResize, { passive: true }) + + const el = document.getElementById('features') + if (el && 'IntersectionObserver' in window) { + self._observer = new IntersectionObserver( + (entries) => { + if (entries[0].isIntersecting && !self._started) { + self._started = true + setState({ progress: 0 }) + } + }, + { threshold: 0.4 } + ) + self._observer.observe(el) + } else { + self._started = true + } + + return () => { + clearInterval(self._tick) + if (self._observer) self._observer.disconnect() + if (self._onArchResize) window.removeEventListener('resize', self._onArchResize) + if (self._drawRaf) cancelAnimationFrame(self._drawRaf) + } + // eslint-disable-next-line react-hooks/exhaustive-deps + }, []) + + return renderVals() +} diff --git a/src/components/landing/pages/useHomeLogic.ts b/src/components/landing/pages/useHomeLogic.ts new file mode 100644 index 000000000..c4ac68ccf --- /dev/null +++ b/src/components/landing/pages/useHomeLogic.ts @@ -0,0 +1,414 @@ +// @ts-nocheck +import { useState, useRef, useEffect, useMemo } from 'react' + +export function useHomeLogic(props = {}) { + const [state, setStateRaw] = useState({ + resourcesOpen: false, + contributorsOpen: false, + productOpen: false, + docsOpen: false, + advantage: 'go' + }) + const stateRef = useRef(state) + stateRef.current = state + const setState = (u) => + setStateRaw((s) => { + const p = typeof u === 'function' ? u(s) : u + return p ? { ...s, ...p } : s + }) + const self = useRef({}).current + + const openProduct = () => { + setState({ productOpen: true }) + } + + const closeProduct = () => { + setState({ productOpen: false }) + } + + const openDocs = () => { + setState({ docsOpen: true }) + } + + const closeDocs = () => { + setState({ docsOpen: false }) + } + + const openResources = () => { + setState({ resourcesOpen: true }) + } + + const closeResources = () => { + setState({ resourcesOpen: false, contributorsOpen: false }) + } + + const openContributors = () => { + setState({ contributorsOpen: true }) + } + + const closeContributors = () => { + setState({ contributorsOpen: false }) + } + + const renderVals = () => { + const isGo = stateRef.current.advantage === 'go' + const activeTab = '#193AE6' + const R = (typeof window !== 'undefined' && window.__resources) || {} + const advImg = + props.advantageImage ?? (R.icebergBg || '/img/landing/shared/iceberg-backdrop.webp') + const advOv = props.advantageOverlay ?? 0.78 + const advWl = props.advantageWaterline ?? 74 + const advantagePos = 'center, center ' + advWl + '%' + const advantageStats = ( + isGo + ? [ + { value: '12.5×', label: 'Faster than traditional tools' }, + { value: '90%', label: 'Cost savings with OSS' } + ] + : [ + { value: '2×', label: 'Faster than Apache Spark' }, + { value: '~50%', label: 'Cheaper per compaction cycle' } + ] + ).map((s) => { + const m = String(s.value).match(/^(\D*)([\d.]+)(\D*)$/) || ['', '', s.value, ''] + const numStr = m[2] || '0' + return { + ...s, + prefix: m[1] || '', + suffix: m[3] || '', + num: parseFloat(numStr) || 0, + decimals: (numStr.split('.')[1] || '').length + } + }) + self._counts = advantageStats.map((s) => ({ + num: s.num, + prefix: s.prefix, + suffix: s.suffix, + decimals: s.decimals + })) + const advantageBg = + 'linear-gradient(160deg, rgba(72,98,235,' + + (advOv * 0.92).toFixed(3) + + '), rgba(25,58,230,' + + advOv.toFixed(3) + + ")), url('" + + advImg + + "')" + // w/h are the files' intrinsic sizes, set on the so the marquee reserves + // space before the logos load. + const logos = [ + { src: '/img/landing/v2/logo-bitespeed.webp', name: 'Bitespeed', w: 340, h: 65 }, + { src: '/img/landing/v2/logo-xeno.webp', name: 'Xeno', w: 176, h: 88 }, + { src: '/img/landing/v2/logo-cordial.webp', name: 'Cordial', w: 244, h: 88 }, + { src: '/img/landing/v2/logo-lendingkart.webp', name: 'Lending Kart', w: 168, h: 88 }, + { src: '/img/landing/v2/logo-astrotalk.webp', name: 'Astro Talk', w: 246, h: 88 }, + { src: '/img/landing/v2/logo-physicswallah.webp', name: 'Physics Wallah', w: 246, h: 88 } + ] + const ticker = [ + 'Postgres', + 'MySQL', + 'MongoDB', + 'Oracle', + 'Kafka', + 'S3', + 'DB2 LUW', + 'MSSQL', + 'Apache Iceberg', + 'Parquet' + ] + + return { + advantageBg, + advantagePos, + advantageStats, + resourcesOpen: stateRef.current.resourcesOpen, + contributorsOpen: stateRef.current.contributorsOpen, + productOpen: stateRef.current.productOpen, + docsOpen: stateRef.current.docsOpen, + openProduct: () => openProduct(), + closeProduct: () => closeProduct(), + openDocs: () => openDocs(), + closeDocs: () => closeDocs(), + openResources: () => openResources(), + closeResources: () => closeResources(), + openContributors: () => openContributors(), + closeContributors: () => closeContributors(), + + terminalLines: [ + { + prefix: '$ ', + color: '#3fa872', + text: 'olake sync --source postgres --dest iceberg', + textColor: '#070911', + delay: 0 + }, + { + prefix: ' ', + color: '#7b84a4', + text: '→ CDC stream connected · 8 tables', + textColor: '#424865', + delay: 0.3 + }, + { + prefix: ' ', + color: '#7b84a4', + text: '→ chunking in parallel ....... done', + textColor: '#424865', + delay: 0.5 + }, + { + prefix: '✓ ', + color: '#3fa872', + text: '1.8B rows → Iceberg on S3', + textColor: '#070911', + delay: 0.7 + }, + { + prefix: '$ ', + color: '#3fa872', + text: 'olake fusion --maintain', + textColor: '#070911', + delay: 1.0 + }, + { + prefix: ' ', + color: '#7b84a4', + text: '→ compaction · cleanup · metadata', + textColor: '#424865', + delay: 1.2 + }, + { + prefix: '✓ ', + color: '#3fa872', + text: 'tables optimized · 2× faster than Spark', + textColor: '#193AE6', + delay: 1.4 + } + ], + + tickerLoop: [...ticker, ...ticker], + + engines: [ + { + tag: '// OLAKE GO', + tagColor: '#193AE6', + glow: 'rgba(25, 58, 230, 0.4)', + borderColor: 'rgba(25, 58, 230, 0.4)', + edgeColor: '#313858', + glyphBg: '#040615', + glyphBorder: 'rgba(25, 58, 230, 0.25)', + glyph: 'G', + href: '/olake-go', + title: 'OLake Go', + body: 'Replicate your databases into Apache Iceberg & Parquet on S3.', + chips: [{ label: 'CDC' }, { label: 'Parallel chunking' }, { label: 'Incremental sync' }], + link: 'Explore OLake Go' + }, + { + tag: '// OLAKE FUSION', + tagColor: '#193AE6', + glow: 'rgba(25, 58, 230, 0.35)', + borderColor: 'rgba(25, 58, 230, 0.4)', + edgeColor: '#292f4b', + glyphBg: '#03040f', + glyphBorder: 'rgba(25, 58, 230, 0.2)', + glyph: 'F', + href: '/olake-fusion', + title: 'OLake Fusion', + body: 'Keep your Apache Iceberg tables consistently performant and scalable.', + chips: [{ label: 'Compaction' }, { label: 'Cleanup', soon: true }], + link: 'Explore OLake Fusion' + } + ], + + // Copies of the logo set; enough to keep the marquee covered on wide screens. + logoGroups: Array.from({ length: 4 }, () => logos), + + selectGo: () => setState({ advantage: 'go' }), + selectFusion: () => setState({ advantage: 'fusion' }), + benchmarkHref: isGo + ? '/docs/benchmarks/ingestion/' + : '/docs/fusion/getting-started/compaction/', + goTabBg: isGo ? activeTab : 'transparent', + goTabColor: isGo ? '#ffffff' : '#000000', + fusionTabBg: !isGo ? activeTab : 'transparent', + fusionTabColor: !isGo ? '#ffffff' : '#000000', + + whyRows: [ + { + stickyTop: 96, + z: 1, + accent: '#193AE6', + num: '01', + kicker: 'FAST', + title: 'Replicate databases at scale', + body: 'Sync MySQL, Postgres, MongoDB, Kafka, and more to Apache Iceberg with parallelised chunking, incremental sync, and CDC.' + }, + { + stickyTop: 138, + z: 2, + accent: '#193AE6', + num: '02', + kicker: 'OPEN', + title: 'Built on open standards', + body: 'Write directly to Apache Iceberg or Parquet, with support for AWS Glue, Hive Metastore, and REST catalogs like Nessie, Polaris, and Unity. ' + }, + { + stickyTop: 180, + z: 3, + accent: '#193AE6', + num: '03', + kicker: 'CONTROLLED', + title: 'Self-hosted, on your infrastructure', + body: 'Deploy entirely within your own cloud or on-prem, keeping full control over where regulated data lives.' + }, + { + stickyTop: 222, + z: 4, + accent: '#3fa872', + num: '04', + kicker: 'MAINTAINED', + title: 'Keep tables fast as data keeps growing', + body: 'Automated compaction and delete-file cleanup keep query performance and storage costs in check as you scale.' + } + ], + + bulletin: [ + { + slotId: 'bulletin-1', + url: props.latestReleasePath || '/docs/release/ingestion', + img: R.bullRelease || '/img/landing/v2/bull-release.webp', + imgPlaceholder: 'Add thumbnail', + tag: 'RELEASE', + title: props.latestReleaseLabel || 'OLake' + }, + { + slotId: 'bulletin-2', + url: '/blog/iceberg-compaction-spark-vs-fusion-benchmark/', + img: R.bullBenchmark || '/img/landing/v2/bull-benchmark.webp', + imgPlaceholder: 'Add thumbnail', + tag: 'BLOG', + title: 'Fusion vs. Spark' + }, + { + slotId: 'bulletin-3', + url: '/blog/schema-evolution-without-breaking-pipelines/', + img: R.bullEngineering || '/img/landing/v2/bull-engineering.webp', + imgPlaceholder: 'Add thumbnail', + tag: 'BLOG', + title: 'Schema evolution' + } + ], + + stories: [ + { + quote: 'Zero Pipeline failure, 50% faster loads.', + company: 'Xeno', + logo: '/img/landing/v2/logo-xeno.webp' + }, + { + quote: "Cordial's Path to an AI-Ready Lakehouse.", + company: 'Cordial', + logo: '/img/landing/v2/logo-cordial.webp' + }, + { + quote: 'From 40-Minute to Sub-Minute Segmentation Queries', + company: 'Bitespeed', + logo: '/img/landing/v2/logo-bitespeed.webp' + } + ], + + resources: [ + { + type: 'DEMO', + title: 'OLake quickstart: your first ingestion pipeline', + href: 'https://youtu.be/IcAJmW72d2A?si=bAmaDOdEDy6vbKt8' + }, + { type: 'WEBINAR', title: 'Iceberg for Agents', href: '/webinar/w-14-iceberg-for-agents' }, + { + type: 'WEBINAR', + title: 'Apache Arrow + ADBC & Apache Iceberg', + href: 'https://www.youtube.com/watch?v=shrS0qdOPis&list=PL0H6rlkVhiiGSaO_xr1xBJ16dQKI-jvF_&index=14' + }, + { + type: 'BLOG', + title: 'Issues with Debezium — and how OLake solves them', + href: '/blog/issues-debezium-kafka/' + } + ] + } + } + + useEffect(() => { + self._fitArch = () => { + const frame = document.getElementById('archFrame') + const canvas = document.getElementById('archCanvas') + if (!frame || !canvas) return + const s = frame.clientWidth / 1680 + canvas.style.transform = 'scale(' + s + ')' + frame.style.height = 452 * s + 'px' + } + self._fitArch() + window.addEventListener('resize', self._fitArch) + setTimeout(self._fitArch, 200) + setTimeout(self._fitArch, 900) + + const card = document.querySelector('[data-countup]') + self._counters = [...document.querySelectorAll('[data-countup]')] + self._animEl = (el, instant) => { + const target = parseFloat(el.getAttribute('data-countup')) || 0 + const decimals = parseInt(el.getAttribute('data-decimals'), 10) || 0 + const prefix = el.getAttribute('data-prefix') || '' + const suffix = el.getAttribute('data-suffix') || '' + const finalText = prefix + target.toFixed(decimals) + suffix + const token = (el._ct = (el._ct || 0) + 1) + if (instant) { + el.textContent = finalText + return + } + const dur = 1100 + let start = null + const step = (ts) => { + if (token !== el._ct) return + if (start === null) start = ts + const p = Math.min((ts - start) / dur, 1) + const eased = 1 - Math.pow(1 - p, 3) + el.textContent = p < 1 ? prefix + (target * eased).toFixed(decimals) + suffix : finalText + if (p < 1) requestAnimationFrame(step) + } + requestAnimationFrame(step) + } + // Toggling the Go/Fusion tab updates the target: show it statically, no roll. + self._mo = new MutationObserver((muts) => { + muts.forEach((m) => self._animEl(m.target, true)) + }) + self._counters.forEach((el) => + self._mo.observe(el, { attributes: true, attributeFilter: ['data-countup'] }) + ) + if (card && 'IntersectionObserver' in window) { + self._io = new IntersectionObserver( + (entries) => { + entries.forEach((e) => { + if (e.isIntersecting) { + self._counters.forEach((el) => self._animEl(el)) + self._io.disconnect() + } + }) + }, + { threshold: 0.4 } + ) + self._io.observe(card.closest('div[style*="border-radius: 22px"]') || card) + } else { + self._counters.forEach((el) => self._animEl(el)) + } + + return () => { + if (self._fitArch) window.removeEventListener('resize', self._fitArch) + if (self._io) self._io.disconnect() + if (self._mo) self._mo.disconnect() + } + // eslint-disable-next-line react-hooks/exhaustive-deps + }, []) + + return renderVals() +} diff --git a/src/components/landing/seo/LandingSeo.tsx b/src/components/landing/seo/LandingSeo.tsx new file mode 100644 index 000000000..83733b570 --- /dev/null +++ b/src/components/landing/seo/LandingSeo.tsx @@ -0,0 +1,70 @@ +import React from 'react' +import Head from '@docusaurus/Head' + +export interface JsonLdSchema { + id: string + data: Record +} + +interface LandingSeoProps { + title: string + description: string + canonicalUrl: string + ogImage: string + twitterTitle?: string + twitterDescription?: string + jsonLdSchemas: JsonLdSchema[] +} + +/** + * Shared SEO head block for the three landing pages, following the pattern + * already used by src/pages/index.jsx and other pages in this repo. + * + * Note: as of this port, `