diff --git a/CHANGELOG.md b/CHANGELOG.md index ca14da1c02..f3723d8a19 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -814,7 +814,7 @@ ### Added -* Add [deploy](https://marquezproject.github.io/marquez/deployment-overview.html) docs for running Marquez on AWS [@wslulciuc](https://github.com/wslulciuc) [@merobi-hub](https://github.com/merobi-hub) +* Add [deploy](https://marquezproject.ai/deployment-overview.html) docs for running Marquez on AWS [@wslulciuc](https://github.com/wslulciuc) [@merobi-hub](https://github.com/merobi-hub) ### Changed @@ -824,10 +824,10 @@ ### Deprecated -* The following endpoints have been deprecated and are **scheduled to be removed in** `0.25.0`. Please use the [`/lineage`](https://marquezproject.github.io/marquez/openapi.html#tag/Lineage/paths/~1lineage/post) endpoint when collecting source, dataset, and job metadata [@wslulciuc](https://github.com/wslulciuc): - * [`/sources`](https://marquezproject.github.io/marquez/openapi.html#tag/Sources/paths/~1sources~1{source}/put) endpoint to collect source metadata - * [`/datasets`](https://marquezproject.github.io/marquez/openapi.html#tag/Datasets/paths/~1namespaces~1{namespace}~1datasets~1{dataset}/put) endpoint to collect dataset metadata - * [`/jobs`](https://marquezproject.github.io/marquez/openapi.html#tag/Jobs/paths/~1namespaces~1{namespace}~1jobs~1{job}/put) endpoint to collect job metadata +* The following endpoints have been deprecated and are **scheduled to be removed in** `0.25.0`. Please use the [`/lineage`](https://marquezproject.ai/openapi.html#tag/Lineage/paths/~1lineage/post) endpoint when collecting source, dataset, and job metadata [@wslulciuc](https://github.com/wslulciuc): + * [`/sources`](https://marquezproject.ai/openapi.html#tag/Sources/paths/~1sources~1{source}/put) endpoint to collect source metadata + * [`/datasets`](https://marquezproject.ai/openapi.html#tag/Datasets/paths/~1namespaces~1{namespace}~1datasets~1{dataset}/put) endpoint to collect dataset metadata + * [`/jobs`](https://marquezproject.ai/openapi.html#tag/Jobs/paths/~1namespaces~1{namespace}~1jobs~1{job}/put) endpoint to collect job metadata ### Fixed @@ -906,7 +906,7 @@ ### Deprecated -* Run API [endpoints](https://marquezproject.github.io/marquez/openapi.html#tag/Jobs) that create or modify a job run (**scheduled to be removed in** `0.19.0`). Please use the **POST** `/api/v1/lineage` [endpoint](https://marquezproject.github.io/marquez/openapi.html#tag/Lineage/paths/~1lineage/post) when collecting job run metadata. [@wslulciuc](https://github.com/wslulciuc) +* Run API [endpoints](https://marquezproject.ai/openapi.html#tag/Jobs) that create or modify a job run (**scheduled to be removed in** `0.19.0`). Please use the **POST** `/api/v1/lineage` [endpoint](https://marquezproject.ai/openapi.html#tag/Lineage/paths/~1lineage/post) when collecting job run metadata. [@wslulciuc](https://github.com/wslulciuc) * Airflow integration, please use the [`openlineage-airflow`](https://pypi.org/project/openlineage-airflow) library instead. [@wslulciuc](https://github.com/wslulciuc) * Spark integration, please use the [`openlineage-spark`](https://search.maven.org/artifact/io.openlineage/openlineage-spark) library instead. [@wslulciuc](https://github.com/wslulciuc) * Write only clients for `java` and `python` (**scheduled to be removed in** `0.19.0`) [@wslulciuc](https://github.com/wslulciuc) diff --git a/README.md b/README.md index 03cd69d86b..3461e77043 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -
+
@@ -64,7 +64,7 @@ Start all services: $ sh ./docker/up.sh ``` -> **Tip:** Use the `--build` flag to build images from source, and/or `--seed` to start Marquez with sample lineage metadata. For a more complete example using the sample metadata, please follow our [quickstart](https://marquezproject.github.io/marquez/quickstart.html) guide. +> **Tip:** Use the `--build` flag to build images from source, and/or `--seed` to start Marquez with sample lineage metadata. For a more complete example using the sample metadata, please follow our [quickstart](https://marquezproject.ai/quickstart.html) guide. > **Note:** Port 5000 is now reserved for MacOS. If running locally on MacOS, you can run `./docker/up.sh --api-port 9000` to configure the API to listen on port 9000 instead. Keep in mind that you will need to update the URLs below with the appropriate port number. @@ -78,7 +78,7 @@ You can open [http://localhost:3000](http://localhost:3000) to begin exploring t **`HTTP API`** -The Marquez [HTTP API](https://marquezproject.github.io/marquez/openapi.html) listens on port `5000` for all calls and port `5001` for the admin interface. The admin interface exposes helpful endpoints like `/healthcheck` and `/metrics`. To verify the HTTP API server is running and listening on `localhost`, browse to [http://localhost:5001](http://localhost:5001). To begin collecting lineage metadata as OpenLineage events, use the [LineageAPI](https://marquezproject.github.io/marquez/openapi.html#tag/Lineage/paths/~1lineage/post) or an OpenLineage [integration](https://openlineage.io/docs/integrations/about). +The Marquez [HTTP API](https://marquezproject.ai/openapi.html) listens on port `5000` for all calls and port `5001` for the admin interface. The admin interface exposes helpful endpoints like `/healthcheck` and `/metrics`. To verify the HTTP API server is running and listening on `localhost`, browse to [http://localhost:5001](http://localhost:5001). To begin collecting lineage metadata as OpenLineage events, use the [LineageAPI](https://marquezproject.ai/openapi.html#tag/Lineage/paths/~1lineage/post) or an OpenLineage [integration](https://openlineage.io/docs/integrations/about). > **Note:** By default, the HTTP API does not require any form of authentication or authorization. @@ -90,7 +90,7 @@ To explore metadata via graphql, browse to [http://localhost:5000/graphql-playgr We invite everyone to help us improve and keep documentation up to date. Documentation is maintained in this repository and can be found under [`docs/`](https://github.com/MarquezProject/marquez/tree/main/docs). -> **Note:** To begin collecting metadata with Marquez, follow our [quickstart](https://marquezproject.github.io/marquez/docs/quickstart.html) guide. Below you will find the steps to get up and running from source. +> **Note:** To begin collecting metadata with Marquez, follow our [quickstart](https://marquezproject.ai/quickstart.html) guide. Below you will find the steps to get up and running from source. ## Versions and OpenLineage Compatibility @@ -112,7 +112,7 @@ Marquez uses a _multi_-project structure and contains the following modules: * [`api`](https://github.com/MarquezProject/marquez/tree/main/api): core API used to collect metadata * [`web`](https://github.com/MarquezProject/marquez/tree/main/web): web UI used to view metadata -* [`clients`](https://github.com/MarquezProject/marquez/tree/main/clients): clients that implement the HTTP [API](https://marquezproject.github.io/marquez/openapi.html) +* [`clients`](https://github.com/MarquezProject/marquez/tree/main/clients): clients that implement the HTTP [API](https://marquezproject.ai/openapi.html) * [`chart`](https://github.com/MarquezProject/marquez/tree/main/chart): helm chart > **Note:** The `integrations` module was removed in [`0.21.0`](https://github.com/MarquezProject/marquez/blob/main/CHANGELOG.md#removed), so please use an OpenLineage [integration](https://openlineage.io/integration) to collect lineage events easily. @@ -168,7 +168,7 @@ By default, Marquez uses the following ports: ```bash $ ./gradlew :api:runShadow ``` -Marquez listens on port `8080` for all API calls and port `8081` for the admin interface. To verify the HTTP API server is running and listening on `localhost`, browse to [http://localhost:8081](http://localhost:8081). We encourage you to familiarize yourself with the [data model](https://marquezproject.github.io/marquez/#data-model) and [APIs](https://marquezproject.github.io/marquez/openapi.html) of Marquez. To run the web UI, please follow the steps outlined [here](https://github.com/MarquezProject/marquez/tree/main/web#development). +Marquez listens on port `8080` for all API calls and port `8081` for the admin interface. To verify the HTTP API server is running and listening on `localhost`, browse to [http://localhost:8081](http://localhost:8081). We encourage you to familiarize yourself with the [data model](https://marquezproject.ai/#data-model) and [APIs](https://marquezproject.ai/openapi.html) of Marquez. To run the web UI, please follow the steps outlined [here](https://github.com/MarquezProject/marquez/tree/main/web#development). > **Note:** By default, the HTTP API does not require any form of authentication or authorization. @@ -178,7 +178,7 @@ Marquez listens on port `8080` for all API calls and port `8081` for the admin i ## Getting Involved -* Website: https://marquezproject.github.io/marquez/ +* Website: https://marquezproject.ai/ * Source: https://github.com/MarquezProject/marquez * Chat: [MarquezProject Slack](https://join.slack.com/t/marquezproject/shared_invite/zt-2iylxasbq-GG_zXNcJdNrhC9uUMr3B7A) * X: [@MarquezProject](https://twitter.com/MarquezProject) diff --git a/clients/python/README.md b/clients/python/README.md index 34706227be..2f4462e16d 100644 --- a/clients/python/README.md +++ b/clients/python/README.md @@ -4,7 +4,7 @@ Python client for [Marquez](https://github.com/MarquezProject/marquez). ## Documentation -See the [API docs](https://marquezproject.github.io/marquez/openapi.html). +See the [API docs](https://marquezproject.ai/openapi.html). ## Requirements diff --git a/docs/blog/2021-07-14-using-marquez-api/index.mdx b/docs/blog/2021-07-14-using-marquez-api/index.mdx index cb6d6ef9be..9eba05cb49 100644 --- a/docs/blog/2021-07-14-using-marquez-api/index.mdx +++ b/docs/blog/2021-07-14-using-marquez-api/index.mdx @@ -546,7 +546,7 @@ curl -s "http://localhost:5000/api/v1/lineage?nodeId=$NODE" | jq –arg N “${N ## Additional Resources -For a list of all the available queries and more information about the API, see the Marquez API Reference: https://marquezproject.github.io/marquez/openapi.html. +For a list of all the available queries and more information about the API, see the Marquez API Reference: https://marquezproject.ai/openapi.html. Interested in contributing to the project? Read our guide for new contributors: https://github.com/MarquezProject/marquez/blob/main/CONTRIBUTING.md. diff --git a/docs/docs/deployment/deployment.mdx b/docs/docs/deployment/deployment.mdx index 90a30ea1c1..a30e047f6a 100644 --- a/docs/docs/deployment/deployment.mdx +++ b/docs/docs/deployment/deployment.mdx @@ -9,7 +9,7 @@ Marquez uses [Helm](https://helm.sh) to manage deployments onto [Kubernetes](htt ## Database -The Marquez [HTTP API](https://marquezproject.github.io/marquez/openapi.html) server relies only on PostgreSQL to store dataset, job, and run metadata allowing for minimal operational overhead. We recommend a cloud provided databases, such as AWS [RDS](https://aws.amazon.com/rds/postgresql), when deploying Marquez onto Kubernetes. +The Marquez [HTTP API](https://marquezproject.ai/openapi.html) server relies only on PostgreSQL to store dataset, job, and run metadata allowing for minimal operational overhead. We recommend a cloud provided databases, such as AWS [RDS](https://aws.amazon.com/rds/postgresql), when deploying Marquez onto Kubernetes. ## Architecture diff --git a/docs/docs/deployment/running-on-aws.mdx b/docs/docs/deployment/running-on-aws.mdx index d8a70f0c3d..b2c24c3201 100644 --- a/docs/docs/deployment/running-on-aws.mdx +++ b/docs/docs/deployment/running-on-aws.mdx @@ -43,7 +43,7 @@ To create an AWS EKS cluster, please follow the steps outlined in the AWS EKS [d ## AWS RDS -Next, create an AWS RDS instance as outlined in the AWS RDS [documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateDBInstance.html). This database will be used to store dataset, job, and run metadata collected as [OpenLineage](https://openlineage.io) events via the Marquez [HTTP API](https://marquezproject.github.io/marquez/openapi.html). +Next, create an AWS RDS instance as outlined in the AWS RDS [documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_CreateDBInstance.html). This database will be used to store dataset, job, and run metadata collected as [OpenLineage](https://openlineage.io) events via the Marquez [HTTP API](https://marquezproject.ai/openapi.html). ##### CREATE AWS RDS DATABASE diff --git a/proposals/2078-optimization-ol-facets.md b/proposals/2078-optimization-ol-facets.md index 34f82a4fb0..f63b9dd597 100644 --- a/proposals/2078-optimization-ol-facets.md +++ b/proposals/2078-optimization-ol-facets.md @@ -8,7 +8,7 @@ Discussion: [https://github.com/MarquezProject/marquez/issues/2078](https://gith ## Overview -[OpenLineage](https://openlineage.io) was initially prototyped using Marquez, with the [initial draft](https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md#010---2021-08-12) of the spec taking inspiration from Marquez's [data model](https://lucid.app/lucidchart/f918ce01-9eb4-4900-b266-49935da271b8/view?page=8xAE.zxyknLQ#). OpenLineage events are collected via [`POST` `/lineage`](https://marquezproject.github.io/marquez/openapi.html#tag/Lineage) calls, and can be queried via the `lineage_events` table using the `run_uuid` associated with the event. The _current_ schema for the `lineage_events` table is defined below: +[OpenLineage](https://openlineage.io) was initially prototyped using Marquez, with the [initial draft](https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md#010---2021-08-12) of the spec taking inspiration from Marquez's [data model](https://lucid.app/lucidchart/f918ce01-9eb4-4900-b266-49935da271b8/view?page=8xAE.zxyknLQ#). OpenLineage events are collected via [`POST` `/lineage`](https://marquezproject.ai/openapi.html#tag/Lineage) calls, and can be queried via the `lineage_events` table using the `run_uuid` associated with the event. The _current_ schema for the `lineage_events` table is defined below: ### Table `lineage_events`