-
Notifications
You must be signed in to change notification settings - Fork 330
FF: restructure and edit #3312
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
kaitlynmichael
wants to merge
6
commits into
main
Choose a base branch
from
DOC-6576
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
FF: restructure and edit #3312
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
61af870
create new files in develop/ai
kaitlynmichael 81df63c
typo
kaitlynmichael 5c1cf53
DOC-6581 DOC-6582
kaitlynmichael b5c5666
DOC-6583
kaitlynmichael 3b69a60
indent list
kaitlynmichael d93c065
review feedback
kaitlynmichael File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| --- | ||
| title: | ||
| description: | ||
| linkTitle: | ||
| weight: 1 | ||
| --- |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| --- | ||
| title: | ||
| description: | ||
| linkTitle: | ||
| weight: 1 | ||
| --- |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| --- | ||
| title: | ||
| description: | ||
| linkTitle: | ||
| weight: 1 | ||
| --- |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| --- | ||
| title: | ||
| description: | ||
| linkTitle: | ||
| weight: 1 | ||
| --- | ||
|
|
||
| ## Manage workspaces | ||
|
|
||
| Use these commands when you need to inspect or change a workspace directly. | ||
|
|
||
| ### Core commands | ||
|
|
||
| ```bash | ||
| ff workspace list | ||
| ff workspace get --name demo-workspace | ||
| ff workspace update <workspace-id> \ | ||
| --name demo-workspace \ | ||
| --description "Updated description" | ||
| ff workspace delete <workspace-id> --force | ||
| ``` | ||
|
|
||
| ### Workspace state to remember | ||
|
|
||
| - workspaces have unique names and descriptions | ||
|
kaitlynmichael marked this conversation as resolved.
Outdated
|
||
| - each workspace tracks `last_applied_version` | ||
| - providers, secret providers, graph state, catalog entries, and serving metadata are workspace-scoped | ||
|
|
||
| Deleting a workspace removes its associated workspace-scoped data. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| --- | ||
| title: | ||
| description: | ||
| linkTitle: | ||
| weight: 1 | ||
| --- |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| --- | ||
| title: | ||
| description: | ||
| linkTitle: | ||
| weight: 1 | ||
| --- |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,200 @@ | ||
| --- | ||
| title: Register providers | ||
| description: Register storage, compute, and catalog providers in a Redis Feature Form workspace, and configure secret backends. | ||
| linkTitle: Register providers | ||
| weight: 30 | ||
| --- | ||
|
|
||
| Register the providers and secret backends a Redis Feature Form workspace needs before you author features or transformations. Providers connect the workspace to external systems for storage, compute, serving, or catalog-backed access, and definitions files reference them by name. | ||
|
kaitlynmichael marked this conversation as resolved.
Outdated
|
||
|
|
||
| ## Prerequisites | ||
|
|
||
| Before you register providers, make sure you have: | ||
|
|
||
| - A workspace. See [Manage workspaces](./manage-workspace.md) for the workspace lifecycle commands. | ||
|
kaitlynmichael marked this conversation as resolved.
Outdated
|
||
| - The `ff` CLI installed and able to reach the Feature Form server. | ||
| - The CLI connects to `localhost:9090` by default; override with `--server <host:port>` or by setting `ServerAddress` in `~/.featureform/config.yaml`. | ||
| - Any environment variables your provider commands reference set **in the Feature Form server's environment**, not in your shell. | ||
| - For example, `--pg-password-secret env:PG_PASSWORD` makes the server resolve `PG_PASSWORD` from its own process environment at runtime. For Helm-based deployments, set these through chart values; for binary deployments, export them where the server starts. | ||
|
kaitlynmichael marked this conversation as resolved.
Outdated
|
||
|
|
||
| The examples on this page use placeholder names like `demo-workspace`, `demo_postgres`, and `spark-main`. Substitute the names you want to use in your own deployment. | ||
|
|
||
| {{< note >}} | ||
| **Best practice:** keep the default health check on. Registration surfaces connectivity and secret-resolution problems at the point you can fix them, rather than as silent failures during materialization or serving. Reserve `--skip-health-check` for cases where you've already validated the provider through another channel. | ||
| {{< /note >}} | ||
|
|
||
| ## Register Postgres for offline storage | ||
|
kaitlynmichael marked this conversation as resolved.
Outdated
|
||
|
|
||
| Use Postgres when the workspace needs an offline store and Postgres-backed SQL execution in the same path. As an `offline-store`, Postgres holds the historical feature values that training sets read from. As a `compute` provider, it runs the SQL transformations that produce those values. | ||
|
kaitlynmichael marked this conversation as resolved.
Outdated
|
||
|
|
||
| The `<release-name>` placeholder in `--pg-host` and in the Redis `--redis-host` stands for your Helm release name. With release name `my-ff`, the bundled Postgres service is `my-ff-featureform-provider-postgres`. If you connect to an external Postgres or Redis instance instead of the bundled chart addons, use that hostname directly. | ||
|
kaitlynmichael marked this conversation as resolved.
Outdated
|
||
|
|
||
| ```bash | ||
| ff provider register demo_postgres \ | ||
| --workspace demo-workspace \ | ||
| --type postgres \ | ||
| --pg-host <release-name>-featureform-provider-postgres \ | ||
| --pg-port 5432 \ | ||
| --pg-database featureform_test \ | ||
| --pg-user testuser \ | ||
| --pg-password-secret env:PG_PASSWORD \ | ||
| --pg-ssl-mode disable | ||
| ``` | ||
|
|
||
| See the [PostgreSQL documentation](https://www.postgresql.org/docs/) for connection and SSL options. | ||
|
|
||
| ## Register Redis as the online store | ||
|
|
||
| Use Redis when the workspace needs an online store for low-latency feature serving. As an `online-store`, Redis holds the latest materialized feature values and serves them to applications at inference time. | ||
|
kaitlynmichael marked this conversation as resolved.
Outdated
|
||
|
|
||
| ```bash | ||
| ff provider register demo_redis \ | ||
| --workspace demo-workspace \ | ||
| --type redis \ | ||
| --redis-host <release-name>-featureform-redis \ | ||
| --redis-port 6379 | ||
| ``` | ||
|
|
||
| In the quickstart definitions file, the feature view references this provider with `inference_store="demo_redis"`. See the [Redis documentation](https://redis.io/docs/latest/) for deployment options. | ||
|
|
||
| ## Register S3 as an offline store | ||
|
|
||
| Use S3 when Feature Form needs an object-storage-backed offline location. As an `offline-store`, S3 holds historical feature values as files (typically Parquet) that training sets read from. Choose S3 when dataset size or retention exceeds what a relational store fits. | ||
|
|
||
| ```bash | ||
| ff provider register data-lake \ | ||
| --workspace demo-workspace \ | ||
| --type s3 \ | ||
| --s3-bucket featureform-data \ | ||
| --s3-region us-west-2 \ | ||
| --s3-access-key-id-secret env:AWS_ACCESS_KEY_ID \ | ||
| --s3-secret-access-key-secret env:AWS_SECRET_ACCESS_KEY | ||
| ``` | ||
|
|
||
| Use `--s3-endpoint` for MinIO or LocalStack-style endpoints when needed. See the [Amazon S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/) for bucket and IAM setup. | ||
|
|
||
| ## Register Spark for compute | ||
|
|
||
| Use Spark when the workspace needs a compute provider for transformation or materialization workloads. As a `compute` provider, Spark runs the transformation and materialization jobs that produce feature values. Choose Spark when dataset size exceeds what a single SQL engine can handle. | ||
|
|
||
| ```bash | ||
| ff provider register spark-main \ | ||
| --workspace demo-workspace \ | ||
| --type spark \ | ||
| --spark-master spark://spark-master:7077 | ||
| ``` | ||
|
|
||
| See the [Apache Spark documentation](https://spark.apache.org/docs/latest/) for cluster and master configuration. | ||
|
|
||
| ## Register an Iceberg catalog | ||
|
|
||
| Use an Iceberg catalog provider when the workspace needs catalog-backed offline storage. As an `offline-store`, the catalog tracks versioned table snapshots over object storage. The workspace reads historical feature values from those tables, with schema evolution and time-travel queries. | ||
|
|
||
| ```bash | ||
| ff provider register iceberg-main \ | ||
| --workspace demo-workspace \ | ||
| --type iceberg_catalog \ | ||
| --iceberg-warehouse s3://featureform-data/warehouse \ | ||
| --iceberg-catalog-name featureform \ | ||
| --iceberg-rest-uri https://iceberg.example.com | ||
| ``` | ||
|
|
||
| This example uses the REST catalog backend; the exact required fields depend on which backend (REST, Hive, Glue, and so on) you choose. See the [Apache Iceberg documentation](https://iceberg.apache.org/docs/latest/) for catalog backend options. | ||
|
|
||
| ## Verify registration | ||
|
|
||
| ```bash | ||
| ff provider list --workspace demo-workspace | ||
| ff provider get demo_postgres --workspace demo-workspace | ||
| ``` | ||
|
|
||
| A successful list returns one row per registered provider: | ||
|
|
||
| ```text | ||
| NAME TYPE WORKSPACE CREATED UPDATED | ||
| demo_postgres postgres demo-workspace 2026-05-12T10:14:02Z 2026-05-12T10:14:02Z | ||
| demo_redis redis demo-workspace 2026-05-12T10:14:18Z 2026-05-12T10:14:18Z | ||
| ``` | ||
|
|
||
| Pass `--output json` or `--output yaml` for machine-readable output. If the list is empty or `get` returns an error, the register command did not complete. Rerun `ff provider register` to see its health-check output, and confirm the provider name and workspace match the ones you registered. | ||
|
|
||
| ## Update or delete a provider | ||
|
|
||
| ```bash | ||
| ff provider update demo_postgres \ | ||
| --workspace demo-workspace \ | ||
| --pg-port 5433 | ||
|
|
||
| ff provider delete demo_postgres --workspace demo-workspace | ||
| ``` | ||
|
|
||
| Use `--force` on `update` when changing values that may break running workloads, such as host, port, or broker addresses. | ||
|
|
||
| ## Configure secret providers | ||
|
kaitlynmichael marked this conversation as resolved.
Outdated
|
||
|
|
||
| Confirm which secret backend a workspace uses, or register an alternate when `env` is not enough. Production deployments typically move off `env` because it mixes secrets with general configuration, offers no rotation or audit, and surfaces values in process listings. Vault, Kubernetes secrets, and AWS Secrets Manager each address those gaps. | ||
|
|
||
| ### Check the built-in `env` provider | ||
|
|
||
| ```bash | ||
| ff secret-provider list --workspace demo-workspace | ||
| ff secret-provider get env --workspace demo-workspace | ||
| ``` | ||
|
|
||
| ### Register another secret provider | ||
|
|
||
| Each backend has different preconditions on the Feature Form server. Pick the one that matches how your server is deployed. | ||
|
|
||
| **Environment provider** — best for local development and bootstrap. The server reads variables from its own process environment. Use a prefix (`--env-prefix FF_`) to avoid collisions with other system variables. | ||
|
|
||
| ```bash | ||
| ff secret-provider register local-env \ | ||
| --workspace demo-workspace \ | ||
| --type env \ | ||
| --env-prefix FF_ | ||
| ``` | ||
|
|
||
| **Vault** — best for shared deployments that need rotation and audit. The server must be able to authenticate to Vault: export `VAULT_TOKEN` for token auth, or configure Kubernetes auth (when the server runs in-cluster) or AppRole. The backend uses the KV v2 secrets engine. | ||
|
|
||
| ```bash | ||
| ff secret-provider register vault-main \ | ||
| --workspace demo-workspace \ | ||
| --type vault \ | ||
| --vault-address https://vault.example.com \ | ||
| --vault-token-path /var/run/secrets/vault-token | ||
| ``` | ||
|
|
||
| **Kubernetes secrets** — best when the server runs inside a Kubernetes cluster and provider credentials are already managed as `Secret` resources. The server's service account needs `get` and `list` permissions on `secrets` in the target namespace. | ||
|
|
||
| ```bash | ||
| ff secret-provider register k8s-main \ | ||
| --workspace demo-workspace \ | ||
| --type k8s \ | ||
| --k8s-namespace featureform \ | ||
| --k8s-secret-name provider-secrets | ||
| ``` | ||
|
|
||
| **AWS Secrets Manager** — best when provider credentials already live in AWS. The server authenticates using the standard AWS credentials chain (IAM role on the host, instance profile, or `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` in the server environment). | ||
|
|
||
| ```bash | ||
| ff secret-provider register aws-main \ | ||
| --workspace demo-workspace \ | ||
| --type aws \ | ||
| --aws-region us-west-2 | ||
| ``` | ||
|
|
||
| ### Update or delete a secret provider | ||
|
|
||
| ```bash | ||
| ff secret-provider update local-env \ | ||
| --workspace demo-workspace \ | ||
| --env-prefix PROD_ | ||
|
|
||
| ff secret-provider delete local-env \ | ||
| --workspace demo-workspace \ | ||
| --yes | ||
| ``` | ||
|
|
||
| ## Next steps | ||
|
|
||
| With providers registered, the workspace is ready to receive feature definitions. See [Define and deploy features](./define-and-deploy-features.md) for authoring a definitions file and running `ff apply`. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| --- | ||
| title: | ||
| description: | ||
| linkTitle: | ||
| weight: 1 | ||
| --- |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Page with content has empty frontmatter fields
Medium Severity
manage-workspace.mdcontains real body content (workspace management commands moved fromstreaming.md) but itstitle,description, andlinkTitlefrontmatter fields are all blank. Unlike the other new placeholder files that are intentionally empty stubs, this page has substantive content. The missing frontmatter means it will render with no page title in navigation and no heading, andregister-providers.mdalready links to it.Reviewed by Cursor Bugbot for commit b5c5666. Configure here.