-
Notifications
You must be signed in to change notification settings - Fork 306
docs: refresh codec server page #4483
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
lennessyy
wants to merge
16
commits into
main
Choose a base branch
from
codec-server-refresh
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 1 commit
Commits
Show all changes
16 commits
Select commit
Hold shift + click to select a range
68fd25e
docs: refresh codec server page
lennessyy 2988738
address review comments
lennessyy 6821361
update diagram
lennessyy 8e6495a
adjust page width
lennessyy 12b16a9
add dark mode diagram
lennessyy 14adf90
Apply suggestions from code review
lennessyy af1aa0e
Apply suggestion from @lennessyy
lennessyy c0c8704
Update docs/encyclopedia/data-conversion/codec-server.mdx
lennessyy ae74001
Merge branch 'main' into codec-server-refresh
lennessyy 2653cbc
minor update
lennessyy 31c2822
merge resolution
lennessyy fd37011
Merge branch 'main' into codec-server-refresh
lennessyy ccdc8cf
Apply suggestions from code review
lennessyy 47ab1d5
address pass access token vs include cros credentials
lennessyy f64a821
spell out acronym
lennessyy 9ab0402
feat: add external storage support to codec server (#4489)
lennessyy File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,19 +2,18 @@ | |
| id: codec-server | ||
| title: Codec Server | ||
| sidebar_label: Codec Server | ||
| description: A Codec Server is an HTTP server that provides remote encoding and decoding for Temporal Payloads. | ||
| description: | ||
| A Codec Server is an HTTP server that provides remote encoding and decoding for Temporal Payloads, enabling the Web UI | ||
| and CLI to display decoded data without exposing encryption keys to the Temporal Service. | ||
| slug: /codec-server | ||
| toc_max_heading_level: 4 | ||
| keywords: | ||
| - encryption | ||
| - explanation | ||
| - keys | ||
| - codec-server | ||
| - payloads | ||
| - secrets | ||
| - data-converters | ||
| - codec-server | ||
| - remote-data-encoding | ||
| tags: | ||
| - codec-server | ||
| - Concepts | ||
| - Encryption | ||
| - Data Converters | ||
|
|
@@ -23,43 +22,96 @@ tags: | |
|
|
||
| import { CaptionedImage } from '@site/src/components'; | ||
|
|
||
| This page discusses [Codec Server](#codec-server). | ||
| A Codec Server is an HTTP/HTTPS server that you host and operate. It runs your [Payload Codec](/payload-codec) logic to | ||
| encode and decode [Payloads](/dataconversion#payload) on behalf of the Temporal CLI and Web UI. The Codec Server is | ||
| independent of the Temporal Service. Encryption keys and codec logic remain in your environment. | ||
|
|
||
| For setup instructions, see [Codec Server setup](/production-deployment/data-encryption#codec-server-setup). | ||
|
|
||
| ## Why use a Codec Server | ||
|
|
||
| When you apply a custom [Payload Codec](/payload-codec) for encryption or compression, data stored in the Temporal | ||
| Service is encoded. The Temporal Service never has access to your encryption keys, so it cannot decode this data. Without | ||
| a Codec Server, the Web UI and CLI display raw encoded payloads. | ||
|
|
||
| ## What is a Codec Server? {#codec-server} | ||
| A Codec Server solves this by giving the Web UI and CLI a way to decode payloads on demand, without exposing keys to the | ||
| Temporal Service. Common reasons to run a Codec Server include: | ||
|
|
||
| A Codec Server is an HTTP/HTTPS server that uses a [custom Payload Codec](/production-deployment/data-encryption) to decode your data remotely through endpoints. | ||
| - **Debugging Workflows.** View decoded Workflow inputs, outputs, and Event History in the Web UI instead of reading | ||
| base64-encoded or encrypted blobs. | ||
| - **Operating from the CLI.** Use commands like `temporal workflow show` and `temporal workflow execute` with readable | ||
| data, even when payloads are encrypted at rest. | ||
| - **Encoding inputs from the UI and CLI.** When you start or signal a Workflow from the Web UI or CLI, the Codec Server | ||
| can encode the input before it reaches the Temporal Service, so sensitive data is never sent in plaintext. | ||
|
lennessyy marked this conversation as resolved.
Outdated
|
||
| - **Compliance and access control.** Because the Codec Server runs in your environment, you control who can decode | ||
| payloads and under what conditions. You can layer authorization on top of the decode endpoint to restrict access per | ||
| user or per Namespace. | ||
|
|
||
| {/* This should not have changed with tctl-to-temporal */} | ||
| ## How a Codec Server works | ||
|
|
||
| A Codec Server follows the Temporal | ||
| [Codec Server Protocol](https://github.com/temporalio/samples-go/tree/main/codec-server#codec-server-protocol). It | ||
| exposes two HTTP POST endpoints: | ||
|
|
||
| - **`/encode`** accepts plaintext payloads and returns encoded payloads. | ||
|
lennessyy marked this conversation as resolved.
Outdated
|
||
| - **`/decode`** accepts encoded payloads and returns decoded payloads. | ||
|
lennessyy marked this conversation as resolved.
Outdated
|
||
|
|
||
| Both endpoints receive and respond with a JSON body containing a `payloads` array of | ||
| [Payload](/dataconversion#payload) objects. The Codec Server passes each payload through your | ||
| [Payload Codec](/payload-codec), which applies the same encoding or decoding logic that your Workers use. | ||
|
|
||
| <CaptionedImage | ||
| src="/diagrams/tctl-diagram-codec-server.svg" | ||
| title="Codec Server" | ||
| width="50%" | ||
| zoom="true" | ||
| src="/diagrams/tctl-diagram-codec-server.svg" | ||
| title="Codec Server" | ||
| width="50%" | ||
| zoom="true" | ||
| /> | ||
|
|
||
| A Codec Server follows the Temporal [Codec Server Protocol](https://github.com/temporalio/samples-go/tree/main/codec-server#codec-server-protocol). | ||
| It implements two endpoints: | ||
| When the Web UI or CLI needs to display decoded data, it sends the encoded payloads to your Codec Server's `/decode` | ||
| endpoint. The Codec Server decodes the payloads and returns them to the client. Decoding happens entirely on the client | ||
| side. The Temporal Service never sees the decoded data. | ||
|
lennessyy marked this conversation as resolved.
Outdated
|
||
|
|
||
| - `/encode` | ||
| - `/decode` | ||
| The `/encode` endpoint works in the other direction. When you start a Workflow or send a Signal from the Web UI or CLI, | ||
| the input is sent to the Codec Server's `/encode` endpoint first, so data reaches the Temporal Service in its encoded | ||
| form. | ||
|
|
||
| Each endpoint receives and responds with a JSON body that has a `payloads` property with an array of [Payloads](/dataconversion#payload). | ||
| The endpoints run the Payloads through a [Payload Codec](/payload-codec) before returning them. | ||
| Your Codec Server should use the same Payload Codec implementation as your Workers to ensure consistent encoding and | ||
| decoding. | ||
|
lennessyy marked this conversation as resolved.
lennessyy marked this conversation as resolved.
|
||
|
|
||
| Most SDKs provide example Codec Server implementation samples, listed here: | ||
| ## Web UI and CLI integration | ||
|
lennessyy marked this conversation as resolved.
Outdated
|
||
|
|
||
| - [Go](https://github.com/temporalio/samples-go/tree/main/codec-server) | ||
| - [Java](https://github.com/temporalio/sdk-java/tree/master/temporal-remote-data-encoder) | ||
| - [.NET](https://github.com/temporalio/samples-dotnet/tree/main/src/Encryption) | ||
| - [Python](https://github.com/temporalio/samples-python/blob/main/encryption/codec_server.py) | ||
| - [TypeScript](https://github.com/temporalio/samples-typescript/blob/main/encryption/src/codec-server.ts) | ||
| To use a Codec Server with the Web UI, configure the Codec Server endpoint in the Web UI settings. For the Temporal CLI, | ||
| pass the `--codec-endpoint` flag. Once configured, the Web UI and CLI route payloads through your Codec Server | ||
| automatically. | ||
|
|
||
| Because the Web UI calls the Codec Server from the browser, you must enable | ||
| [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) on your Codec Server. At a minimum, allow the origin of | ||
| your Web UI, the `POST` method, and the `Content-Type` and `X-Namespace` headers. Each request from the Web UI includes | ||
| an `X-Namespace` header identifying the Namespace the payloads belong to. | ||
|
|
||
| For detailed configuration steps, CORS setup, and authorization options, see | ||
| [Codec Server setup](/production-deployment/data-encryption#codec-server-setup). | ||
|
|
||
| ## Authorization | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This looks good but somebody from security, perhaps Kent Gruber, should have a look. |
||
|
|
||
| Access to a Codec Server should be restricted because it can decode sensitive data with a single API call. Common | ||
|
lennessyy marked this conversation as resolved.
Outdated
|
||
| approaches include: | ||
|
|
||
| #### Usage | ||
| - **Network-level restrictions.** Run the Codec Server on `localhost` or behind a VPN. The Web UI can communicate with a | ||
| Codec Server that is only accessible locally. | ||
| - **Token-based authorization.** Integrate your organization's authentication provider (OAuth, Auth0, or similar). | ||
| Temporal Cloud can pass access tokens (JWT) to your Codec Server with each request. Verify these tokens against the | ||
| [Temporal Cloud JWKS endpoint](https://login.tmprl.cloud/.well-known/jwks.json). | ||
|
|
||
| When you apply custom encoding with encryption or compression on your Workflow data, it is stored in the encrypted/compressed format on the Temporal Server. For details on what data is encoded, see [Securing your data](/production-deployment/data-encryption). | ||
| You may also need a [key management infrastructure](/key-management) to share encryption keys between your Workers and | ||
| the Codec Server. | ||
|
|
||
| To see decoded data when using the Temporal CLI or Web UI to perform some operations on a Workflow Execution, configure the Codec Server endpoint in the Web UI and the Temporal CLI. | ||
| When you configure the Codec Server endpoints, the Temporal CLI and Web UI send the encoded data to the Codec Server, and display the decoded data received from the Codec Server. | ||
| ## SDK Codec Server samples | ||
|
|
||
| For details on creating your Codec Server, see [Codec Server Setup](/production-deployment/data-encryption#codec-server-setup). | ||
| Most Temporal SDKs provide example Codec Server implementations: | ||
|
|
||
| - [Go](https://github.com/temporalio/samples-go/tree/main/codec-server) | ||
| - [Java](https://github.com/temporalio/sdk-java/tree/master/temporal-remote-data-encoder) | ||
| - [Python](https://github.com/temporalio/samples-python/blob/main/encryption/codec_server.py) | ||
| - [TypeScript](https://github.com/temporalio/samples-typescript/blob/main/encryption/src/codec-server.ts) | ||
| - [.NET](https://github.com/temporalio/samples-dotnet/blob/main/src/Encryption/CodecServer/Program.cs) | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.