Repository navigation
Add MSK IAM authentication via SASL/OAUTHBEARER - #24
Merged
Merged
Conversation
When KAFKA_CONSUMER_SASL_MECHANISMS=OAUTHBEARER is set, automatically registers an oauth_cb that generates MSK IAM tokens via aws-msk-iam-sasl-signer-python. Both AdminClient and Consumer get the callback through _base_kafka_config. - Lazy import of signer (clear error if not installed) - Warns if singular sasl.mechanism is set (librdkafka ignores it) - Logs token generation failures for prod debugging - Optional dep: pip install millpond[msk-iam] (always in Docker image) - Documents credential isolation (IRSA for Kafka, static for DuckDB S3)
There was a problem hiding this comment.
Pull request overview
Adds MSK IAM authentication support to Millpond’s Kafka clients by wiring a SASL/OAUTHBEARER token callback into the shared Kafka client configuration, plus packaging/docs updates to keep AWS credentials isolated between Kafka (IRSA) and DuckDB S3 (static keys).
Changes:
- Introduces
_maybe_attach_oauth_cb()and hooks it into_base_kafka_config()so bothAdminClientandConsumerreceive the OAUTHBEARER callback. - Adds an optional dependency extra (
msk-iam) and installs it in the Docker image by default. - Updates unit tests and documentation around SASL config naming (
sasl.mechanisms) and AWS credential isolation.
Reviewed changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
millpond/consumer.py |
Adds MSK IAM OAUTHBEARER callback wiring via shared Kafka config helper. |
pyproject.toml |
Defines msk-iam optional dependency extra. |
uv.lock |
Locks aws-msk-iam-sasl-signer-python and its dependency set. |
Dockerfile |
Installs the msk-iam extra in image builds via uv sync --extra msk-iam. |
tests/unit/test_consumer.py |
Adds unit tests for OAUTHBEARER callback behavior and config-key warning. |
tests/unit/test_config.py |
Updates env var usage to KAFKA_CONSUMER_SASL_MECHANISMS → sasl.mechanisms. |
README.md |
Documents Kafka vs DuckDB S3 credential isolation rationale. |
AGENT.md |
Captures credential-isolation design decision and optional-dep rationale. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 8 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
oauth_cbfor MSK IAM token generation whenKAFKA_CONSUMER_SASL_MECHANISMS=OAUTHBEARERaws-msk-iam-sasl-signer-python(optional dep, always installed in Docker image)_base_kafka_configsasl.mechanism(singular) is used — librdkafka only readssasl.mechanisms(plural)Test plan
KAFKA_CONSUMER_SASL_MECHANISMS=OAUTHBEARERandKAFKA_CONSUMER_SECURITY_PROTOCOL=SASL_SSL