Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .claude/rules/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,26 @@ tests/
| `@pytest.mark.monitoring` | CloudWatch/DLQ verification | Skip with `-m "not monitoring"` |
| `@pytest.mark.snapshots` | Usage snapshot verification | Skip with `-m "not snapshots"` |

## Async Fixture Scoping (pytest-asyncio)

When using module or class-scoped async fixtures, **both** the fixture AND test markers must specify matching `loop_scope`:

```python
# Fixture
@pytest_asyncio.fixture(scope="module", loop_scope="module")
async def shared_repo(...):
...

# Test
@pytest.mark.asyncio(loop_scope="module")
async def test_something(shared_repo):
...
```

Without matching `loop_scope`, you'll get: `RuntimeError: Task got Future attached to a different loop`

**Data isolation pattern:** Use `unique_entity_prefix` fixture for per-test entity ID prefixes within shared infrastructure.

## Running Tests

```bash
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/ci-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,8 @@ jobs:
AWS_DEFAULT_REGION: us-east-1
AWS_ENDPOINT_URL: http://localhost:4566
- name: Run integration tests
run: uv run pytest -m integration -v --cov=zae_limiter --cov-report=xml --junitxml=junit.xml -o junit_family=legacy
# --durations=10 shows the 10 slowest tests for performance analysis
run: uv run pytest -m integration -v --durations=10 --cov=zae_limiter --cov-report=xml --junitxml=junit.xml -o junit_family=legacy
env:
AWS_ACCESS_KEY_ID: test
AWS_SECRET_ACCESS_KEY: test
Expand Down
47 changes: 47 additions & 0 deletions docs/adr/121-module-scoped-test-fixtures.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# ADR-121: Module-scoped Test Fixtures for Integration Tests

**Status:** Proposed
**Date:** 2026-02-02
**Issue:** [#253](https://github.com/zeroae/zae-limiter/issues/253)

## Context

Integration tests against LocalStack create and destroy DynamoDB tables for each test function. With 19 tests distributed across 4 pytest-xdist workers, this results in multiple table creation/deletion cycles per worker. The cumulative overhead causes the integration test suite to run for approximately 13 minutes.

The tests are isolated at the data level—each test creates distinct entities and buckets—but share no infrastructure within each worker. This isolation pattern is overly conservative since tests do not interfere with each other's data when using unique entity identifiers.

## Decision

Integration tests must use module-scoped pytest fixtures (`localstack_repo_module`, `localstack_limiter_module`) that create infrastructure once per test module, with per-test data isolation achieved through unique entity ID prefixes (`unique_entity_prefix` fixture).

## Consequences

**Positive:**
- Integration test runtime reduced by ~20% (from ~13 minutes to ~10 minutes with xdist)
- Same isolation guarantees maintained through unique entity prefixes
- Pattern aligns with E2E tests which already use class-scoped fixtures
- Cleaner fixture naming convention with scope in name

**Negative:**
- Tests must use `loop_scope="module"` on both fixtures and test markers for async compatibility
- Debugging failures requires understanding that data from other tests exists in the shared table
- Tests that require clean infrastructure state must use function-scoped fixtures explicitly

**Observed Limitation:**
The improvement is modest (~20%) because CI runs with pytest-xdist (`-n auto --dist loadscope`), which distributes test modules across 4 workers. Each worker creates its own module-scoped fixtures, so the total stack count equals the worker count rather than 1.

Disabling xdist to maximize fixture sharing was tested and rejected—sequential execution took ~26 minutes vs ~10 minutes with xdist. Parallelization benefits outweigh stack creation overhead.

## Alternatives Considered

### Disable xdist for integration tests
Rejected because: Sequential execution (~26 min) is significantly slower than parallel execution with per-worker stacks (~10 min). The parallelization benefit exceeds the stack creation overhead.

### Class-scoped fixtures
Rejected because: Module scope maximizes sharing across test classes within each xdist worker, reducing infrastructure creation from once-per-class to once-per-module.

### Session-scoped fixtures
Rejected because: Session scope would share infrastructure across test modules, making test isolation harder to reason about and potentially causing cross-module interference in CI parallel execution.

### Per-test table creation (status quo)
Rejected because: The ~13 minute runtime creates friction for development iteration. Even a 20% improvement is worthwhile.
54 changes: 49 additions & 5 deletions tests/benchmark/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,10 @@
localstack_endpoint,
minimal_stack_options,
sync_localstack_limiter,
unique_entity_prefix,
unique_name,
unique_name_class,
unique_name_module,
)
from tests.unit.conftest import (
_patch_aiobotocore_response,
Expand All @@ -37,10 +39,13 @@
"minimal_stack_options",
"aggregator_stack_options",
"sync_localstack_limiter",
"sync_localstack_limiter_module",
"sync_localstack_limiter_no_cache",
"sync_localstack_limiter_with_aggregator",
"unique_entity_prefix",
"unique_name",
"unique_name_class",
"unique_name_module",
"capacity_counter",
"benchmark_entities",
]
Expand Down Expand Up @@ -200,20 +205,25 @@ def test_example(self, sync_limiter, capacity_counter):
yield counter


@pytest.fixture
@pytest.fixture(scope="class")
def sync_localstack_limiter_with_aggregator(
localstack_endpoint, aggregator_stack_options, unique_name
localstack_endpoint, aggregator_stack_options, unique_name_class
):
"""SyncRateLimiter with Lambda aggregator for benchmark tests.

Creates a LocalStack-based limiter with aggregator Lambda enabled
for benchmarking cold start and warm start latency.

This fixture is primarily used for Lambda cold/warm start benchmarks
that require the aggregator function to be deployed.
This fixture is class-scoped to share a single stack across all tests
within TestLambdaColdStartBenchmarks. This is intentional:
- First test in class = cold start (fresh Lambda)
- Subsequent tests = warm start (reused container)

Tests MUST use unique_entity_prefix for entity ID isolation.
See issue #253 for details.
"""
limiter = SyncRateLimiter(
name=unique_name,
name=unique_name_class,
endpoint_url=localstack_endpoint,
region="us-east-1",
stack_options=aggregator_stack_options,
Expand Down Expand Up @@ -309,3 +319,37 @@ def sync_localstack_limiter_no_cache(localstack_endpoint, minimal_stack_options)
limiter.delete_stack()
except Exception as e:
print(f"Warning: Stack cleanup failed: {e}")


# ---------------------------------------------------------------------------
# Module-scoped fixtures for shared infrastructure (issue #253)
# ---------------------------------------------------------------------------


@pytest.fixture(scope="module")
def sync_localstack_limiter_module(localstack_endpoint, minimal_stack_options, unique_name_module):
"""SyncRateLimiter with minimal stack (module-scoped, shared across all tests).

Creates CloudFormation stack once per test module. Tests MUST use
unique_entity_prefix for entity ID isolation within the shared table.

This fixture reduces benchmark time by avoiding repeated stack creation/deletion.
See issue #253 for details.

Note: Do not use for benchmarks that require fresh infrastructure (e.g., Lambda
cold start benchmarks). Use function-scoped sync_localstack_limiter instead.
"""
limiter = SyncRateLimiter(
name=unique_name_module,
endpoint_url=localstack_endpoint,
region="us-east-1",
stack_options=minimal_stack_options,
)

with limiter:
yield limiter

try:
limiter.delete_stack()
except Exception as e:
print(f"Warning: Module-scoped stack cleanup failed: {e}")
Loading
Loading