Add composite index main_detection(occurrence_id, timestamp DESC)#1251
Add composite index main_detection(occurrence_id, timestamp DESC)#1251
Conversation
Adds a composite index to speed up the /captures/ list endpoint, which joins main_detection and main_occurrence, filters by project_id, and sorts by timestamp DESC. Without this index PostgreSQL falls back to a nested-loop join scanning ~517k detection rows before filtering. Measured impact (per external perf report): Before: ~1,470 ms (nested loop, full timestamp sort) After: ~251 ms (parallel hash join on the filtered result) ~83% reduction on cold cache. Uses django.contrib.postgres.operations.AddIndexConcurrently with atomic=False so the index builds without holding a table-level write lock — safe for online prod deployment on a table this size. Co-Authored-By: Claude <noreply@anthropic.com>
✅ Deploy Preview for antenna-ssec canceled.
|
✅ Deploy Preview for antenna-preview canceled.
|
📝 WalkthroughWalkthroughThis pull request introduces a composite database index on the Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Adds a PostgreSQL composite index on main_detection (occurrence_id, timestamp DESC) to improve performance of the /captures/ hot-path query patterns that join detections to occurrences and sort by newest detection timestamp.
Changes:
- Add a composite ordered index definition to the
Detectionmodel (occurrence,-timestamp). - Add a non-atomic migration that creates the index using
CREATE INDEX CONCURRENTLYviaAddIndexConcurrently.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
ami/main/models.py |
Declares the new composite index on Detection to support join + timestamp DESC ordering patterns. |
ami/main/migrations/0084_index_detection_occurrence_timestamp.py |
Applies the index online with AddIndexConcurrently and atomic = False for production safety on a large table. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@coderabbitai review this please if you haven't already |
|
✅ Actions performedReview triggered.
|
|
I have tested locally on a large dataset, will test in staging & demo env next |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
ami/main/migrations/0084_index_detection_occurrence_timestamp.py (1)
1-21: Concurrent index migration looks correct; consider documenting recovery for invalid-index failures.
AddIndexConcurrently+atomic = Falseis the right approach for a largemain_detectiontable to avoid anACCESS EXCLUSIVElock. Index definition matches the modelMeta.indexesentry (same name and field order), somakemigrations --checkshould remain clean.Operational note (per the PR description): if
CREATE INDEX CONCURRENTLYfails mid-build, Postgres leaves an INVALID index that must be dropped manually (DROP INDEX CONCURRENTLY detection_occurrence_ts_desc;) before the migration can be retried — otherwise the retry will fail on the duplicate relation. Worth calling out in the deploy runbook, and verifying on staging before prod.Also, Ruff RUF012 warnings on
dependencies/operationsare false positives — these are standard Django migration class attributes.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@ami/main/migrations/0084_index_detection_occurrence_timestamp.py` around lines 1 - 21, The migration uses AddIndexConcurrently with atomic = False to avoid locks, but you must document operational recovery for a failed CONCURRENTLY build: add a brief comment in the migration (near AddIndexConcurrently / atomic = False) and/or update the deploy runbook explaining that a mid-build failure leaves an INVALID index named detection_occurrence_ts_desc and operators must run DROP INDEX CONCURRENTLY detection_occurrence_ts_desc before retrying the migration; you can also note that Ruff RUF012 warnings on dependencies/operations are false positives so no code change is needed for those.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@ami/main/migrations/0084_index_detection_occurrence_timestamp.py`:
- Around line 1-21: The migration uses AddIndexConcurrently with atomic = False
to avoid locks, but you must document operational recovery for a failed
CONCURRENTLY build: add a brief comment in the migration (near
AddIndexConcurrently / atomic = False) and/or update the deploy runbook
explaining that a mid-build failure leaves an INVALID index named
detection_occurrence_ts_desc and operators must run DROP INDEX CONCURRENTLY
detection_occurrence_ts_desc before retrying the migration; you can also note
that Ruff RUF012 warnings on dependencies/operations are false positives so no
code change is needed for those.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 8493ad21-4fe2-429b-aac0-38fddbd13b9e
📒 Files selected for processing (2)
ami/main/migrations/0084_index_detection_occurrence_timestamp.pyami/main/models.py
Summary
Add a composite index on
main_detection (occurrence_id, timestamp DESC)to speed up the/captures/list endpoint, which joinsmain_detectionandmain_occurrence, filters byproject_id, and sorts by timestamp DESC.Without this index PostgreSQL falls back to a nested-loop join that scans ~517k detection rows before filtering by project, then sorts the full set.
Measured impact (external perf report, warm DB, caching disabled)
This is a straight-up plan change: the query planner picks a parallel hash join because it can read detections ordered by timestamp for each occurrence directly from the index, instead of collecting all qualifying detections and sorting.
Implementation
ami/main/models.py— addMeta.indexes = [Index(fields=[\"occurrence\", \"-timestamp\"], name=\"detection_occurrence_ts_desc\")]onDetection.ami/main/migrations/0084_index_detection_occurrence_timestamp.py— usesdjango.contrib.postgres.operations.AddIndexConcurrentlywithatomic = False.Verified locally that the migration emits the correct SQL:
…and the resulting index is visible on the table:
Why AddIndexConcurrently
main_detectionis one of the largest tables in the system. A plainCREATE INDEXtakes anACCESS EXCLUSIVE-level table lock while it builds, blocking writes for the duration.CREATE INDEX CONCURRENTLYtrades a longer build for an online build that only needs aSHARE UPDATE EXCLUSIVElock — safe during production traffic.Gotcha:
AddIndexConcurrentlycan't run inside a transaction, so the migration setsatomic = False. If the build fails partway, the index is left in an invalid state and must be dropped manually (DROP INDEX CONCURRENTLY) before re-running — standard for concurrent index builds.Test plan
python manage.py migrate mainapplies cleanly in CI stacksqlmigrateemitsCREATE INDEX CONCURRENTLY\\d main_detectionshows the composite indexEXPLAIN (ANALYZE, BUFFERS)on/captures/query with a large project, compare plan before/afterRelated
Pairs with #1250 (require
project_idon hot list endpoints) as the second half of the "fix the major culprits now" pass. Neither depends on the other — can merge independently.🤖 Generated with Claude Code
Summary by CodeRabbit