A practical audit checklist for LLM-based systems with persistent memory.
A 98-item audit checklist for any LLM-based assistant or agent that accumulates memory across sessions — dialog, episodic, semantic, vector, graph, or multimodal.
Covers the full memory loop: ingestion → storage → retrieval → context assembly → model → output → feedback.
Every item is backed by a real production incident. This is not theory — it's distilled operational experience from running 12+ AI assistants with multi-layered memory architectures over multiple years.
- Developers building AI assistants with persistent memory
- Teams maintaining long-running conversational AI systems
- Architects designing memory layers for LLM agents
- Anyone who has experienced silent memory degradation and wants to prevent it
If your AI agent or LLM assistant has any of these symptoms, this checklist will help you find and fix the root cause:
- Agent repeats itself — says the same thing every conversation, stuck in a loop
- Agent forgets everything after restart — personality resets, context lost
- Memory grows uncontrollably — database doubles in size after every reindex
- Agent "hallucinates" its own memories — confidently recalls things that never happened
- Context degradation — responses get worse over time, slowly, silently
- Feedback loops — agent analyzes its own reports, writes summaries of summaries
- Stale information persists — agent keeps mentioning outdated facts despite corrections
- Embedding search returns garbage — after model change, semantic search breaks silently
- Truncated context — memories cut off mid-sentence,
text[:limit]destroys meaning - Log files grow to hundreds of MB — blocking diagnostics and agent operations
- Tests pass but agent is broken — unit tests green, production behavior degraded
- Agent can't distinguish its own output from user input — self-contamination loop
Each symptom maps to specific checklist items. See Section K (Quick Diagnostics) in the standard for the full symptom → defect mapping.
New here? Start with the Quick Check: 13 critical items with a symptom, a five-minute check and a pass criterion for each, about one hour in total.
- Download
STANDARD.md(English) ortranslations/STANDARD_RU.md(Russian) - Start with Section 0 (System Map) — without it, other sections are unreliable
- Work through sections A–K, marking each item:
yes/partial/no/n/a - For each failure, record evidence (file, line, SQL query, log entry)
- Use the Verdict Protocol at the end to assess pass/fail
- Record the results in the audit report template — see also a completed real-world audit report (anonymized) and an anonymized re-audit aggregate
| Section | Focus | Items |
|---|---|---|
| 0. System Map | Preparation: identify all stores, paths, keys, baselines | 10 |
| A. Input Validation | What enters memory — and what shouldn't | 7 |
| B. Write Integrity | Idempotency, deduplication, transactions | 6 |
| C. Growth Management | Limits, retention, cache invalidation | 8 |
| D. Retrieval | Ranking, gating, embedding compatibility | 11 |
| E. Context Assembly | Summarization, budgets, position awareness, artifact separation | 13 |
| F. Feedback Loop | Anti-recursion, echo prevention, authority laundering | 8 |
| G. Isolation | Trust boundaries, domain separation, negative fixtures | 7 |
| H. Concurrency | Migrations, races, test isolation | 6 |
| I. Observability | Monitoring, recovery, diagnostics, warning dedup, escalation | 16 |
| J. Change Management | One change at a time, rollback, baselines | 6 |
| K. Quick Diagnostics | Symptom → probable defect mapping | — |
| Total | 98 |
- [CRIT] — Direct risk of data loss, poisoning, leakage, or uncontrolled growth. Failure = blocker.
- [IMP] — Risk of silent quality degradation.
- [REC] — Maturity and maintainability improvement.
- Memory stores state, not just data. Deletion is personality damage.
- Blind truncation is prohibited. Summarization only.
- One change per observation interval. Simultaneous modifications make regression non-localizable.
- Diagnostics cannot be part of the sick system. External tools see structure; the agent sees context. Neither alone is complete.
- The feedback loop is always closed. Anything stored in memory returns to the model and can reproduce itself.
The standard includes a mandatory preparation phase for live systems (Section 2.1):
- Backup before any action
- Inventory all stores and paths
- Verify tables and relationships
- Migrate, don't delete
- Green tests before auditing
- Isolate tests from production
- Check log sizes — uncontrolled growth blocks the auditing agent
Post-audit: remediation via roadmap, one wave at a time, with commits and status updates after each.
docs/ADMISSION_CRITERIA.md— admission gate for new checklist itemsdocs/AUDIT_GUIDE.md— how to conduct, deliver, and report an audit (effective from v1.8)docs/patterns/TEST_EVIDENCE_PATTERNS.md— non-normative test-evidence patterns: MAQS classifies what to prove, the patterns show how
This standard was developed alongside a series of practical articles:
- Memory-Safe AI Development — How to design memory systems
- Silent Failures, Part 1 — Five patterns of internal degradation
- Silent Failures, Part 2 — Environmental and structural degradation
Derived from multi-year operational experience with 12+ deployed AI assistants across different architectures (Llama, Gemini, Claude, DeepSeek, Mistral, GPT-4o, Qwen). Each checklist item traces to a real incident: self-contamination loops, silent amnesia, storage poisoning, embedding mismatches, feedback loops, personality loss on restart, and more.
Anonymized incident descriptions are published in the companion articles above (Silent Failures, Parts 1–2); the full incident-to-item mapping is maintained by the author and available as audit evidence.
Q: Do I need this if my agent only uses JSON files, not databases? A: Yes. Sections 0 (System Map) and F (Feedback Loop) apply regardless of storage format. JSON files are stores too — they grow, they accumulate duplicates, they can be poisoned by model output.
Q: How long does an audit take? A: First audit: 4–8 hours (including preparation). Subsequent audits of the same system: 1–2 hours (focus on changed sections).
Q: My system passes all unit tests. Do I still need this? A: Unit tests verify code correctness. This standard verifies memory integrity. Shadow config, orphan execution, stale seed, and feedback loops all pass unit tests while silently degrading the system.
Q: Is this compatible with OWASP LLM Top 10? A: They are complementary, not competing. OWASP covers security threats (prompt injection, data poisoning). MAQS covers memory quality (recursion, degradation, context integrity). A robust system needs both.
MAQS operates in a rapidly growing research space. Key 2026 publications that address overlapping concerns:
- "Silent Failure in LLM Agent Systems: The Entropy Principle" (Liu, 2026) — taxonomy of 22 intrinsic properties across six lifecycle layers, including memory persistence. arxiv.org/html/2606.08162v1
- "When Errors Become Narratives" (2026) — longitudinal study of silent failures in a production LLM agent runtime; 70% of failures caught by human observation, not tests. arxiv.org/pdf/2606.14589
- "Memory as Infrastructure" (Helwig, 2026) — months-scale instrumented record of persistent agent memory in production; introduces "silent memory death." arxiv.org/pdf/2609.05510
- "Memory as Metabolism" (2026) — governance framework for user-coupled drift in personal LLM memory systems. arxiv.org/pdf/2604.12034
- "Mnemonic Sovereignty" (2026) — survey on security of long-term LLM agent memory; notes that benign-persistence failures remain sparsely studied. arxiv.org/html/2604.16548v1
- "Memory for Autonomous LLM Agents" (2026) — survey of mechanisms and evaluation; documents summarization drift and silent paging failures. arxiv.org/html/2603.07670v1
These works taxonomize and study memory failures. MAQS provides the actionable audit checklist to detect them before they drift.
See CONTRIBUTING.md for guidelines.
Contributions welcome:
- Translations to other languages
- New checklist items with incident backing
- Audit report templates
- Tool integrations (CI/CD, automated checks)
- Case studies and experience reports
This work is licensed under Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0). Copyright (c) 2026 Aleksandr Kossarev
You are free to share, adapt, and build upon this standard — even commercially — as long as you give appropriate credit and distribute your contributions under the same license.
Aleksandr Kossarev — Jõgeva, Estonia
- Dev.to: @aleksandr_kossarev_e23623
- GitHub: @alexk202
- Project: Arche Iscrin
If this standard helped you prevent a memory incident — consider starring the repo. If it didn't prevent one — consider contributing the incident as a new checklist item.
Keywords (for search engines)
LLM memory management, AI agent persistent memory, chatbot memory leak, LLM context degradation, AI assistant forgets context, persistent memory recursion, feedback loop prevention, AI memory poisoning, LLM agent memory architecture, context window optimization, memory-safe AI development, AI agent audit checklist, LLM memory quality standard, chatbot repeats itself fix, AI agent context loss, embedding model mismatch, silent failures AI agents, LLM memory anti-recursion, persistent memory feedback loop, AI assistant personality loss restart, memory truncation vs summarization, AI agent diagnostic checklist