Docs: align repo docs to 3.1.0, drop AI phrasing; harden bounded file reads

Author: DyNooob

ARCHITECTURE.md

@@ -59,9 +59,9 @@ The registry propagates a known source SHA-256 from observation metadata into ev
 
 `READ_ONLY < STATEFUL < DESTRUCTIVE`.
 
-The default policy allows only `READ_ONLY` and disallows network traffic. In 2.1:
+The default policy allows only `READ_ONLY` and disallows network traffic. In 3.1:
 
-- 65 tools are `READ_ONLY`;
+- 67 tools are `READ_ONLY` (68 registered in total);
 - `archive.extract_safe` is `STATEFUL` because it writes a derived workspace while leaving the source archive unchanged;
 - no registered tool is `DESTRUCTIVE`;
 - network-capable tools are separately gated.

RELEASE_NOTES_3.1.md

@@ -1,6 +1,6 @@
 # DFTK 3.1.0 release notes
 
-DFTK 3.1 keeps the project boundary introduced in 3.0: DFTK is a deterministic forensic capability layer, not an autonomous Agent. The release adds a native Agent transport and strengthens the existing case workspace without changing the public Observation/Evidence contract.
+DFTK 3.1 keeps the project boundary introduced in 3.0: DFTK is a deterministic forensic toolkit, not an autonomous Agent. The release adds a native Agent transport and strengthens the existing case workspace without changing the public Observation/Evidence contract.
 
 ## Native MCP
 

SECURITY.md

@@ -28,8 +28,8 @@ Only the latest published release line receives security fixes.
 
 | Version | Supported |
 |---------|-----------|
-| 2.1.x   | ✅ Yes |
-| < 2.1   | ❌ No |
+| 3.1.x   | ✅ Yes |
+| < 3.1   | ❌ No |
 
 ## Scope notes
 

skills/dftk/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: dftk
 description: Evidence-preserving digital forensics (DFIR) toolkit for agents. 68 read-only/stateful tools plus a unified-timeline and case-correlation workflow across Android, Linux, Windows, network, email, and crypto, behind a safety-gated Observation/Evidence contract. Use when an agent must collect, hash, parse, analyze, or correlate forensic artifacts without mutating evidence.
-version: 3.0.0
+version: 3.1.0
 author: DyNooob @ DigiForensics
 license: Apache-2.0
 tags:
@@ -14,7 +14,7 @@ tags:
 
 # dftk — Digital Forensics Toolkit (Agent Skill)
 
-dftk is a **capability layer**, not an autonomous forensic agent: a set of
+dftk is a **read-only forensic toolkit**, not an autonomous forensic agent: a set of
 evidence-preserving primitives and composable recipes for analysts, automation,
 and agents. Every tool returns a structured `Observation` (status + facts +
 evidence with source provenance).

src/dftk/SKILL.md

@@ -13,7 +13,7 @@ tags:
 ---
 # DFTK — Digital Forensics Toolkit
 
-DFTK is a **forensic capability layer**, not the investigator. You are the investigator.
+DFTK is a **read-only forensic toolkit**, not the investigator. You are the investigator.
 Use DFTK to obtain deterministic, source-traceable observations; use reasoning to decide what must be proven, which capability can prove it, whether the evidence is sufficient, and when to stop.
 
 The governing rule is:

src/dftk/core/helpers.py

@@ -55,3 +55,41 @@ def bounded_files(root: Path, *, max_files: int = 10000):
             n += 1
             if n >= max_files:
                 return
+
+# In-memory read ceiling for tools that must load the whole file. Forensic
+# evidence can legitimately be large, but loading multi-GB inputs unconditionally
+# risks out-of-memory; per-tool overrides are allowed.
+DEFAULT_MAX_READ = 4 * 1024 * 1024 * 1024
+
+def read_file_bounded(path: str | os.PathLike[str], max_bytes: int = DEFAULT_MAX_READ):
+    """Read a file fully, refusing inputs above ``max_bytes`` to avoid OOM.
+
+    Returns ``(data, None)`` on success, or ``(None, err)`` where ``err`` is a
+    dict with ``kind`` of ``"too_large"`` or ``"stat"`` and supporting fields.
+    The caller is responsible for converting ``err`` into an ``Observation``.
+    """
+    p = Path(path)
+    try:
+        size = p.stat().st_size
+    except OSError as e:
+        return None, {"kind": "stat", "error": str(e)}
+    if size > max_bytes:
+        return None, {"kind": "too_large", "size": size, "limit": max_bytes}
+    return p.read_bytes(), None
+
+def read_file_bounded_observation(name: str, path: str | os.PathLike[str], max_bytes: int = DEFAULT_MAX_READ):
+    """Like :func:`read_file_bounded`, but returns ``(data, None)`` or ``(None, Observation)``.
+
+    On oversized input the Observation is ``UNSUPPORTED``; on a stat/read failure it is ``ERROR``.
+    Both carry the source SHA-256 so the caller can return them directly.
+    """
+    from .models import Observation, Status
+    data, err = read_file_bounded(path, max_bytes)
+    if err is None:
+        return data, None
+    if err["kind"] == "too_large":
+        return None, Observation(name, Status.UNSUPPORTED, "File exceeds in-memory read limit",
+                                 errors=[f"size={err['size']} > limit={err['limit']}"],
+                                 meta={"source_sha256": sha256_file(path)})
+    return None, Observation(name, Status.ERROR, "File read failed", errors=[err["error"]],
+                             meta={"source_sha256": sha256_file(path)})

src/dftk/primitives/android.py

@@ -18,7 +18,7 @@
 import struct, zipfile, hashlib, re
 from dftk.core.registry import registry
 from dftk.core.models import Observation, Evidence, Status, SafetyLevel
-from dftk.core.helpers import sha256_file
+from dftk.core.helpers import sha256_file, read_file_bounded_observation
 
 class DexFormatError(ValueError): pass
 
@@ -324,7 +324,8 @@ def _apk_signing_block(data:bytes)->dict:
 def apk_signing_inventory(path:str)->Observation:
     p=Path(path)
     if not p.is_file() or not zipfile.is_zipfile(p): return Observation('android.apk_signing_inventory',Status.UNSUPPORTED,'Input is not an APK/ZIP')
-    data=p.read_bytes(); block=_apk_signing_block(data)
+    data,err=read_file_bounded_observation('android.apk_signing_inventory',p,1024*1024*1024); if err: return err
+    block=_apk_signing_block(data)
     with zipfile.ZipFile(p) as z:
         names=z.namelist(); v1=[n for n in names if n.upper().startswith('META-INF/') and n.upper().endswith(('.RSA','.DSA','.EC','.SF'))]
     schemes=[]

src/dftk/primitives/binary.py

@@ -17,7 +17,7 @@
 import struct,re
 from dftk.core.registry import registry
 from dftk.core.models import Observation,Evidence,Status,SafetyLevel
-from dftk.core.helpers import printable_strings,sha256_file
+from dftk.core.helpers import printable_strings, sha256_file, read_file_bounded_observation
 
 MACHINES={3:"x86",40:"ARM",62:"x86-64",183:"AArch64",243:"RISC-V"}
 
@@ -64,7 +64,7 @@ def name_at(off):
 def elf_inventory(path:str,strings:bool=True,string_limit:int=2000)->Observation:
     p=Path(path)
     if not p.is_file(): return Observation("binary.elf_inventory",Status.ERROR,"ELF file not found",errors=[str(p)])
-    data=p.read_bytes()
+    data,err=read_file_bounded_observation('binary.elf_inventory',p,512*1024*1024); if err: return err
     try: info=parse_elf(data)
     except ElfError as e: return Observation("binary.elf_inventory",Status.UNSUPPORTED,"ELF parsing failed",errors=[str(e)],meta={"source_sha256":sha256_file(p)})
     rows=printable_strings(data,4)[:string_limit] if strings else []
@@ -80,7 +80,7 @@ def elf_inventory(path:str,strings:bool=True,string_limit:int=2000)->Observation
 def pe_inventory(path:str,strings:bool=False,string_limit:int=2000)->Observation:
     p=Path(path)
     if not p.is_file(): return Observation('binary.pe_inventory',Status.ERROR,'PE file not found',errors=[str(p)])
-    data=p.read_bytes(); h=sha256_file(p)
+    data,err=read_file_bounded_observation('binary.pe_inventory',p,512*1024*1024); if err: return err; h=sha256_file(p)
     if len(data)<0x40 or data[:2]!=b'MZ': return Observation('binary.pe_inventory',Status.UNSUPPORTED,'Not an MZ/PE file',meta={'source_sha256':h})
     peoff=struct.unpack_from('<I',data,0x3c)[0]
     if peoff+24>len(data) or data[peoff:peoff+4]!=b'PE\0\0': return Observation('binary.pe_inventory',Status.UNSUPPORTED,'PE signature not found',meta={'source_sha256':h})
@@ -106,7 +106,7 @@ def pe_inventory(path:str,strings:bool=False,string_limit:int=2000)->Observation
 def native_indicator_scan(path:str,limit:int=1000)->Observation:
     p=Path(path)
     if not p.is_file(): return Observation('binary.native_indicator_scan',Status.ERROR,'Binary not found',errors=[str(p)])
-    rows=printable_strings(p.read_bytes(),4); categories={
+    data,err=read_file_bounded_observation('binary.native_indicator_scan',p,512*1024*1024); if err: return err; rows=printable_strings(data,4); categories={
         'jni':re.compile(r'(?:Java_[A-Za-z0-9_]+|JNI_OnLoad|RegisterNatives)'),
         'crypto':re.compile(r'(?i)(?:AES|DES|RC4|ChaCha|EVP_|SHA(?:1|224|256|384|512)|MD5|Cipher|encrypt|decrypt)'),
         'url':re.compile(r'(?i)https?://[^\s"\']+'),

src/dftk/primitives/database.py

@@ -32,7 +32,7 @@ def sqlite_inventory(path:str,count_rows:bool=True,table_limit:int=500)->Observa
         for typ,name,tbl,sql in rows:
             row={"type":typ,"name":name,"table":tbl,"sql":sql}
             if count_rows and typ=='table' and not name.startswith('sqlite_'):
-                try: row['row_count']=con.execute(f'SELECT COUNT(*) FROM "{name.replace(chr(34),chr(34)*2)}"').fetchone()[0]
+                try: row['row_count']=con.execute(f'SELECT COUNT(*) FROM "{name.replace(chr(34), chr(34)*2)}"').fetchone()[0]
                 except sqlite3.DatabaseError as e: row['row_count_error']=str(e)
             objects.append(row); ev.append(Evidence(str(p),'sqlite_schema',sql or name,locator=f"sqlite_master:{typ}:{name}"))
         qc=con.execute('PRAGMA quick_check').fetchone()[0]
@@ -93,7 +93,7 @@ def sql_dump_inventory(path:str,max_bytes:int=256*1024*1024,statement_limit:int=
                 if not raw: break
                 scanned+=len(raw)
                 line=raw.decode('utf-8','replace')
-                if not buf and line.lstrip().startswith(('--','#')): continue
+                if line.lstrip().startswith(('--','#')): continue
                 buf+=line
                 # Process complete-ish statements at semicolon boundaries. This is intentionally a schema/activity inventory, not a full SQL parser.
                 while ';' in buf and statements<statement_limit:

src/dftk/primitives/email_auth.py

@@ -21,7 +21,7 @@
 import re, hashlib, os
 from dftk.core.registry import registry
 from dftk.core.models import Observation,Evidence,Status,SafetyLevel
-from dftk.core.helpers import sha256_file
+from dftk.core.helpers import sha256_file, read_file_bounded_observation
 
 
 def _decode(v:str)->str:
@@ -58,7 +58,7 @@ def _parse_dkim_domains(headers:list[str]):
 def auth_analyze(path:str)->Observation:
     p=Path(path)
     if not p.is_file(): return Observation("email.auth_analyze",Status.ERROR,"EML file not found",errors=[str(p)])
-    raw=p.read_bytes()
+    raw,err=read_file_bounded_observation('email.auth_analyze',p,256*1024*1024); if err: return err
     try: msg=BytesParser(policy=policy.default).parsebytes(raw)
     except Exception as e: return Observation("email.auth_analyze",Status.ERROR,"Email parse failed",errors=[str(e)],meta={"source_sha256":sha256_file(p)})
     from_name,from_addr=parseaddr(_decode(msg.get('From','')))
@@ -96,7 +96,7 @@ def dkim_verify(path:str)->Observation:
     if not p.is_file(): return Observation("email.dkim_verify",Status.ERROR,"EML file not found",errors=[str(p)])
     try: import dkim
     except ImportError: return Observation("email.dkim_verify",Status.UNSUPPORTED,"dkimpy is not installed",errors=["install optional dependency: pip install 'dftk[email]'"],meta={"source_sha256":sha256_file(p)})
-    raw=p.read_bytes()
+    raw,err=read_file_bounded_observation('email.dkim_verify',p,256*1024*1024); if err: return err
     try:
         ok=bool(dkim.verify(raw))
     except Exception as e:
@@ -122,7 +122,8 @@ def mime_inventory(path:str,include_body_preview:bool=False,preview_chars:int=10
     import hashlib
     p=Path(path)
     if not p.is_file(): return Observation('email.mime_inventory',Status.ERROR,'Email file not found',errors=[str(p)])
-    try: msg=BytesParser(policy=policy.default).parsebytes(p.read_bytes())
+    data,err=read_file_bounded_observation('email.mime_inventory',p,256*1024*1024); if err: return err
+    try: msg=BytesParser(policy=policy.default).parsebytes(data)
     except Exception as e: return Observation('email.mime_inventory',Status.ERROR,'Email parsing failed',errors=[f'{type(e).__name__}: {e}'],meta={'source_sha256':sha256_file(p)})
     headers={k:str(v) for k,v in msg.items()}; parts=[]; attachments=[]; ev=[]
     iterable=list(msg.walk()) if msg.is_multipart() else [msg]