Saving graded outcomes

PLAN.md

@@ -0,0 +1,88 @@
+# Future: semantic (embeddings) matching for the grade cache
+
+Status: **not started**. Written 2026-08-19 as a placeholder so this isn't forgotten once
+there's enough real usage to justify it.
+
+## Context
+
+`/api/grade` already has an exact-match cache (`supabase/migrations/20260819000000_add_grade_cache.sql`,
+`find_cached_grade` RPC, wired into `src/app/api/grade/route.ts`). It matches on
+`(sentence_zh, strictness_used, normalize_answer(user_answer))`, where `normalize_answer` just
+lowercases/trims/collapses whitespace — a cache hit only ever fires for text that's byte-identical
+after that normalization, so it has **zero risk of serving a wrong grade**. It misses true
+paraphrases ("I want to buy this book" vs "I'd like to buy this book").
+
+We discussed extending this to fuzzy/character-similarity matching (edit distance, trigram
+similarity, a "70% similar" threshold) and **rejected it**: string-similarity metrics can't tell a
+paraphrase from a meaning-flipping edit. "I don't like apples" vs "I do like apples" score as
+highly similar by edit distance despite being opposite in meaning — the same mechanism that would
+catch "buy" ≈ "purchase" also catches "don't" ≈ "do", so there's no threshold that gets the gain
+without the risk. Not worth the correctness regression for a grading feature.
+
+Embeddings are the sound version of the same idea: they're trained to place semantically different
+sentences far apart even when textually close (so negation pairs score *less* similar, not more),
+so cosine similarity on embeddings is a meaningfully better signal for "does this mean the same
+thing" than any character-level metric. Still not perfect — don't treat it as risk-free.
+
+## Step 0 — measure first, before building anything
+
+Do **not** start this on a total-row-count trigger — total rows spread across ~1,050 seeded
+sentences × 3 strictness levels tells you almost nothing. What matters is concentration: how many
+attempts land on the *same* `(sentence_zh, strictness_used)` bucket, since that's the granularity
+the cache matches on. `selectStaticSentence` (`src/lib/staticSentences.ts`) narrows to the top-5
+lowest-accuracy candidates among a user's *unlocked* vocab, and early on that vocab is small (20
+words at signup), so traffic concentrates onto a small set of easy/common sentences before vocab
+has grown out — expect a Zipfian pileup on a handful of sentences well before the corpus as a
+whole has meaningful volume.
+
+Run this (static mode only — that's the mode this cache protects financially; AI-mode sentences
+are per-user generated and won't repeat across users at all):
+
+```sql
+select sentence_zh, strictness_used, count(*) as attempts,
+       count(distinct user_answer_normalized) as distinct_answers
+from sentence_attempts
+group by sentence_zh, strictness_used
+order by attempts desc
+limit 30;
+```
+
+`attempts / distinct_answers` on the top rows is a direct read on the exact-match cache's real hit
+rate where it matters most. Eyeball the `distinct_answers` for a top bucket — if most of them look
+like genuine near-duplicates a human would call "the same answer" (not just typos, which exact-match
+already catches), that's the signal semantic matching would help. As an order-of-magnitude starting
+point: check in once you're around a few hundred total static-mode grade attempts — that's roughly
+when the top few sentences should have accumulated double digits of repeat attempts, enough to read
+something from the ratio instead of noise. If the top buckets' near-duplicate rate is low, skip this
+feature — it isn't worth the added cost/complexity/risk.
+
+## If the data justifies building it
+
+1. **Enable `pgvector`** on the Supabase project (`CREATE EXTENSION IF NOT EXISTS vector;`).
+2. **Add an embedding column** to `sentence_attempts`, e.g. `user_answer_embedding vector(N)`
+   (dimension depends on the embedding model chosen), populated at insert time in
+   `src/app/api/grade/route.ts` alongside the existing `insertAttempt` write.
+3. **Call an embedding API** for the incoming answer before the cache lookup (cheap/fast relative
+   to a grading call, but it's a real added cost+latency on every grade request — factor that in).
+4. **Replace (or supplement) the exact-match RPC** with a similarity query: still filter on
+   `sentence_zh = ... AND strictness_used = ...` (those must still match exactly — only the answer
+   comparison becomes fuzzy), then order by cosine distance to the new answer's embedding
+   (`<=>` operator with an `ivfflat` or `hnsw` index) and only accept a match above a conservative
+   threshold. Start the threshold high (few false positives) and only loosen it based on observed
+   accuracy — don't guess a number up front.
+5. **No backfill needed for old rows** — same pattern as `feedback`: only rows with an embedding
+   populated are eligible matches, historical rows just never match until re-graded (fine).
+6. **Mirror the existing `find_cached_grade` SECURITY DEFINER pattern** (see
+   `supabase/migrations/20260806171015_drop_p_user_id_use_auth_uid.sql` for the REVOKE/GRANT
+   convention this project requires on every new function) rather than inventing a new access
+   pattern.
+
+## Open questions to resolve when this is picked up
+
+- Which embedding provider/model, and what's the actual per-call cost at expected volume — needs
+  to stay well below the cost of the Anthropic grading call it's trying to avoid, or the whole
+  feature is pointless.
+- What similarity threshold actually holds up — needs eyeballing real (sentence, candidate-match)
+  pairs near the cutoff, not a number picked in the abstract.
+- Whether to keep exact-match as a fast-path before falling through to the embedding lookup
+  (probably yes — it's free and zero-risk, only pay for an embedding call on an exact-match miss).

src/app/api/generate/route.ts

@@ -29,7 +29,19 @@ export async function POST(req: NextRequest) {
   if (auth instanceof NextResponse) return auth
   const { user, supabase } = auth
 
-  const rateLimit = await checkRateLimit(user.id, 'generate')
+  // practice_mode is read before rate limiting so the budget can be chosen
+  // per mode — 'ai' spends the user's own key (no budget protection needed,
+  // just a light anti-hammering cap) while 'static' only costs Supabase
+  // reads (same light cap, for a different reason).
+  const { data: userSettings } = await supabase
+    .from('settings')
+    .select('practice_mode')
+    .eq('user_id', user.id)
+    .single()
+
+  const practiceMode = userSettings?.practice_mode ?? 'static'
+
+  const rateLimit = await checkRateLimit(user.id, practiceMode === 'ai' ? 'generate_ai' : 'generate_static')
   if (!rateLimit.success) {
     return NextResponse.json(
       { error: 'Too many requests' },
@@ -47,14 +59,6 @@ export async function POST(req: NextRequest) {
       py: r.py.slice(0, MAX_RECENT_FIELD_LENGTH),
     }))
 
-  const { data: userSettings } = await supabase
-    .from('settings')
-    .select('practice_mode')
-    .eq('user_id', user.id)
-    .single()
-
-  const practiceMode = userSettings?.practice_mode ?? 'static'
-
   if (practiceMode === 'static') {
     // The client's variety-tracking sends recently-served {zh, py} pairs, not
     // sentence_bank ids — resolve them to ids here so selectStaticSentence can

src/app/api/grade/route.ts

@@ -122,7 +122,35 @@ export async function POST(req: NextRequest) {
 
   const truncatedAnswer = user_answer.slice(0, 500)
 
-  const prompt = `You are grading a Chinese-to-English translation exercise.
+  // Cross-user cache: an identical (sentence, strictness, normalized answer)
+  // triple may already have been graded by someone else, especially likely
+  // in static mode where a shared, finite sentence_bank means the same wrong
+  // answers recur across users. A hit skips the Anthropic call entirely — a
+  // lookup failure just falls through to a normal miss, never aborts the
+  // request.
+  // Project has no generated Supabase Database types, so .rpc() results are
+  // untyped — narrow to the shape find_cached_grade actually returns.
+  const { data: cached, error: cacheError } = await supabase
+    .rpc('find_cached_grade', {
+      p_sentence_zh: sentence_zh,
+      p_strictness: strictness,
+      p_user_answer: truncatedAnswer,
+    })
+    .maybeSingle() as { data: { score: number; correct_answer: string; feedback: string } | null; error: { message: string } | null }
+
+  if (cacheError) console.error('find_cached_grade error:', cacheError.message)
+
+  let parsed: GradeResponse
+
+  if (cached) {
+    parsed = {
+      correct: false, // recomputed below from score, never trusted from a cached row
+      score: cached.score,
+      feedback: cached.feedback,
+      correct_answer: cached.correct_answer,
+    }
+  } else {
+    const prompt = `You are grading a Chinese-to-English translation exercise.
 
 Chinese sentence: ${sentence_zh}
 Pinyin: ${sentence_py}
@@ -138,43 +166,46 @@ Grading mode: ${STRICTNESS[strictness] ?? STRICTNESS[2]}
 Respond with ONLY valid JSON, no markdown:
 {"correct":true or false,"score":0-100,"feedback":"one concise sentence","correct_answer":"the most natural English translation"}`
 
-  try {
-    const parsed = await callClaudeJson(prompt, 200, isGradeResponse, anthropicClient)
-    parsed.correct = parsed.score >= 70
-
-    // Tracking writes must never block or fail the grade response, but they also
-    // can't be truly fire-and-forget — Vercel freezes the function once the
-    // response is sent, which was silently dropping these. waitUntil keeps the
-    // invocation alive until they finish without making the client wait for them.
-    const recordAttempts = vocab_used?.length
-      ? Promise.all(vocab_used.map((zh: string) =>
-          supabase.rpc('record_word_attempt', {
-            p_word_zh: zh,
-            p_correct: parsed.correct,
-          })
-        )).catch(err => console.error('record_word_attempt error:', err))
-      : Promise.resolve()
-
-    const insertAttempt = supabase.from('sentence_attempts').insert({
-      user_id:         user.id,
-      sentence_zh,
-      sentence_py,
-      user_answer:     body.user_answer,
-      correct_answer:  parsed.correct_answer,
-      score:           parsed.score,
-      correct:         parsed.correct,
-      strictness_used: strictness,
-      vocab_used:      vocab_used ?? [],
-    }).then(({ error }) => {
-      if (error) console.error('sentence_attempts insert error:', error.message)
-    })
-
-    waitUntil(Promise.allSettled([recordAttempts, insertAttempt]))
-
-    return NextResponse.json(parsed)
-  } catch (err) {
-    console.error('Grade error:', err)
-    const status = err instanceof ClaudeResponseError ? 502 : 500
-    return NextResponse.json({ error: 'Grading failed' }, { status })
+    try {
+      parsed = await callClaudeJson(prompt, 200, isGradeResponse, anthropicClient)
+    } catch (err) {
+      console.error('Grade error:', err)
+      const status = err instanceof ClaudeResponseError ? 502 : 500
+      return NextResponse.json({ error: 'Grading failed' }, { status })
+    }
   }
+
+  parsed.correct = parsed.score >= 70
+
+  // Tracking writes must never block or fail the grade response, but they also
+  // can't be truly fire-and-forget — Vercel freezes the function once the
+  // response is sent, which was silently dropping these. waitUntil keeps the
+  // invocation alive until they finish without making the client wait for them.
+  const recordAttempts = vocab_used?.length
+    ? Promise.all(vocab_used.map((zh: string) =>
+        supabase.rpc('record_word_attempt', {
+          p_word_zh: zh,
+          p_correct: parsed.correct,
+        })
+      )).catch(err => console.error('record_word_attempt error:', err))
+    : Promise.resolve()
+
+  const insertAttempt = supabase.from('sentence_attempts').insert({
+    user_id:         user.id,
+    sentence_zh,
+    sentence_py,
+    user_answer:     body.user_answer,
+    correct_answer:  parsed.correct_answer,
+    score:           parsed.score,
+    correct:         parsed.correct,
+    feedback:        parsed.feedback,
+    strictness_used: strictness,
+    vocab_used:      vocab_used ?? [],
+  }).then(({ error }) => {
+    if (error) console.error('sentence_attempts insert error:', error.message)
+  })
+
+  waitUntil(Promise.allSettled([recordAttempts, insertAttempt]))
+
+  return NextResponse.json(parsed)
 }

src/lib/api/ratelimit.ts

@@ -2,7 +2,14 @@ import { Ratelimit } from '@upstash/ratelimit'
 import { Redis } from '@upstash/redis'
 
 const BUDGETS = {
-  generate: { limit: 15, window: '1 m' },
+  // 'ai' mode spends the user's own Anthropic key, so this cap isn't a cost
+  // control — it just stops the route itself from being hammered.
+  generate_ai: { limit: 30, window: '1 m' },
+  // 'static' mode only costs Supabase reads, so it gets the same light cap.
+  generate_static: { limit: 30, window: '1 m' },
+  // Grading always spends the app's own Anthropic key regardless of mode, so
+  // it keeps the tight budget — this is the one endpoint that costs real
+  // money on every call.
   grade: { limit: 15, window: '1 m' },
   speak: { limit: 10, window: '1 m' },
 } as const satisfies Record<string, { limit: number; window: `${number} ${'ms' | 's' | 'm' | 'h' | 'd'}` }>

src/types/index.ts

@@ -42,6 +42,23 @@ export interface Progress {
   updated_at: string
 }
 
+export interface SentenceAttempt {
+  id: string
+  user_id: string
+  round_summary_id: string | null
+  sentence_zh: string
+  sentence_py: string
+  user_answer: string
+  user_answer_normalized: string
+  correct_answer: string
+  feedback: string | null
+  score: number
+  correct: boolean
+  strictness_used: 1 | 2 | 3
+  vocab_used: string[]
+  attempted_at: string
+}
+
 export interface CorpusWord {
   zh: string
   py: string

supabase/migrations/20260819000000_add_grade_cache.sql

@@ -0,0 +1,78 @@
+-- Turns sentence_attempts into a cross-user grading cache for /api/grade.
+-- Static-mode grading always spends the app's own ANTHROPIC_API_KEY, and the
+-- static sentence_bank is shared across all users, so an exact-match cache on
+-- (sentence_zh, strictness_used, normalized user_answer) can skip a real
+-- Anthropic call whenever the same wrong answer has already been graded
+-- before, by any user.
+
+-- feedback was never persisted by /api/grade's insert — required to
+-- reconstruct a complete GradeResponse from a cached row.
+ALTER TABLE public.sentence_attempts ADD COLUMN feedback text;
+
+-- Normalization lives in exactly one place (SQL), reused by the generated
+-- column below and by the lookup RPC, so application code never duplicates
+-- this logic.
+CREATE FUNCTION public.normalize_answer(p text)
+RETURNS text
+LANGUAGE sql
+IMMUTABLE
+PARALLEL SAFE
+AS $$
+  SELECT lower(trim(regexp_replace(p, '\s+', ' ', 'g')))
+$$;
+
+-- Indexable, always-consistent normalized form of user_answer.
+ALTER TABLE public.sentence_attempts
+  ADD COLUMN user_answer_normalized text
+  GENERATED ALWAYS AS (public.normalize_answer(user_answer)) STORED;
+
+-- Cross-user cache lookup. SECURITY DEFINER to read across all users' rows
+-- for this one narrow, controlled purpose — bypasses the
+-- sentence_attempts_select RLS policy (auth.uid() = user_id) deliberately.
+-- Returns only derived grading fields, never user_id or another user's raw
+-- answer text (the caller already knows their own answer — they just
+-- submitted it in this same request).
+CREATE FUNCTION public.find_cached_grade(
+  p_sentence_zh text,
+  p_strictness   smallint,
+  p_user_answer  text
+)
+RETURNS TABLE(score smallint, correct_answer text, feedback text)
+LANGUAGE sql
+SECURITY DEFINER
+STABLE
+AS $$
+  SELECT score, correct_answer, feedback
+  FROM public.sentence_attempts
+  WHERE sentence_zh = p_sentence_zh
+    AND strictness_used = p_strictness
+    AND user_answer_normalized = public.normalize_answer(p_user_answer)
+    AND feedback IS NOT NULL
+  ORDER BY attempted_at DESC
+  LIMIT 1
+$$;
+
+-- Covering partial index matching the RPC's predicates exactly, so the
+-- lookup is index-only (score/correct_answer/feedback via INCLUDE) and the
+-- ORDER BY ... LIMIT 1 needs no separate sort. Partial on feedback IS NOT
+-- NULL keeps the index small (excludes pre-migration rows) and matches the
+-- RPC's own filter so the planner will actually pick it.
+CREATE INDEX sentence_attempts_cache_lookup_idx
+  ON public.sentence_attempts (sentence_zh, strictness_used, user_answer_normalized, attempted_at DESC)
+  INCLUDE (score, correct_answer, feedback)
+  WHERE feedback IS NOT NULL;
+
+-- This project has `ALTER DEFAULT PRIVILEGES FOR ROLE postgres IN SCHEMA
+-- public GRANT ALL ON FUNCTIONS TO anon` in effect (see
+-- 20260806171015_drop_p_user_id_use_auth_uid.sql), so every new CREATE
+-- FUNCTION is auto-granted to anon unless explicitly revoked — every
+-- existing RPC in this project follows with this same REVOKE/GRANT block.
+REVOKE ALL ON FUNCTION public.normalize_answer(text) FROM PUBLIC;
+REVOKE ALL ON FUNCTION public.normalize_answer(text) FROM anon;
+GRANT EXECUTE ON FUNCTION public.normalize_answer(text) TO authenticated;
+GRANT EXECUTE ON FUNCTION public.normalize_answer(text) TO service_role;
+
+REVOKE ALL ON FUNCTION public.find_cached_grade(text, smallint, text) FROM PUBLIC;
+REVOKE ALL ON FUNCTION public.find_cached_grade(text, smallint, text) FROM anon;
+GRANT EXECUTE ON FUNCTION public.find_cached_grade(text, smallint, text) TO authenticated;
+GRANT EXECUTE ON FUNCTION public.find_cached_grade(text, smallint, text) TO service_role;