diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 0000000..0bb8ac9 --- /dev/null +++ b/PLAN.md @@ -0,0 +1,121 @@ +# Future: semantic (embeddings) matching for the grade cache + +Status: **not started**. Written 2026-08-19 as a placeholder so this isn't forgotten once +there's enough real usage to justify it. + +## Context + +`/api/grade` already has an exact-match cache (`supabase/migrations/20260819000000_add_grade_cache.sql`, +`find_cached_grade` RPC, wired into `src/app/api/grade/route.ts`). It matches on +`(sentence_zh, strictness_used, normalize_answer(user_answer))`, where `normalize_answer` just +lowercases/trims/collapses whitespace — a cache hit only ever fires for text that's byte-identical +after that normalization, so it has **zero risk of serving a wrong grade**. It misses true +paraphrases ("I want to buy this book" vs "I'd like to buy this book"). + +We discussed extending this to fuzzy/character-similarity matching (edit distance, trigram +similarity, a "70% similar" threshold) and **rejected it**: string-similarity metrics can't tell a +paraphrase from a meaning-flipping edit. "I don't like apples" vs "I do like apples" score as +highly similar by edit distance despite being opposite in meaning — the same mechanism that would +catch "buy" ≈ "purchase" also catches "don't" ≈ "do", so there's no threshold that gets the gain +without the risk. Not worth the correctness regression for a grading feature. + +Embeddings are the sound version of the same idea: they're trained to place semantically different +sentences far apart even when textually close (so negation pairs score *less* similar, not more), +so cosine similarity on embeddings is a meaningfully better signal for "does this mean the same +thing" than any character-level metric. Still not perfect — don't treat it as risk-free. + +Before ever serving a semantic-match cache hit to a real user: build a small labeled evaluation set +spanning the failure modes that actually matter here — negation ("I like it" / "I don't like it"), +entity substitution ("I bought an apple" / "I bought a pear"), scope/quantifier differences ("I want +all of them" / "I want some of them"), and genuine valid paraphrases (the true positives this +feature exists to catch) — and run the matcher in shadow mode first: compute what it *would* have +returned against real production traffic, log it, but keep serving the real Anthropic grade +regardless, so false positives can be measured against ground truth before anything is ever actually +served from cache. Pick a concrete false-positive-rate threshold below which this is worth shipping, +and a rollback trigger if live false positives exceed it after launch — don't ship on vibes just +because the offline number looked reasonable. + +## Step 0 — measure first, before building anything + +Do **not** start this on a total-row-count trigger — total rows spread across ~1,050 seeded +sentences × 3 strictness levels tells you almost nothing. What matters is concentration: how many +attempts land on the *same* `(sentence_zh, strictness_used)` bucket, since that's the granularity +the cache matches on. `selectStaticSentence` (`src/lib/staticSentences.ts`) narrows to the top-5 +lowest-accuracy candidates among a user's *unlocked* vocab, and early on that vocab is small (20 +words at signup), so traffic concentrates onto a small set of easy/common sentences before vocab +has grown out — expect a Zipfian pileup on a handful of sentences well before the corpus as a +whole has meaningful volume. + +Run this. `sentence_attempts` has no `practice_mode` column, so this can't filter on mode directly +— joining against `sentence_bank.sentence_zh` approximates "static-sourced" without a schema change +(imperfect: a coincidentally-identical AI-generated sentence would also match, but that's rare and +this is a one-off diagnostic, not something that needs to be exact). Also restricted to +`feedback IS NOT NULL`, matching `find_cached_grade`'s actual eligibility — pre-migration rows +without feedback were never real cache candidates and would otherwise inflate the counts: + +```sql +select sa.sentence_zh, sa.strictness_used, count(*) as attempts, + count(distinct sa.user_answer_normalized) as distinct_answers +from sentence_attempts sa +where sa.feedback is not null + and sa.sentence_zh in (select sentence_zh from sentence_bank) +group by sa.sentence_zh, sa.strictness_used +order by attempts desc +limit 30; +``` + +This is a repetition metric, not a directly-measured hit rate — no hit/miss instrumentation exists +in `grade/route.ts` today. `(attempts - distinct_answers) / attempts` on the top rows is a closer +proxy for what the exact-match cache's actual hit rate would look like (bounded 0–1: the fraction of +attempts that repeat an already-seen normalized answer for that bucket), but treat it as an +approximation, not a measured number. Eyeball the `distinct_answers` for a top bucket — if most of +them look like genuine near-duplicates a human would call "the same answer" (not just typos, which +exact-match already catches), that's the signal semantic matching would help. As an +order-of-magnitude starting point: check in once you're around a few hundred total static-mode grade +attempts — that's roughly when the top few sentences should have accumulated double digits of repeat +attempts, enough to read something from the ratio instead of noise. If the top buckets' +near-duplicate rate is low, skip this feature — it isn't worth the added cost/complexity/risk. + +## If the data justifies building it + +1. **Enable `pgvector`** on the Supabase project (`CREATE EXTENSION IF NOT EXISTS vector;`). +2. **Add an embedding column** to `sentence_attempts`, e.g. `user_answer_embedding vector(N)` + (dimension depends on the embedding model chosen), plus a `user_answer_embedding_model text` + column recording which embedding model/version produced it — both populated together at insert + time in `src/app/api/grade/route.ts` alongside the existing `insertAttempt` write. Vectors from + different models/versions aren't comparable; this column is what lets a future model swap + coexist with old data instead of silently corrupting similarity results. +3. **Only call the embedding API on an exact-match miss** — keep `find_cached_grade`'s existing + exact-match lookup as the free first-pass fast path, unchanged. Only generate an embedding for + the incoming answer (cheap/fast relative to a grading call, but still a real added cost+latency) + when that lookup returns nothing. Never spend an embedding call on a request the exact-match + cache already resolved. +4. **Add a second, semantic-match RPC that runs only after an exact-match miss**: filter on + `sentence_zh = ... AND strictness_used = ... AND feedback IS NOT NULL AND + user_answer_embedding_model = ` (those + must still match exactly, the eligibility guard stays the same as `find_cached_grade` — only a + row with a complete grading result can ever be replayed — and the model-version guard keeps + comparisons confined to a single embedding space; only the answer comparison becomes fuzzy), + then order by cosine distance to the new answer's embedding (`<=>` operator with an `ivfflat` or + `hnsw` index) and only accept a match above a conservative threshold. Return the same + derived-field-only projection as `find_cached_grade` (`score, correct_answer, feedback` — never + `user_id` or another user's raw answer text). Start the threshold high (few false positives) and + only loosen it based on observed accuracy — don't guess a number up front. Only fall through to + the real Anthropic grading call if this also misses. +5. **No backfill needed for old rows** — same pattern as `feedback`: only rows with an embedding + populated *and* matching the current embedding model version are eligible matches. Historical + rows, and rows from a prior model version after any future model swap, just never match until + re-graded (fine) — existing populated vectors are never deleted or rewritten, just naturally + excluded from matching a different model's queries. +6. **Mirror the existing `find_cached_grade` SECURITY DEFINER pattern** (see + `supabase/migrations/20260806171015_drop_p_user_id_use_auth_uid.sql` for the REVOKE/GRANT + convention this project requires on every new function) rather than inventing a new access + pattern. + +## Open questions to resolve when this is picked up + +- Which embedding provider/model, and what's the actual per-call cost at expected volume — needs + to stay well below the cost of the Anthropic grading call it's trying to avoid, or the whole + feature is pointless. +- What similarity threshold actually holds up — needs eyeballing real (sentence, candidate-match) + pairs near the cutoff, not a number picked in the abstract. diff --git a/src/app/api/generate/route.ts b/src/app/api/generate/route.ts index 3612688..137c587 100644 --- a/src/app/api/generate/route.ts +++ b/src/app/api/generate/route.ts @@ -29,7 +29,30 @@ export async function POST(req: NextRequest) { if (auth instanceof NextResponse) return auth const { user, supabase } = auth - const rateLimit = await checkRateLimit(user.id, 'generate') + // Cheap gate before the settings lookup below, so a client hammering this + // route can't force an unbounded number of Supabase reads while we don't + // yet know its mode (and therefore which real budget applies). + const preflight = await checkRateLimit(user.id, 'generate_preflight') + if (!preflight.success) { + return NextResponse.json( + { error: 'Too many requests' }, + { status: 429, headers: { 'Retry-After': String(preflight.retryAfterSeconds) } } + ) + } + + // practice_mode is read before the real rate-limit check so the budget can + // be chosen per mode — 'ai' spends the user's own key (no budget + // protection needed, just a light anti-hammering cap) while 'static' only + // costs Supabase reads (same light cap, for a different reason). + const { data: userSettings } = await supabase + .from('settings') + .select('practice_mode') + .eq('user_id', user.id) + .single() + + const practiceMode = userSettings?.practice_mode ?? 'static' + + const rateLimit = await checkRateLimit(user.id, practiceMode === 'ai' ? 'generate_ai' : 'generate_static') if (!rateLimit.success) { return NextResponse.json( { error: 'Too many requests' }, @@ -47,14 +70,6 @@ export async function POST(req: NextRequest) { py: r.py.slice(0, MAX_RECENT_FIELD_LENGTH), })) - const { data: userSettings } = await supabase - .from('settings') - .select('practice_mode') - .eq('user_id', user.id) - .single() - - const practiceMode = userSettings?.practice_mode ?? 'static' - if (practiceMode === 'static') { // The client's variety-tracking sends recently-served {zh, py} pairs, not // sentence_bank ids — resolve them to ids here so selectStaticSentence can diff --git a/src/app/api/grade/route.ts b/src/app/api/grade/route.ts index ff51f85..ba917d6 100644 --- a/src/app/api/grade/route.ts +++ b/src/app/api/grade/route.ts @@ -122,7 +122,36 @@ export async function POST(req: NextRequest) { const truncatedAnswer = user_answer.slice(0, 500) - const prompt = `You are grading a Chinese-to-English translation exercise. + // Cross-user cache: an identical (sentence, strictness, normalized answer) + // triple may already have been graded by someone else, especially likely + // in static mode where a shared, finite sentence_bank means the same wrong + // answers recur across users. A hit skips the Anthropic call entirely — a + // lookup failure just falls through to a normal miss, never aborts the + // request. + // Project has no generated Supabase Database types, so .rpc() results are + // untyped — narrow to the shape find_cached_grade actually returns. + const { data: cached, error: cacheError } = await supabase + .rpc('find_cached_grade', { + p_sentence_zh: sentence_zh, + p_sentence_py: sentence_py, + p_strictness: strictness, + p_user_answer: truncatedAnswer, + }) + .maybeSingle() as { data: { score: number; correct_answer: string; feedback: string } | null; error: { message: string } | null } + + if (cacheError) console.error('find_cached_grade error:', cacheError.message) + + let parsed: GradeResponse + + if (cached) { + parsed = { + correct: false, // recomputed below from score, never trusted from a cached row + score: cached.score, + feedback: cached.feedback, + correct_answer: cached.correct_answer, + } + } else { + const prompt = `You are grading a Chinese-to-English translation exercise. Chinese sentence: ${sentence_zh} Pinyin: ${sentence_py} @@ -138,43 +167,46 @@ Grading mode: ${STRICTNESS[strictness] ?? STRICTNESS[2]} Respond with ONLY valid JSON, no markdown: {"correct":true or false,"score":0-100,"feedback":"one concise sentence","correct_answer":"the most natural English translation"}` - try { - const parsed = await callClaudeJson(prompt, 200, isGradeResponse, anthropicClient) - parsed.correct = parsed.score >= 70 - - // Tracking writes must never block or fail the grade response, but they also - // can't be truly fire-and-forget — Vercel freezes the function once the - // response is sent, which was silently dropping these. waitUntil keeps the - // invocation alive until they finish without making the client wait for them. - const recordAttempts = vocab_used?.length - ? Promise.all(vocab_used.map((zh: string) => - supabase.rpc('record_word_attempt', { - p_word_zh: zh, - p_correct: parsed.correct, - }) - )).catch(err => console.error('record_word_attempt error:', err)) - : Promise.resolve() - - const insertAttempt = supabase.from('sentence_attempts').insert({ - user_id: user.id, - sentence_zh, - sentence_py, - user_answer: body.user_answer, - correct_answer: parsed.correct_answer, - score: parsed.score, - correct: parsed.correct, - strictness_used: strictness, - vocab_used: vocab_used ?? [], - }).then(({ error }) => { - if (error) console.error('sentence_attempts insert error:', error.message) - }) - - waitUntil(Promise.allSettled([recordAttempts, insertAttempt])) - - return NextResponse.json(parsed) - } catch (err) { - console.error('Grade error:', err) - const status = err instanceof ClaudeResponseError ? 502 : 500 - return NextResponse.json({ error: 'Grading failed' }, { status }) + try { + parsed = await callClaudeJson(prompt, 200, isGradeResponse, anthropicClient) + } catch (err) { + console.error('Grade error:', err) + const status = err instanceof ClaudeResponseError ? 502 : 500 + return NextResponse.json({ error: 'Grading failed' }, { status }) + } } + + parsed.correct = parsed.score >= 70 + + // Tracking writes must never block or fail the grade response, but they also + // can't be truly fire-and-forget — Vercel freezes the function once the + // response is sent, which was silently dropping these. waitUntil keeps the + // invocation alive until they finish without making the client wait for them. + const recordAttempts = vocab_used?.length + ? Promise.all(vocab_used.map((zh: string) => + supabase.rpc('record_word_attempt', { + p_word_zh: zh, + p_correct: parsed.correct, + }) + )).catch(err => console.error('record_word_attempt error:', err)) + : Promise.resolve() + + const insertAttempt = supabase.from('sentence_attempts').insert({ + user_id: user.id, + sentence_zh, + sentence_py, + user_answer: body.user_answer, + correct_answer: parsed.correct_answer, + score: parsed.score, + correct: parsed.correct, + feedback: parsed.feedback, + strictness_used: strictness, + vocab_used: vocab_used ?? [], + }).then(({ error }) => { + if (error) console.error('sentence_attempts insert error:', error.message) + }) + + waitUntil(Promise.allSettled([recordAttempts, insertAttempt])) + + return NextResponse.json(parsed) } diff --git a/src/lib/api/ratelimit.ts b/src/lib/api/ratelimit.ts index 2df1d34..5fd4ccf 100644 --- a/src/lib/api/ratelimit.ts +++ b/src/lib/api/ratelimit.ts @@ -2,7 +2,19 @@ import { Ratelimit } from '@upstash/ratelimit' import { Redis } from '@upstash/redis' const BUDGETS = { - generate: { limit: 15, window: '1 m' }, + // Cheap first-pass gate for /api/generate, checked before the settings + // lookup that determines mode — protects that Supabase read itself from + // being hammered by a client the real per-mode budgets below haven't even + // classified yet. Not a replacement for them, just a pre-filter. + generate_preflight: { limit: 30, window: '1 m' }, + // 'ai' mode spends the user's own Anthropic key, so this cap isn't a cost + // control — it just stops the route itself from being hammered. + generate_ai: { limit: 30, window: '1 m' }, + // 'static' mode only costs Supabase reads, so it gets the same light cap. + generate_static: { limit: 30, window: '1 m' }, + // Grading always spends the app's own Anthropic key regardless of mode, so + // it keeps the tight budget — this is the one endpoint that costs real + // money on every call. grade: { limit: 15, window: '1 m' }, speak: { limit: 10, window: '1 m' }, } as const satisfies Record diff --git a/src/types/index.ts b/src/types/index.ts index 58f3551..74c004a 100644 --- a/src/types/index.ts +++ b/src/types/index.ts @@ -42,6 +42,23 @@ export interface Progress { updated_at: string } +export interface SentenceAttempt { + id: string + user_id: string + round_summary_id: string | null + sentence_zh: string + sentence_py: string + user_answer: string + user_answer_normalized: string + correct_answer: string + feedback: string | null + score: number + correct: boolean + strictness_used: 1 | 2 | 3 + vocab_used: string[] + attempted_at: string +} + export interface CorpusWord { zh: string py: string diff --git a/supabase/migrations/20260819000000_add_grade_cache.sql b/supabase/migrations/20260819000000_add_grade_cache.sql new file mode 100644 index 0000000..0df081b --- /dev/null +++ b/supabase/migrations/20260819000000_add_grade_cache.sql @@ -0,0 +1,78 @@ +-- Turns sentence_attempts into a cross-user grading cache for /api/grade. +-- Static-mode grading always spends the app's own ANTHROPIC_API_KEY, and the +-- static sentence_bank is shared across all users, so an exact-match cache on +-- (sentence_zh, strictness_used, normalized user_answer) can skip a real +-- Anthropic call whenever the same wrong answer has already been graded +-- before, by any user. + +-- feedback was never persisted by /api/grade's insert — required to +-- reconstruct a complete GradeResponse from a cached row. +ALTER TABLE public.sentence_attempts ADD COLUMN feedback text; + +-- Normalization lives in exactly one place (SQL), reused by the generated +-- column below and by the lookup RPC, so application code never duplicates +-- this logic. +CREATE FUNCTION public.normalize_answer(p text) +RETURNS text +LANGUAGE sql +IMMUTABLE +PARALLEL SAFE +AS $$ + SELECT lower(trim(regexp_replace(p, '\s+', ' ', 'g'))) +$$; + +-- Indexable, always-consistent normalized form of user_answer. +ALTER TABLE public.sentence_attempts + ADD COLUMN user_answer_normalized text + GENERATED ALWAYS AS (public.normalize_answer(user_answer)) STORED; + +-- Cross-user cache lookup. SECURITY DEFINER to read across all users' rows +-- for this one narrow, controlled purpose — bypasses the +-- sentence_attempts_select RLS policy (auth.uid() = user_id) deliberately. +-- Returns only derived grading fields, never user_id or another user's raw +-- answer text (the caller already knows their own answer — they just +-- submitted it in this same request). +CREATE FUNCTION public.find_cached_grade( + p_sentence_zh text, + p_strictness smallint, + p_user_answer text +) +RETURNS TABLE(score smallint, correct_answer text, feedback text) +LANGUAGE sql +SECURITY DEFINER +STABLE +AS $$ + SELECT score, correct_answer, feedback + FROM public.sentence_attempts + WHERE sentence_zh = p_sentence_zh + AND strictness_used = p_strictness + AND user_answer_normalized = public.normalize_answer(p_user_answer) + AND feedback IS NOT NULL + ORDER BY attempted_at DESC + LIMIT 1 +$$; + +-- Covering partial index matching the RPC's predicates exactly, so the +-- lookup is index-only (score/correct_answer/feedback via INCLUDE) and the +-- ORDER BY ... LIMIT 1 needs no separate sort. Partial on feedback IS NOT +-- NULL keeps the index small (excludes pre-migration rows) and matches the +-- RPC's own filter so the planner will actually pick it. +CREATE INDEX sentence_attempts_cache_lookup_idx + ON public.sentence_attempts (sentence_zh, strictness_used, user_answer_normalized, attempted_at DESC) + INCLUDE (score, correct_answer, feedback) + WHERE feedback IS NOT NULL; + +-- This project has `ALTER DEFAULT PRIVILEGES FOR ROLE postgres IN SCHEMA +-- public GRANT ALL ON FUNCTIONS TO anon` in effect (see +-- 20260806171015_drop_p_user_id_use_auth_uid.sql), so every new CREATE +-- FUNCTION is auto-granted to anon unless explicitly revoked — every +-- existing RPC in this project follows with this same REVOKE/GRANT block. +REVOKE ALL ON FUNCTION public.normalize_answer(text) FROM PUBLIC; +REVOKE ALL ON FUNCTION public.normalize_answer(text) FROM anon; +GRANT EXECUTE ON FUNCTION public.normalize_answer(text) TO authenticated; +GRANT EXECUTE ON FUNCTION public.normalize_answer(text) TO service_role; + +REVOKE ALL ON FUNCTION public.find_cached_grade(text, smallint, text) FROM PUBLIC; +REVOKE ALL ON FUNCTION public.find_cached_grade(text, smallint, text) FROM anon; +GRANT EXECUTE ON FUNCTION public.find_cached_grade(text, smallint, text) TO authenticated; +GRANT EXECUTE ON FUNCTION public.find_cached_grade(text, smallint, text) TO service_role; diff --git a/supabase/migrations/20260819010000_grade_cache_match_pinyin.sql b/supabase/migrations/20260819010000_grade_cache_match_pinyin.sql new file mode 100644 index 0000000..05184b2 --- /dev/null +++ b/supabase/migrations/20260819010000_grade_cache_match_pinyin.sql @@ -0,0 +1,47 @@ +-- find_cached_grade previously matched only on (sentence_zh, strictness_used, +-- normalized answer). Chinese has genuine heteronyms (多音字) — the same +-- character string can carry more than one reading/meaning (e.g. 还 hái +-- "still" vs huán "return") — and sentence_py is shown directly to the user +-- when show_pinyin is 'always'/'tap', so it can carry the actual intended +-- reading for a given sentence_zh. Widening the match key to require +-- sentence_py too closes the (narrow, but real) risk of reusing a grade +-- computed for a different intended reading of the same characters. + +DROP FUNCTION IF EXISTS public.find_cached_grade(text, smallint, text); + +CREATE FUNCTION public.find_cached_grade( + p_sentence_zh text, + p_sentence_py text, + p_strictness smallint, + p_user_answer text +) +RETURNS TABLE(score smallint, correct_answer text, feedback text) +LANGUAGE sql +SECURITY DEFINER +STABLE +AS $$ + SELECT score, correct_answer, feedback + FROM public.sentence_attempts + WHERE sentence_zh = p_sentence_zh + AND sentence_py = p_sentence_py + AND strictness_used = p_strictness + AND user_answer_normalized = public.normalize_answer(p_user_answer) + AND feedback IS NOT NULL + ORDER BY attempted_at DESC + LIMIT 1 +$$; + +-- Same default-privileges auto-grant-to-anon concern as the original +-- function — DROP removed the old grants along with the old signature, so +-- the new one needs this block again. +REVOKE ALL ON FUNCTION public.find_cached_grade(text, text, smallint, text) FROM PUBLIC; +REVOKE ALL ON FUNCTION public.find_cached_grade(text, text, smallint, text) FROM anon; +GRANT EXECUTE ON FUNCTION public.find_cached_grade(text, text, smallint, text) TO authenticated; +GRANT EXECUTE ON FUNCTION public.find_cached_grade(text, text, smallint, text) TO service_role; + +DROP INDEX IF EXISTS public.sentence_attempts_cache_lookup_idx; + +CREATE INDEX sentence_attempts_cache_lookup_idx + ON public.sentence_attempts (sentence_zh, sentence_py, strictness_used, user_answer_normalized, attempted_at DESC) + INCLUDE (score, correct_answer, feedback) + WHERE feedback IS NOT NULL;