Math KG — System architecture

TL;DR

Three write-path pipelines converge on a shared Cloudflare D1 database and R2 blob store, and one portal reads it all.

1. Content extraction pulls every published lesson from school-api, produces pedagogical briefs at three altitudes (asset · lesson · chapter), and writes them to R2 + D1.
2. LO authoring creates Learning Objectives per (curriculum, thread, grade) from framework sources (NCERT for CBSE, ATP for Merdeka) via the /author-los skill, with server-side validation and audit.
3. Tagging maps each extracted asset to zero-or-more LOs via /tag-grade: a matcher pass with per-chapter fanout, then adversarial verification, then gap detection.

A SME review layer sits on top of tagging — humans approve/edit/reject via the KG MCP; the portal shows the overlay live. The compute is many parallel Claude Code sessions (not a hosted service); state is fully in the cloud so any session can be rebooted from any login without losing progress. Everything is served through a static + live-overlay portal at kg.classrootsedu.in. A parallel LO knowledge graph lives in Neo4j Aura for graph queries.

System diagram

Three pipelines write to shared D1 + R2 state (color-coded arrows: blue = extraction, purple = LO authoring / SME review, orange = tagging). Dashed arrows = live/manual (Neo4j is loaded manually from D1 for graph queries; portal live overlay reads D1 on every load). Dashed box = SME review layer that overlays on top of AI tagging.

State stores

Cloudflare R2 — bucket math-kg-state

Account 71519131ff2aeeda400f7e429873fe06 (Vinay). Blobs only — no query API. Access via wrangler r2 object.

Cloudflare D1 — database cbse-kg-portal-reviews

SQLite at the edge. Database id 59d0a9ac-3370-4882-aa82-8c08b6619a75. All mutation via wrangler d1 execute or via the Pages Functions API.

Neo4j Aura

Managed graph database. Currently loaded: Merdeka fractions v1 tags (562 TAGGED_TO edges over 290 Asset nodes and 46 LO nodes). v3 tag reload pending. CBSE side: 67 LO nodes exist, no Asset nodes yet (school-api has no CBSE content).

Query surface: Neo4j Aura Browser (creds in prior scratchpad neo4j.env) OR via the kg-mcp Python MCP server (19 tools including list_los, create_lo, bulk_create_tags, list_gaps).

Portal static assets

Files at curriculum/cbse/fractions/portal/data/<curriculum>/<thread>/. Deployed to Cloudflare Pages on every wrangler pages deploy.

Content extraction pipeline

Per-lesson flow, orchestrated by /extract-batch at the chapter or grade level:

1. Discovery. /extract-batch queries D1 for pending extraction_claims in scope.
2. Atomic claim. Optimistic UPDATE flips rows to status='claimed' with a 2-hour timeout. Losers of the race skip.
3. Parallel fanout. Workflow spawns one sub-agent per claimed lesson (auto-capped at ~10 concurrent). Each sub-agent invokes /extract-lesson --as-subagent.
4. Per-lesson work (each sub-agent):
   • mcp__school-api__slide_list for the lesson's slide manifest.
   • For each video slide: download mp4 → Whisper transcribe → upload transcript to raw/whisper/ → compose video brief (arc as array of phase objects, transcript_ref pointer).
   • For each simulation slide: fetch data.js → parse into content buckets → zstd-upload raw to raw/sim_datajs/ → compose sim brief.
   • For each image slide: vision-LLM → single-sentence pedagogical_intent → delete image bytes. Only the intent line persists.
   • Compose slides brief and lesson brief bottom-up (lesson brief needs all children as input).
   • Self-validate against the reference schema (13 lesson keys with minimum char/count thresholds; 5 identifier fields per asset brief; arc as array). Return {lesson_id, validated, gaps, r2_keys_written, brief_url}.
5. Central commit. The /extract-batch parent walks the returned results. validated:true → D1 status='done' + write brief_index rows. validated:false → reset D1 to status='pending' with the gap list in error_summary. Sub-agent crash → same reset with gap='sub-agent-crashed'.
6. Portal refresh. Post-batch, run build_skeletons.py to pull the newly-done briefs from R2 into assets_index.json, then wrangler pages deploy. Live D1 status shows up in the portal without redeploy via /api/extraction-status.

LO authoring pipeline

Learning Objectives are the atomic units of the knowledge graph — one canonical statement per skill a learner acquires at a given grade. LOs are authored by Curriculum SMEs (currently Vinay, Arjun, Mani) using the /author-los skill against a framework source. LOs are NOT auto-derived from extracted content; they exist independently, and are what content later gets tagged to.

Trigger + inputs

• Trigger: Skill(skill='author-los', args='<curriculum> <thread> <grade>'). Runs one SME's session end-to-end for one (curriculum, thread, grade).
• Framework source: authoritative curriculum text.
  • CBSE: NCERT Ganita Prakash + Mathematics textbooks (local corpus at ~/Desktop/Byjus AI/TextBooks/).
  • Merdeka: ATP (Alur Tujuan Pembelajaran) PDFs for each grade from Penerbit Erlangga's ESPS series, plus DoE 2022 Kemendikbudristek textbooks. Shared Drive folder is the canonical location.
• Rule book: three Notion pages under 📘 Math KG Rule Books (parent id 3964e013-82d2-8146-9619-d4c4b7292281) are the source of truth: LO Creation Rules, LO Authoring Prompt, Tagging & Verification Rules. Skills fetch these at run start via mcp__notion__notion-fetch; local curriculum/cbse/LO_CREATION_RULES.md is a stale fallback.

Flow

1. SME opens a fresh Claude Code session, runs /join to bootstrap identity + MCP setup, then /author-los.
2. Skill loads current LO catalogue for the thread (via mcp__kg__list_los) so no duplicates are authored.
3. Skill loads the framework PDFs + prior-grade LO catalogue as context, invokes the LO Authoring Prompt from Notion.
4. For each LO the model produces: canonical statement, bloom_verb, bloom_level, conceptual_object, primary_thread, prerequisite edges, cross-thread prereqs, source citations (framework + clause_ref), misconception references.
5. Each LO write goes through mcp__kg__create_lo → server-side validation (banned verbs, canonical uniqueness, grade order for edges, orphan detection) → D1 los + los_changes (audit).
6. Edges go through mcp__kg__create_edge and mcp__kg__create_cross_thread_prereq. Cross-thread edges support PLACEHOLDER targets for un-authored target threads.
7. Skill outputs companion artifacts per grade to disk: scope_g<N>.md, coverage_matrix_g<N>.md, appends to authoring_questions.md. These are for human review, not portal consumption.

Server-side validation (what rejects bad LOs)

Bad LOs bounce at the API with 422 <rule citation> — never reach Neo4j.

Current state

Tagging pipeline

Tagging maps extracted assets (sims / videos / lessons / chapters) to authored LOs. Runs after both content extraction and LO authoring are done for a grade. Three stages, each with different fanout characteristics.

Trigger + inputs

• Trigger: Skill(skill='tag-grade', args='<curriculum> <thread> <grade>').
• Inputs: extracted deep corpus (from corpus/deep/<chapter_id>.json in R2 or portal static), authored LOs for the grade + prior grades (from D1 los).
• Rule book: Notion "Tagging & Verification Rules" page, refetched at run start.

Stage A · Matcher (per-chapter Workflow fanout)

• Fanout: chapters with ≤20 assets run inline; chapters >20 assets fan out one sub-agent per chapter via Workflow.
• Per asset, the LLM proposes 0-1 primary LO tag with confidence + evidence + rationale.
• Governed by v3 rules R1-R7 from the Notion rule book:
  1. R1 single-primary: each asset gets at most one primary LO tag (no primary/secondary/weak fanout).
  2. R2 last-taught-wins: when an asset teaches a concept spanning multiple grades, tag to the highest grade LO in scope.
  3. R3 story-video-fallback: pure story-hook videos with no math vocabulary tag to the closest concept LO in the lesson's arc, not null.
  4. R4 null-if-no-fit: rather than force-tag, leave untagged and let gap detection surface it.
  5. R5 bloom-verb-match: the asset's demonstrated action must match the LO's bloom_verb.
  6. R6 cross-grade: an asset in grade N may tag to a grade M < N LO if the concept is grade-M level.
  7. R7 concept-over-method: when a sim demonstrates concept X via method Y, tag to concept X's LO, not the method LO.
• Output shape per tag: {tag_id, asset_id, lo_id, confidence, evidence, rationale, applied_rule, tag_origin='ai_matched'}.

Stage B · Adversarial verification

• For each verify-band tag (confidence in the middle range), spawn 3 skeptic sub-agents in parallel with the prompt "try to refute this tag."
• Each skeptic returns {refuted: bool, reason}.
• If ≥ 2 of 3 refute, the tag is downgraded to review_status='flagged'.
• Prevents plausible-but-wrong tags from silently entering the KG.

Stage C · Gap detection

• For every LO in the target grade, count non-flagged tags.
• 0 non-flagged tags → LO is a gap. Categorize the reason: no_tag (no asset teaches it), all_flagged (all tags refuted in Stage B), ambiguous (multiple candidates, no clear primary).
• Gap list is written to disk (gaps_g<N>.json) and can be queried via GET /api/gaps?curriculum=&thread=&grade=.

Commit

All tags land in D1 ai_tags via POST /api/tags/bulk (batched, per-row validated, cap 5000/call). The commit is atomic per batch. Every tag carries a batch_id for provenance.

Current state

SME review workflow

AI tagging is not final. Every tag lands as review_status='pending' (or 'flagged' if Stage B refuted it); a Tagging Reviewer walks the queue and applies one of four actions.

Portal is read-only. Writes go through KG MCP.

This is a deliberate architectural choice — as of 2026-07-07 the portal at kg.classrootsedu.in has NO interactive tag-modification UI. It shows current tag state (raw AI tags + review overlays) as read-only badges. Reviewers write via the kg-mcp Python MCP server exposing four tag-review tools:

Overlay model

The portal computes displayed tags at load time by overlaying tag_reviews onto ai_tags:

• AI tag exists + no review → display as pending.
• AI tag exists + approve review → display as approved, badge = ✓ SME.
• AI tag exists + edit review → display with SME-supplied confidence/rationale, badge = ✎ SME edit.
• AI tag exists + reject review → hidden on the LO card, still queryable in the coverage report's Rejected filter.
• SME-added tag → display with badge = SME added.

Overlay is computed live via GET /api/reviews — no redeploy needed. When an SME runs mcp__kg__approve_tag, the portal reflects the badge on the next page load.

Auth

Every MCP write goes through Authorization: Bearer <sme-token>. Tokens are issued via curriculum/pipeline/issue_sme_token.py <email>, stored hashed at rest in sme_tokens, and audit-logged on every use. Portal-side auth uses Cloudflare Access with a @classrootsedu.com allowlist.

Current state

Portal architecture

Two data planes

The portal reads from two sources on every page load:

• Static: per-thread assets_index.json baked at build time. Contains the chapter tree, lesson metadata, and any successfully-validated brief content. Fast, cache-friendly, browser-parseable in a single fetch.
• Live overlay: /api/extraction-status + /api/ai-tags + /api/reviews. Queried per portal load, D1 truth. When Session A flips a D1 status to done, the pill flips within seconds on next reload — no redeploy needed.

Portal Discipline · No snapshot JSON for mutable state

Rule. If a portal surface renders state that mutates at runtime — tag counts, coverage gaps, extraction status, retirement status, SME review verdicts, freshly-uploaded assets — it reads live via /api/*. It never reads a bundled JSON snapshot for that state, even as a fallback. A snapshot that ships at build time is authoritative only for state that changes at build time (chapter tree structure, lesson metadata, static brief content).

Why. Every "portal disagrees with live data" bug ultimately traces to a bundled snapshot that outlived the state it captured. Two examples caught during 2026-07-09 Round-2 review: retired-edge graph draws (edge status went stale between deploys) and Coverage Report gaps disagreeing with list_gaps (the panel computed against a snapshot that predated the last SME approval sweep). Both would have been avoided by writing the surface as a live-first fetch from day one.

How to apply. When adding a new surface, ask: can this state change between deploys? If yes, put it on a /api/* endpoint that reads D1 or R2 live. Client-side compute over already-fetched live data is acceptable when the computation is deterministic and the inputs are themselves live-sourced (e.g. computing coverage from a live /api/ai-tags pull is fine — the drift risk is on the server side, not in the client formula). Bundled JSON for mutable state is not.

Exception. A static baseline may ship alongside a live endpoint as a first-paint accelerator, but only if the live endpoint overrides it on hit and the code path treats the static as a stale hint, not truth. See assets_index.json + /api/lesson-full for the reference pattern.

Pages Functions API

Auth: Cf-Access-Authenticated-User-Email header (browser path) OR Authorization: Bearer <sme-token> (MCP path). Bearer tokens hashed at rest in sme_tokens.

Filter model

Portal filters cascade in this order: curriculum (Merdeka / CBSE) → thread (5 ATP Elemen, or all) → strand (18 pedagogical sub-groupings, populated per-thread) → grade (G1-G10) → status (done / incomplete / in-progress / not-started / errored). Any filter combination narrows both the chapter list and the lesson list within each expanded chapter.

Knowledge graph

Neo4j Aura holds the LO relationships; the D1 los and edges tables are the write-path (any LO mutation writes to D1 first and is periodically loaded into Aura). Aura is the read-path for graph traversal queries.

Skills catalog

Each skill is a self-contained project-scoped slash command at .claude/commands/<name>.md. Invoke via Skill(skill='<name>', args='...') from any session.

Coordination model

No scheduler, no central orchestrator. Sessions coordinate via optimistic D1 claims. Two rules keep it consistent:

1. Atomic claim. Every /extract-batch or sub-agent claim goes through a single SQL UPDATE ... WHERE status='pending'. The database serializes; the losing session gets 0 rows changed and picks a different lesson.
2. 2-hour timeout. A claimed row auto-expires if not marked done within 2 hours. Any subsequent claim treats expired rows as reclaimable. If a session dies mid-work, work returns to the pool automatically.

Session identity = the email associated with the KG MCP bearer token (issued via curriculum/pipeline/issue_sme_token.py). Every mutation carries the actor email; every D1 write is traceable.

Parallelism cap: Workflow auto-caps at min(16, CPU cores − 2) concurrent sub-agents per session, so passing 47 lessons to parallel() runs them in waves of ~10 without any explicit batching in the skill.

Observability

Every operational question maps to one of these commands:

How many lessons are in each state?

curl -s "https://kg.classrootsedu.in/api/extraction-status?curriculum=merdeka&thread=numbers" \
  | jq '.counts'
# → {pending: 424, claimed: 3, done: 4, errored: 0}

What is Session A currently working on?

wrangler d1 execute cbse-kg-portal-reviews --remote --command="
  SELECT lesson_id, chapter_id, claimed_by_email, claimed_at, claim_expires
    FROM extraction_claims
   WHERE status='claimed'
   ORDER BY claimed_at DESC LIMIT 20;"

What errored + why?

wrangler d1 execute cbse-kg-portal-reviews --remote --command="
  SELECT lesson_id, error_summary
    FROM extraction_claims
   WHERE status='errored'
      OR (status='pending' AND error_summary IS NOT NULL)
   ORDER BY completed_at DESC LIMIT 20;"

What R2 objects exist for a lesson?

# List all briefs written for a lesson
wrangler d1 execute cbse-kg-portal-reviews --remote --command="
  SELECT altitude, r2_url, authored_at, authored_by, content_hash
    FROM brief_index
   WHERE lesson_id='<uuid>'
   ORDER BY authored_at;"

# Pull the actual brief content
wrangler r2 object get math-kg-state/briefs/lesson/<lesson_id>.json --pipe --remote

Live progress rollup by chapter

wrangler d1 execute cbse-kg-portal-reviews --remote --command="
  SELECT chapter_id, grade,
         SUM(status='done')      AS done,
         SUM(status='claimed')   AS claimed,
         SUM(status='pending')   AS pending,
         SUM(status='errored')   AS errored,
         COUNT(*)                AS total
    FROM extraction_claims
   WHERE curriculum='merdeka' AND thread='numbers'
   GROUP BY chapter_id, grade
   ORDER BY grade, chapter_id;"

The portal Asset inspector renders the same data visually — chapter progress counters (0/47 · 3⚠), per-lesson status pills, and clickable schema-gap panels when a brief fails validation.

LO catalogue — what's authored where?

wrangler d1 execute cbse-kg-portal-reviews --remote --command="
  SELECT curriculum, primary_thread AS thread, grade, COUNT(*) AS los
    FROM los
   WHERE status='active'
   GROUP BY curriculum, primary_thread, grade
   ORDER BY curriculum, primary_thread, grade;"

Tag coverage per LO — what's tagged, flagged, or a gap?

curl -s "https://kg.classrootsedu.in/api/gaps?curriculum=merdeka&thread=numbers&grade=G5" \
  | jq '{total_los: (.los | length), gaps: (.gaps | length), reasons: (.gaps | group_by(.reason) | map({(.[0].reason): length}) | add)}'
# → {total_los: 15, gaps: 2, reasons: {no_tag: 1, all_flagged: 1}}

SME review actions — who's approving what?

wrangler d1 execute cbse-kg-portal-reviews --remote --command="
  SELECT reviewer_email, action, COUNT(*) AS n
    FROM tag_reviews
   WHERE curriculum='merdeka'
   GROUP BY reviewer_email, action
   ORDER BY reviewer_email, action;"

LO change audit — full mutation history for one LO

wrangler d1 execute cbse-kg-portal-reviews --remote --command="
  SELECT changed_at, actor_email, before_json, after_json
    FROM los_changes
   WHERE lo_id='<lo-id>'
   ORDER BY changed_at;"

What's not wired yet

• CBSE extraction. school-api has no CBSE content. CBSE side of the portal shows an "extraction pending" banner. Any CBSE flow needs a separate content source (partner API or manual authoring), not a code change.
• Auto Neo4j sync post-tagging. After a tagging run, LO / Asset / TAGGED_TO nodes need to be reloaded into Neo4j Aura. Currently a human step — see docs/superpowers/handoffs/ for prior manual runs.
• Portal auto-deploy per grade completion. When a grade finishes extracting, build_skeletons.py + wrangler pages deploy currently run manually. Should be wired into /rebuild-portal-index as a post-batch hook.
• Live D1-driven lesson brief rendering. Currently briefs are baked into assets_index.json at build time. Live status flips on load, but brief content still requires a redeploy. Making lesson-brief content itself live would remove the last redeploy dependency.
• Slack / external progress broadcast. Explicitly disabled — sessions report in-session only. If team-scale visibility is needed later, an /api/status-summary polled by a small notifier bot would be the cleanest add.
• Character registry / speaker diarization for videos. Currently every character name is extracted per-video. A registry would let cross-video reuse become explicit.

Snapshot: 2026-07-09. Owner: Vinay (vinay@classrootsedu.com). Access-gated to @classrootsedu.com. For deeper history, see docs/superpowers/handoffs/HANDOVER-2026-07-08-mass-math-asset-extraction.md and the earlier fractions-pilot handovers.