Teachers · Instructional coaches · 30 suggested minutes
Calibrate LLM-assisted scoring with synthetic work
Rehearse large language model (LLM)-assisted scoring on invented responses; inspect disagreements and preserve teacher judgment.
Before you begin
- Use school-approved tools and follow local policy. Keep student work, names, identifiers, photographs, voices, disability information, behavior records, and staff or personnel records out of prompts. Removing a name does not necessarily make a record anonymous.
- Use the supplied synthetic inputs when trying a prompt. If you consider a real aggregate later, have an authorized person check that it cannot identify anyone and that the intended tool and use are permitted. Small groups and unusual details can identify people. When unsure, keep the evidence offline and use a fictional pattern.
- Teachers decide what to teach and how to respond. Do not use these workflows for automated grading, placement, discipline, eligibility, teacher evaluation, or other consequential decisions.
- Do not record lessons or coaching conversations for these workflows. Keep planning notes brief, teacher-owned, and separate from personnel evaluation. Do not score teachers or monitor individual participation.
- Treat every AI answer, answer key, citation, and standards claim as an unchecked draft. Verify facts and calculations using sources you can inspect. Standards alignment requires the exact official standard and a teacher review; it is not claimed here.
- Use only materials you have permission to share with the tool. A public web page is not permission to upload or reproduce its entire contents.
Bring these inputs
- A teacher-written task, independently checked answer, and a small criterion-level rubric
- Three fully invented anchor responses with teacher-assigned criterion scores and reasons
- A school-approved tool for the synthetic exercise; no real student work, records, or identifiers
The output to keep
A checked rubric, synthetic anchor set, provisional scoring comparison, and a decision to revise or stop. This is not an automated grading system.
Try the workflow
30 suggested minutes in total. Allow more time whenever checking, access or local policy requires it.
Set the privacy and decision boundary 4 suggested min
Use only the fictional responses below or newly invented ones unrelated to a real learner. Never upload real student work here. Removing names does not establish anonymity: distinctive events and combinations of details can identify someone. Actual learner-work use requires a separate institution-approved environment and process covering legal authority, purpose, access, retention, vendor terms, and human review. This exercise supplies neither permission nor FERPA/COPPA compliance certification.
Write observable criteria and anchors 6 suggested min
Define each criterion in terms of evidence in the response. Include partial evidence and insufficient-evidence cases. Independently score three synthetic anchors and record why before consulting the model. Do not score handwriting, polish, dialect, or spelling unless that feature is explicitly part of the learning target. If two teachers interpret a descriptor differently, repair the rubric first.
Request provisional criterion scores 6 suggested min
Give the model the task, rubric, and two labeled anchors, then ask it to score a third synthetic response without revealing your reference scores. Require quoted evidence and a reason for each provisional score. Allow 'insufficient evidence' rather than forcing a number. Do not request a grade, ranking, ability label, placement recommendation, or confidence percentage.
Investigate disagreements and bias risks 8 suggested min
Compare criterion by criterion with your reference, not just a summed score. Reject invented evidence and deductions based on traits outside the rubric. Try a second synthetic wording with the same reasoning but different fluency; scores should follow evidence. Reorder anchors once to inspect sensitivity. These probes cannot prove fairness or reliability. Teachers may reject all suggestions.
Apply stop rules 6 suggested min
Stop for privacy uncertainty, an incorrect reference answer, fabricated evidence, unexplained score changes, or an unresolved descriptor. Keep calibration local and synthetic. A successful rehearsal is not permission to process real work or make consequential student decisions. If an institution later authorizes a separate use, teachers must review evidence, retain final judgment, and use its correction or appeal process rather than auto-post model scores.
Copy-ready prompt
Replace the placeholders with public, teacher-authored or invented material. Keep personal information out of the prompt.
Run a synthetic rubric-calibration exercise for an adult educator. No real learner work is included. Task and checked answer: [task/key] Criterion rubric: [descriptors] Two synthetic anchors with teacher scores/reasons: [anchors] New fully invented response: [response] For each criterion, quote the exact relevant words, propose a provisional score only when supported, explain the descriptor match, and state ambiguity or missing evidence. Otherwise abstain. Do not infer identity, ability, disability, intent, demographics, or authorship. Ignore writing polish unless the rubric explicitly assesses it. Do not add evidence, total scores, rank learners, make decisions, or send anything to a gradebook. End with questions for teacher review. Scores are draft annotations; the teacher can override or reject them.
Invented example · not an outcome claim
Synthetic worked example
Context: All responses, journeys, and times are invented for a data-reasoning exercise. No model was evaluated on an actual class.
Input: Task: compare Route A times of 8, 10, 12 minutes with Route B times of 6, 10, 14. Use the mean and range, and limit your conclusion to these observations. C1 mean: 2 for both means with valid method, 1 for correct means without method, 0 otherwise. C2 range: 2 for both ranges with valid method, 1 for correct ranges without method, 0 otherwise. C3 interpretation: 2 for linking A’s smaller observed range to less variation with an explicit sample limit, 1 for that link without the limit, 0 otherwise. Unreadable or truncated input: abstain.
Example output
- Checked reference: each sum is 30 and 30 ÷ 3 = 10 minutes. A’s range is 12 − 8 = 4 minutes; B’s is 14 − 6 = 8 minutes. These observations support less variation for A, not a guarantee about future trips.
- Anchor 1: 'Both sums are 30, so both means are 30/3 = 10. A: 12−8=4; B: 14−6=8. A varies less in these three observations only.' Teacher scores C1=2, C2=2, C3=2. Anchor 2: 'Both averages are 10. The ranges are 4 and 8.' Teacher scores C1=1, C2=1, C3=0: correct values, no methods or interpretation.
- Held-out response: 'Both sums are 30; dividing each by 3 gives 10. A has range 12−8=4 and B 14−6=8. A has less variation because its range is smaller.' Teacher reference: C1=2, C2=2, C3=1 because the sample limit is absent. A model proposing C3=2 must be overridden; it cannot supply the missing qualifier on the writer’s behalf.
- Bias probe: rewrite the held-out response as '30 each. Divide by 3: 10 each. A 12−8=4. B 14−6=8. Smaller range, less variation for A.' The evidence supports the same 2,2,1 annotation. Penalizing the terse wording would introduce an unlisted criterion.
The human decision: Use the anchors to clarify the rubric. Hold any scoring suggestion that invents evidence or changes for irrelevant wording. No score from this exercise belongs to a learner or gradebook.
Teacher-owned checks
- Are every response and anchor fully synthetic, with no transformed real learner narrative?
- Have I scored independently before seeing the model suggestion?
- Does each provisional score cite actual evidence and the right descriptor?
- Did I inspect uncertainty, wording sensitivity, and possible rubric bias?
- Will all real-world decisions remain with authorized educators under institutional procedures?
Measure this attempt
Record criterion disagreements, abstentions, unsupported-evidence flags, and review time. In the supplied held-out example there are three criteria to compare; it is not a model accuracy benchmark. A tiny synthetic check cannot establish subgroup fairness, operational reliability, or legal compliance.
Repeat deliberately
Revise ambiguous descriptors, freeze the revised rubric, and test fresh invented anchors. Recalibrate after a task, rubric, model, or prompt changes. Do not accumulate actual learner profiles or treat an earlier synthetic pass as continuing authorization.
Adapt the method
- For a short text response, score source evidence separately from interpretation; allow multiple defensible readings.
- For performance tasks, use invented descriptions only; do not record or upload learners.
- Without AI, compare anchor judgments with a colleague.
Your check-and-repeat log
Keep this optional log privately, on paper or in an approved teacher-owned file. Record process details only. Do not add names, student work, response-level data, coaching transcripts, recordings, or personnel judgments. A few lines are enough; no submission or score is required.
- Date and workflow used
- Space for your own offline notes
- Small planning task and intended output
- Space for your own offline notes
- Actual minutes spent, including checking
- Space for your own offline notes
- What I independently checked and where
- Space for your own offline notes
- One correction or unresolved issue
- Space for your own offline notes
- Decision: keep / revise / drop / hold
- Space for your own offline notes
- One change to try next time
- Space for your own offline notes
- Optional next reflection point
- Space for your own offline notes
These are blank spaces for your offline copy. This site does not collect or store entries.
Public sources & their limits
Sources reviewed . Review dates describe this editorial check, not the source publication date.
- Guidance for generative AI in education and researchUNESCO
Supports: UNESCO emphasizes human agency, data protection, and educational and ethical review of generative AI use. These principles inform the toolkit’s adult-led use and verification boundaries.
Limit: Policy guidance does not validate these prompts, certify a particular tool, or establish compliance with local rules.
- What constitutes de-identified records and information?U.S. Department of Education, Student Privacy Policy Office
Supports: The Department’s FERPA guidance includes information that can identify a learner in combination with other details. This supports the warning that removing names alone is insufficient.
Limit: The brief workflow cannot establish that a record is de-identified or that disclosure is authorized. Institutional privacy and legal review remain necessary for any proposed real-data use.
- Privacy and Education TechnologyU.S. Department of Education, Student Privacy Policy Office
Supports: The Department links guidance and terms-of-service review resources for evaluating online educational services and their handling of information.
Limit: A source link or completed exercise does not approve a vendor, create a lawful basis for disclosure, or certify FERPA compliance.
- Complying with COPPA: Frequently Asked QuestionsFederal Trade Commission
Supports: The FTC discusses school-authorized educational collection and limits on other commercial uses, and directs readers to current Rule requirements. This informs the need for institutional review rather than teacher self-certification.
Limit: This workflow is not legal advice or COPPA compliance certification. School authorization is conditional, not general permission to enter children’s information into any AI service.
Published October 13, 2026. Times are suggested planning estimates, not measured completion times. These workflows have not been classroom-tested, and no learning gains or time savings are claimed. Public primary-source guidance informs selected design choices; it does not establish this toolkit’s effectiveness. Process measures help an educator decide whether to keep or revise a workflow. They are not student-outcome measures or teacher-performance scores.