Building a speech evaluation pipeline for an LMS

Speech in an LMS is a different problem from chat. The input is messy, long, and emotionally loaded for the student. The output has to be gradable: consistent enough that a teacher trusts it, fast enough that a cohort can submit in the same window.

On Educollab we built an automated speech evaluation pipeline as part of a broader GenAI push — alongside an intelligent notification classification model. This is the shape that held up.

Don't start with the model card

The product question is not "which ASR is best." It's:

• What does a teacher actually mark?
• What is allowed to be probabilistic vs. what must be deterministic?
• What happens when audio is quiet, clipped, or in a second language?

We split the pipeline: capture and normalize audio, transcribe, then evaluate against a rubric the course already owned. The LLM (or classifier) sees a constrained artifact — transcript plus rubric — not a raw blob and a hope.

That constraint is the product. Unbounded "grade this however" prompts drift. Rubrics don't.

Classification is a product feature

Notifications in an LMS are a firehose. An intelligent classification model sounds like a nice-to-have until you realize teachers ignore the stream, and students miss the one message that matters.

We treated classification as routing: what is actionable, what is informational, what can wait. Same discipline as speech eval — labeled outcomes, not vibes.

Production is the rest of the system

SSO, cron microservices, and third-party syncs don't belong in a model demo, but they belong in the LMS. Speech jobs have to enqueue, retry, and land in the same identity story as the rest of the app. Background work that isn't owned becomes a support ticket.

GenAI features that survive are the ones that look boring in architecture diagrams: queues, indexes, auth, and a rubric someone already agreed to.