Jiaqi Gan

Small LM Post-Training for AI Call Agents

ScamAI · 2026

Synthetic training data and one LoRA recipe for six small LMs that answer calls.

Problem

A call-screening agent must reply within its owner's authority during a call and summarize the call afterward. Small on-device models do both poorly, and real calls come with no reference replies or summaries to train on.

My contribution

  • Built the data pipeline that produced 3,395 synthetic calls and 15,102 training targets (11,707 replies, 3,395 summaries).
  • Fine-tuned six 0.6–4B models (Qwen3, Llama 3.2, Gemma 3), one rank-16 LoRA adapter per model for both tasks.
  • Evaluated 4,608 outputs on 192 held-out calls with a custom five-criterion LLM rubric.

Approach

  1. Caller author

    Sees a private scenario brief

    Writes the caller's turns

  2. Reply author

    Sees the conversation up to the current caller turn

    Writes each agent reply

  3. Summary author

    Sees the completed transcript only

    Writes the owner summary

Gate Dialogue review summary review: a call is kept only if both accept it

3,395 calls

11,707 reply targets

3,395 summary targets

Train One rank-16 LoRA adapter per model, both tasks sampled equally, loss on target tokens only; six 0.6–4B models (Qwen3, Llama 3.2, Gemma 3)

Evaluate Released vs. adapted checkpoint on 192 held-out calls: 4,608 outputs

How training data is constructed. Each author sees only what its role permits, so reply targets cannot rely on information the agent has not heard yet, and summaries can rely only on what was said.

Results

  • 93.75–96.88% reply qualification after training, up from 20.83–53.65% (range over six models)
  • 43.23–72.40% summary qualification after training, up from 0–6.25%

Qualification = an output passes all five criteria of a custom LLM rubric. It is not accuracy, human acceptance or deployment success.

Limits

  • No ablations; the gain reflects the data and the training recipe together.
  • The same LLM evaluator screened the training data and scored the outputs; human ratings are not reported.
  • Text inputs from synthetic calls, not a live voice system.