Skip to content
    Back to Blog
    August 31, 20261 min read

    By Brian Hanson · Updated Nov 11, 2026

    How to Evaluate New Language Models for Customer Service

    Judge the response, not the architecture label

    A new model design is a reason to investigate, not a reason to replace your customer-service system immediately. Ask whether a candidate answers your actual questions correctly and knows when to involve a person.

    Sander Dieleman’s explanation of continuous diffusion language models describes generating text by progressively denoising a continuous representation. That mechanism alone does not establish better conversation, fewer errors, lower costs or higher sales in your business.

    Practical exercise: build a support test set

    1. Collect representative questions. Include routine requests, ambiguous wording, missing order information and cases your team finds difficult. Remove personal data from examples.
    2. Write acceptable answers. Use your current policies and identify questions that require a handoff rather than an invented answer.
    3. Compare candidates consistently. Give each system the same approved information and review the results against the same criteria.
    4. Check accuracy before tone. A friendly reply that invents a refund or delivery promise should fail your test.
    5. Measure the complete process. Include response time, review, corrections and escalation, not just how quickly text appears.

    Keep a route to human help

    Start with a limited trial and assign someone to review failures. Keep the existing service route available while you assess the results. Choose the system that meets your requirements rather than assuming a technical label guarantees a more natural customer experience.