Skip to main content
Simulations let you check how an assistant behaves before a real caller does. Define a caller persona and a goal, and an LLM plays that caller against your assistant — either a fast text conversation or a real voice call — while a second LLM judges the result against the criteria you set.

​
Where to find Simulations

Open an assistant, select the Test button in the header, and choose Simulations. The panel opens next to the editor, so you can adjust the prompt and rerun a test without leaving the page.
Assistant Test menu with Simulations

Open the assistant’s Test menu and choose Simulations to create a test.

​
Creating a test

Select New test and step through:
1

Start

Begin blank, or apply a persona Milian suggested — see Generate with Milian below.
2

Details

Name the test and write the caller’s persona (identity and personality) in plain text.
Simulation Details with example caller persona

New test → Details: name the test and edit the caller persona. This example comes from the appointment-booking suggestion.

3

Goal

Describe what the caller is trying to accomplish, for example: “Your primary objective is to book a consultation for next week.”
Simulation Goal with an appointment-booking example

New test → Goal: describe the outcome and caller behavior to test. The shown appointment and phone number are example text.

4

Judge

Choose Chat or Voice mode, add the success criteria the judge should check, optionally list tools the assistant is expected to call, and set the maximum number of caller turns (up to 20, default 12).
Under Expected tools, only selected tools are listed. Choose Add tools, search the available tools, and use the checkboxes to require a tool call. Remove a listed tool with its remove button. An empty list adds no tool-call requirements. This selection changes the test expectations, not the tools connected to your assistant. For a function missing from the picker, expand Enter an exact tool name and enter its exact callable name. Save the test to keep your changes.
Select Generate with Milian instead of starting from a blank test. Milian proposes a batch of ready-made personas and success criteria for the assistant — review them and create the ones you want with one click.

​
Chat vs. voice

  • Chat — a fast, text-only conversation between the caller LLM and your assistant. No call is placed, and it’s the cheaper way to iterate on a prompt.
  • Voice — places a real call: a simulated caller with a synthesized voice talks to your assistant over the same path a live call takes, so the whole speech pipeline gets exercised. It takes longer than chat, costs more, and shows up in History like any other call; pick the caller’s voice when you switch to this mode.
When a knowledge base is linked, chat simulations run real knowledge searches and answer from the retrieved sources. Search successes and failures appear in the tool-call results. Other tools remain simulated and do not send messages, create bookings, or perform external actions. Flow branches are simplified. Chat simulations load the API tools and enabled MCP functions assigned to the assistant, including their argument schemas. External responses use synthetic test data that stays consistent across follow-up calls. The current-time tool uses the assistant’s timezone. Write realistic preconditions in the scenario, such as a full class or a failed lookup. A successful simulation checks the conversation and tool usage; verify provider permissions and actual data separately in Test Chat. Run selected uses the same runner as an individual chat simulation. End-call and assistant-handoff inputs are supported. Built-in tools with additional runtime workflows report an unsupported-tool error if invoked, rather than a synthetic success; verify those workflows separately in a controlled Test Chat. Voice simulations execute the assistant’s configured tools. Bookings, messages and other connected actions can be real. Use test data and test destinations when testing those actions. Reopening the panel resumes updates for active voice runs. Closing it stops Run selected from launching further tests; an already running call continues. If required evaluation data is unavailable, the run shows Error and keeps the transcript and available findings.

​
Creating a test from a real call

Open a finished call in History and select Create Simulation Test. Famulor writes a persona, goal, and default success criteria (accurate information, professional tone, proper resolution or routing) from that call’s transcript and analysis, saves them as a new test, and takes you to that assistant’s Simulations panel — review and edit it there before the first run. It’s the fastest way to turn a real conversation that went wrong into a regression test. See Post-call analysis.

​
Running tests and reading results

Choose a test from the side list on larger screens or the selector on mobile. Switch between Results, Transcript, Response times, and Tools. Expand a success criterion to read its reasoning. Select Start simulation, then choose Chat or Voice from the menu to start that run. Use the test actions menu to edit or delete a test. Run one test at a time, or use the checkboxes to choose a set. Select all selects every test, Clear selection clears it, and Run selected runs only the checked tests in order. Checkboxes appear on hover or keyboard focus and remain visible for selected tests; on touch screens, open Select tests. Each run reports:
  • a pass or fail per success criterion, with the judge’s reasoning;
  • the full transcript, labeled Caller / Assistant;
  • tool calls made during the run, checked against anything you marked as an expected tool;
  • for voice runs, a latency breakdown — end-to-end, time to first LLM token, time to first synthesized audio, speech-recognition latency, and perceived response time — plus a View call link to the resulting call in History.

​
Understanding the timings

Voice results show median (p50) and 95th percentile response times. Speech recognition, first response token and first synthesized audio describe different pipeline stages; they can overlap and should not be added together. Caller-observed timing is a separate measurement. A dash means no measurement is available, not zero latency. Refresh results when a call has just ended: measurements can arrive after the evaluation, and both older and new runs use the available call measurements. Chat results show the time to generate the complete simulated text reply, including tool processing. They are not a benchmark of your live voice assistant or a measurement of the first streamed token, and have no voice speed rating. Use View call to open the voice run in History for per-reply timing, recorded interruptions, transcript and available audio. Live chat, messaging and email share History’s conversation, usage and automation views; audio-only measurements apply to voice calls.
Simulations is text- or audio-based conversation testing, not a validator for your prompt’s wording alone — the caller LLM can improvise within the persona and goal you set, so a test result reflects how the conversation actually unfolds.

​
Availability and cost

Simulations is a plan feature. If it isn’t included, the panel says Simulation testing is not in your plan — compare plans under Settings → Plan. Every workspace member can open the panel and read past runs; creating, editing, and running tests is limited to workspace owners and admins. Simulations cost extra credits: every message in the run’s transcript is charged at your workspace’s Call simulator message rate — both in chat and in voice mode. Your current rate is listed on the Usage page under the credit rates (see How usage is billed), and each run appears there as a Simulation run transaction. Famulor checks your balance before a run starts and only charges for the messages actually produced. A voice run additionally consumes normal voice time for the call itself, since it exercises the full speech pipeline.

​
Overview and suggested fixes

Open Overview to see the latest result for each scenario and the overall criteria score: passed criteria divided by evaluated criteria, including expected-tool checks. Errors, running tests and unrun tests are counted separately. Audio-reference quality and conversation-quality metrics remain Not measured when unavailable. Tool calls appear as expandable cards in the transcript and Tools tab. Synthetic actions remain labeled Simulated; missing input data is shown as Not recorded. For a failed test, choose Review with Milian to open the assistant sidebar with a prepared diagnosis request, scenario, criteria, transcript and tool evidence. Review and send the draft. Milian can propose changes; the handoff does not save changes or launch another test. The scenario comparison chart switches between criteria scores, median chat response times and median voice response times. Chat and voice measurements stay separate; missing values are omitted. Select a bar to open the scenario. Tool activity separates successful real calls, simulated calls and failures, and shows how many finished runs contain tool records.

​
API & MCP

​
CLI