Testing recipes
Run the tests of a recipe version in your sandbox workspace, read the result, and test with real app accounts on a long-lived sandbox agent.
A test run checks a recipe version before you apply it for customers. The platform builds a real agent from the version, sends it each test prompt of the spec, records the answers, and removes the agent again.
A test run does not judge the answers. It gives you each answer, the tools the agent called and how long it took. Your own code, or a person, decides whether the version is good.
All paths below are relative to https://sokko.ai/api/v1.
The sandbox workspace
Test runs happen in one workspace that the platform keeps for your account: the sandbox. No customer data is in it.
- It is created with your first test run. Its
externalIdis_sandbox, and its name isSandbox. GET /accountshows its id assandboxWorkspaceId.- It is left out of
GET /workspacesunless you sendincludeSandbox=true. - It does not count against your workspace limit. It cannot be renamed and it
cannot be deleted (403
forbidden). - It has shared memory when memory is enabled for your account, so a recipe with memory can be tested.
- Credits. It starts at zero, like every workspace. A test agent's model
calls are paid from the sandbox's credits, so
move credits to the sandbox first. A run that
finds none ends as
failedwithcredits_exhausted. - A seat. A test agent uses one place in your agent pool, and one of the
sandbox's
limits.maxAgentsPerWorkspaceplaces, while the run lasts. A run that finds no free place ends asfailedwithplan_limit. A long-lived agent you keep in the sandbox takes one of those places for good.
Write tests
Tests are part of the spec:
{
"tests": [
{ "name": "Opening hours", "prompt": "Are you open on Sunday?", "timeoutSeconds": 120 },
{ "name": "Order cutoff", "prompt": "Can I still order for tomorrow at 17:00?" }
],
"testVariables": { "business_name": "Test Bakery", "house_rules": "Closed on Mondays." }
}| Field | Rules |
|---|---|
tests[].name | 1 to 80 characters, unique in the spec. |
tests[].prompt | 1 to 4,000 characters. The message sent to the agent. No variables. |
tests[].timeoutSeconds | 30 to 600, default 180. |
testVariables | Values for text and block variables, used by a test run. |
A spec holds up to 10 tests. Each test is sent in a conversation of its own, so a test cannot depend on the one before it.
A secret variable has no test value in the spec, because a spec is stored
and returned. You give secrets when you start the run.
A test agent is always an operator agent, whatever the recipe's audience
is. So a test run does not show what the stricter rules of an end_users
agent change: the tools that are refused, and the refusal of agent memory.
Start a test run
POST /recipes/{recipeId}/test-runs| Field | Type | Rules |
|---|---|---|
version | integer | Optional. The version to test. Without it: the latest. |
variables | object | Optional. Values for this run. They go over the spec's testVariables. Give the secret values here. |
The body is optional: a request with no body tests the latest version with
the spec's testVariables.
curl https://sokko.ai/api/v1/recipes/order-desk/test-runs \
-H "Authorization: Bearer $SOKKO_PLATFORM_KEY" \
-H "Content-Type: application/json" \
-d '{ "version": 3, "variables": { "orders_token": "mcp_test_5e4d3c2b1a" } }'The answer is 202 with the test run, in the status queued:
{
"id": "e8a0c2e4-6f8b-4d0a-b2c4-e6f8a0b2c4d6",
"recipeId": "1c3e5a7b-9d0f-4b2c-8e4a-6c8e0a2c4e6f",
"version": 3,
"status": "queued",
"results": [],
"integrations": [{ "app": "googlecalendar", "status": "not_connected" }],
"startedAt": null,
"finishedAt": null
}Tokens of MCP servers
Every MCP server of the recipe that has a token needs the value of its
token variable in variables, in every run. The run registers the servers in
the sandbox and removes them again at its end, so no token of an earlier run
is there to reuse. A run without one answers 422 invalid_variables.
Use a token of a test environment. The test agent calls your servers for real. A token you give for a test is stored only for the duration of the run. It is removed when the run ends, with the servers the run registered, and it is in no answer of the API. If a run could not remove everything at its end, the platform tries again in the background until it is gone.
One run at a time
An account runs one test run at a time. A new run waits as queued until
the one before it has ended. At most 5 runs wait: a sixth answers 429
test_queue_full.
Errors
| Status | Code | When |
|---|---|---|
| 400 | invalid_request | A bad field. |
| 402 | billing_inactive | Your subscription is not active, or a payment is past due. |
| 404 | not_found | No such recipe or version in your account. |
| 409 | recipe_archived | The recipe is archived. |
| 409 | no_tests | The version has no tests. |
| 422 | invalid_variables | A required variable has no value, a value does not fit, or the token of an MCP server is missing. |
| 429 | test_queue_full | Five runs already wait. Retry-After is 60. |
| 503 | unavailable | A token could not be stored. Nothing was queued: send the request again. |
What a run does
| Status | What happens |
|---|---|
queued | The run waits for its turn. |
preparing | The version is applied in the sandbox, as an operator agent, with no apps. |
provisioning | The agent starts. A run whose agent is not ready after 10 minutes fails. |
running | Each test prompt is sent in a new conversation, one after the other. |
cleaning | The test conversations and the agent are deleted, and so are the MCP servers this run registered. |
completed | Every prompt was sent and has a result. |
failed | The run could not finish. error says why. |
cancelled | You cancelled the run. |
For each prompt:
- The run waits for the turn to end, up to the test's
timeoutSeconds. After that it stops the turn and records the resulttimed_out. - A turn that was interrupted is tried once more, in a new conversation. A
second interruption is recorded as
interrupted. - If the agent asks a question in place of an answer, the result is
waitingandanswerholds what the agent wrote. Nobody answers the question. - The answer is stored up to 64 KB. A longer one is cut, with
"truncated": true.toolsholds the first 200 tool calls.
A whole run has 45 minutes. Every run ends with cleaning, also a failed or a
cancelled one, so a test agent does not stay in your pool. The skills that the
run added to the sandbox's library stay there.
completed means the run did its work. It does not mean the tests passed:
read results.
Read a test run
GET /test-runs/{testRunId}
GET /recipes/{recipeId}/test-runsThe first reads one run. The second lists the runs of a recipe.
curl https://sokko.ai/api/v1/test-runs/e8a0c2e4-6f8b-4d0a-b2c4-e6f8a0b2c4d6 \
-H "Authorization: Bearer $SOKKO_PLATFORM_KEY"{
"id": "e8a0c2e4-6f8b-4d0a-b2c4-e6f8a0b2c4d6",
"recipeId": "1c3e5a7b-9d0f-4b2c-8e4a-6c8e0a2c4e6f",
"version": 3,
"status": "completed",
"results": [
{
"name": "Opening hours",
"prompt": "Are you open on Sunday?",
"status": "completed",
"answer": "Yes. We are open on Sunday from 7:00 to 18:00.",
"truncated": false,
"tools": [],
"durationMs": 5210
},
{
"name": "Order cutoff",
"prompt": "Can I still order for tomorrow at 17:00?",
"status": "completed",
"answer": "No. Orders for the next day close at 16:00.",
"truncated": false,
"tools": [{ "name": "orders_cutoff", "state": "completed" }],
"durationMs": 8934
}
],
"integrations": [{ "app": "googlecalendar", "status": "not_connected" }],
"startedAt": "2026-11-21T10:14:03.118Z",
"finishedAt": "2026-11-21T10:16:47.902Z"
}| Field | Meaning |
|---|---|
status | The status of the run, from the table above. |
results | One entry per test, in the order of the spec. Empty until the run reaches its tests. |
results[].status | How the turn ended: completed, failed, timed_out, interrupted or waiting. |
results[].answer | The agent's answer text. |
results[].truncated | true when the answer was cut at 64 KB. |
results[].tools | The tools the agent called, each with name and state. |
results[].durationMs | How long the turn took. |
results[].error | Only for a test that did not complete: { "code", "message" }. |
integrations | The apps the recipe names. Always not_connected: a test run connects no apps. |
error | Only on a failed run: { "code", "message" }, why the run itself could not finish. |
The object holds no id of the sandbox workspace, the test agent or its conversations: they are gone when the run has ended.
Codes of a run's error:
error.code | Meaning |
|---|---|
credits_exhausted | The sandbox workspace has no credits. |
plan_limit | No free place for the test agent in your pool or in the sandbox. |
agent_not_ready | The test agent was not ready after 10 minutes. |
run_timeout | The run did not finish in 45 minutes. |
apply_failed | The version could not be applied in the sandbox. |
internal | A fault on the platform's side. |
A run can also fail with the code of the refusal that an
apply of the version would get,
for example skill_conflict, unprocessable or memory_not_enabled.
To know when a run ends, subscribe to the event test_run.completed, or read
the run every few seconds. The event is sent for every end of a run
(completed, failed or cancelled). It carries recipeId, testRunId and
status, and no answer text. See
Webhooks.
A typical check in a deployment pipeline: publish the version, start a run,
wait for the event, read the run, compare each answer and tools with what
you expect, and only then upgrade your customers' agents.
Errors: 404 not_found.
Cancel a test run
POST /test-runs/{testRunId}/cancelcurl -X POST https://sokko.ai/api/v1/test-runs/e8a0c2e4-6f8b-4d0a-b2c4-e6f8a0b2c4d6/cancel \
-H "Authorization: Bearer $SOKKO_PLATFORM_KEY"The answer is 202 with the test run. A queued run is cancelled at once.
A run in progress goes to cleaning, removes what it made, and then has the
status cancelled. The results it already has stay. A run that is already
cleaning or has ended is answered as it is.
Errors: 404 not_found.
Test with real app accounts
A test run connects no apps. Its agent lives for a few minutes, and nobody is there to sign in. A prompt that needs a connected app shows you what the agent does without the app, which is worth a test of its own, and nothing more.
To test a recipe with real accounts, keep a long-lived agent in the sandbox workspace:
Create the agent once
Apply the recipe in the sandbox workspace like in any workspace. Use the
sandboxWorkspaceId of GET /account. It is null until your first test
run, so start one run first.
curl https://sokko.ai/api/v1/workspaces/$SANDBOX/agents \
-H "Authorization: Bearer $SOKKO_PLATFORM_KEY" \
-H "Content-Type: application/json" \
-d '{
"recipe": "order-desk",
"externalId": "order-desk-staging",
"variables": { "business_name": "Test Bakery", "orders_token": "mcp_test_5e4d3c2b1a" }
}'Connect test accounts
Run the connect flow for this agent, and sign in with accounts you own for testing: a test mailbox, a test calendar. Do this once. The connections stay with the agent.
Talk to it
Open conversations with the agent and send
the prompts that need the apps. Read the answers and the tools of each turn.
Upgrade it first
When you publish a new version, upgrade this agent before any customer's agent. An upgrade keeps the app connections, so the same test accounts serve every version.
This agent uses one place in your agent pool for as long as it exists. Suspend it between releases if you want it to do no work: it keeps its place, its files and its connections.
Recipes
Describe your standard agent once, publish it as a numbered version, apply it for each customer with that customer's values, and upgrade agents to a newer version.
Data and privacy
Where the conversations of your end users are stored, who can read them, how to delete the data of one end user, and what that call does not remove.