Sokko Docs
Platform API

Testing recipes

Run the tests of a recipe version in your sandbox workspace, read the result, and test with real app accounts on a long-lived sandbox agent.

A test run checks a recipe version before you apply it for customers. The platform builds a real agent from the version, sends it each test prompt of the spec, records the answers, and removes the agent again.

A test run does not judge the answers. It gives you each answer, the tools the agent called and how long it took. Your own code, or a person, decides whether the version is good.

All paths below are relative to https://sokko.ai/api/v1.

The sandbox workspace

Test runs happen in one workspace that the platform keeps for your account: the sandbox. No customer data is in it.

  • It is created with your first test run. Its externalId is _sandbox, and its name is Sandbox.
  • GET /account shows its id as sandboxWorkspaceId.
  • It is left out of GET /workspaces unless you send includeSandbox=true.
  • It does not count against your workspace limit. It cannot be renamed and it cannot be deleted (403 forbidden).
  • It has shared memory when memory is enabled for your account, so a recipe with memory can be tested.
  • Credits. It starts at zero, like every workspace. A test agent's model calls are paid from the sandbox's credits, so move credits to the sandbox first. A run that finds none ends as failed with credits_exhausted.
  • A seat. A test agent uses one place in your agent pool, and one of the sandbox's limits.maxAgentsPerWorkspace places, while the run lasts. A run that finds no free place ends as failed with plan_limit. A long-lived agent you keep in the sandbox takes one of those places for good.

Write tests

Tests are part of the spec:

{
  "tests": [
    { "name": "Opening hours", "prompt": "Are you open on Sunday?", "timeoutSeconds": 120 },
    { "name": "Order cutoff", "prompt": "Can I still order for tomorrow at 17:00?" }
  ],
  "testVariables": { "business_name": "Test Bakery", "house_rules": "Closed on Mondays." }
}
FieldRules
tests[].name1 to 80 characters, unique in the spec.
tests[].prompt1 to 4,000 characters. The message sent to the agent. No variables.
tests[].timeoutSeconds30 to 600, default 180.
testVariablesValues for text and block variables, used by a test run.

A spec holds up to 10 tests. Each test is sent in a conversation of its own, so a test cannot depend on the one before it.

A secret variable has no test value in the spec, because a spec is stored and returned. You give secrets when you start the run.

A test agent is always an operator agent, whatever the recipe's audience is. So a test run does not show what the stricter rules of an end_users agent change: the tools that are refused, and the refusal of agent memory.

Start a test run

POST /recipes/{recipeId}/test-runs
FieldTypeRules
versionintegerOptional. The version to test. Without it: the latest.
variablesobjectOptional. Values for this run. They go over the spec's testVariables. Give the secret values here.

The body is optional: a request with no body tests the latest version with the spec's testVariables.

curl https://sokko.ai/api/v1/recipes/order-desk/test-runs \
  -H "Authorization: Bearer $SOKKO_PLATFORM_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "version": 3, "variables": { "orders_token": "mcp_test_5e4d3c2b1a" } }'

The answer is 202 with the test run, in the status queued:

{
  "id": "e8a0c2e4-6f8b-4d0a-b2c4-e6f8a0b2c4d6",
  "recipeId": "1c3e5a7b-9d0f-4b2c-8e4a-6c8e0a2c4e6f",
  "version": 3,
  "status": "queued",
  "results": [],
  "integrations": [{ "app": "googlecalendar", "status": "not_connected" }],
  "startedAt": null,
  "finishedAt": null
}

Tokens of MCP servers

Every MCP server of the recipe that has a token needs the value of its token variable in variables, in every run. The run registers the servers in the sandbox and removes them again at its end, so no token of an earlier run is there to reuse. A run without one answers 422 invalid_variables.

Use a token of a test environment. The test agent calls your servers for real. A token you give for a test is stored only for the duration of the run. It is removed when the run ends, with the servers the run registered, and it is in no answer of the API. If a run could not remove everything at its end, the platform tries again in the background until it is gone.

One run at a time

An account runs one test run at a time. A new run waits as queued until the one before it has ended. At most 5 runs wait: a sixth answers 429 test_queue_full.

Errors

StatusCodeWhen
400invalid_requestA bad field.
402billing_inactiveYour subscription is not active, or a payment is past due.
404not_foundNo such recipe or version in your account.
409recipe_archivedThe recipe is archived.
409no_testsThe version has no tests.
422invalid_variablesA required variable has no value, a value does not fit, or the token of an MCP server is missing.
429test_queue_fullFive runs already wait. Retry-After is 60.
503unavailableA token could not be stored. Nothing was queued: send the request again.

What a run does

StatusWhat happens
queuedThe run waits for its turn.
preparingThe version is applied in the sandbox, as an operator agent, with no apps.
provisioningThe agent starts. A run whose agent is not ready after 10 minutes fails.
runningEach test prompt is sent in a new conversation, one after the other.
cleaningThe test conversations and the agent are deleted, and so are the MCP servers this run registered.
completedEvery prompt was sent and has a result.
failedThe run could not finish. error says why.
cancelledYou cancelled the run.

For each prompt:

  • The run waits for the turn to end, up to the test's timeoutSeconds. After that it stops the turn and records the result timed_out.
  • A turn that was interrupted is tried once more, in a new conversation. A second interruption is recorded as interrupted.
  • If the agent asks a question in place of an answer, the result is waiting and answer holds what the agent wrote. Nobody answers the question.
  • The answer is stored up to 64 KB. A longer one is cut, with "truncated": true. tools holds the first 200 tool calls.

A whole run has 45 minutes. Every run ends with cleaning, also a failed or a cancelled one, so a test agent does not stay in your pool. The skills that the run added to the sandbox's library stay there.

completed means the run did its work. It does not mean the tests passed: read results.

Read a test run

GET /test-runs/{testRunId}
GET /recipes/{recipeId}/test-runs

The first reads one run. The second lists the runs of a recipe.

curl https://sokko.ai/api/v1/test-runs/e8a0c2e4-6f8b-4d0a-b2c4-e6f8a0b2c4d6 \
  -H "Authorization: Bearer $SOKKO_PLATFORM_KEY"
{
  "id": "e8a0c2e4-6f8b-4d0a-b2c4-e6f8a0b2c4d6",
  "recipeId": "1c3e5a7b-9d0f-4b2c-8e4a-6c8e0a2c4e6f",
  "version": 3,
  "status": "completed",
  "results": [
    {
      "name": "Opening hours",
      "prompt": "Are you open on Sunday?",
      "status": "completed",
      "answer": "Yes. We are open on Sunday from 7:00 to 18:00.",
      "truncated": false,
      "tools": [],
      "durationMs": 5210
    },
    {
      "name": "Order cutoff",
      "prompt": "Can I still order for tomorrow at 17:00?",
      "status": "completed",
      "answer": "No. Orders for the next day close at 16:00.",
      "truncated": false,
      "tools": [{ "name": "orders_cutoff", "state": "completed" }],
      "durationMs": 8934
    }
  ],
  "integrations": [{ "app": "googlecalendar", "status": "not_connected" }],
  "startedAt": "2026-11-21T10:14:03.118Z",
  "finishedAt": "2026-11-21T10:16:47.902Z"
}
FieldMeaning
statusThe status of the run, from the table above.
resultsOne entry per test, in the order of the spec. Empty until the run reaches its tests.
results[].statusHow the turn ended: completed, failed, timed_out, interrupted or waiting.
results[].answerThe agent's answer text.
results[].truncatedtrue when the answer was cut at 64 KB.
results[].toolsThe tools the agent called, each with name and state.
results[].durationMsHow long the turn took.
results[].errorOnly for a test that did not complete: { "code", "message" }.
integrationsThe apps the recipe names. Always not_connected: a test run connects no apps.
errorOnly on a failed run: { "code", "message" }, why the run itself could not finish.

The object holds no id of the sandbox workspace, the test agent or its conversations: they are gone when the run has ended.

Codes of a run's error:

error.codeMeaning
credits_exhaustedThe sandbox workspace has no credits.
plan_limitNo free place for the test agent in your pool or in the sandbox.
agent_not_readyThe test agent was not ready after 10 minutes.
run_timeoutThe run did not finish in 45 minutes.
apply_failedThe version could not be applied in the sandbox.
internalA fault on the platform's side.

A run can also fail with the code of the refusal that an apply of the version would get, for example skill_conflict, unprocessable or memory_not_enabled.

To know when a run ends, subscribe to the event test_run.completed, or read the run every few seconds. The event is sent for every end of a run (completed, failed or cancelled). It carries recipeId, testRunId and status, and no answer text. See Webhooks.

A typical check in a deployment pipeline: publish the version, start a run, wait for the event, read the run, compare each answer and tools with what you expect, and only then upgrade your customers' agents.

Errors: 404 not_found.

Cancel a test run

POST /test-runs/{testRunId}/cancel
curl -X POST https://sokko.ai/api/v1/test-runs/e8a0c2e4-6f8b-4d0a-b2c4-e6f8a0b2c4d6/cancel \
  -H "Authorization: Bearer $SOKKO_PLATFORM_KEY"

The answer is 202 with the test run. A queued run is cancelled at once. A run in progress goes to cleaning, removes what it made, and then has the status cancelled. The results it already has stay. A run that is already cleaning or has ended is answered as it is.

Errors: 404 not_found.

Test with real app accounts

A test run connects no apps. Its agent lives for a few minutes, and nobody is there to sign in. A prompt that needs a connected app shows you what the agent does without the app, which is worth a test of its own, and nothing more.

To test a recipe with real accounts, keep a long-lived agent in the sandbox workspace:

Create the agent once

Apply the recipe in the sandbox workspace like in any workspace. Use the sandboxWorkspaceId of GET /account. It is null until your first test run, so start one run first.

curl https://sokko.ai/api/v1/workspaces/$SANDBOX/agents \
  -H "Authorization: Bearer $SOKKO_PLATFORM_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "recipe": "order-desk",
    "externalId": "order-desk-staging",
    "variables": { "business_name": "Test Bakery", "orders_token": "mcp_test_5e4d3c2b1a" }
  }'

Connect test accounts

Run the connect flow for this agent, and sign in with accounts you own for testing: a test mailbox, a test calendar. Do this once. The connections stay with the agent.

Talk to it

Open conversations with the agent and send the prompts that need the apps. Read the answers and the tools of each turn.

Upgrade it first

When you publish a new version, upgrade this agent before any customer's agent. An upgrade keeps the app connections, so the same test accounts serve every version.

This agent uses one place in your agent pool for as long as it exists. Suspend it between releases if you want it to do no work: it keeps its place, its files and its connections.

On this page