Deterministic testing for MCP tools with AI-powered analysis and test generation.
The MCP Developer Test system provides two complementary approaches to testing:
Run tests from the command line using the probecodex-agent:
| Name | Category | Hardware | Description |
|---|---|---|---|
| system-info | system | No | MCP version, disk space |
| license-status | license | No | License validation |
| emulator-check | emulator | No | QEMU, Renode installation |
| connection-basic | connection | Yes | Connect and read registers |
| memory-read | memory | Yes | Read RAM and Flash |
| core-control | core | Yes | Halt, run, reset CPU |
| rtos-freertos | rtos | Yes | FreeRTOS tasks and heap |
Access the MCP Tests page from More → MCP Tests in the Portal navigation.
View, upload, download test definition JSON files
View history, download input/output, trigger analysis
Generate test JSON from natural language
Describe what you want to test in natural language, and AI will generate the test JSON for you.
{
"name": "memory-read-ram",
"description": "Test RAM read at 0x20000000",
"tests": [
{
"name": "Read 64 bytes from RAM",
"tool": "mem_read",
"params": { "addr": "0x20000000", "len": 64 },
"expect": {
"success": true,
"exists": ["hex", "ascii"]
}
}
]
}After generation, you can edit the JSON, download it, or save directly to the Portal.
When tests fail, AI analyzes the results and correlates them with session telemetry to provide actionable insights.
{
"healthScore": 65,
"summary": "2 of 6 tests failed due to target disconnection",
"issues": [
{
"severity": "high",
"title": "Memory read timeout",
"details": "mem_read at 0x20001000 timed out after 5000ms",
"telemetryCorrelation": "Target disconnect event at 14:32:15",
"testName": "Read extended RAM"
}
],
"recommendations": [
"Check debug probe connection stability",
"Verify target is halted before memory reads"
],
"nextSteps": [
"Run connection-basic test to verify probe connection",
"Check OpenOCD logs for timeout reasons"
]
}Test suites are defined as JSON with the following structure:
{
"name": "my-test-suite",
"description": "What this test validates",
"tests": [
{
"name": "Human readable step name",
"tool": "mcp_tool_name",
"params": { "arg1": "value1" },
"expect": {
"success": true,
"type": "object",
"exists": ["field1", "field2"],
"min": 0,
"max": 100
}
}
]
}| Type | Description | Example |
|---|---|---|
| success | Tool call succeeded | "success": true |
| exists | Fields exist in result | "exists": ["pc", "sp"] |
| type | Result type check | "type": "object" |
| min/max | Numeric range | "min": 0, "max": 100 |
| Endpoint | Method | Description |
|---|---|---|
| /api/mcp-tests/suites | GET | List test suites |
| /api/mcp-tests/suites/:id/download | GET | Download test definition JSON |
| /api/mcp-tests/suites/upload | POST | Upload new test suite |
| /api/mcp-tests/suites/generate | POST | AI generate test from description |
| /api/mcp-tests/runs | GET | List test runs |
| /api/mcp-tests/runs/:id/download-input | GET | Download test run input JSON |
| /api/mcp-tests/runs/:id/download-output | GET | Download test run output JSON |
| /api/mcp-tests/runs/:id/analyze | POST | Trigger AI analysis |
| /api/mcp-tests/runs/:id/analysis | GET | Get existing analysis |
Run no-hardware tests in your CI/CD pipeline:
# In CI pipeline - run only no-hardware tests probecodex-agent mcp test --no-hardware --output=results.json # Check exit code (0 = all passed, 1 = failures) if [ $? -ne 0 ]; then echo "MCP tests failed!" exit 1 fi
Results are automatically submitted to the Portal and will trigger AI analysis if tests fail.