Mistakes Claude Makes When Your Board Farm Is Broken

How a hardware-in-the-loop lab fails

You know the list:

• A USB hub resets. /dev/ttyUSB0 comes back as ttyUSB1, and everything not addressed through /dev/serial/by-id moves with it.
• The host installs an OS update nobody planned, or just reboots, and one unit does not come back. The board is fine. Nothing can reach it.
• The SD-Mux stays on the host side after a hub reset. The board boots into nothing.
• A USB Ethernet adapter or an HDMI grabber drops off. Network tests time out, display tests compare a black frame.
• The card has thousands of flashes behind it and fails to boot one time in twenty.
• The supply sags under load, the board resets, and the reset reads as a test failure.

All of these fail quietly and land in the same bucket as real test failures. CI calls the split infrastructure failure (CircleCI has a job status for it); a hardware lab needs it one level lower: firmware or cable? And since a quiet fault stays until someone walks over, availability is set by mean time to recovery, not by failure rate.

Claude does not stop at a broken API. It goes around it.

A Jenkins job with the same broken file transfer goes red at the copy step. Someone reads the log and walks to the rack.

An agent treats every tool you expose as a workaround for every other tool. Each step in the log above is reasonable from where it sits, and wrong for the rack:

• The fault is buried under “Done”. No report to the owner of the board farm.
• In the worst case the board boots the zImage as it should, but Claude creates a new memory note telling it to choose this path next time. The next session does not even try the file transfer.
• Twenty times the tool calls and tokens. In the device’s activity timeline: one console session opened.

The base64 transfer has siblings:

The agent cannot tell “the lab is broken” from “I used the tool wrong”. It has no hallway, no red LED, no colleague saying “box 4 is weird today”. It has the API. A bigger model does not change that: a NeurIPS 2026 paper injected faults into MCP tool calls, task scores fell by about 60 %, and the larger model held up no better than the small one.

Board farm health checks, honest errors, server-side limits

What we do about it in fernsteuerung.io.

A health check per board

LAVA has done this for years with a known-good job per device. We do it on the node, with less machinery:

$ curl -s http://<node>/health
{
  "status": "fail",
  "checks": [
    {"name": "node",    "status": "ok",   "detail": "v0.9.0, up 3d"},
    {"name": "console", "status": "ok",   "detail": "/dev/serial/by-id/usb-FTDI_..., last rx 3s ago"},
    {"name": "power",   "status": "ok",   "detail": "on"},
    {"name": "sdmux",   "status": "ok",   "detail": "dut"},
    {"name": "usb",     "status": "fail", "detail": "port 1-1.3: no such device"}
  ]
}

• The checks only look. Console: the device node exists, is a char device and can be opened. The node also watches udev for its USB serial adapter, so a disconnect, a re-enumeration or errors on the bus show up in the check, even when the port is back by the time you ask. Power: the read command exits 0. SD-Mux: usbsdmux get answers. USB: the board’s port is on the bus. Nothing switches power, moves the mux or takes the tty from a client that holds it. A card in the middle of a flash reports busy.
• Call it when you need it: an agent that is unsure about a state, a CI job before it starts, a cron job every morning. The cron job is the one that finds the host that rebooted overnight.
• For hardware the node does not know (a PDU, a relay board, a hub with switchable port power), health_command in device.yaml runs your script: exit 0 is ok, anything else fails with its first output line as the detail.

power:
  health_command: /usr/local/bin/pdu-ping --port 4

An error the agent can follow

The error text is the agent’s only view of the rack, and agents follow instructions in it far better than they interpret a raw failure:

# invites a workaround
file_put failed: 500 exit status 1

# one the agent follows
file_put unavailable: lab fault (health: usb fail, port 1-1.3:
no such device). Do not work around it. Report it and end the task.

A lab fault must look different from a usage error, carry the reason, and say what to do next. That is MCP tool error handling for a lab. The same goes for tool descriptions: “interactive console input, not a file transfer path” costs one line.

Limits in the server, not in the prompt

A prompt is a suggestion. Every lease has a TTL the server enforces, so a dead agent session (closed laptop, full context, killed process) cannot hold a board overnight.

Small, visible faults

One node per board with its own power, console and mux: a crash takes one board, not the rack. One activity timeline per device for users, CI and agents. One health verdict for all of them: if the agent and Jenkins disagree about box 3, that is a bug.

What stays the same

None of this is about AI. Split infrastructure failures from test failures, find broken slots before they are used, keep faults to one board. If your CI board farm is stable, on labgrid, LAVA or a shelf of scripts, you are most of the way there.

The one new rule: an agent routes around a broken lab and calls it success. So the lab has to say “broken” in the API, where the agent is listening.

Fix the rack first. Then blame Claude.

Frequently asked questions

Give an agent one board, with the rack in view

Related reading