09Troubleshooting

Every entry here was hit on the reference robot. Symptom → cause → fix. Logs live under /tmp/mabel_*.log on whichever machine runs the stack; the cockpit tails the important ones live.

Connectivity

SymptomCause → fix
Headset gets video but says disconnected (Mac sim) macOS firewall blocks incoming :9090 for Python. Allow it: System Settings ▸ Firewall ▸ Options ▸ allow python3. The launcher detects and prints the exact command.
Port in use at launch a previous run's listener survived — lsof -ti :9090 :8080 | xargs kill -9 (the launcher normally reaps these itself).
Robot pingable but every app "connects to nothing" the bring-up wedged before opening its ports — check /tmp/mabel_bridge.log and restart real_run.sh.
LAN dead but Tailscale fine (Jetson) Wi-Fi NIC power save. Disable powersave on the wlan interface (NetworkManager wifi.powersave 2).
App shows a Sim badge on the real robot the server identity contains "sim" — make sure the real launcher set MABEL_DISPATCH_TARGET=real:ros (it does by default).

Base & Teensys

SymptomCause → fix
Base Teensy unreachable the launcher probes every carrier-up wired port (ARP) and pins a route automatically. If it still fails: ethtool <port> — "Link detected: no" means cable or Teensy power; then ping 192.168.124.10 per-port. 100M link = Teensy direct; 10G = it's behind the Ethernet expansion board.
Teensy reachable yesterday, gone today after re-plugging ports a stale pinned /32 route on the old port blackholes the IP — the launcher now strips these itself; manually: nmcli device modify <old-port> -ipv4.routes ….
Permanent heading drift when driving straight steer zero offsets not captured, or a swerve module mounted twisted — recalibrate; if one module is mechanically rotated, re-square it (02 · Base).

Arms & hands

SymptomCause → fix
Arms limp after connecting by design. Arms stay limp until an operator enables torque from a client. Not a fault.
Left and right hands swapped the two CH340 adapters are identical — register each adapter's USB serial to its side in hardware_bridge/mabel_hw/config.py; never rely on port order.
One whole hand dead all 17 servos share one TTL chain — a single bad connector drops the hand. Scan the chain (precheck.py) and check the first unreachable ID's connector.
Fingers miss their targets after a rebuild tendon slack read as calibration error. Re-tension, then re-run the hand end-stop calibration — in that order.
Hands render contorted in a viewer while real fingers are fine the wire streams absolute joint positions; clients must render qpos − ref for the 14 ORCA joints with nonzero reference. Fix the client, never the stream.

Cameras & sensing

SymptomCause → fix
RealSense on USB but no /dev/video*, SDK sees 0 devices (Jetson) librealsense's RSUSB udev rule unbinds uvcvideo. Permanent fix: sudo mv /etc/udev/rules.d/99-realsense-rsusb-unbind.rules{,.disabled} && sudo udevadm control --reload && sudo udevadm trigger. The watchdog also self-heals each plug.
Wrist cameras swap sides across reboots identical UVC devices enumerate by arrival — pin them by hub port on the powered hub (udev rules in hardware_bridge/install/).
Cameras dark in the sim an empty scene — pass a real scene: ./run.sh -s path/to/scene.xml.

Performance

SymptomCause → fix
Joints laggy / control loop slow while video streams camera encode threads starving the control loop — the launcher already pins cameras to utility QoS; don't run extra viewers on the robot host, and prefer LAN over relay for video.
Teleop feels 100+ ms behind check transport first (cockpit's status row shows each client's route). Relay adds an internet round trip; LAN/Tailscale-direct is the low-latency path.
Sim robot won't move under hand teleop both hands must be tracked, then anchor the clutch (Enter in the cockpit / recenter in the headset). Watch palm err L/R → 0 in the log.

When stuck