Every entry here was hit on the reference robot. Symptom → cause → fix.
Logs live under /tmp/mabel_*.log on whichever machine runs the
stack; the cockpit tails the important ones live.
The four that bite everyone
Tap a panel for the long answer — the figure, the commands, and where in the
wiki it is written up.
Connectivity
Symptom
Cause → fix
Headset gets video but says disconnected (Mac sim)
macOS firewall blocks incoming :9090 for Python. Allow it:
System Settings ▸ Firewall ▸ Options ▸ allow python3.
The launcher detects and prints the exact command.
Port in use at launch
a previous run's listener survived —
lsof -ti :9090 :8080 | xargs kill -9 (the launcher
normally reaps these itself).
Robot pingable but every app "connects to nothing"
the bring-up wedged before opening its ports — check
/tmp/mabel_bridge.log and restart
real_run.sh.
LAN dead but Tailscale fine (Jetson)
Wi-Fi NIC power save. Disable powersave on the wlan interface
(NetworkManager wifi.powersave 2).
App shows a Sim badge on the real robot
the server identity contains "sim" — make sure the real launcher set
MABEL_DISPATCH_TARGET=real:ros (it does by default).
Base & Teensys
Symptom
Cause → fix
Base Teensy unreachable
the launcher probes every carrier-up wired port (ARP) and pins a
route automatically. If it still fails:
ethtool <port> — "Link detected: no" means cable
or Teensy power; then
ping 192.168.124.10 per-port. 100M link = Teensy direct;
10G = it's behind the Ethernet expansion board.
Teensy reachable yesterday, gone today after re-plugging ports
a stale pinned /32 route on the old port blackholes the IP — the
launcher now strips these itself; manually:
nmcli device modify <old-port> -ipv4.routes ….
Permanent heading drift when driving straight
steer zero offsets not captured, or a swerve module mounted
twisted — recalibrate; if one module is mechanically rotated,
re-square it (02 · Base).
Arms & hands
Symptom
Cause → fix
Arms limp after connecting
by design. Arms stay limp until an operator enables torque
from a client. Not a fault.
Left and right hands swapped
the two CH340 adapters are identical — register each adapter's USB
serial to its side in hardware_bridge/mabel_hw/config.py;
never rely on port order.
One whole hand dead
all 17 servos share one TTL chain — a single bad connector drops the
hand. Scan the chain (firmware/orca_hand/feetech/scripts/scan.py) and check the first
unreachable ID's connector.
Fingers miss their targets after a rebuild
tendon slack read as calibration error. Re-tension, then
re-run the hand end-stop calibration — in that order.
Hands render contorted in a viewer while real fingers are fine
the wire streams absolute joint positions; clients must render
qpos − ref for the 14 ORCA joints with nonzero
reference. Fix the client, never the stream.
Cameras & sensing
Symptom
Cause → fix
RealSense on USB but no /dev/video*, SDK sees 0 devices (Jetson)
librealsense's RSUSB udev rule unbinds uvcvideo. Permanent fix:
sudo mv /etc/udev/rules.d/99-realsense-rsusb-unbind.rules{,.disabled}
&& sudo udevadm control --reload && sudo udevadm trigger.
The watchdog also self-heals each plug.
Wrist cameras swap sides across reboots
identical UVC devices enumerate by arrival — pin them by hub port on
the powered hub (udev rules in hardware_bridge/install/).
Cameras dark in the sim
an empty scene — pass a real scene:
./run.sh -s path/to/scene.xml.
Performance
Symptom
Cause → fix
Joints laggy / control loop slow while video streams
camera encode threads starving the control loop — the launcher
already pins cameras to utility QoS; don't run extra viewers on the
robot host, and prefer LAN over relay for video.
Teleop feels 100+ ms behind
check transport first (cockpit's status row shows each client's
route). Relay adds an internet round trip; LAN/Tailscale-direct is
the low-latency path.
Sim robot won't move under hand teleop
both hands must be tracked, then anchor the clutch (Enter
in the cockpit / recenter in the headset). Watch
palm err L/R → 0 in the log.
When stuck
Logs: /tmp/mabel_real_hw.log, /tmp/mabel_bridge.log,
/tmp/mabel_hw_bridge.log, /tmp/mabel_real_slam.log —
the launcher prints which log belongs to which subsystem at start.
Per-bus liveness: the watchdog line (every 10 s) tells you exactly which
device stopped answering.
Open an issue at
robotmabel/open_mabel
with the relevant log tail and your build tier.
FIX!
Do I need hardware to start?
No — and you shouldn't start with it. The entire stack runs against the
MuJoCo twin on a laptop: cd server && ./run.sh boots physics, cameras,
the teleop gateway and a browser control studio. Phones, headsets and browsers
connect to the twin exactly as they later connect to the robot — same
ports, same wire contract, and the clients can't tell which is live.
Prove every motion in the twin first, then order parts. 05 · Software →
A motor won't respond
Work the bus, not the motor:
① Bus up? ./can_up.sh then ./can_scan.sh — 1 Mbit everywhere.
② Termination: ~60 Ω measured across CAN-H/L (two 120 Ω ends in parallel).
③ ID clash: assign IDs one device powered at a time — expect 6 on the base,
7 per arm, 17 per hand chain.
④ First unreachable ID in a chain = check that connector; everything
downstream vanishes with it.
The two wrist boards are identical, so USB enumerates them in arrival
order — left becomes right after a reboot. Pin each camera by its hub
port (udev rules ship in hardware_bridge/install/) and keep
both on one powered USB-3 hub.
Slow or stuttering? Drop to 640×480 — teleop wants latency, not pixels; the
measured glass-to-glass budget is dominated by encode, not resolution.
The MJCF (mabel_full.xml) is the single source of truth —
the URDF, the web GLB, the phone/headset rigs and the tip-over constants are all
generated from it. Edit the model and skip the regeneration, and every
viewer silently keeps showing the old body.
Re-run the pipeline in order (MJCF → URDF → GLB → rigs → mirror),
then re-check the CoM constants. Never hand-edit a derived file.