Low-Cost Robotic Manipulation

An affordable, language-driven robotic manipulation system built to make assistive pick-and-place accessible on a hobby budget. We show that reliable manipulation does not require a $20,000–$35,000 commercial arm — the SO-101 arm itself is ~$400, and a full system with a depth camera and edge compute added comes in around $600–1,000.

The problem

When a spinal or neurological injury impairs upper-body motor function, everyday tasks like eating, drinking, or retrieving a dropped object can become major challenges. Assistive robot arms exist, but adoption is limited by high cost and by the persistent unreliability of grasping on affordable hardware. Two failure modes dominate low-cost systems: object disambiguation (grabbing the wrong thing in a cluttered scene) and grasp execution uncertainty (attempting a grasp that was never going to work). The SO-101 has no force/torque sensing at all, so the system cannot feel a bad grasp coming — it has to predict one before it happens.

System architecture — verify before act

Rather than learn the whole task end-to-end in one model, we split it into two sequential, independently-testable stages: disambiguate, then verify. That decomposition is the core design decision of the project — it turns “the robot failed” into a diagnosable question of which stage failed.

Stage 1 — disambiguation. OpenAI Whisper transcribes the spoken command (keyboard fallback if speech isn’t recognized). YOLO-World, an open-vocabulary detector, matches the requested attributes (“blue cup,” “on the left”) directly against the scene with no fixed class list, so it doesn’t need retraining for new objects. We evaluated OWL-ViT as an alternative and rejected it — its 0.5–0.7 s per-frame latency was an order of magnitude too slow for live teleoperation, versus YOLO-World’s 35–50 ms. The detected bounding box center is then back-projected to a 3D robot-frame position using the Intel RealSense’s hardware depth stream and factory-calibrated intrinsics, and that position is injected directly into the language prompt SmolVLA receives.

Stage 2 — verify, then act. Before any motion command is issued, a fine-tuned Dex-Net GQ-CNN scores the candidate grasp for quality Q. If Q ≥ 0.5 the system proceeds — the target bounding box is drawn onto the RGB frame as a visual overlay and SmolVLA executes. If Q < 0.5, the arm repositions and the pipeline re-evaluates rather than attempting a grasp it already predicts will fail. This is the “verify-before-act” idea in one sentence: don’t let the robot find out a grasp was bad by dropping the object.

Perception stage — open-vocabulary detection fused with RealSense depth to localize the target in 3D before any motion.

The architecture change that mattered: 6D → 11D state

SmolVLA’s default input to the SO-101 is a 6-DoF joint-position vector — the arm’s own proprioception, nothing about the scene. Trained on that alone, the policy has no way to distinguish “the cup is on the left” from “the cup is on the right” except by memorizing whatever was most common in the demonstration data. That is exactly what happened: our first baseline, trained without any spatial grounding signal, collapsed to always moving left regardless of where the cup actually was — 50% success, indistinguishable from a coin flip.

The fix — and the project’s central architectural contribution — is to append the YOLO-World bounding box of the language-grounded target (center, width, height, confidence — 5 numbers, normalized to the image frame) onto the standardized 6-D joint state, producing an 11-D “state-plus-detection” vector. That vector replaces the original state at the input of SmolVLA’s existing state-token projection; nothing else about the transformer, the flow- matching action expert, or the training recipe changes. It is a five-number addition to one input tensor, and it is the highest-leverage change in the whole system.

Mechanistically, behavior cloning over teleoperated demonstrations tends to collapse toward the average trajectory when the training distribution is asymmetric — the flow-matching model minimizes denoising error by learning “go left” instead of learning “go to the target,” because that’s the lower-error answer on average. Conditioning on the live bounding box breaks that collapse: every demonstration now carries a distinct input signature tied to the actual target location, so the network is forced to learn the mapping from detection to action instead of a location-independent placement habit. One small implementation detail turned out to matter a lot here: when YOLO-World fails to detect anything, the missing bbox has to be signaled with a sentinel value. An all-zeros sentinel seemed natural but was actively harmful — after normalization it lands at roughly −2.5 standard deviations, which overlaps real extreme-position detections and produced a distinctive “arm dancing” failure. Switching to an all‑−1 sentinel (≈ −2.8σ, a value no real detection can produce) fixed it cleanly.

The result: bbox conditioning didn’t just improve accuracy, it eliminated wrong-object grasps entirely (50% → 0% wrong-object rate) and took overall task success from 50% to 88–90% on a single-object baseline, before any of the later camera or routing improvements.

Model routing: an expert per object, not one model for all

The natural next step — a single unified model for both objects — failed repeatedly. Across five iterations (different sentinel encodings, different bbox-injection strategies), a unified two-object policy consistently collapsed to a right-side placement bias, because the training data happened to place objects on the right slightly more often. This isn’t a SmolVLA-specific limitation; it’s a property of behavior cloning itself, and it would affect diffusion policies or ACT just as much.

The fix we shipped is structural rather than architectural: a voice- controlled routing wrapper that loads a separate single-object expert policy per target class and selects the correct one at runtime from the Whisper transcript and the YOLO-World detection. Each policy only ever sees one task, so it never has the chance to develop a cross-object placement bias.

Beyond fixing the bias, routing turned out to have practical advantages we hadn’t fully anticipated:

• Memory footprint scales with one task, not all tasks. Only the active expert needs to be resident in GPU memory, which is what makes this run on a single consumer GPU instead of requiring the union of every task’s capacity.
• Cheaper, faster training per expert. Each policy trains on task-specific data only, converges faster, and avoids the cross-task interference that degrades joint training.
• Incremental expansion. Adding a new object means training and registering one more expert, not retraining the whole system — the same logic as a plugin architecture in software.

Grasp verification: fine-tuning Dex-Net’s GQ-CNN

The verification gate is a Grasp-Quality CNN from Berkeley’s Dex-Net 4.0: a dual-stream network that reads a rotated 32×32 depth patch through a 4-layer CNN encoder, embeds the scalar approach depth through a small FC layer, concatenates the two, and outputs a single success probability.

Dex-Net’s public weights were trained on 6.7 million synthetic depth patches, and it shows on real hardware: the pretrained model behaves almost like a blind classifier on the D435i, assigning high quality to nearly everything (precision 0.31, recall 1.00 — it says yes to almost every grasp). We fine-tuned it on just 142 real grasp attempts collected on the SO-101 + D435i rig, freezing the convolutional encoder (179,520 parameters) and updating only the pose-embedding and classification head (17.85M parameters). Freezing the backbone preserves the low-level depth-edge features Dex-Net already learned at scale, while re-calibrating the decision boundary to our camera’s actual depth distribution. That lifted precision to 0.89 and AUC-ROC from 0.50 to 0.78 — a near-random classifier became a genuinely useful pre-execution gate, from 142 labeled examples.

Two smaller engineering fixes were needed to make the gate usable live rather than just accurate offline. The GQ-CNN samples 256 candidate grasps per frame and reports the best one, which is noisy frame-to-frame from depth jitter and bbox wobble — so we added a per-class lock-on: once a candidate clears Q ≥ 0.30, its pose is held and re-scored with EMA smoothing rather than re-sampled from scratch every frame, and released only if the target moves out of frame, quality collapses, or a clearly better candidate appears. Separately, on elongated objects like bottles the model would sometimes lock onto a physically-wrong parallel-closure axis; a geometry-aware angle prior biases candidate angles toward the bounding box’s short axis whenever its aspect ratio exceeds 1.2, which stopped that failure mode.

Hardware & environment

The full stack runs on the open-source SO-101 arm (~$400, 6-DOF, Feetech STS3215 servos) — the arm itself is the cheap part. A realistic full-system cost also has to include a wrist-mounted RGB camera, an elevated Intel RealSense depth camera (D435i, later upgraded to D456), and the edge compute needed to run detection and inference on-device, which brings a deployable system closer to $600–1,000 — still a fraction of a $20,000+ commercial wheelchair-mounted arm. The depth camera sits behind and slightly to the side of the arm. That placement wasn’t the first thing we tried — an earlier front-mounted depth camera suffered from the arm’s own links occluding its view mid-grasp, and moving the camera to an elevated side angle solved it outright. SmolVLA (500M parameters, 16 VLM layers, flow-matching action expert) was fine-tuned per task on an NVIDIA A100 via Google Colab and run locally on an RTX 5090 laptop GPU at 20–60 Hz effective control through LeRobot’s asynchronous inference. Voice commands go through OpenAI Whisper (large), and detection runs on YOLO-World’s yolov8s-worldv2 checkpoint.

Results: nine iterations to a working system

The reported numbers are the endpoint of an iterative process, not a single training run. A no-conditioning baseline scored 50% (positional bias). Adding the 11-D bbox state to a single-object policy jumped that to 88–90% with zero wrong-object grasps. Attempts at one unified two-object model failed repeatedly on the placement-bias problem described above — including one run where 3 corrupted double-length recording episodes out of 101 (3% of the dataset) were enough on their own to make the policy learn an oscillating, unusable trajectory, which is now a mandatory data-quality check before any training run. Splitting into two routed expert policies fixed the bias: cup 90%, bottle 74% on the original 15 fps webcam setup. Upgrading only the base camera to an Intel RealSense D456 at 30 fps — no changes to the model or training procedure — pushed that to cup 100%, bottle 92%, for a combined 96/100 (96%) task success across two objects and two spatial positions. Camera quality, in this project, mattered as much as anything in the model.

SmolVLA policy training (orange-cup task) — loss, gradient norm, and learning-rate schedule converging over 40,000 steps to a final loss of 0.016.

What we learned

A few findings surprised us enough that they changed how we built the rest of the system:

• The bbox is the whole trick. A five-number input change did more for reliability than any change to the underlying policy architecture.
• Unified multi-object policies fail on bias, not capacity. More parameters wouldn’t have fixed the right-side placement collapse — it’s a behavior-cloning property, and per-object routing sidesteps it entirely.
• Corrupted episodes are silently catastrophic. A 3% contamination rate in a flow-matching dataset was enough to produce an oscillating, unusable policy — verifying episode frame counts before training is now non-negotiable.
• Camera quality can outweigh model quality. The single largest jump in bottle-grasp accuracy (74% → 92%) came from a camera upgrade, not a model change.
• Confidence thresholds need tuning against real prompt overlap. At YOLO-World’s default confidence threshold, semantically similar prompts like “small cup” and “large cup” would sometimes both fire on the same physical object; raising the threshold to 0.40 resolved it.

Honest scope

The grasp-verification dataset is 142 labeled attempts — enough to move the GQ-CNN from useless to genuinely informative, but still small: the persistent gap between training and validation accuracy across fine-tuning runs is a textbook small-dataset overfitting signature. At the 0.5 decision threshold the fine-tuned model is also conservative to a fault, with 33% recall — it rejects roughly two-thirds of grasps that would have actually succeeded, which is safer than the alternative but leaves real performance on the table (a lower 0.35–0.40 threshold trades some of that precision back for usable recall). Per-object-class performance is uneven, too: bottle and small-cup detections separate cleanly, but the marker class — underrepresented in the 142 samples — shows almost no discriminative signal at all. The SO-101 has no force or torque feedback, so every one of these numbers comes from vision alone; results on a different arm’s kinematics and gripper are not guaranteed to transfer, and no formal usability studies with people who have upper-limb impairments have been run yet — that’s the natural next step, not something this prototype claims to have already validated.

Code

Code, datasets, and trained models are open-sourced in the Hummingbird repository.