Why cloud-controlled robots still need local intelligence

Key takeaways

  • Phillip Isola of MIT published “Robot-use agents,” arguing that a language model could use a robot much as it uses a calculator or computer. In his proposed setup, the robot would expose its sensor and actuator APIs to the agent, sending sensor data to the agent and receiving actuator commands from it.

  • Anthropic’s Frontier Red Team measured the latency problem. Real-time control of its quadruped setup would require roughly 83 Hz, whereas current non-reasoning inference runs at about 0.2-0.4 Hz. Typical turns took 2 to 8 seconds for text and 5 to 15 seconds with images, rising to 15 to 60 seconds with high reasoning.

  • The interface matters as much as the model. Direct manipulation succeeded on only 0-5.5% of tasks, while a pretrained policy improved every model tested. The combined systems still trailed the policy on standard tasks, but the strongest models improved performance on novel tasks that the policy could not solve on its own.

Most robot intelligence today runs on the robot itself, with models and controllers developed around a particular type of hardware. Intelligence is customized for different robots, integrated with the machine, and deployed incrementally. That makes adding new capabilities relatively slow because changes have to fit the robot, its sensors, its actuators, and its existing control stack.

On September 7, Phillip Isola of MIT published a short essay describing another path. Just as a language model can use a calculator, web search, or an entire computer, it could use a robot as a tool. Isola describes the model as a “puppeteer” controlling a robot body and argues that any internet-connected robot could potentially become accessible to a general-purpose agent. The intelligence could run in the cloud without being trained specifically for one robot. In a footnote, he describes the required interface: expose the robot’s sensor and actuator APIs, send sensor data to the model, and return actuator commands to the machine. 

The main constraint is speed. Anthropic’s Frontier Red Team estimated that real-time control of a Unitree Go2 requires roughly 83 Hz, while current non-reasoning inference runs at only 0.2 to 0.4 Hz. In direct-control tests, the simulator had to pause between model calls because the models were too slow for physics loops running at 10-125 Hz. Text-only turns typically took 2 to 8 seconds, rising to 5 to 15 seconds with images. High-reasoning runs on Claude Opus 4.6 and 4.7 usually took 15 to 60 seconds, with some taking up to 180 seconds.

The study also shows how much the interface matters. In direct manipulation tests on a 7-DoF Franka Panda arm, full task success ranged from 0 to 5.5%. When the model instead supervised a pretrained vision-language-action policy, success improved across all models tested. On the standard LIBERO benchmark, every combined system still trailed the policy running alone, but on three novel tasks where the policy failed on its own, Opus 4.5, Opus 4.6, and Gemini 3.1 supervised it to success. Smaller interface changes also had large effects. An interactive cursor raised Mythos Preview’s success from 6% to 32% on a ten-task subset, while a simple compass outperformed other perceptual aids in quadruped navigation.

The results point toward two control loops running at different speeds. A local controller handles balance, contact, obstacle response, and other time-sensitive actions, while the cloud model interprets goals, reasons over observations, and supervises higher-level behavior. Anthropic does not prescribe this architecture, but its experiments show why it works. Current models are too slow for continuous physical control and perform better when a dedicated policy handles the low-level physics.

That split also defines the safety boundary. The robot needs to know what to do if the connection slows or drops, while the remote agent should only be allowed to control specific objects, tasks, or areas. Sending sensor data to the cloud also raises questions around bandwidth, privacy, and reliability. The same boundary can provide an audit trail showing what the local policy suggested, what the remote model changed, and which action the robot ultimately took.

In one run, Grok 4.1 Fast mistook a table reflected in a glass door for the actual target and moved toward the door. In another, a model misjudged a trash can, collided with it, and dragged it several meters. No model completed a full office loop using vision alone.

Isola is describing where robotics could go, while Anthropic’s results show the engineering constraint that matters today. The key question is which decisions can tolerate seconds of latency and which must happen within milliseconds. Balance, contact, and immediate safety need to remain local. Planning, interpretation, and supervision can run in the cloud. As models improve, that boundary will move, but deciding where to draw it will remain one of the most important choices in connected robotics.

The Field Report
  • Hai Robotics has been selected to automate a new e-commerce fulfillment center for an unnamed European fashion retailer, in what the company says will be the world’s largest deployment of rack-climbing warehouse robots. The project will use more than 1,500 HaiPick Climb robots across a 30,000-square-meter operation with 1.2 million double-deep storage locations. Hai Robotics says the system will support throughput of more than 24,000 totes per hour and will use a goods-to-person model designed for high-volume omnichannel fulfillment.

  • General Robotics said its GRID platform now automates its own engineering work, which cuts the time to get a new robot into production from about a month to as little as 2 hours. In laboratory tests, the system built and transferred manipulation skills between Flexiv Rizon and UR5e arms, generated synthetic training data, incorporated human demonstrations, and diagnosed hardware issues during deployment. In one test, GRID identified a 143 mm and 6.4° mismatch between its assumed robot model and controller measurements and reduced the discrepancy to 5.7 mm and 0.68° through iterative correction.

  • Vention has introduced MachineAgent, an agentic AI layer for designing, programming, monitoring, and troubleshooting industrial automation systems. Users can generate automation cell layouts from plain-language prompts, create or debug MachineLogic applications, query production performance, and pull machine logs to diagnose issues such as sensor or cable faults. MachineAgent works through Vention’s CLI and MCP server, allowing teams to use coding agents including Claude Code, Cursor, VS Code, and OpenAI Codex.

  • Antioch has raised $32 million in new funding, bringing its total funding to $40.5 million, as it expands a development platform for robotics and other physical AI systems. The platform allows engineering teams to test perception models, planners, controllers, sensors, and hardware across large numbers of simulated scenarios before deploying them to physical systems. Antioch says customers are using the platform for continuous evaluation, synthetic data generation, and failure analysis across manufacturing, drones, autonomous vehicles, and robotics. The company is also integrating with NVIDIA Omniverse, Isaac Sim, and Isaac Lab, while working with Nebius on simulation infrastructure.

  • Stellantis Pro One and UQI Robotics are collaborating on Box-on-Wheels, an autonomous logistics vehicle designed for last-mile delivery. The project combines Stellantis’ commercial-vehicle platform with UQI Robotics’ autonomous-driving, AI, and robotics technologies. The companies will test how autonomous operation, electrification, connectivity, and related services can work together in commercial logistics. The vehicle is scheduled for its public debut at IAA Transportation in Hanover.

Built on NATS

Form3 operates a cloud-native Payments-as-a-Service platform that connects banks with external payment networks, including SWIFT, BACS, Faster Payments, and SEPA. Each network has its own message formats, retry logic, and security requirements, so Form3 provides a single API that translates customer payment requests into the formats required by the underlying rails. The application is written in Go and runs as loosely coupled microservices on Kubernetes, with Linkerd handling service routing and observability.

Form3 originally used AWS SNS and SQS together as its pub-sub layer, but that architecture created constraints as the company expanded. Form3 guarantees payment processing within 500 milliseconds, while some customers also wanted protection against the loss of a single cloud provider. Faster Payments introduced another requirement because parts of that service had to run in dedicated physical infrastructure connected through leased lines. Form3 therefore needed an event bus that could deliver low-latency messaging across public clouds while remaining lightweight enough to run within resource-constrained physical data centers.

The engineering team tested NATS and NATS Streaming as replacements for the SNS and SQS combination and found that NATS met its requirements for latency, reliability, resilience, and compute usage. Form3 subsequently replaced both AWS messaging services with NATS, using JetStream for persistent messaging. Message latency fell from around 300 milliseconds to below 50 milliseconds, while smaller NATS instances processed more transactions than the previous architecture.

JetStream became particularly important for Faster Payments, where Form3 needed configurable persistence within its dedicated data center environment. Its relatively small CPU and memory footprint allowed the company to run persistent messaging within the limited infrastructure available there while supporting a higher payment volume. JetStream also gives Form3 control over delivery and persistence for payment-related message streams that cannot rely on the at-most-once delivery model used by NATS Core.

Form3 also began using NATS as the foundation for a cloud-agnostic architecture. The plan combines JetStream with NATS Leaf Nodes to connect deployments across different environments, allowing workloads to move between clouds and supporting an active-active model without tying payment availability to a single provider's SLA.

For The Commute

Jan Liphardt on Building the AI Brain for Humanoid Robots (Automated with Brian Heater)

OpenMind founder Jan Liphardt explains why humanoid robots can look highly capable in demos but still struggle to do chores in everyday environments. After buying a humanoid robot for his own home, he found that getting it to stand was easy compared with making it understand people, navigate unpredictable surroundings, and respond naturally. He argues that the final few percent of deployment often requires most of the work, because robots have to deal with everything from traffic and animals to body language and changing environments. The conversation also explores how robotics companies can make robot decisions easier to inspect and debug, and why different physical tasks may require very different approaches to training and data.