Our industry typically describes a device’s intelligence in terms of the inference it serves and, in the case of vision, how many frames per second those models process. As physical AI moves from machines that see and think to machines that take action in the real world, we need to focus on a machine’s decision-making process. This process must be considered during the system’s overall design phase, before anyone ever thinks about choosing a model or a processor.
Let’s think about a fictional example that will help illustrate this core argument. Imagine an inspection station positioned along a conveyor belt whose job is to check for defects in packages as they move past a mounted camera. The camera examines each package, a vision model verifies whether the seal is intact or any other anomalies exist, and if a container comes through with a problem, the model flags it. Once this happens, a mechanical diverter arm activates and pushes the package off the line and into a bin where it can be double-checked later by a line worker.
But what about in the real world?
Now let’s imagine the system experiences a delay during communications between the model and the mechanical arm, or fails to remedy errors. When latency or incomplete systems like this exist, a container might come through with a torn seal or a puncture in the side wall, and even though the model does its job and flags this package correctly as defective, the issues in the system create lag and low adaptability in its response. By the time the reject command (which would be correct) reaches the diverter, the container has already passed by. In this case, the system’s log would show a correct decision, and everything would appear to have functioned properly during an audit or review. However, the defective container would still be moving down the line unchecked. This forces us to take a close look at how we define success in these systems, which requires us to study and develop them in a continuous cycle.
In this example, locating a defect is simply one step in a much more complex undertaking. The system must also successfully match each image to the correct container and decide whether the image contains enough information to justify a decision. It must ensure that the option of rejecting the container is allowed at that moment, send the command quickly enough for the diverter arm to actuate, and then confirm the container’s location after all of this has occurred. In theory, all these steps can be executed correctly, and consistently, because the underlying technology is mature and well-tested. But in the real world, some image inputs will be unclear, and there may not be a second camera angle or other verification method included. These issues can then be compounded or obscured by other failures in the system, such as the diverter mechanism itself having a problem, which would require a maintenance alert and a plan for the containers that keep arriving while the arm is incapacitated. Problems like these are commonplace in complex automated systems today. Which means that all these cases must be planned for during the system’s overall design phase, because accounting for them after the system is installed is significantly more difficult and costly.
When designing systems like the ones we build today in our industry, I find it helps to work backward. Start with the real-world (not ideal) conditions the system will encounter during its deployment, including the precise costs of each mistake it might make and exactly how much time it is given to respond. Then trace your way back, starting from the outcome you need to achieve and ending with the specific information the system must have at its disposal in order to achieve it. That thought-exercise will result in what we can loosely consider the decision path. That path should begin with what the machine’s sensors capture and conclude with the action the machine (or other component in the system) will take. To ensure safe and predictable AI, it absolutely must cover the unclear situations, the predictable problems, the unexpected anomalies, the so-called “edge cases” that have often been treated as “out of scope” during previous generations of system deployments (refer to The Edge Cases Were Never Edge Cases for an example). The path must include a plan for these edge cases to be handed off to a more capable model or to a person, a process that could be described as “handoff escalation” or simply escalation.
Our earlier article, The AI That Runs the Physical World Looks Nothing Like Your Favorite Chatbot, argued that an edge AI system should be judged by what its models accomplish together, as a group. The idea of a “decision path,” which I am describing here, carries that argument one step further, to the software constellation that exists around the models. That software keeps track of what the system currently knows about the scene and schedules each task. It holds the rules about which actions are allowed, sends the commands that carry out each action, and validates the result afterward. When a team plans all this together, an operator can check the correctness of each outcome, such as—per our example—validating which container was rejected and where it ended up going.
A short list of answers will still leave room for errors
A compact decision model can be a good fit for a task like the example I described in the introduction. For the purposes of this discussion, a decision model is a classification model built to choose one answer from a short, fixed list of options given some input state, producing probabilistic point estimate of the model’s confidence for each option. For the inspection station in my example, the list might be (a) accept, (b) reject or (c) send for additional review. Because the list is brief and fixed, the model can be correspondingly simpler to connect to the rest of the system, and it often requires less computing power than a model designed to output a paragraph’s worth of explanatory text.
A decision model as defined above can still, of course, make a mistake by selecting an incorrect answer from its limited list of alternatives. The system installed in the inspection station may, for example, become confused when it encounters a package it hasn’t been specifically trained to detect, or it may base its answer on an input image that is a few frames too old. The possibility of errors like these—where the system picks the wrong option—means that the software running on the device that carries out decisions, which we call “the runtime” in this series, must be required to compare each answer the model proposes against what is happening in real-time, in the real-world environment, at that moment, before rendering a final verdict. When we design software in this way, we ensure that a model that selects, say, one of three answers will still seek the runtime’s approval before making a decision to take action.
To be clear, the confidence score that comes with each answer must be checked too. It is tempting to read a score of 0.9 as a 90 percent chance that the answer is correct; however, in “On Calibration of Modern Neural Networks,” presented at the International Conference on Machine Learning in 2017, Chuan Guo and colleagues showed that modern neural networks are often significantly overconfident (they can be underconfident as well). Engineers typically refer to the alignment between a decision model’s confidence and how often it is correct as “calibration.” A model is well calibrated when, of 100 decisions it makes at 90 percent confidence, about 90 turn out to be correct. It must be emphasized that calibration must be measured on data and images captured from the real-world work environment, because a model that was well calibrated on clean test images in a lab scenario may not stay that way when encountering challenges such as glare and dust. Even a well-calibrated score describes the odds across numerous decisions, and the next decision may still serve as the exception. This means that the design team for a given system must decide in advance which decisions the machine is empowered to make independently and which ones will require additional screening, whether that occurs via a more capable model or via human review.
The difficult cases always have a cost to consider
It is helpful to remember that each step in the decision path will run at a particular defined pace. Perception can run on every frame delivered by the camera, while the decision step itself will trigger only when a new situation arises (e.g., a new package arriving on the conveyor belt). And that same system will only call a more capable reasoning model in cases where the small model cannot make an accurate determination on its own, and a human being may intervene in an even smaller number of situations. Each step has its own unique characteristics, but what they have in common is that every step competes for processor time, memory, and (in cases where humans become involved) staff attention. This means that the design team for any given system must budget for these realities, including the inevitable moment when several challenging scenarios occur simultaneously.
Research on “selective classification” is concerned with analyzing models that are given the autonomy to decline cases they are unsure about. When discussing this concept, it is beneficial to separate two numerical values that are easy to conflate. The first is coverage, the share of cases the system decides on its own. The second is the error rate on the cases that the system decides. A system that declines a higher number of cases usually makes fewer mistakes on the ones it does choose to manage, but we cannot forget that every case a system declines has to go somewhere and be dealt with. Either a person reviews it, a more capable model handles it, or the system pauses and waits. Each of those possibilities has an associated cost measured in time, computing power or staff hours. I say all this simply to emphasize that the idea of escalation must always be considered part of the compute budget, and any time spent by human beings in the loop must be accounted for as well.
A low average escalation rate may seem positive, but it can obscure what happens in the real world, such as when several challenging cases arrive in quick succession. Continuing my example from the introduction, let’s imagine that a supplier unexpectedly switches to a reflective film on its packaging (remember we are using small edge models which require more fine-tuning maintenance), which creates a situation where dozens of containers in a row become hard to read. Suddenly, cases accumulate in the review queue, and packages back up on the conveyor belt as the shared processor is busy running the more capable model on the first item in line. The design team must determine exactly how the system will behave during unconventional but widespread situations like these. The team might, for example, build a system that relies on a second image captured from another angle, or one that diverts unclear containers immediately to a secondary holding lane, or one that slows down the conveyor belt automatically as standard operating procedure under certain predefined conditions.
Looking at where the decision path meets the underlying silicon
When we choose to build a device centered around a single system-on-chip, it means that multiple processes such as image capture, image processing, AI inference, object tracking, video encoding and the execution of decisions will all draw from a single pool of shared (and sometimes extremely limited) resources. It means that each execution step will produce data that the next step will rely upon for its own successful execution. And it means that a model that appears efficient when examined independently, outside the context of the overall system, may actually be extraordinarily inefficient in practice, if its output must be copied across the chip several times, for example, or if it consistently interrupts another task with a tighter deadline.
For decades, Ambarella has designed its processors around the practical, real-world costs of moving data within devices that are deployed in the field, where hard constraints influence performance. In When the Edge Is 400 Kilometers Up, we described the architectural designs that tend to create successful outcomes in power-limited devices like these. These architectures often include AI accelerators that are “purpose-built around the dataflow of neural network execution,” and their image signal processing and video encoding are “integrated tightly with the accelerator to keep off-chip memory bandwidth, the dominant driver of power consumption, under control.” In other words, the most successful architectures recognize that every round trip to the memory block, usually DRAM, includes a very real cost measured in both time and power consumption. The article called this design philosophy “the engineering thesis Ambarella has been pursuing for more than a decade,” and described Ambarella’s CVflow® architecture, now in its third generation, as one that “concentrates compute on the operations that dominate neural network execution while minimizing the DRAM traffic that drives power consumption in conventional designs.”
Les Kohn, Ambarella’s chief technology advisor, explained this line of reasoning in an August 2025 EE Times article, “Will Robots Use Reasoning At The Edge?” Today’s neural networks do most of their work through multiply-accumulates, in which the chip multiplies two numbers and adds the result to a running total. A vision network can repeat that step millions or billions of times for a single image, and as a consequence, many designers spend significant effort attempting to speed up their AI performance by reducing this count. However, Kohn pointed out the catch: “there may be things that you do to reduce the number of multiply-accumulates, but they create other bottlenecks in the process. It could be accessing DRAM or different bottlenecks getting data and moving data around the chip.” He went on to describe Ambarella’s design philosophy as algorithm-first: “When we’re designing, let’s say, the next generation of hardware, we are looking at the problems we ran into with all the algorithms we ran, so that when we implement the next generation of algorithms, we can take advantage of understanding how the hardware works.”
Put simply, Ambarella designs each new generation of its chips around what it learned running real algorithms on the previous one, which is the same principle I am advocating here for the system as a whole. A chip designed this way is built for the specific problem a decision path creates, in which many steps share one chip, one pool of memory and one power budget. When applied to a decision path, the algorithm-first approach leads to a series of vital questions such as the following: Which intermediate results must stay in memory so the next step can use them? Which tasks must run concurrently? How much memory is remaining when the more capable model is called to manage a challenging scenario? What happens if that model takes too long to respond? The answers to questions like these will help a design team determine whether a processor under consideration can capably support their application as a single coordinated system under real load, in the situations it will encounter in the field.
Measuring your desired outcome on the chip being evaluated
Bob Kunz, Ambarella’s chief architect, made the following point in Model Bets: Will Your Model Hold Up on Your Data?: “the numbers that inform a hardware decision must come from the hardware itself.” The same post warns of the risks of using benchmark conditions to evaluate model performance, offering the example of a person-detector trained on well-lit street scenes that “may behave differently on a warehouse floor at night,” going on to describe evaluation tools in Ambarella’s software development kit that “run directly on our silicon.” When building a decision path like those I describe in this article, designers must run the full workload on the target chip being evaluated, with perception and video processing included, and then observe how the system behaves over hours of operation, under the actual constraints of memory and power it will encounter when deployed.
Returning to the inspection station from the introduction, a test like this one is how the system’s designer would catch the delay between the model and the diverter arm before the system is installed, when correcting it is much simpler. Yes, the system’s log would show a correct detection and nothing that would reveal the underlying failure, which means a test that measured only the model would pass as successful. But a test of the whole decision path demands more engineering rigor than that. It asks whether each container ended up in the correct location before its deadline passed, and it accounts for the total elapsed time from the perception step’s processing of the critical image to the moment the diverter arm moved. Unclear cases, missed deadlines, and required actions that never completed must each be recorded separately, because each one calls for a different remedy.
Working this way provides everyone involved, from model developers to product managers, with a shared methodology for evaluating performance, selecting hardware, and defining application requirements with accuracy. Each choice must improve the machine’s ability to complete its assigned task under these predefined conditions. So before selecting a model or a chip, ask yourself what specific actions the machine is required to accomplish, and exactly what should happen when accomplishing these actions is impossible. The next post in this series will discuss which of the machine’s decisions should run on the device itself and which can be sent to a server nearby or to the cloud.