Measure performance on your own data and real silicon before you commit.
Start an edge AI project by searching through a list of interestingly named models (e.g., YOLO, EfficientDet, RT-DETR, SqueezeNet) until you find one that performs your task with achievable accuracy numbers. Then, over the following weeks, build the application around that model, and hope when you get to hardware integration that you can reproduce the accuracy you expected on your target hardware. Fail this critical test, and you’ve lost a bet. The only options left to you are to retrain, move to a larger chip, or swap models and rework everything, all of which cost significant time and money. These failed bets are how embedded AI projects miss deadlines and go well over budget. You lacked critical information to make a better decision.
Teams are betting with the wrong information. On the cloud, published accuracy figures are usually measured at full precision on a GPU with an overabundance of memory and power overwhelming. This yields very specific data, but it answers a different question and won’t translate well to real-world conditions. That same model placed in an embedded accelerator must be quantized, sometimes pruned, and processed through a compiler stack that makes its own judgments about how operations correspond to the hardware. Each of those steps can affect both accuracy and latency, and the effects are especially pronounced in edge products requiring aggressive tradeoffs. Published numbers don’t predict a given model’s true performance at the edge.
The Cooper Model Garden on Ambarella’s Developer Zone addresses this very problem by giving developers a true picture of model performance when they select their models. The Garden consists of pre-trained models representing a broad spectrum of task categories that most edge applications rely on, including image classification, object detection, semantic segmentation, pose estimation, depth estimation, face detection and recognition, and vision-language models (VLMs). A developer building a people-counting application, for example, doesn’t need a PhD in object detection literature to find viable candidate models. Ambarella engineers vet and curate the models in the Garden, so developers start with models whose performance and accuracy on our hardware are known quantities.
Each model in the Garden is quantized and optimally pruned to improve performance and minimize the memory footprint. Removing weights from a trained network can make most developers nervous because pruning can reduce accuracy on most architectures. This intuition is correct. Most accelerators support fixed-pattern sparsity, so pruning must follow a rigid block-based structure. Eliminating blocks of weights to insert sparsity increases the chance that a really important weight is zeroed out to meet the hardware constraint.
If, instead, we eliminate only insignificant weights, then we have high confidence that the final model will remain accurate. Our CVflow architecture provides an unstructured weight sparsity solution in hardware. Thus, pruning can focus on removing weights that a given network can afford to lose rather than being forced to adhere to a pattern imposed by the accelerator. Networks on our architecture retain better accuracy at a given sparsity level than with a block-based solution. The benefits extend beyond a smaller model file and reduced DRAM transfer cost. The hardware bypasses zeros during computation, reducing latency and power. This solution is rare because it is difficult to implement well in silicon, but we solved it after working through many unexpected technical challenges and using some clever engineering tricks.
The best hardware requires the best software. Pruning accurately is not simply setting a compiler switch. Finding the right balance between data format, dynamic range, pruning budget, and memory allocation for a single model requires a lot of software development, experimentation, and investigation. While networks contain a lot of similar computational structures, each network has its own weights grown from a complex and opaque training process. Knowing which weights can be safely clipped off is a really interesting gardening task. Fortunately, we’ve already done this work for every model in the Garden.
We publish real-world performance benchmarks for every model. A developer can see how a given model performs on a specific Ambarella part before downloading anything, which means the shortlist of candidate models (and often the shortlist of candidate parts) forms from measured numbers rather than from figures produced on someone else’s hardware for an unrelated use case.
Even with our best efforts, there is one missing piece of information: how a model will run on a customer’s specific data. For example, a person-detector trained on well-lit street scenes may behave differently on a warehouse floor at night. The most efficient hardware, the best software, and the most bespoke curation cannot resolve that issue.
Enter Ambarella’s SDK, which includes evaluation utilities that run directly on our silicon rather than in a proxy environment. A developer can test a model on a specific board and measure accuracy under the production conditions, using the same runtime, the same compiler decisions, and the same memory constraints. Numbers produced this way measure reality in a way that simulated numbers from a desktop cannot match. Host-side testing has value early on when a developer needs to iterate quickly and wants fast feedback. To make the right bet, the numbers that inform a hardware decision must come from the hardware itself.
For VLMs, our utilities support standardized benchmarks including VQA, COCO retrieval, phrase grounding, and open-vocabulary detection, along with domain-specific metrics such as grounding precision and retrieval recall. VLMs are harder to evaluate than a detector, where a bounding box is either right or wrong.
An earlier post in this blog series argued that embedded AI projects lose the most time when code that looks correct fails to run correctly on the target device, and that avoiding this outcome requires verifying code against real device behavior instead of trusting a tool’s confidence in its own output. This same principle applies to model selection, which is why the evaluation process uses actual Ambarella silicon.
Why isn’t this evaluation methodology standard practice? Access to hardware is the core obstacle. Traditionally, running a real-world measurement meant having the evaluation kit in hand, which required a purchase order, a shipment, and a bring-up process, all before anyone knew whether the part was the right choice. Teams usually had to commit to the hardware first, then spend time and money determining whether they had made the right choice.
The Virtual AI Model Performance Benchmark Lab, built in collaboration with DeepEdge.ai, changes the traditional evaluation process by allowing developers to upload their own dataset and run benchmarks against Ambarella hardware through a secure device farm, without ordering a kit. The performance results are derived from real Ambarella silicon running the developer’s own data. This service is available at no cost to Ambarella customers under a freemium model.
DeepEdge.ai brought its end-to-end AI development platform to the collaboration, and the Lab pairs that tooling with access to Ambarella’s CVflow portfolio. A developer can establish measured performance on their own data, on the specific part they are considering, and then decide whether to commit. Evaluation stops being a way to confirm a decision that has already been made and becomes the basis for making it.
Ambarella’s edge AI parts share the CVflow architecture and a common SDK, which means that much of the work performed on one Ambarella part can move to another as a developer’s needs change. For example, a team that invested time in quantizing, tuning, and measuring a model on a mid-range chip can transfer that effort to a higher-end part smoothly. The model may need to be recompiled and the numbers confirmed, but the team is adjusting a known quantity rather than starting the evaluation process over from scratch. Scaling a VLM model higher or lower may require a different device sized appropriately for the network, but what carries forward is the accumulated understanding of how models behave on CVflow, which informs the new selection even when the model itself changes.
This flexibility and knowledge transfer matter more than they might appear. A developer’s product lines never remain static and must adapt as technology and markets change. A camera family, for example, may span several price and performance tiers that run the same core software, while a robotics platform may prototype on a development board but ship on a lower-power part. Developers require the ability to adjust their part choice quickly when a design shifts or a new business arrangement calls for a different variant. Ambarella’s shared architecture helps ensure that adjustments like these don’t require starting the evaluation from the beginning.
The alternative is to pay the full cost of a failed bet: lost development work and a slipping schedule. Move to a different chip vendor, or even change generations from the same vendor. Start over with a different accelerator paradigm, a different compiler, a different set of operators and limitations, and a completely new collection of quirks and bugs. That cost is easy to overlook when evaluating a single part in isolation, and ignoring it can cause product roadmap schedules to shift unexpectedly.
During a panel at the Embedded Vision Summit in May, I said that the architecture behind a compelling demo is almost never the architecture that ends up in production. Demos show that something is possible, but real products must work. They have to fit within a defined power budget, perform across a range of operating conditions, and sell at a price people will pay with a workable margin. A model that has been quantized to run on the target part, tested against the customer’s own data, and benchmarked on real silicon has answered questions that a demo setup isn’t designed to answer. The sooner this type of evaluation happens in a project, the more valuable it becomes. Every week of application work built on an unverified assumption is a week that may be lost. With Ambarella’s Cooper Model Garden and the evaluation tools in the Developer Zone, a team knows what they are committing to before they ever commit. Developers can browse the models, review the published benchmarks, and run their own evaluations at the Ambarella Developer Zone.