Cactus Is Putting AI Back in Your Pocket
A two-founder startup out of Y Combinator's Summer 2025 batch is betting that the smartest place to run AI isn't a datacenter - it's the phone already in your hand.
The pitch for the last five years of artificial intelligence has pointed in one direction: up. Bigger models, bigger clusters, bigger bills. Cactus, a company that went through Y Combinator's Summer 2025 batch, is making the opposite argument. The most useful place to run a model, its founders say, is not a rack of GPUs a thousand miles away. It is the phone in your pocket, the watch on your wrist, or the glasses on your face - hardware that is already paid for and already in your hand.
Cactus is an open-source AI inference engine, written from scratch in C and C++, that runs language, speech, and vision models directly on smartphones, laptops, wearables, and other low-power devices. It does not need a server to think. A model loaded through Cactus answers in well under a tenth of a second, keeps the user's data on the device, and keeps working when the signal drops. For a developer, that is four hard problems - latency, privacy, offline access, and cost - solved by moving the computation rather than adding to it.
The IdeaWhy run AI on the device at all?
Start with the frustration every AI app shares. You type a question, the app sends it to a data center, the data center runs the model, and the answer travels back. That round trip takes time - often more than a second - and it repeats for every request. Each one is a paid API call, so the bill grows in lockstep with the number of users. And the whole scheme collapses the moment the network does: on a plane, in a tunnel, in a rural clinic with one bar of signal.
Cactus removes the round trip. The model sits on the device, so the answer is generated locally. The company reports that this drops latency from over a second to under 100 milliseconds, the difference between an app that feels sluggish and one that feels instant. Because nothing leaves the device, privacy stops being a policy and becomes a property of the architecture. And because there is no per-call server cost, the economics stay flat as an app scales from a thousand users to a million.
The ProductOne engine, many devices
The core of Cactus is the inference engine. It can load any GGUF model - the common format used to package open models like Qwen, Gemma, Llama, DeepSeek, Phi, and Mistral - and run it from full precision down to two-bit quantization, the aggressive compression that lets a capable model fit on a modest phone. A technique called zero-copy memory mapping keeps RAM use low and model loading close to instant, which matters when the hardware you are targeting is a mid-range Android rather than a server.
Around that engine sit the SDKs, and this is where Cactus courts the people who will actually use it. There are bindings for Flutter, React Native, Kotlin and Java, and Swift and Objective-C, so a mobile developer can add on-device AI to an app with a few lines of code in the framework they already work in. The SDK also exposes MCP tool-calls, letting a local model reach device-native functions - setting a reminder, searching the photo gallery, sending a message - so the AI can do things, not just talk.
Any GGUF model from Hugging Face, from FP32 to 2-bit.
Flutter, React Native, Kotlin, or Swift bindings.
Text, speech, and vision inference, fully local.
Hard tasks route to the cloud, automatically.
The engineering choices point at the same target from different angles. Writing the engine in C and C++ rather than a higher-level language is a deliberate cost: it is harder to build and harder to maintain, but it is the only way to wring usable speed out of a phone's processor and its NPU. Supporting quantization all the way down to two bits is another. Each drop in precision shrinks the model and speeds it up at some cost to quality, and giving developers the full dial - rather than one blessed setting - lets them trade accuracy for footprint based on the device they are actually shipping to.
That last step is the pragmatic part. Cactus is not dogmatic about keeping everything local. It ships hybrid routing that sends simple or sensitive tasks to the device and complex ones to the cloud, with an optional on-device-only mode for anyone who wants a hard guarantee that no data leaves. In practice, the company says, more than 80 percent of production transcription and language inference is handled on-device, with the cloud reserved for the tasks that genuinely need it.
The company has also started building its own small models to prove how far down the ladder this can go. The clearest example is Needle, a 26-million-parameter tool-calling model distilled from Google's Gemini - small enough to run on a smartwatch. A model that size will not write an essay, but it can reliably decide which function to call and with what arguments, which is exactly the job a wearable assistant needs done on a tight power budget.
Needle is a 26M-parameter model distilled from Gemini. For scale, that is thousands of times smaller than the frontier models it borrows from - shrunk deliberately so it can run on a watch, not a data center.
The FoundersTwo people who left comfortable jobs
Cactus is run by Roman Shemet and Henry Ndubuaku. Shemet, the chief executive, came from a background in quant finance and economics, with product and data engineering work along the way; outside the company he is a pilot, a triathlete, and a chess player. Ndubuaku, the chief technical officer, left an AI and machine-learning role at Nvidia - a company whose entire business is selling the cloud compute that Cactus is trying to make less necessary - to help build the alternative. The broader team draws from Oxford, Google, AWS, Salesforce, and MIT, which reads less like a marketing line and more like a description of who you need to write a fast inference engine in C++.
The BusinessFree to start, licensed to scale
Cactus follows the open-core model that has worked for a generation of developer tools. The framework itself is free and open source - a plus for adoption, since developers can read every line, and the project has drawn more than 4,200 stars on GitHub. Hobbyists, students, non-profits, and small businesses use it at no cost. Commercial and enterprise users pay for a license, reported as custom pricing on a per-monthly-active-device basis, with enterprise tiers adding hardware acceleration and cloud telemetry on top.
It is a clean fit with the product's own pitch. The whole reason a company adopts Cactus is to escape a metered cloud bill that scales with usage; charging per active device, rather than per inference, keeps the vendor's incentive aligned with the customer's. The more an app leans on local inference, the more both sides save.
The early traction is the kind investors like to see before a product is widely known: deployments already running Cactus serve more than 500,000 inference tasks a week, with named implementations including AnythingLLM and KinAI. Those are real apps doing real work on users' own hardware, not demo counters. The company raised on Y Combinator's standard Summer 2025 terms and is based in San Francisco.
The MarketCompeting with the platform owners
On-device AI is not an empty field. Cactus sits alongside heavyweight tooling: llama.cpp, the open-source project that popularized running LLMs on consumer hardware; Google's AI Edge and LiteRT stack; Apple's on-device Foundation Models; PyTorch's ExecuTorch; ONNX Runtime Mobile; and MLC LLM. Several of those are backed by the companies that also make the phones and chips. Competing there means being genuinely better at the boring, hard part - squeezing performance out of constrained silicon across many devices - and being easier to adopt than a platform-locked SDK.
| What matters | Cloud AI | Cactus (on-device) |
|---|---|---|
| Latency | >1 second round trip | Under 100ms, local |
| Privacy | Data leaves the device | Data stays on the device |
| Offline | Needs a connection | Works with no signal |
| Cost at scale | Per-call API bill | Flat, per active device |
| Reach | Any client | Phones, laptops, wearables, edge |
Cactus's answer is breadth and openness. One engine covers iOS, Android, macOS, and wearables from a single toolkit, it is model-agnostic across the GGUF ecosystem rather than tied to one vendor's zoo, and its code is fully auditable. For a startup shipping AI to millions of phones, where every cloud call is effectively a tax and every second of latency costs engagement, that combination is less a nice-to-have than a line item that decides whether the product's unit economics survive contact with scale.
What You Can BuildFrom smart glasses to a clinic with no Wi-Fi
The uses fall out naturally from the constraints Cactus removes. A voice assistant that answers in under 150 milliseconds because it is not waiting on a server. A meeting-transcription tool on a laptop that never uploads what was said. Live captions on smart glasses, running on a chip that sips power. A smart-home command router that keeps working when the internet is down. Health and finance apps whose entire compliance story is that the sensitive data physically cannot leave the phone.
There is a strategic lesson buried in Cactus's choice, and it is worth stealing. When a market gets crowded, the durable position is often inside the constraint everyone else is trying to escape. Cloud AI is where the attention and the capital have gone. The edge - small models, tight memory, no network, cheap silicon - is harder, less glamorous, and far less contested. Cactus walked toward the hard part on purpose, and turned the constraint into the moat.
Whether that bet pays off depends on the same thing every infrastructure bet depends on: whether enough builders adopt it that it becomes the default. The signals so far - an open-source project with a real star count, half a million weekly inferences in production, and a founding team that left the cloud's biggest beneficiary to build the alternative - suggest the wager is a serious one. The phone in your pocket has been a supercomputer for years. Cactus is one of the companies finally treating it like one.
It also arrives at a moment when the hardware is ready to meet it. The neural processing units now standard in flagship phones were, until recently, underused silicon looking for a job. Cactus is one of the tools that gives them one, and its own benchmarks show the payoff: a four-billion-parameter model running on an NPU where a CPU would have struggled. As those chips trickle down from flagships into mid-range devices over the next few product cycles, the ceiling on what can run locally rises with them - and a framework already tuned for that hardware is positioned to ride the curve rather than chase it.