Software used to have the decency to break in recognizable ways. A button failed. A server returned an error. A test turned red and sent an engineer back to the offending line. Large language models introduced a more polished species of failure: the answer arrives on time, sounds certain, and may be magnificently wrong. Jeffrey Ip has built his career around making that uncertainty less theatrical and more measurable.
Ip is the co-founder and CEO of Confident AI, the San Francisco company behind DeepEval, an open-source framework for testing LLM applications. The shorthand is "pytest for LLMs," which is both accurate and insufficient. Pytest asks whether known software did the known thing. DeepEval must judge systems that vary their wording, take different routes through a task, call tools, retrieve documents, and occasionally produce nonsense wearing a tie.
The premise is sober: AI quality should be part of the engineering loop. Define what good looks like. Assemble test cases. Measure outputs. Compare prompts and models. Catch regressions before release. Then watch production, because an application that behaved yesterday has not signed a contract to behave tomorrow.
The education of a reliability founder
Before founding Confident AI, Ip worked inside two companies where scale is not a slogan but a physical condition. At Microsoft, he helped build recommendation systems for Office 365 users. At Google, he worked on infrastructure for YouTube's Creator Studio. One job dealt with suggestions across a productivity suite; the other supported tools used by millions of creators. Both placed him close to the invisible machinery that lets a mass-market product feel ordinary.
His public education record lists Imperial College London from 2019 to 2022. The more revealing curriculum came afterward: distributed systems, recommendation software, and the operational discipline required when tiny errors encounter enormous audiences. By the time generative AI apps began appearing in every pitch deck, Ip had seen what grown-up infrastructure looked like. The new applications were dazzling. Their quality controls were often artisanal.
DeepEval first appeared in 2023. Rather than ask developers to adopt an alien quality process, it borrowed the grammar of software testing. Tests could live near code. Evaluations could run in continuous integration. Teams could use built-in measures for answer relevance, faithfulness, hallucination, tool use, task completion, and other behaviors, or define criteria for their own products. The interface mattered because habits travel more easily in familiar clothes.
The evaluation loop
That familiarity concealed a new kind of work. An ordinary assertion might compare two numbers. An LLM evaluator may need to decide whether a response answered the user's actual question, stayed grounded in supplied context, selected the right tools, and completed a multi-step job without wandering. Some checks can be deterministic. Others use language models as judges. The challenge is to make those judgments repeatable enough to guide engineering decisions rather than decorate a dashboard.
The product is an argument for moving AI teams from vibes to evidence.
Open source as the opening move
Ip and co-founder Kritin Vongthongsri built the business in two layers. DeepEval is the open-source foundation, available for engineers to inspect, run locally, and place in their workflows. Confident AI is the collaborative platform around it: datasets, experiment reports, observability, red teaming, and governance for teams that need a shared quality system. The arrangement gives developers a useful tool before asking their company to buy anything.
It also creates a demanding audience. Open-source users do not merely applaud. They file issues, challenge abstractions, ask for integrations, and reveal the cases a small internal team would never imagine. That pressure becomes product research. DeepEval's repository now spans metrics for retrieval systems and agents, synthetic data generation, tracing, benchmarks, prompt optimization, and integrations with popular AI frameworks. DeepTeam extends the thesis into adversarial testing: reliability includes knowing how a system behaves when somebody tries to make it misbehave.
The public trail shows Ip moving between founder and maintainer modes. One day brings a product explanation about automated error analysis. Another brings release work on tracing, simulation, model support, or documentation. His writing is technical without pretending the technology is settled. His author bio jokes about an "unhealthy LLM evals addiction," a compact admission that category building requires repeating the same question until the market learns to ask it too.
Five days, fifty-five calls
Confident AI entered Y Combinator's Winter 2025 batch with something more persuasive than a theoretical market: developers were already using the open-source project. Ahead of Demo Day, Ip scheduled 55 investor calls. His stated objective was wonderfully free of perfume: close the round quickly, avoid raising more than the company could sensibly use, and return to building.
The week began on Sunday night with three calls and $500,000 in commitments. Ip briefly imagined the process might be easy. Monday corrected him. Twenty calls produced no yeses. Investors who had promised quick replies sent rejections that same evening. Tuesday offered one hopeful partner meeting, so Ip and Vongthongsri rented a conference room on New Montgomery Street for $100 an hour. They got the commitment. Ip, with the pattern-recognition skills of a founder under pressure, credited the room.
The deeper tension was the company's open-source history. DeepEval's steady adoption had helped Confident AI get into YC, but some investors preferred the sudden revenue curve of a top-down enterprise company. A year spent earning developer trust could look slow beside a startup engineered for an eight-week graph. Then one investor arrived having studied the project, its community, and its technical writing. Ip did not need to show a pitch-deck slide. The work had already given the presentation.
Momentum returned on Wednesday. More commitments arrived Thursday and Friday. By 3 p.m. Friday, the target was covered; the company later added one more fund and announced an oversubscribed $2.2 million seed round. Participants included Y Combinator, Flex Capital, Liquid 2 Ventures, January Capital, Rebel Fund, Vermilion Cliffs Ventures, and angel investor Oliver Jung. Ip's account is useful because it preserves the emotional whiplash: early swagger, a trough, a good room, revived ego, and the inevitable deflation of that ego once the paperwork was done.
A company built around the red test
Fundraising supplied time. The product still had to earn its place in a crowded AI infrastructure stack. Confident AI's answer has been to widen the loop without abandoning the engineer who started it. Teams can curate datasets, run evaluations during development, compare model and prompt changes, inspect traces, monitor production traffic, generate reports, and simulate adversarial conversations. The open-source metrics remain the common language between local code and the organizational view.
This matters because AI quality is not one number. A customer-service agent can be factually grounded and still fail to finish the task. A retrieval system can find the right document and compose a poor answer. A tool-using agent can reach the desired outcome by taking a reckless path. An executive wants a trend. An engineer needs the failing trace. A reviewer needs examples. A compliance team needs an auditable record. Confident AI is trying to keep those perspectives in the same room without making each participant learn everybody else's software.
Recent releases make that ambition concrete. The company has added automated error analysis, configurable reporting, agent-focused metrics, simulation controls, and self-hosting options. In August 2026 it announced availability through AWS Marketplace for customers that want Confident AI to run inside their own AWS account and region. Procurement and deployment are less romantic than model benchmarks, which is precisely why they mark a product growing up.
Studies at Imperial College London, followed by software-engineering work at Microsoft and Google.
DeepEval enters the open-source world with a testing vocabulary built for LLM applications.
Confident AI is founded in San Francisco to add collaboration and production infrastructure around evaluation.
The company joins YC W25 and completes its $2.2 million seed round.
DeepEval expands its agent-testing surface while Confident AI adds reporting, error analysis, and enterprise deployment options.
The confidence business
Ip's choice of company name contains a useful paradox. Confidence is exactly what language models perform so effortlessly and what engineers should grant them so reluctantly. The point of evaluation is not to produce certainty where none exists. It is to replace unexamined confidence with bounded confidence: this model passed these tests, on this dataset, under these conditions, with these known failures.
His path from Microsoft and YouTube to a small AI tooling company also reflects a broader shift in software. The scarce resource is no longer merely model capability. It is the ability to turn capability into a dependable product. The companies selling spectacular demos get the first conversation. The companies building measurement, observability, security, and governance stay for the difficult second conversation: what happens when real users arrive?
Ip has said the next phase is to improve DeepEval while teaching more people how to evaluate LLM systems in practice. Education is not an accessory to the product. A testing framework succeeds when teams change their release habits. That requires code, documentation, examples, and enough public argument to make quality feel like a daily responsibility rather than a clean-up project.
There is something appealingly unfashionable in the mission. Generative AI is sold through surprise. Ip is building for repetition. The same input should be examined again after the prompt changes, after the model changes, after a new tool is attached, and after production discovers the edge cases the launch plan missed. The red test is not a failure of imagination. It is a small act of honesty.
And honesty, in this corner of technology, may be the most useful infrastructure of all.