llm-testing
(3)
Confident AI
Confident AI is a San Francisco startup (YC W25) that builds the evaluation and testing layer for LLM applications. Its open-source frameworks DeepEval and DeepTeam - often described as 'Pytest for LLMs' - let engineers unit-test, benchmark, red team, and monitor chatbots, agents, and RAG pipelines with research-backed metrics. The cloud platform adds dataset management, observability, adversarial testing, and governance so teams can prove AI quality before and after shipping.
Haize Labs
Haize Labs is a New York-based AI safety and reliability startup that automates the red-teaming, stress-testing, and evaluation of large language models. Founded in 2024 by a trio of Harvard-trained researchers, the company builds algorithms that hunt for the inputs that make AI models misbehave - jailbreaks, failure modes, and edge cases - so they can be fixed before real users find them. Its 'haizing suite' and multi-turn attack engine Cascade are used by frontier model labs including OpenAI and Anthropic, alongside enterprises like Deloitte and MongoDB.
Distributional
Distributional is an enterprise AI testing platform that uses adaptive, statistical analysis of production traces to catch behavioral drift and regressions in AI agents and LLM applications. Founded in 2023 by former Intel AI software GM Scott Clark, the company helps Fortune 500 teams ship generative AI with confidence by treating reliability as a continuous test problem rather than a one-shot benchmark.