llm-as-a-judge
(3)
Confident AI
Confident AI is a San Francisco startup (YC W25) that builds the evaluation and testing layer for LLM applications. Its open-source frameworks DeepEval and DeepTeam - often described as 'Pytest for LLMs' - let engineers unit-test, benchmark, red team, and monitor chatbots, agents, and RAG pipelines with research-backed metrics. The cloud platform adds dataset management, observability, adversarial testing, and governance so teams can prove AI quality before and after shipping.
Judgment Labs
Judgment Labs is a San Francisco applied-research startup building the continuous-improvement layer for AI agents. Its platform and open-source SDK, Judgeval, trace, evaluate, and monitor 'deep agents' - systems that reason, plan, use tools, and remember - turning messy production data into measurable, repeatable improvements. Founded in 2026 by three childhood friends out of Stanford, it raised $32M in combined seed and Series A funding led by Lightspeed Venture Partners.
TrustLab
TrustLab is a San Mateo-based trust and safety software company building AI-with-human-in-the-loop tools that detect, evaluate, and enforce against harmful online content - from misinformation and hate speech to identity fraud. Founded in 2020 by former trust and safety leaders from Google, YouTube, TikTok, and Reddit, it sells content-safety and AI-oversight products (DetectAI, ModAI, SuperviseAI) to social platforms, marketplaces, and governments, and was selected by the European Commission to independently measure disinformation across major platforms under its Code of Practice.