Supabase Evals

Supabase Evals is an open-source benchmark and testing framework designed to evaluate how effectively AI coding agents can build, deploy, and troubleshoot projects using the Supabase platform. It provides a standardized way to measure agent performance across real-world developer tasks, such as schema design, debugging Edge Functions, and managing Row Level Security (RLS) policies.

Supabase Evals is an open-source benchmark and testing framework designed to evaluate how effectively AI coding agents can build, deploy, and troubleshoot projects using the Supabase platform. It provides a standardized way to measure agent performance across real-world developer tasks, such as schema design, debugging Edge Functions, and managing Row Level Security (RLS) policies.

• 01Automated benchmarking of AI coding agents against real Supabase tasks
• 02Support for both 'Tools' and 'Local-stack' (Docker-based) evaluation runtimes
• 03Comprehensive scoring system using SQL checks, client calls, and LLM-as-judge reviews
• 04Regression suite for tracking and fixing specific agent failure modes
• 05Integration with Supabase CLI and MCP (Model Context Protocol) tools