← all releases
Microsoft·Sep 11, 2026·v0.1.0·open source

ThinkingBox

ThinkingBox is an open-source sandbox and benchmarking framework designed to evaluate the reliability of AI agents in stateful business workflows. It separates the execution framework from the benchmark package, allowing for independent updates to the harness and test scenarios.

featured on blogAI AgentsBenchmarkingOpen SourceModel Context Protocol
visit website github ↗
on the blogMicrosoft Open-Sources ThinkingBox to Benchmark AI Agent Side Effects 4 min
overview

ThinkingBox is an open-source sandbox and benchmarking framework designed to evaluate the reliability of AI agents in stateful business workflows. It separates the execution framework from the benchmark package, allowing for independent updates to the harness and test scenarios.

key features
  • 01Isolated, stateful tool environments using MCP (Model Context Protocol)
  • 02Verification of backend state and side effects rather than just transcripts
  • 03ThinkingBox-Bench dataset with 507 executable business tasks
  • 04Support for multi-turn interactions with simulated users
  • 05Configurable evaluation harness for pass@1, pass@20, and pass^20 metrics
Ad
Master AI Marketing: Work Smarter, Not Harder

Master AI Marketing: Work Smarter, Not Harder

Unlock the power of AI to automate workflows and scale your results. Join 1.5M+ professionals and stay ahead of the curve with the latest AI insights.

Sponsored
related productsbrowse all →
MiroThinkeropen sourceMiroThinker is an open-source search agent model developed by MiroMindAI, designed for tool-augmented reasoning and real-world information seeking. It aims to match the deep research capabilities of leading AI models like OpenAI's Deep Research and Google's Gemini Deep Research.Letta Code SDKopen sourceThe Letta Code SDK is a software development kit that enables developers to build deeply personalized agents with persistent memory that learn over time. It serves as the interface to Letta Code, facilitating the creation of stateful agents capable of continuous learning and improvement.OB-1open sourceOB-1 is a self-improving coding agent developed by OpenBlock Labs, designed to autonomously handle the full development lifecycle, from project management to pull requests. It integrates seamlessly into existing workflows, enhancing productivity and code quality.
Master AI Marketing: Work Smarter, Not Harder

Master AI Marketing: Work Smarter, Not Harder

Unlock the power of AI to automate workflows and scale your results. Join 1.5M+ professionals and stay ahead of the curve with the latest AI insights.

Sponsored