SanityHarness

A lightweight evaluation harness for coding agents that runs high-signal, compact but challenging problems in isolated Docker containers. Evaluate agents across 26 tasks in 6 languages with weighted scoring, integrity verification, and detailed reporting.

Note: v1.8.x is now live, both in releases and the leaderboard. Use v1.6.1 with --legacy flag if you want to compare run data to the old legacy leaderboard.

Table of Contents

• Features
• Quick Start
• Usage
• Available Tasks
• Configuration
• Agents
• How It Works
• Output
• Architecture
• Version History
• Contributing
• License

Features

• Isolated Execution: Each task runs in a dedicated Docker container
• Multi-Language Support: Go, Rust, TypeScript, Kotlin, Dart, and Zig (26 tasks)
• 19 Built-in Agents: Gemini, Claude, OpenCode, Codex, Goose, and more
• Weighted Scoring: Empirically-derived difficulty factors for fair comparison
• BLAKE3 Verification: Cryptographic integrity checks for submissions
• Watch Mode: Automatically re-run tests on file changes
• Hidden Tests: Additional validation applied only during eval
• Parallel Eval: Run multiple tasks concurrently with --parallel
• Agent Sandboxing: Bubblewrap isolation restricts agents to their workspace
• Persistent Caches: Speed up builds with .sanity-cache/ mounts

Quick Start

Prerequisites

• Go 1.25+
• Docker (running daemon)
• bubblewrap (optional, for agent sandboxing)

Installation

git clone https://github.com/lemon07r/sanityharness.git
cd sanityharness
make tools    # Install dev tools (first-time only)
make build    # Build the CLI

Global Flags

Flag	Short	Description
--config		Config file path (default: ./sanity.toml)
--tasks-dir		External tasks directory
--verbose	-v	Enable debug logging

Usage

List Tasks

./sanity list                        # List all tasks
./sanity list --json                 # JSON output
./sanity list --language go          # Filter by language
./sanity list --tier core            # Filter by tier
./sanity list --difficulty hard      # Filter by difficulty

Initialize Workspace

./sanity init go/bank-account        # Create workspace with stub files
./sanity init go/bank-account -o ./my-dir

Run a Task

./sanity run go/bank-account         # Run tests once
./sanity run go/bank-account --watch # Re-run on file changes
./sanity run go/bank-account -w ./my-impl --timeout 60

Evaluate an Agent

./sanity eval --agent gemini                          # Evaluate against core tasks
./sanity eval --agent gemini --model gemini-3-pro     # Specify model
./sanity eval --agent gemini --tier all --parallel 4  # All tasks, 4 concurrent
./sanity eval --agent gemini --dry-run                # Preview without running
./sanity eval --agent droid --reasoning high          # Set reasoning effort
./sanity eval --agent gemini --use-mcp-tools          # Enable MCP tools
./sanity eval --agent opencode --use-skills           # Enable Agent Skills mode
./sanity eval --agent opencode --disable-mcp          # Disable MCP tools / currently only supported for opencode
./sanity eval --agent opencode --keep-workspaces      # Keep workspaces for debugging
./sanity eval --agent gemini --no-sandbox             # Disable bubblewrap sandbox
./sanity eval --agent gemini --legacy                 # Legacy mode (hidden tests visible to agent)
./sanity eval --resume ./eval-results/2026-01-07T120000-gemini  # Resume interrupted eval

View Results

./sanity show sessions/go-bank-account-2026-01-15T143022-a1b2c3d4
./sanity show sessions/go-bank-account-2026-01-15T143022-a1b2c3d4 --json

Verify Submission

./sanity verify ./eval-results/2026-01-07T120000-gemini

Clean Up

./sanity clean                # Interactive cleanup
./sanity clean --all --force  # Clean everything

Version

./sanity version              # Show version, commit, build date

Task References

Tasks can be referenced as:

• Canonical: <language>/<slug> (e.g., go/bank-account) - always unambiguous
• Bare slug: bank-account - works if unique across languages

Available Tasks

26 tasks across 6 languages with varying difficulty:

Language	Tasks	Tiers	Difficulty
Go	6	4 core, 2 extended	Hard - Expert
Rust	6	4 core, 2 extended	Hard - Expert
TypeScript	5	4 core, 1 extended	Hard
Kotlin	3	3 extended	Hard
Dart	3	3 extended	Hard
Zig	3	3 extended	Hard - Expert

See docs/TASKS.md for complete task listings and metadata.

Configuration

Create sanity.toml in your project root (see sanity.toml.example):

[harness]
max_attempts = 10
default_timeout = 60
session_dir = "sessions"

[docker]
go_image = "ghcr.io/lemon07r/sanity-go:latest"
auto_pull = true

Config files are searched in order:

1. ./sanity.toml
2. ~/.sanity.toml
3. ~/.config/sanity/config.toml

See docs/CONFIGURATION.md for all options.

Agents

Built-in Agents

Agent	Description
gemini	Google Gemini CLI
kilocode	Kilo Code CLI
opencode	OpenCode CLI
claude	Anthropic Claude Code
codex	OpenAI Codex CLI
kimi	Moonshot Kimi CLI
crush	Crush CLI
copilot	GitHub Copilot CLI
droid	Factory Droid CLI
iflow	iFlow CLI
qwen	Qwen Code CLI
amp	Sourcegraph Amp CLI (modes: smart, rush)
codebuff	Codebuff CLI (modes: max, lite)
vibe	Mistral Vibe CLI
goose	Block Goose CLI
junie	JetBrains Junie CLI
ccs	Claude Code Switch (profiles: agy, glm, kimi, etc.)
cline	Cline CLI
pi	Pi CLI

Custom Agents

[agents.my-agent]
command = "/path/to/my-agent"
args = ["--auto-approve", "{prompt}"]
model_flag = "-m"
env = { API_KEY = "xxx" }

See docs/CONFIGURATION.md#agent-configuration for full schema.

Workspace isolation: During sanity eval, each agent runs in an isolated temporary workspace under /tmp rather than inside eval-results/. This prevents agents from reading other eval results, sibling task solutions, or their own agent.log. After the agent finishes, files are copied back to eval-results/ for validation. Combined with the bubblewrap sandbox (which uses --tmpfs /tmp), agents have zero visibility into other evaluations.

Sandbox note: sanity eval runs agents inside a bubblewrap sandbox where $HOME is read-only by default. A configurable allowlist is mounted read/write ([sandbox] shared_readwrite_dirs) and read-only ([sandbox] shared_readonly_dirs), with additional writable paths available via [sandbox] writable_dirs. Non-allowlisted top-level home directories are masked, and extra sensitive paths can be masked with [sandbox] readable_denylist. Use --no-sandbox to disable.

Legacy mode: Prior to v1.6.0, a bug caused hidden tests to be included in the workspace during sanity eval, making them visible to agents. The --legacy flag reproduces this behavior so that older evaluation runs can be fairly compared or resumed. When --legacy is active, hidden test files are written to the workspace at init time (instead of being overlaid just before validation), and the hidden-test overlay step is skipped. Use this flag when resuming runs that were originally executed with the buggy behavior.

How It Works

1. Container Strategy: Containers run sleep infinity; commands execute via docker exec for fast reuse
2. Workspace Mounting: Your code is mounted at /workspace in the container
3. User Permissions: Runs as your host UID:GID to avoid root-owned files
4. Cache Persistence: Language caches mount from .sanity-cache/ for faster builds
5. Embedded Tasks: Task files are compiled into the binary for zero-dependency distribution

Output

Session Output

Each sanity run creates:

sessions/<session-id>/
├── result.json      # Structured results
├── report.md        # Markdown summary
├── logs/            # Per-attempt logs
└── workspace/       # Final code

Eval Output

Each sanity eval creates:

eval-results/<timestamp>-<agent>/
├── summary.json       # Complete results with weighted scores
├── attestation.json   # BLAKE3 hashes for verification
├── report.md          # Human-readable report
├── submission.json    # Leaderboard format
├── run-config.json    # Config for resume capability
└── <task>/
    ├── agent.log      # Agent output during task execution (includes HARNESS timeout footer)
    ├── validation.log # Test runner output + HARNESS validation footer (always non-empty)
    ├── integrity.json # Present on integrity violations; forensic metadata
    ├── integrity-files/ # Present on integrity violations; expected/actual file copies
    └── integrity-diff/  # Present on integrity violations; per-file diffs

Resume interrupted evals: If interrupted (CTRL+C), the harness saves partial results and prints a resume command. Use ./sanity eval --resume <dir> to continue.

See docs/SCORING.md for scoring details and output schemas.

Architecture

sanityharness/
├── cmd/sanity/          # CLI entry point
├── internal/
│   ├── cli/             # Cobra commands
│   ├── config/          # TOML configuration
│   ├── errors/          # Error summarization
│   ├── result/          # Session/attempt types
│   ├── runner/          # Docker execution
│   └── task/            # Task loading
├── tasks/               # Embedded task files
└── containers/          # Dockerfiles

See docs/DEVELOPMENT.md for architecture details.

Version History

For a full summary of all changes since v1.6.1 (the entire 1.7.x line and further), see docs/ROAD-TO-V2-Overhaul.md. Current stable release: v1.8.10.

Contributing

Contributions are welcome! Please see docs/CONTRIBUTING.md for guidelines.

Quick start:

make pre-commit  # Run before committing
make test        # Run tests

License

MIT License