Skip to main content
Why traditional benchmarks fall short

The Problem with AI Benchmarks

Most benchmarks test answers that already exist in training data. Models can achieve high scores through memorization rather than reasoning.

The Solution

Trial outcomes do not exist until the data lands. No memorization, no leakage, and a full time series of how each model updated over time.

What We're Testing

Can AI models reason about noisy clinical evidence and make accurate predictions about the future?

The five-step evaluation process
1

Publish Eligible Database Markets

The current season evaluates database markets linked to live, bettable trial questions with pending outcomes. Markets missing required linked trial fields are skipped rather than filled with placeholders.

2

Open a Daily Model Run

From Admin AI, Run AI + Trade requests a fresh decision from every selected model for each eligible database market. Each click supersedes completed decisions for that America/New_York run date, then automatically executes the resulting eligible database AMM trades.

3

Record Decision Snapshots

Provider API workers store each model decision snapshot from frozen trial facts, database AMM price, model cash, held YES/NO shares, and trade caps. The prompt requires an intrinsic YES forecast from trial fields first, then a market action after price and portfolio context.

4

Auto-Execute Ready Trades

When all decisions for a model-day are ready, the desk automatically opens a trade-execution task using stored decisions only. Before execution it refreshes database market state, reapplies portfolio caps, caps sells to current holdings, and commits the AMM trade transactionally.

5

Resolve and Rank

Public rankings use the current-season AI leaderboard, sorted by database equity, accuracy, correct count, and model name. Model stats are derived from each model account's net position on resolved markets: more YES shares than NO shares is a YES call, more NO than YES is a NO call, and unresolved or tied positions stay pending. Confidence is derived from YES/NO share dominance. Stored decision snapshots remain available for first/final pre-outcome analysis.

Current-season database runtime

Market Venue

Markets use database balances and YES/NO positions with a transactional AMM. Prices, positions, trades, and history are read directly from the main Postgres database.

Model Accounts

Season 8 gives each AI model a database account with 100,000 simulated dollars. P&L is current database equity minus that starting balance.

Read-only Audience

People can follow trials and inspect model decisions. Only configured AI model accounts can submit simulated Season 8 trades.

The models we compare

Claude Opus 4.7

Anthropic

claude-opus-4-7

Request path
Anthropic Messages API
Web access
Enabled
Anthropic web_search_20250305 server tool (max_uses: 7)
Reasoning
Provider default
No explicit thinking or reasoning parameter is sent
Output settings
2,000 output tokens
JSON Schema output_configProvider-default sampling

GPT-5.6 Sol

OpenAI

gpt-5.6-sol

Request path
OpenAI Responses API
Web access
Enabled
OpenAI web_search tool
Reasoning
High effort
reasoning.effort = high
Output settings
8,000 output tokens
Responses JSON Schema text formatProvider-default sampling

GPT-5.6 Terra

OpenAI

gpt-5.6-terra

Request path
OpenAI Responses API
Web access
Enabled
OpenAI web_search tool
Reasoning
High effort
reasoning.effort = high
Output settings
8,000 output tokens
Responses JSON Schema text formatProvider-default sampling

GPT-5.6 Luna

OpenAI

gpt-5.6-luna

Request path
OpenAI Responses API
Web access
Enabled
OpenAI web_search tool
Reasoning
High effort
reasoning.effort = high
Output settings
8,000 output tokens
Responses JSON Schema text formatProvider-default sampling

Grok 4.3

xAI

grok-4.3

Request path
xAI Responses API
Web access
Enabled
xAI web_search tool with returned source metadata
Reasoning
Provider default
No explicit reasoning effort parameter is sent
Output settings
4,000 output tokens
Responses JSON Schema text formatProvider-default sampling

Gemini 3.1 Pro

Google

gemini-3.1-pro-preview

Request path
Google Gen AI generateContent
Web access
Enabled
Google Search grounding tool
Reasoning
Automatic thinking budget
thinkingConfig.thinkingBudget = -1
Output settings
16,000 output tokens
application/json with responseJsonSchemaProvider-default sampling

DeepSeek-V4-Pro

Fireworks

accounts/fireworks/models/deepseek-v4-pro

Request path
Fireworks Chat Completions API
Web access
Not available
No web-search tool is sent
Reasoning
Medium effort
reasoning_effort = medium
Output settings
4,096 output tokens
Chat Completions JSON Schema response_formattemperature = 0.6One structured-parse retry at temperature 0 and medium reasoning effort

GLM-5.2

Fireworks

accounts/fireworks/models/glm-5p2

Request path
Fireworks Chat Completions API
Web access
Not available
No web-search tool is sent
Reasoning
Medium effort
reasoning_effort = medium
Output settings
4,096 output tokens
Chat Completions JSON Schema response_formattemperature = 0.6One structured-parse retry at temperature 0 and medium reasoning effort

Qwen3.7 Plus

Fireworks

accounts/fireworks/models/qwen3p7-plus

Request path
Fireworks Chat Completions API
Web access
Not available
No web-search tool is sent
Reasoning
Disabled
reasoning_effort = none
Output settings
4,096 output tokens
Chat Completions JSON Schema response_formattemperature = 0.6One structured-parse retry at temperature 0 with reasoning disabled

GPT-OSS 120B

Fireworks

accounts/fireworks/models/gpt-oss-120b

Request path
Fireworks Chat Completions API
Web access
Not available
No web-search tool is sent
Reasoning
Medium effort
reasoning_effort = medium
Output settings
4,096 output tokens
Chat Completions JSON Schema response_formattemperature = 0.6One structured-parse retry at temperature 0 and medium reasoning effort

Kimi K2.6

Fireworks

accounts/fireworks/models/kimi-k2p6

Request path
Fireworks Chat Completions API
Web access
Not available
No web-search tool is sent
Reasoning
Disabled
reasoning_effort = none
Output settings
4,096 output tokens
Chat Completions JSON Schema response_formattemperature = 0.2One structured-parse retry at temperature 0 with reasoning disabled

MiniMax M3

Fireworks

accounts/fireworks/models/minimax-m3

Request path
Fireworks Chat Completions API
Web access
Not available
No web-search tool is sent
Reasoning
Low effort
reasoning_effort = low
Output settings
4,096 output tokens
Chat Completions JSON Schema response_formattemperature = 0.7One structured-parse retry at temperature 0 and low reasoning effort
Model Decision Prompt
Generated from the runtime decision prompt builder
You are an expert biotech trial analyst and prediction-market decision maker.

First estimate the intrinsic probability that the live trial question resolves YES from the trial facts alone. Then compare that view to the current market price and choose the best allowed action under the provided portfolio constraints.

Your task has two ordered stages.

Stage 1: Intrinsic forecast
- Use only the trial fields.
- Do not use market or portfolio fields when estimating intrinsic YES odds.
- Produce:
  - yesProbability: a number from 0 to 1
  - binaryCall: yes if yesProbability >= 0.5, otherwise no
  - confidence: integer from 50 to 100
  - reasoning: specific and decision-useful, at least 20 characters, target at most 400 characters, hard max 600 characters

Stage 2: Market action
- After forming the intrinsic forecast, compare it to the market price.
- Use market and portfolio fields only in this stage.
- Choose exactly one action from allowedActions.
- Use HOLD when the pricing gap is small, uncertainty is high, or constraints make the trade unattractive.
- amountUsd must be non-negative and must not exceed the relevant cap:
  - buy actions: maxBuyUsd
  - SELL_YES: maxSellYesUsd
  - SELL_NO: maxSellNoUsd
- If a sell action is not feasible, use HOLD.
- Size every action using only this market's price and the provided portfolio caps.
- action.explanation must be plain language and at most 220 characters.

General rules
- Output valid JSON only.
- No markdown.
- No extra keys.
- Do not restate the input.
- Keep forecast.reasoning focused on trial design, patient population, endpoint quality, prior data, operational execution, and disclosure risk.
- Keep forecast.reasoning at or under 400 characters when possible and never above 600 characters.
- Keep action.explanation focused on valuation and trade logic.

Input JSON:
{
  "meta": {
    "eventId": "trial-acme-ab101-phase-2",
    "trialQuestionId": "question-acme-ab101-positive-topline",
    "marketId": "market-acme-ab101-positive-topline",
    "modelId": "gpt-5.6-sol",
    "asOf": "2026-07-15T14:30:00.000Z",
    "runDateIso": "2026-07-15T14:30:00.000Z"
  },
  "trial": {
    "displayTitle": "AB-101 Phase 2 topline readout",
    "sponsorName": "Acme Bio",
    "sponsorTicker": "ACME",
    "exactPhase": "Phase 2",
    "estPrimaryCompletionDate": "2026-08-31T00:00:00.000Z",
    "daysToPrimaryCompletion": 47,
    "indication": "Moderate-to-severe ulcerative colitis",
    "intervention": "AB-101 oral small molecule",
    "protocolPrimaryEndpoint": "Clinical remission at week 12",
    "marketPrimaryEndpoint": "Clinical remission at week 12",
    "primaryEndpoint": "Clinical remission at week 12",
    "currentStatus": "Active, not recruiting",
    "briefSummary": "Randomized placebo-controlled Phase 2 study evaluating AB-101 in adults with ulcerative colitis who had inadequate response to standard therapy.",
    "nctNumber": "NCT01234567",
    "questionPrompt": "Will AB-101 show a positive result on clinical remission at week 12?"
  },
  "market": {
    "yesPrice": 0.43,
    "noPrice": 0.57
  },
  "portfolio": {
    "cashAvailable": 100000,
    "yesSharesHeld": 0,
    "noSharesHeld": 0,
    "maxBuyUsd": 100000,
    "maxSellYesUsd": 0,
    "maxSellNoUsd": 0
  },
  "constraints": {
    "allowedActions": [
      "BUY_YES",
      "BUY_NO",
      "SELL_YES",
      "SELL_NO",
      "HOLD"
    ],
    "explanationMaxChars": 220
  }
}

Return exactly:
{
  "forecast": {
    "yesProbability": 0.0,
    "binaryCall": "no",
    "confidence": 50,
    "reasoning": "string"
  },
  "action": {
    "type": "HOLD",
    "amountUsd": 0,
    "explanation": "string"
  }
}
Expected JSON Response
{
  "forecast": {
    "yesProbability": 0.61,
    "binaryCall": "yes",
    "confidence": 68,
    "reasoning": "Prior inflammatory bowel disease signal, endpoint clarity, and placebo-controlled design support a modest edge versus the current market line, though execution and durability risk remain material."
  },
  "action": {
    "type": "BUY_YES",
    "amountUsd": 100,
    "explanation": "Intrinsic odds look modestly above the current YES price."
  }
}
Runtime JSON schema (shape + constraints)
{
  "type": "object",
  "additionalProperties": false,
  "required": [
    "forecast",
    "action"
  ],
  "properties": {
    "forecast": {
      "type": "object",
      "additionalProperties": false,
      "required": [
        "yesProbability",
        "binaryCall",
        "confidence",
        "reasoning"
      ],
      "properties": {
        "yesProbability": {
          "type": "number",
          "minimum": 0,
          "maximum": 1
        },
        "binaryCall": {
          "type": "string",
          "enum": [
            "yes",
            "no"
          ]
        },
        "confidence": {
          "type": "integer",
          "minimum": 50,
          "maximum": 100
        },
        "reasoning": {
          "type": "string",
          "minLength": 20,
          "maxLength": 600
        }
      }
    },
    "action": {
      "type": "object",
      "additionalProperties": false,
      "required": [
        "type",
        "amountUsd",
        "explanation"
      ],
      "properties": {
        "type": {
          "type": "string",
          "enum": [
            "BUY_YES",
            "BUY_NO",
            "SELL_YES",
            "SELL_NO",
            "HOLD"
          ]
        },
        "amountUsd": {
          "type": "number",
          "minimum": 0
        },
        "explanation": {
          "type": "string",
          "minLength": 1,
          "maxLength": 220
        }
      }
    }
  }
}