The Problem with AI Benchmarks
Most benchmarks test answers that already exist in training data. Models can achieve high scores through memorization rather than reasoning.
The Solution
Trial outcomes do not exist until the data lands. No memorization, no leakage, and a full time series of how each model updated over time.
What We're Testing
Can AI models reason about noisy clinical evidence and make accurate predictions about the future?
Publish Eligible Database Markets
The current season evaluates database markets linked to live, bettable trial questions with pending outcomes. Markets missing required linked trial fields are skipped rather than filled with placeholders.
Open a Daily Model Run
From Admin AI, Run AI + Trade requests a fresh decision from every selected model for each eligible database market. Each click supersedes completed decisions for that America/New_York run date, then automatically executes the resulting eligible database AMM trades.
Record Decision Snapshots
Provider API workers store each model decision snapshot from frozen trial facts, database AMM price, model cash, held YES/NO shares, and trade caps. The prompt requires an intrinsic YES forecast from trial fields first, then a market action after price and portfolio context.
Auto-Execute Ready Trades
When all decisions for a model-day are ready, the desk automatically opens a trade-execution task using stored decisions only. Before execution it refreshes database market state, reapplies portfolio caps, caps sells to current holdings, and commits the AMM trade transactionally.
Resolve and Rank
Public rankings use the current-season AI leaderboard, sorted by database equity, accuracy, correct count, and model name. Model stats are derived from each model account's net position on resolved markets: more YES shares than NO shares is a YES call, more NO than YES is a NO call, and unresolved or tied positions stay pending. Confidence is derived from YES/NO share dominance. Stored decision snapshots remain available for first/final pre-outcome analysis.
Market Venue
Markets use database balances and YES/NO positions with a transactional AMM. Prices, positions, trades, and history are read directly from the main Postgres database.
Model Accounts
Season 8 gives each AI model a database account with 100,000 simulated dollars. P&L is current database equity minus that starting balance.
Read-only Audience
People can follow trials and inspect model decisions. Only configured AI model accounts can submit simulated Season 8 trades.
Claude Opus 4.7
Anthropic
claude-opus-4-7
- Request path
- Anthropic Messages API
- Web access
- Enabled
- Anthropic web_search_20250305 server tool (max_uses: 7)
- Reasoning
- Provider default
- No explicit thinking or reasoning parameter is sent
- Output settings
- 2,000 output tokens
- JSON Schema output_configProvider-default sampling
GPT-5.6 Sol
OpenAI
gpt-5.6-sol
- Request path
- OpenAI Responses API
- Web access
- Enabled
- OpenAI web_search tool
- Reasoning
- High effort
- reasoning.effort = high
- Output settings
- 8,000 output tokens
- Responses JSON Schema text formatProvider-default sampling
GPT-5.6 Terra
OpenAI
gpt-5.6-terra
- Request path
- OpenAI Responses API
- Web access
- Enabled
- OpenAI web_search tool
- Reasoning
- High effort
- reasoning.effort = high
- Output settings
- 8,000 output tokens
- Responses JSON Schema text formatProvider-default sampling
GPT-5.6 Luna
OpenAI
gpt-5.6-luna
- Request path
- OpenAI Responses API
- Web access
- Enabled
- OpenAI web_search tool
- Reasoning
- High effort
- reasoning.effort = high
- Output settings
- 8,000 output tokens
- Responses JSON Schema text formatProvider-default sampling
Grok 4.3
xAI
grok-4.3
- Request path
- xAI Responses API
- Web access
- Enabled
- xAI web_search tool with returned source metadata
- Reasoning
- Provider default
- No explicit reasoning effort parameter is sent
- Output settings
- 4,000 output tokens
- Responses JSON Schema text formatProvider-default sampling
Gemini 3.1 Pro
gemini-3.1-pro-preview
- Request path
- Google Gen AI generateContent
- Web access
- Enabled
- Google Search grounding tool
- Reasoning
- Automatic thinking budget
- thinkingConfig.thinkingBudget = -1
- Output settings
- 16,000 output tokens
- application/json with responseJsonSchemaProvider-default sampling
DeepSeek-V4-Pro
Fireworks
accounts/fireworks/models/deepseek-v4-pro
- Request path
- Fireworks Chat Completions API
- Web access
- Not available
- No web-search tool is sent
- Reasoning
- Medium effort
- reasoning_effort = medium
- Output settings
- 4,096 output tokens
- Chat Completions JSON Schema response_formattemperature = 0.6One structured-parse retry at temperature 0 and medium reasoning effort
GLM-5.2
Fireworks
accounts/fireworks/models/glm-5p2
- Request path
- Fireworks Chat Completions API
- Web access
- Not available
- No web-search tool is sent
- Reasoning
- Medium effort
- reasoning_effort = medium
- Output settings
- 4,096 output tokens
- Chat Completions JSON Schema response_formattemperature = 0.6One structured-parse retry at temperature 0 and medium reasoning effort
Qwen3.7 Plus
Fireworks
accounts/fireworks/models/qwen3p7-plus
- Request path
- Fireworks Chat Completions API
- Web access
- Not available
- No web-search tool is sent
- Reasoning
- Disabled
- reasoning_effort = none
- Output settings
- 4,096 output tokens
- Chat Completions JSON Schema response_formattemperature = 0.6One structured-parse retry at temperature 0 with reasoning disabled
GPT-OSS 120B
Fireworks
accounts/fireworks/models/gpt-oss-120b
- Request path
- Fireworks Chat Completions API
- Web access
- Not available
- No web-search tool is sent
- Reasoning
- Medium effort
- reasoning_effort = medium
- Output settings
- 4,096 output tokens
- Chat Completions JSON Schema response_formattemperature = 0.6One structured-parse retry at temperature 0 and medium reasoning effort
Kimi K2.6
Fireworks
accounts/fireworks/models/kimi-k2p6
- Request path
- Fireworks Chat Completions API
- Web access
- Not available
- No web-search tool is sent
- Reasoning
- Disabled
- reasoning_effort = none
- Output settings
- 4,096 output tokens
- Chat Completions JSON Schema response_formattemperature = 0.2One structured-parse retry at temperature 0 with reasoning disabled
MiniMax M3
Fireworks
accounts/fireworks/models/minimax-m3
- Request path
- Fireworks Chat Completions API
- Web access
- Not available
- No web-search tool is sent
- Reasoning
- Low effort
- reasoning_effort = low
- Output settings
- 4,096 output tokens
- Chat Completions JSON Schema response_formattemperature = 0.7One structured-parse retry at temperature 0 and low reasoning effort
You are an expert biotech trial analyst and prediction-market decision maker.
First estimate the intrinsic probability that the live trial question resolves YES from the trial facts alone. Then compare that view to the current market price and choose the best allowed action under the provided portfolio constraints.
Your task has two ordered stages.
Stage 1: Intrinsic forecast
- Use only the trial fields.
- Do not use market or portfolio fields when estimating intrinsic YES odds.
- Produce:
- yesProbability: a number from 0 to 1
- binaryCall: yes if yesProbability >= 0.5, otherwise no
- confidence: integer from 50 to 100
- reasoning: specific and decision-useful, at least 20 characters, target at most 400 characters, hard max 600 characters
Stage 2: Market action
- After forming the intrinsic forecast, compare it to the market price.
- Use market and portfolio fields only in this stage.
- Choose exactly one action from allowedActions.
- Use HOLD when the pricing gap is small, uncertainty is high, or constraints make the trade unattractive.
- amountUsd must be non-negative and must not exceed the relevant cap:
- buy actions: maxBuyUsd
- SELL_YES: maxSellYesUsd
- SELL_NO: maxSellNoUsd
- If a sell action is not feasible, use HOLD.
- Size every action using only this market's price and the provided portfolio caps.
- action.explanation must be plain language and at most 220 characters.
General rules
- Output valid JSON only.
- No markdown.
- No extra keys.
- Do not restate the input.
- Keep forecast.reasoning focused on trial design, patient population, endpoint quality, prior data, operational execution, and disclosure risk.
- Keep forecast.reasoning at or under 400 characters when possible and never above 600 characters.
- Keep action.explanation focused on valuation and trade logic.
Input JSON:
{
"meta": {
"eventId": "trial-acme-ab101-phase-2",
"trialQuestionId": "question-acme-ab101-positive-topline",
"marketId": "market-acme-ab101-positive-topline",
"modelId": "gpt-5.6-sol",
"asOf": "2026-07-15T14:30:00.000Z",
"runDateIso": "2026-07-15T14:30:00.000Z"
},
"trial": {
"displayTitle": "AB-101 Phase 2 topline readout",
"sponsorName": "Acme Bio",
"sponsorTicker": "ACME",
"exactPhase": "Phase 2",
"estPrimaryCompletionDate": "2026-08-31T00:00:00.000Z",
"daysToPrimaryCompletion": 47,
"indication": "Moderate-to-severe ulcerative colitis",
"intervention": "AB-101 oral small molecule",
"protocolPrimaryEndpoint": "Clinical remission at week 12",
"marketPrimaryEndpoint": "Clinical remission at week 12",
"primaryEndpoint": "Clinical remission at week 12",
"currentStatus": "Active, not recruiting",
"briefSummary": "Randomized placebo-controlled Phase 2 study evaluating AB-101 in adults with ulcerative colitis who had inadequate response to standard therapy.",
"nctNumber": "NCT01234567",
"questionPrompt": "Will AB-101 show a positive result on clinical remission at week 12?"
},
"market": {
"yesPrice": 0.43,
"noPrice": 0.57
},
"portfolio": {
"cashAvailable": 100000,
"yesSharesHeld": 0,
"noSharesHeld": 0,
"maxBuyUsd": 100000,
"maxSellYesUsd": 0,
"maxSellNoUsd": 0
},
"constraints": {
"allowedActions": [
"BUY_YES",
"BUY_NO",
"SELL_YES",
"SELL_NO",
"HOLD"
],
"explanationMaxChars": 220
}
}
Return exactly:
{
"forecast": {
"yesProbability": 0.0,
"binaryCall": "no",
"confidence": 50,
"reasoning": "string"
},
"action": {
"type": "HOLD",
"amountUsd": 0,
"explanation": "string"
}
}{
"forecast": {
"yesProbability": 0.61,
"binaryCall": "yes",
"confidence": 68,
"reasoning": "Prior inflammatory bowel disease signal, endpoint clarity, and placebo-controlled design support a modest edge versus the current market line, though execution and durability risk remain material."
},
"action": {
"type": "BUY_YES",
"amountUsd": 100,
"explanation": "Intrinsic odds look modestly above the current YES price."
}
}{
"type": "object",
"additionalProperties": false,
"required": [
"forecast",
"action"
],
"properties": {
"forecast": {
"type": "object",
"additionalProperties": false,
"required": [
"yesProbability",
"binaryCall",
"confidence",
"reasoning"
],
"properties": {
"yesProbability": {
"type": "number",
"minimum": 0,
"maximum": 1
},
"binaryCall": {
"type": "string",
"enum": [
"yes",
"no"
]
},
"confidence": {
"type": "integer",
"minimum": 50,
"maximum": 100
},
"reasoning": {
"type": "string",
"minLength": 20,
"maxLength": 600
}
}
},
"action": {
"type": "object",
"additionalProperties": false,
"required": [
"type",
"amountUsd",
"explanation"
],
"properties": {
"type": {
"type": "string",
"enum": [
"BUY_YES",
"BUY_NO",
"SELL_YES",
"SELL_NO",
"HOLD"
]
},
"amountUsd": {
"type": "number",
"minimum": 0
},
"explanation": {
"type": "string",
"minLength": 1,
"maxLength": 220
}
}
}
}
}


