Rating Criteria

Overview

Caliber evaluates markets using multiple criteria, each assessing a different aspect of market definition quality.

Gate Criteria

These criteria must pass for the rating to proceed. If any gate fails, the final score is 0.

Source Reachabilitystatic

Verifies that all provided source URLs are accessible.

Source Blocklistedstatic

Checks that no sources are on the blocklist of unreliable domains.

Score Criteria

These criteria contribute weighted scores to the final rating. Weights sum to 100%.

Source Count20%static

Evaluates whether enough sources are provided for reliable resolution.

Source Agreement20%static

Assesses whether the min_agreement threshold is achievable.

Source History10%static

Checks the historical reliability of the provided sources.

Prompt Subjectivity10%llm

Analyzes how clear, specific, and unambiguous the prompt is.

Temporal Soundness10%llm

Evaluates whether time-based constraints are logically valid.

Source Relevancy20%llm

Determines if sources are likely to contain the required information.