Create Evaluation
Create a new model evaluation using an existing benchmark.
This endpoint initiates a model evaluation task using a benchmark from your library. Supports both single model evaluation and comparison mode where two models are evaluated side-by-side.
Request Body:
model_id: Primary model to evaluate (required)benchmark_id: ID of the benchmark to use for evaluation (required)model_id_2(optional): Second model ID for comparison modecustom_metrics(optional): Custom evaluation metrics configuration
Returns:
evaluation_id: Unique identifier for tracking the evaluationmodel_id: The primary model being evaluatedstatus: Initial status (alwayspending)benchmark_id: The benchmark being usedcreated_at: Timestamp when evaluation was createdmessage: Confirmation message
Example Request (Single Model):
POST /api/v3/evaluations
Headers: {"Authorization": "Bearer <api_key>"}
{
"model_id": "claude-3-sonnet-20240229",
"benchmark_id": "task-abc123",
"custom_metrics": ["accuracy", "relevance"]
}
Example Request (Comparison Mode):
POST /api/v3/evaluations
Headers: {"Authorization": "Bearer <api_key>"}
{
"model_id": "claude-3-sonnet-20240229",
"model_id_2": "gpt-4",
"benchmark_id": "task-abc123"
}
Example Response:
{
"evaluation_id": "eval-xyz789",
"model_id": "claude-3-sonnet-20240229",
"status": "pending",
"benchmark_id": "task-abc123",
"created_at": "2024-01-15T10:30:00Z",
"message": "Evaluation created successfully and queued for execution"
}
Notes:
- Single model mode: Evaluates one model against the benchmark
- Comparison mode: Provide
model_id_2to compare two models side-by-side - Evaluation runs asynchronously - use the returned
evaluation_idto check status - Use
/evaluations/{evaluation_id}/statusto track progress
Authorizations
Bearer authentication header of the form Bearer <token>, where <token> is your auth token.
Body
Request schema for creating a new model evaluation
Response
Returns a unique identifier for the initiated evaluation along with the initial evaluation status. This endpoint starts an asynchronous evaluation process using a specified benchmark and model(s), allowing users to track progress and retrieve results once completed.
Response schema for evaluation creation
Unique identifier for the evaluation
ID of the model being evaluated
Current status of the evaluation
Benchmark ID used
ISO timestamp of evaluation creation
Status message