> ## Documentation Index > Fetch the complete documentation index at: https://docs.omni.co/llms.txt > Use this file to discover all available pages before exploring further. # Start an eval run > Create and start a new run against an existing prompt set. The run enqueues one agentic job per prompt and begins executing immediately. Returns the newly created run with its initial per-prompt result rows. ## OpenAPI ````yaml /api/openapi.yaml post /v1/ai/eval/runs openapi: 3.1.0 info: title: Omni API description: > The Omni REST API provides programmatic access to your Omni instance for managing users, documents, queries, schedules, and more. version: 1.0.0 contact: name: Omni Support url: https://docs.omni.co servers: - url: https://{instance}.omniapp.co/api description: Production variables: instance: default: blobsrus description: Your production Omni instance subdomain - url: https://{instance}.playground.exploreomni.dev/api description: Playground variables: instance: default: blobsrus description: Your playground Omni instance subdomain security: - bearerAuth: [] - orgApiKey: [] tags: - name: Who Am I description: Inspect your own user permissions - name: AI description: AI-powered query generation - name: AI Credit Controls description: Manage organization-level AI credit usage - name: AI Eval description: >- AI evaluation: manage prompt sets and runs used to score AI quality against curated prompt suites. - name: AI Model Suggestions description: Manage AI-generated suggestions for shared models - name: AI Routines description: >- Manage AI Routines: schedule recurring AI-powered tasks to run automatically on your data. - name: Documents v2 description: > A draft-based workflow for creating and editing documents: create a document, patch a draft, then publish. Replaces the one-shot `PUT`/`PATCH` v1 document write endpoints. - name: Documents description: Create, retrieve, and manage documents - name: API Tokens description: >- Manage API tokens (Organization keys, Personal Access Tokens, MCP OAuth grants) - name: Connections description: Manage database connections - name: Connection environments description: Manage connection environments database connections - name: Content description: Unified content retrieval (documents and folders) - name: Content migration description: Export and import dashboards - name: Content validator description: Validate content against models and perform find/replace operations - name: Dashboard downloads description: Download dashboards and tiles as PDF, PNG, XLSX, CSV, or JSON files - name: Dashboard filters and controls description: Read and update dashboard filter and control default values - name: dbt description: Manage dbt configuration for connections - name: Document favorites description: Favorite and unfavorite documents - name: Document labels description: Apply and manage labels on documents - name: Document permissions description: Manage document-level access - name: Labels description: > Manage labels in an organization. Labels can be applied to documents and folders to help organize and categorize content. **Label types:** - **Basic labels**: Can be created and managed by any user - **Verified labels**: Indicate curated or officially sanctioned content. Admin-only. - **Homepage labels**: Appear on the organization homepage. Admin-only. - name: Folders description: Create and organize content folders - name: Folder permissions description: Manage folder-level access - name: Jobs description: Check status of asynchronous jobs - name: Models description: Create and manage data models - name: Model branches description: Manage model branches and merge changes - name: Model git configuration description: Manage git configuration for shared models - name: Queries description: Execute workbook queries - name: Schedules description: Create and manage scheduled tasks - name: Schedule recipients description: Manage schedule recipients - name: Schema refresh schedules description: Manage automated schema refresh schedules for connections - name: Topics description: Retrieve topic information from models - name: Uploads description: Manage file uploads - name: Users description: Manage users - name: User attributes description: Manage user attribute definitions - name: User groups description: Manage user groups - name: User model roles description: Manage model and connection role assignments for users - name: User group model roles description: Manage model and connection role assignments for user groups - name: Uploads description: Manage CSV and spreadsheet uploads paths: /v1/ai/eval/runs: post: tags: - AI Eval summary: Start an eval run description: >- Create and start a new run against an existing prompt set. The run enqueues one agentic job per prompt and begins executing immediately. Returns the newly created run with its initial per-prompt result rows. operationId: aiEvalRunsCreate requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/EvalRunsCreateBody' responses: '201': description: Run created and jobs enqueued. content: application/json: schema: $ref: '#/components/schemas/EvalRunsCreateResponse' '400': description: Invalid request body. content: application/json: schema: $ref: '#/components/schemas/EvalApiError' example: detail: 'Bad Request: name: Required' status: 400 '401': description: Missing or invalid API key. content: application/json: schema: $ref: '#/components/schemas/EvalApiError' example: detail: 'Unauthorized: Missing or invalid API key' status: 401 '403': description: >- Insufficient permissions. The caller must have at least the Querier role on the prompt set's model. content: application/json: schema: $ref: '#/components/schemas/EvalApiError' example: detail: AI eval requires at least Querier access on the model status: 403 '404': description: >- The prompt set was not found, or `run_config.branch_id` does not match an existing branch in the organization. content: application/json: schema: $ref: '#/components/schemas/EvalApiError' example: detail: Prompt set not found status: 404 '422': description: '`run_config.branch_id` does not belong to the prompt set''s model.' content: application/json: schema: $ref: '#/components/schemas/EvalApiError' example: detail: A prompt being updated does not belong to this prompt set status: 422 '429': description: Per-user active-run cap reached. content: application/json: schema: $ref: '#/components/schemas/EvalApiError' example: detail: Too many active runs; wait for an in-flight run to finish status: 429 '503': description: AI eval is paused for this organization. content: application/json: schema: $ref: '#/components/schemas/EvalApiError' example: detail: AI eval is paused for this organization status: 503 components: schemas: EvalRunsCreateBody: type: object properties: description: type: - string - 'null' maxLength: 1024 description: >- Optional human-readable description for the run. Omit or pass `null` to leave it unset. Max 1024 characters. example: Re-running after switching to gpt-4o for query generation prompt_set_id: type: string format: uuid description: The prompt set to execute. example: 550e8400-e29b-41d4-a716-446655440000 run_config: type: object properties: branch_id: type: string format: uuid description: >- Optional branch ID to run against. Must be a branch of the prompt set's model. example: 440e8400-e29b-41d4-a716-446655440006 description: Per-run configuration. Optional — omit if no overrides. required: - prompt_set_id EvalRunsCreateResponse: type: object properties: job_count: type: integer description: >- Number of per-prompt agentic jobs created for this run (one per prompt that fanned out successfully). Enqueue onto the work queue happens after creation and is best-effort, so this count reflects jobs created, not necessarily those successfully enqueued. example: 12 run: $ref: '#/components/schemas/EvalRunDetail' required: - job_count - run EvalApiError: type: object description: Error response returned by the AI Eval endpoints. properties: detail: type: string description: Human-readable error message describing what went wrong. status: type: integer description: HTTP status code of the error. required: - detail - status EvalRunDetail: type: object properties: branch_id: type: - string - 'null' format: uuid description: >- Optional branch ID the run was executed against. Null when run against the main shared model. example: null branch_name: type: - string - 'null' description: Display name for the branch, if `branch_id` is set. example: null completed_at: type: - string - 'null' description: ISO 8601 timestamp when the run reached a terminal state. example: null created_at: type: - string - 'null' description: ISO 8601 timestamp when the run was created. example: '2025-01-15T10:00:00.000Z' description: type: - string - 'null' description: Optional human-readable description for the run. example: null id: type: string format: uuid description: Unique identifier for the run. example: 660e8400-e29b-41d4-a716-446655440001 is_archived: type: boolean description: Whether the run has been archived. example: false model_id: type: string format: uuid description: The shared model this run was executed against. example: 880e8400-e29b-41d4-a716-446655440003 prompt_set_id: type: string format: uuid description: The prompt set this run was created from. example: 550e8400-e29b-41d4-a716-446655440000 results: type: array items: $ref: '#/components/schemas/EvalRunResult' description: >- Per-prompt results for this run, ordered by their creation order in the prompt set. run_number: type: integer description: Sequential, per-prompt-set run number. example: 3 status: type: string enum: - RUNNING - COMPLETE - CANCELLED description: >- Run-level lifecycle. Flips to a terminal state (COMPLETE or CANCELLED) exactly once. example: RUNNING required: - branch_id - branch_name - completed_at - created_at - description - id - is_archived - model_id - prompt_set_id - results - run_number - status description: The newly created run with its initial results. EvalRunResult: type: object properties: agentic_job: $ref: '#/components/schemas/EvalRunResultAgenticJob' cost: type: - number - 'null' description: Total LLM cost (USD) for this prompt, if available. example: 0.0021 error_reason: type: - string - 'null' description: Failure reason string for prompts whose underlying job failed. example: null id: type: string format: uuid description: Unique identifier for the run result row. example: aa0e8400-e29b-41d4-a716-446655440005 prompt: type: string description: The prompt text that was evaluated. example: What are the top 5 products by revenue? score: type: - number - 'null' description: Numeric judge score for this prompt result, if scoring ran. example: 0.9 scoring_cost: type: - number - 'null' description: Total LLM cost (USD) for scoring this prompt result. example: 0.0004 timing_ms: type: - integer - 'null' description: Wall-clock duration of the underlying job in milliseconds. example: 4321 required: - agentic_job - cost - error_reason - id - prompt - score - scoring_cost - timing_ms EvalRunResultAgenticJob: type: object properties: conversation_id: type: - string - 'null' format: uuid description: Conversation the agentic job belongs to. example: 770e8400-e29b-41d4-a716-446655440002 id: type: string format: uuid description: Agentic job identifier. example: 990e8400-e29b-41d4-a716-446655440004 state: type: string enum: - CANCELLED - COMPLETE - DELIVERING - EXECUTING - FAILED - QUEUED description: Current state of the agentic job that ran this prompt. example: COMPLETE required: - conversation_id - id - state securitySchemes: bearerAuth: type: http scheme: bearer bearerFormat: JWT description: > Can be either an [Organization API Key](/api/authentication#organization-api-keys) or [Personal Access Token (PAT)](/api/authentication#token-types). Include in the `Authorization` header as: `Bearer YOUR_TOKEN` orgApiKey: type: http scheme: bearer bearerFormat: JWT description: > Requires an [Organization API Key](/api/authentication#organization-api-keys). Personal Access Tokens (PATs) are not supported for this endpoint. Include in the `Authorization` header as: `Bearer ORGANIZATION_API_KEY` ````