Generate conversational responses with streaming support and multimodal capabilities.
This endpoint generates chat completions using language models, supporting both text-only conversations and vision inputs. Includes support for function calling, streaming responses, and automatic conversation tracking.
Request Body:
model: Model ID (required). Use your aligned model id (e.g. model_01kmqm4nrn9fw6r); base text models (e.g. model_01kmqm4nrn9fw6r) also work if you have not aligned a model yet. For reserved capacity pass ptm_id:model_id
messages: Array of message objects (required, minimum 1), each containing:
role: Message role (system, user, or assistant)
content: Text string OR array of content objects (for multimodal)
Tip: For prompts where you expect a long output (e.g., max_tokens > 500), prefer stream: true over stream: false to avoid HTTP 504 gateway timeouts on slow or large responses.
tools (optional): List of tools/functions available to the model
tool_choice (optional): Tool selection mode (auto, none, or specific tool)
top_p (optional): Nucleus sampling parameter
top_k (optional): Top-k sampling parameter
n (optional): Number of completions to generate (default: 1)
exclude: Set true to exclude reasoning tokens from response
enabled: Enable reasoning with default parameters
Optional Headers:
X-Session-ID: Session identifier for multi-turn conversation tracking
Returns:
Non-streaming mode -
id: Unique identifier for the response
object: Object type (always chat.completion)
created: Unix timestamp when completion was created
model: Model used for chat completion
choices: List of completion choices, each containing:
index: Index of the choice
message: Response message with:
role: Role of the author (always assistant)
content: Generated response text
tool_calls (optional): Tool calls made by the model (for function calling)
finish_reason: Reason model stopped (stop for natural stop, length if max tokens reached)
usage: Token usage statistics:
prompt_tokens: Number of tokens in the prompt
completion_tokens: Number of tokens generated
total_tokens: Total tokens used (prompt + completion)
confidence_score (optional): Confidence score, 0-100, on aligned models; null on base models
Streaming mode - ChatCompletionChunk stream with:
id: Unique identifier
created: Timestamp
model: Model ID
choices: List of chunk choices with:
index: Choice index
delta: Delta content with role and content
finish_reason: Reason for stopping (only in final chunk)
usage (optional): Only present in final chunk
Example Request (Text Chat):
POST /api/v3/inference/chat/completionsHeaders: {"Authorization": "Bearer <api_key>"}{ "model": "model_01kmqm4nrn9fw6r", "messages": [ { "role": "system", "content": "You are a helpful assistant." }, { "role": "user", "content": "What is the capital of France?" } ], "max_tokens": 500, "temperature": 0.7}
Example Response (Text Chat):
{ "id": "nugen-abc123", "object": "chat.completion", "created": 1704123600.0, "model": "model_01kmqm4nrn9fw6r", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "The capital of France is Paris." }, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 20, "completion_tokens": 8, "total_tokens": 28 }, "confidence_score": 89.5454}
Example Request (Using Domain-Aligned Model):
POST /api/v3/inference/chat/completionsHeaders: {"Authorization": "Bearer <api_key>"}{ "model": "model_01kmqm4nrn9fw6r", "messages": [ { "role": "user", "content": "How do I return a product?" } ], "max_tokens": 500, "temperature": 0.7}
Example Response (Domain-Aligned Model):
{ "id": "nugen-abc456", "object": "chat.completion", "created": 1704123700.0, "model": "model_01kmqm4nrn9fw6r", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "To return a product, please visit our returns portal within 30 days of purchase with your order number and receipt." }, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 15, "completion_tokens": 25, "total_tokens": 40 }, "confidence_score": 95.8}
How to get your aligned model ID:
Create an alignment project using POST /api/v3/alignment-projects/create
Poll GET /api/v3/alignment-projects/{id}/status until status is COMPLETED
Use the model_id field from the response (e.g., "model_01kmqm4nrn9fw6r")
Deploy the model using POST /api/v3/models/{model_id}/deployment (optional - for evaluation/production use)
Use the model_id in the model parameter for inference requests
Example Request (Vision - Image URL):
POST /api/v3/inference/chat/completionsHeaders: {"Authorization": "Bearer <api_key>"}{ "model": "model_01kmqm4nrn9fw6r", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe what you see in this image." }, { "type": "image_url", "image_url": { "url": "https://example.com/image.jpg" } } ] } ], "max_tokens": 1000}
POST /api/v3/inference/chat/completionsHeaders: {"Authorization": "Bearer <api_key>"}{ "model": "model_01kmqm4nrn9fw6r", "messages": [ {"role": "user", "content": "What is the status of INC-4821?"} ], "tools": [ { "type": "function", "function": { "name": "get_incident", "description": "Get incident status", "parameters": { "type": "object", "properties": { "incident_id": {"type": "string"} } } } } ], "tool_choice": "auto"}
Example Request (Streaming):
POST /api/v3/inference/chat/completionsHeaders: {"Authorization": "Bearer <api_key>"}{ "model": "model_01kmqm4nrn9fw6r", "messages": [{"role": "user", "content": "Tell me a story"}], "stream": true}
Example Response (Streaming):
data: {"id":"nugen-abc123","created":1704123600.0,"model":"model_01kmqm4nrn9fw6r","choices":[{"index":0,"delta":{"role":"assistant","content":"Once"},"finish_reason":null}]}data: {"id":"nugen-abc123","created":1704123600.0,"model":"model_01kmqm4nrn9fw6r","choices":[{"index":0,"delta":{"content":" upon"},"finish_reason":null}]}data: {"id":"nugen-abc123","created":1704123600.0,"model":"model_01kmqm4nrn9fw6r","choices":[{"index":0,"delta":{"content":" a time"},"finish_reason":"stop"}],"usage":{"prompt_tokens":10,"completion_tokens":5,"total_tokens":15}}data: [DONE]
Provisioned throughput:
Reserve inference capacity for production workloads. Pass your provisioned id in model as ptm_id:model_id in place of the plain model_id. The request and response surface is identical, so your application code stays as it is once capacity is reserved. Available for aligned models on the Enterprise Edition.
Notes:
Supports both text-only and vision/multimodal inputs depending on the model
Using Domain-Aligned Models: Pass the model_id from your completed alignment project as the model parameter. Get your aligned model IDs from GET /api/v3/alignment-projects/{id}/status or GET /api/v3/models/aligned
Function calling enables models to invoke external tools via tools and tool_choice parameters
confidence_score (0-100) comes from inference-time alignment: uncertainty is quantified across the generation trajectory, flagging any drift from your domain. Available on aligned models only; always null for base models. In streaming mode, confidence arrives as its own {"object": "confidence", ...} events interleaved with content chunks, one per generated span, rather than a single value on the final chunk
For streaming, usage statistics are included only in the final chunk
Conversations are automatically saved to S3 if enabled in user settings
Include X-Session-ID header for multi-turn conversation tracking
Use prompt_truncate_len to control context window usage for long conversations
The reasoning parameter enables advanced reasoning capabilities for supported models