Custom LLMs
Custom LLMs lets you connect your own language model providers to InteractiveAI. Once connected, these models become available in the Playground for prompt testing and in Evaluators for automated quality assessment. Your provider charges you directly based on usage, while InteractiveAI simply routes requests through your credentials.
This section serves two purposes. Connections store your API keys for providers like OpenAI, Anthropic, and Google AI Studio. Configurations define pricing and tokenization settings so InteractiveAI can accurately calculate costs when you use models through direct API integrations rather than the InteractiveAI Router.
Connections
Connections store the credentials InteractiveAI uses to communicate with external LLM providers. Each connection links a provider name to an API key and endpoint configuration.
Viewing Connections
The Connections tab displays all configured providers in a table showing:
Provider
Display name you assigned to this connection
Adapter
The provider type (e.g., openai, google-ai-studio)
Base URL
API endpoint (default uses the provider's standard endpoint)
API Key
Masked key showing the last few characters
Adding a Connection
Click the + button in the top-right corner to open the Add LLM Connection modal.
Provider name
A display name to identify this connection within InteractiveAI (e.g., "OpenAI Production", "Gemini")
LLM adapter
The provider type that determines the API schema. Options include openai, google-ai-studio, anthropic, and others
API Base URL
Leave as default to use the provider's standard endpoint, or enter a custom URL for self-hosted or proxy configurations
API Key
Your provider's API key. Stored encrypted in the database
Extra Headers
Optional HTTP headers to include with requests (also stored encrypted)
You can create multiple connections for the same provider. This is useful for separating production and development keys, or for connecting to different accounts.
Configurations
Configurations define model metadata for cost tracking and tokenization. When you use models through direct API integrations (not through the InteractiveAI Router), these definitions tell the platform how to calculate costs based on token usage.
Viewing Configurations
The Configurations tab displays all model definitions in a table showing:
Model Name
Identifier for the model (e.g., gemini-2.5-pro, gpt-5)
Prices per unit
Number of pricing rules configured
Provider
Who maintains this definition (User or System)
Match Pattern
Regex pattern used to identify this model in traces
Tokenizer
Tokenization method for counting tokens
Created
When the definition was added
Last Used
Most recent usage of this model
Adding a Model Definition
Click Add Model Definition to open the configuration modal.
Model Details
Model Name
The model identifier as it appears in API calls (e.g., gpt-4-turbo, claude-3-opus)
Match Pattern
A regex pattern to match this model in your traces. For example, (?i)^(gpt-5)$ matches "gpt-5" case-insensitively
Tokenizer
The tokenization method used to count tokens. Select the appropriate tokenizer for accurate cost calculation
Prices
Set prices per usage type. Usage types must exactly match the keys in your ingested usage details.
For OpenAI and compatible providers, typical usage types are:
input— Price per input tokenoutput— Price per output token
For Anthropic models, you may also configure:
input— Price per input tokenoutput— Price per output tokencache_read— Price per cached input token
Click + Add Price to add additional usage types as needed.
Price Preview
The modal displays a live preview showing your configured prices at different scales:
input
$0.000001
$0.001
$1
output
$0.000002
$0.002
$2
This helps you verify that pricing is configured correctly before saving.
When to Use Custom LLMs
Custom LLMs are essential when you want to:
Use the Playground: Test prompts interactively against your preferred models
Run Evaluators: Power LLM-as-a-Judge evaluations with your own model credentials
Track costs accurately: Define pricing for models used through direct integrations so dashboards reflect actual spending
If you're using the InteractiveAI Router for model access, you don't need to configure Custom LLMs, the Router handles provider connections and cost tracking automatically.
Last updated
Was this helpful?