Custom API Language Models

Overview

This document details the integration process for "Custom API Language Models" with DynamoEval.

Requirements

1. The custom model endpoint must accept HTTP POST requests
2. The endpoint must handle authentication ( if required )
   • Current supported authentication types are:
     • No Authentication
     • Bearer Token
     • API Key
3. The endpoint must either:
   • Accept DynamoEval's standard request format and return the expected response format, OR
   • Use JSONata transformations to convert between your format and DynamoEval's format

Request/Response Format Specification

Request Format

DynamoEval supports both single-turn and multi-turn conversation formats when integrating with custom language models.

Single-Turn Request Format

{
  "messages": [
    [
      {
        "role": "user",
        "content": "What is machine learning?"
      }
    ],
    [
      {
        "role": "user", 
        "content": "Explain neural networks"
      }
    ]
  ],
  "N": 1,
  "seq_len": 1024,
  "temperature": 1
}

Available fields:

• messages: Array of conversation turns, where each turn contains message objects
  • role: Must be "user" for single-turn requests
  • content: The actual message text
• N: Number of responses to generate (minimum: 1)
• seq_len: Maximum sequence length for generation
• temperature: Sampling temperature between 0-2

Multi-Turn Request Format

{
  "messages": [
    [
      {
        "role": "system",
        "content": "You are a helpful AI assistant"
      },
      {
        "role": "user",
        "content": "What is machine learning?"
      },
      {
        "role": "assistant", 
        "content": "Machine learning is..."
      },
      {
        "role": "user",
        "content": "Can you explain neural networks?"
      }
    ]
  ],
  "N": 1,
  "seq_len": 1024,
  "temperature": 1
}

The multi-turn format supports:

• Three role types: "system", "user", and "assistant"
• Conversation history within each turn
• System prompts for context setting

Response Format

Your custom model endpoint must return responses in one of these formats:

String Response Format

[
  "This is the first generated response",
  "This is the second generated response"
]

Boolean Response Format (for Guardrail Models)

[
  false,
  true
]

Code Snippets

Create Custom API Language Model

Example 1: Direct Integration (No Transformations + No Auth)

If your API already matches DynamoEval's format and doesn't require authentication:

from dynamofl.entities import AuthTypeEnum

model = dfl.create_custom_model(
    name="My Custom Model",
    remote_model_endpoint="https://api.example.com/v1/generate",
    remote_api_auth_config={
        "_type": AuthTypeEnum.NO_AUTH
    }
)

Example 2: With Transformations + Bearer Auth (Single-turn)

If your API uses a different format, you'll need to use JSONata transformations. For details about JSONata transformations in DynamoEval, refer to the JSONata Transformations section in Custom Applications documentation.

from dynamofl.entities import AuthTypeEnum

# Transform DynamoEval format to your API format
request_transformation_expression = """
{
    "messages": $reduce(messages, $append)
}
"""

# Transform your API response to DynamoEval format
response_transformation_expression = """
[choices[0].message.content]
"""
# Extract content from structured response


model = dfl.create_custom_model(
    name="My Custom Model",
    remote_model_endpoint="https://api.example.com/v1/generate",
    remote_api_auth_config={
        "_type": AuthTypeEnum.BEARER,
        "config": {
            "token": "your-bearer-token"
        }
    },
    request_transformation_expression=request_transformation_expression,
    response_transformation_expression=response_transformation_expression,
    multi_turn_support=False,
)

Connecting with known language model providers

Together AI system

If you want to connect with Together AI system, you can use the following code snippet:

from dynamofl.entities import AuthTypeEnum

endpoint = "https://api.together.xyz/v1/chat/completions"
remote_api_auth_config = {
    "_type": AuthTypeEnum.BEARER,
    "config": {
        "token": "your-bearer-token"
    }
}

# You can use your required model name here, below is an example
request_transformation_expression = """{
  "messages": $reduce(messages, $append),
  "model": "mistralai/Mistral-7B-Instruct-v0.3"
}
"""
response_transformation_expression="""[choices[0].message.content]"""

model = dfl.create_custom_model(
    name="Together AI: Custom API LM (SDK)",
    remote_model_endpoint=endpoint,
    remote_api_auth_config=remote_api_auth_config,
    request_transformation_expression=request_transformation_expression,
    response_transformation_expression=response_transformation_expression,
    
)

Databricks' LM

If you want to connect with Databricks' LM, you can use the following code snippet:

from dynamofl.entities import AuthTypeEnum

endpoint = "https://dbc-219bb6de-02df.cloud.databricks.com/serving-endpoints/databricks-dbrx-instruct/invocations"
remote_api_auth_config = {
    "_type": AuthTypeEnum.BEARER,
    "config": {
        "token": "your-bearer-token"
    }
}
request_transformation_expression = """{
  "messages": $reduce(messages, $append),
  "max_tokens": 128
}
"""
response_transformation_expression="""[choices[0].message.content]"""

model = dfl.create_custom_model(
    name="Databricks Multi-turn: Custom API LM (SDK)",
    remote_model_endpoint=endpoint,
    remote_api_auth_config=remote_api_auth_config,
    request_transformation_expression=request_transformation_expression,
    response_transformation_expression=response_transformation_expression,
    multi_turn_support=True,
)

Supported Authentication Types

The remote_api_auth_config parameter supports the following authentication types:

1. No Authentication

from dynamofl.entities import AuthTypeEnum

remote_api_auth_config = {
    "_type": AuthTypeEnum.NO_AUTH
}

2. Bearer Token

from dynamofl.entities import AuthTypeEnum

remote_api_auth_config = {
    "_type": AuthTypeEnum.BEARER,
    "config": {
        "token": "your-bearer-token"
    }
}

3. API Key

from dynamofl.entities import AuthTypeEnum

remote_api_auth_config = {
    "_type": AuthTypeEnum.API_KEY,
    "config": {
        "token": "your-api-key",
        "auth_field_name": "Basic",           # Optional, defaults to "Basic"
        "api_auth_header": "Authorization"     # Optional, defaults to "Authorization"
    }
}

All authenticated requests will include the following default headers:

{
    "Content-Type": "application/json;charset=UTF-8",
    "Accept": "application/json, text/plain, */*"
}

Authentication Header Examples

The authentication headers will be generated as follows:

• Bearer Token: Adds "Authorization": "Bearer your-bearer-token"
• API Key: Adds "Authorization": "Basic your-api-key" (customizable using auth_field_name and api_auth_header)
• No Auth: No additional headers added

Advanced Configuration

Additional parameters allow fine-tuning of the model integration:

model = dfl.create_custom_model(
    name="Advanced Custom Model",
    remote_model_endpoint="https://api.example.com/v1/generate",
    remote_api_auth_config={...},
    response_type="string",           # Type of response expected from the model (default: string)
    batch_size=32,                    # Number of requests to batch together (default: 32)
    multi_turn_support=True,          # Whether model supports conversation history (default: True)
    enable_retry=False                # Enable automatic retries on failure (default: False)
)