Make inference requests

You can run inference for a target use case (and optionally model) using the Adaptive Python SDK, or the OpenAI Python library. The Adaptive SDK adopts a messages format similar to OpenAI’s Python library.

If you do not set a model, the Adaptive client routes your request to the model you’ve set as default for the client’s use case, or one of the models included in an active A/B test.

Interactions (pairs of [messages, completion] resulting from chat requests) are logged and saved to the Adaptive Interaction Store by default.

response = client.chat.create(
  use_case="use_case_key",
  model="[optional_model_key]",
  messages=[
    {"role": "system", "content": "You are a good bot."},
    {"role": "user", "content": "Hello model!"}
  ],
  labels={"label_key": "label_value"}
)

response = client.chat.create(
  use_case="use_case_key",
  model="[optional_model_key]",
  messages=[
    {"role": "system", "content": "You are a good bot."},
    {"role": "user", "content": "Hello model!"}
  ],
  labels={"label_key": "label_value"}
)

response = oai_client.chat.completions.create(
  model="use_case_key/[optional_model_key]",
  messages=[
    {"role": "system", "content": "You are a good bot."},
    {"role": "user", "content": "Hello model!"}
  ],
  metadata={"label_key": "label_value"}
)

import requests 

headers = {"Authorization": "Bearer ADAPTIVE_API_KEY"}
payload = { 
  "model": "use_case_key/[optional_model_key]",
  "messages": [
    {"role": "system", "content": "You are a good bot."},
    {"role": "user", "content": "Hello model!"}
  ],
  "labels": {"label_key": "label_value"}
}


response = requests.post(
  url="ADAPTIVE_URL/api/v1/chat/completions",
  json=payload,
  headers=headers
)

curl "$ADAPTIVE_URL/api/v1/chat/completions" \
  -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $ADAPTIVE_API_KEY" \
  -d '{
    "model":"use_case_key/[optional_model_key]",
    "messages":[{"role":"user","content":"Hello model!"}],
    "labels":{"label_key":"label_value"}
  }'

You can optionally tag requests with labels, as illustrated above. Labels are useful to organize and categorize completions. The labels parameter is a dictionary of user-defined key-value pairs, for example labels = {"project": "RAG Bot", "topic": "Industrial Tools"}. If you are using the OpenAI Python library, labels is not a supported parameter, but you can pass your labels in metadata, and they will be logged as any other label in Adaptive Engine.

Although the OpenAI Python library can be used to make inference requests to your Adaptive Engine deployment, not all of Adaptive Engine’s input parameters are supported by OpenAI’s library and vice-versa. See Chat API Reference for a list of supported parameters.

Both the Adaptive SDK’s and OpenAI Python library’s return types are Pydantic Models, which help with autocompletion within your editor. You can access the model’s response text with:

completion_text = response.json()["choices"][0]["message"]["content"]