Integrating LiteLLM Proxy with Yandex Cloud AI Studio and Monium Traces
LiteLLM Proxy
LiteLLM Proxy simplifies multi-model workflows by providing centralized application access management, rate limiting, cost tracking, and trace generation for request analysis.
You can monitor your requests using Monium Traces, a distributed tracing system. A trace aggregates data about individual operations, such as their sequence, duration, and errors.
Traces help you:
- Identify the exact step, service, or application component where an error or latency occurs.
- Analyze prompts, responses, and token spend. For example, you can view the exact context provided to the model, generation time, and prompt/response token counts.
Integrating LiteLLM Proxy with the tracing system allows you to collect this data on the proxy side: you do not need to add telemetry code to your application.
In this tutorial, you will deploy LiteLLM Proxy and configure trace export to Monium Traces via OpenTelemetry v2
Required paid resources
The solution support costs include:
- VM computing resources and a disk (see Yandex Compute Cloud pricing).
- Public IP address and outgoing traffic (see Yandex Virtual Private Cloud pricing).
- Text generation (see Yandex Cloud AI Studio pricing
). - Trace logging to Monium Traces (see Yandex Monium pricing).
Steps to follow
- Get your cloud ready.
- Deploy and configure the LiteLLM Proxy instance.
- Send requests to the model and check the traces.
If you deployed this solution only for testing, you can delete the created resources after completing the tutorial.
Getting started
Sign up for Yandex Cloud and create a billing account:
- Navigate to the management console
and log in to Yandex Cloud or create a new account. - On the Yandex Cloud Billing
page, make sure you have a billing account linked and it has theACTIVEorTRIAL_ACTIVEstatus. If you do not have a billing account, create one and link a cloud to it.
If you have an active billing account, you can create or select a folder for your infrastructure on the cloud page
Set up your environment
Create a VM from the Ubuntu 24.04 LTS public image.
Warning
When creating the VM, specify at least 4 GB of RAM.
Create API keys
Create two service accounts with API keys (one per service) so LiteLLM Proxy can access Yandex Cloud AI Studio and send traces to Monium Traces:
-
Create a service account to enable LiteLLM Proxy to access Yandex Cloud AI Studio:
- In the management console
, select the folder where you are going to create your infrastructure. - Navigate
to Identity and Access Management. - Click Create service account.
- In the window that opens, enter the service account name:
litellm-ai-studio-sa. - In the Roles in the folder field, click
Add role and select roleai.languageModels.user. - Click Create.
- In the management console
-
Create an API key for LiteLLM Proxy access to Yandex Cloud AI Studio:
- In the list of service accounts, select
litellm-ai-studio-sa. - At the top of the page, click
and select Create new key → Create API key. - In the window that opens, select a scope
yc.ai.languageModels.executeand click Create. - Save the secret part of the API key. You will not be able to view it again after closing the dialog window.
- In the list of service accounts, select
-
Create a service account and API key to enable LiteLLM Proxy to access Yandex Monium. Specify the following properties:
- Service account name:
litellm-monium-sa. - Role:
monium.traces.writer. - API key scope:
yc.monium.traces.write.
- Service account name:
Deploy and configure the LiteLLM Proxy instance
To deploy LiteLLM Proxy on the VM you created, install Docker, prepare the configuration files, and start the container:
-
Connect to the VM over SSH or via OS Login depending on the connection method you selected when creating your resources.
-
Install and configure Docker:
-
Install the required dependencies:
sudo apt update sudo apt install ca-certificates curl gnupg -
Install the Docker GPG
key:sudo install -m 0755 -d /etc/apt/keyrings sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc sudo chmod a+r /etc/apt/keyrings/docker.asc -
Add the Docker repository to your
aptpackage sources:echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \ $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ sudo tee /etc/apt/sources.list.d/docker.list > /dev/null -
Install Docker:
sudo apt update sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -
Add the current user to the
dockergroup and open a new shell to apply the changes:sudo usermod -aG docker $USER newgrp docker -
Make sure Docker is installed and operational:
docker run hello-worldResult:
Unable to find image 'hello-world:latest' locally latest: Pulling from library/hello-world 4f55086f7dd0: Pull complete d5e71e642bf5: Download complete Digest: sha256:5e23090353324d887c48ad5e5c56d294eab81588df9605b07d1afe89******** Status: Downloaded newer image for hello-world:latest Hello from Docker! This message shows that your installation appears to be working correctly. ...
-
-
In your home directory, create a project directory for Docker Compose and navigate into it:
mkdir ~/litellm-proxy && cd ~/litellm-proxy -
Create the required configuration files for the project:
touch .env touch config.yaml touch docker-compose.yaml -
In a text editor, open the
.envfile:nano .env -
Add the following environment variables to the
.envfile:YANDEX_AI_STUDIO_API_KEY=<ai_studio_key_secret_part> MONIUM_API_KEY=<monium_key_secret_part> MONIUM_PROJECT=folder__<folder_ID> LITELLM_MASTER_KEY=sk-local-litellmWhere:
-
YANDEX_AI_STUDIO_API_KEY: Secret part of the API key for thelitellm-ai-studio-saservice account. -
MONIUM_API_KEY: Secret part of the API key for thelitellm-monium-saservice account. -
MONIUM_PROJECT: Monium project ID. It consists of thefolder__prefix followed by your folder ID where the VM and service accounts were created. Here is an example:folder__b1gt6g8ht345********. -
LITELLM_MASTER_KEY: Key that applications use to access LiteLLM Proxy. You define it in the.envfile so the proxy can authenticate incoming requests. For this tutorial, keep thesk-local-litellmvalue. This value is used in the code of theagent_example.pytest application you will create in the next step.Tip
For production environments, run
openssl rand -hex 32to generate a random string and prepend it withsk-.If you updated
LITELLM_MASTER_KEYin the.envfile after starting the container, recreate the container by runningdocker compose up -d --force-recreate. If using a custom key, make sure to set the exact same value in theLITELLM_PROXY_KEYenvironment variable in your terminal before running theagent_example.pytest application:export LITELLM_PROXY_KEY='<key_from_env_file>'The
openailibrary inside the test application automatically attaches this key to theAuthorizationheader using theBearerscheme. The header value must match the key configured for the proxy in.env.
-
-
In a text editor, open the
config.yamlfile:nano config.yaml -
Add the following configuration to the
config.yamlfile:# LiteLLM proxy: Yandex AI Studio (OpenAI-compatible) → traces to Yandex Monium. # Model URI: gpt://<folder_ID>/deepseek-v4-flash model_list: - model_name: deepseek-v4-flash litellm_params: model: openai/gpt://<folder_ID>/deepseek-v4-flash api_base: https://ai.api.cloud.yandex.net/v1 api_key: os.environ/YANDEX_AI_STUDIO_API_KEY litellm_settings: drop_params: true set_verbose: false # Both this callback and LITELLM_OTEL_V2=true are required. callbacks: - otel general_settings: master_key: os.environ/LITELLM_MASTER_KEYWhere
modelis the model ID in LiteLLM: theopenai/prefix and the Yandex Cloud AI Studio model URI containing your folder ID. Here is an example:openai/gpt://b1gt6g8ht345********/deepseek-v4-flash.The
os.environ/prefix indicates that the parameter value should be retrieved from the respective environment variable. -
In a text editor, open the
docker-compose.yamlfile:nano docker-compose.yaml -
Add the Docker Compose project configuration into the
docker-compose.yamlfile:services: litellm: image: docker.litellm.ai/berriai/litellm:main-latest container_name: litellm-proxy restart: unless-stopped # Host network: on this class of host net.ipv4.ip_forward=0 and bridge NAT drops connections. network_mode: host volumes: - ./config.yaml:/app/config.yaml:ro environment: - LITELLM_OTEL_V2=true - OTEL_EXPORTER=otlp_http - OTEL_ENDPOINT=https://ingest.monium.yandex.cloud/otlp - OTEL_SERVICE_NAME=litellm-proxy - OTEL_ENVIRONMENT_NAME=local - OTEL_HEADERS=Authorization=Api-Key ${MONIUM_API_KEY:?MONIUM_API_KEY},x-monium-project=${MONIUM_PROJECT:?MONIUM_PROJECT},x-monium-cluster=${MONIUM_CLUSTER:-default},x-monium-service=litellm-proxy - OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_only - YANDEX_AI_STUDIO_API_KEY=${YANDEX_AI_STUDIO_API_KEY:?YANDEX_AI_STUDIO_API_KEY} - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY:-sk-local-litellm} command: - --config - /app/config.yaml - --port - "4000" - --host - "0.0.0.0"Configuration parameters:
-
environment: Environment variables:OTEL_EXPORTER=otlp_http: Configures trace delivery via OTLP over HTTP.OTEL_ENDPOINT=https://ingest.monium.yandex.cloud/otlp: Sets the base endpoint address. LiteLLM Proxy appends/v1/tracesto it and sends traces tohttps://ingest.monium.yandex.cloud/otlp/v1/traces.x-monium-service=litellm-proxy: Name to assign to the service in the Monium UI.OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_only: Enables capturing model request and response text inside spans.
-
network_mode: host: Runs the container in the host's network.This allows the container to access Yandex Cloud AI Studio and Yandex Monium even if IP packet forwarding is disabled on the host (
net.ipv4.ip_forward = 0). LiteLLM Proxy listens for incoming connections on VM port4000.
-
-
Run the container:
docker compose up -dResult:
[+] up 27/27 ✔ Image docker.litellm.ai/berriai/litellm:main-latest Pulled 82.4s ✔ Container litellm-proxy Started 43.1s -
To check that LiteLLM Proxy is ready to accept requests, run this command in the VM terminal:
for _ in $(seq 1 30); do curl -sf http://127.0.0.1:4000/health/liveliness >/dev/null && echo "proxy is up" && break sleep 1 doneResult:
proxy is up
Send requests to the model and check the traces
To test trace delivery, create and run a test application on the same VM where LiteLLM Proxy is running. The application will route requests to the model through the proxy, which in turn will send the traces to Monium Traces. You can then find these traces in Monium to review the request details.
Create and run a test application
The Python-based test application simulates a customer conversation with a support agent. It uses the openai library to send requests to the model via LiteLLM Proxy. Install this library inside a dedicated virtual environment to isolate the application dependencies.
To create and run the application:
-
Navigate to the
~/litellm-proxydirectory you created when deploying LiteLLM Proxy on the VM:cd ~/litellm-proxy -
Install Python and the
venvmodule for virtual environment management:sudo apt install python3 python3.12-venv -
Create and activate the
litellm-venvvirtual environment:python3 -m venv ~/litellm-venv source ~/litellm-venv/bin/activate -
Install the
openailibrary:pip install 'openai==2.54.0' -
Create and open the
agent_example.pytest application file in a text editor:touch agent_example.py nano agent_example.py -
Add this code to the
agent_example.pyfile:Application code
"""Customer support agent that talks to a local LiteLLM proxy only. No client-side telemetry: tracing is handled by the proxy (OTEL v2 → Monium). """ from __future__ import annotations import argparse import json import os import random import sys from openai import OpenAI DEFAULT_PROXY_BASE = "http://127.0.0.1:4000/v1" DEFAULT_PROXY_KEY = "sk-local-litellm" DEFAULT_MODEL = "deepseek-v4-flash" class KnowledgeBase: """Simple knowledge base for customer support.""" def __init__(self) -> None: self.knowledge = { "shipping": { "standard": "Standard shipping takes 5-7 business days and costs $5.99", "express": "Express shipping takes 2-3 business days and costs $12.99", "overnight": "Overnight shipping delivers next business day and costs $24.99", }, "returns": { "policy": "You can return items within 30 days of purchase for a full refund", "process": "To return an item, go to your account, find the order, and click 'Return Item'", "condition": "Items must be in original condition with tags attached", }, "account": { "password_reset": "Click 'Forgot Password' on the login page and follow the email instructions", "update_info": "You can update your account information in the 'My Account' section", "delete_account": "Contact customer service to delete your account permanently", }, "products": { "warranty": "All products come with a 1-year manufacturer warranty", "availability": "Check product availability on the product page", "specifications": "Product specifications are listed in the product description", }, } def search(self, query: str) -> str: query_lower = query.lower() if any( word in query_lower for word in ["ship", "delivery", "shipping", "deliver"] ): if "express" in query_lower or "fast" in query_lower: return self.knowledge["shipping"]["express"] if "overnight" in query_lower or "next day" in query_lower: return self.knowledge["shipping"]["overnight"] return self.knowledge["shipping"]["standard"] if any(word in query_lower for word in ["return", "refund", "exchange"]): if "how" in query_lower or "process" in query_lower: return self.knowledge["returns"]["process"] if "condition" in query_lower: return self.knowledge["returns"]["condition"] return self.knowledge["returns"]["policy"] if any( word in query_lower for word in ["account", "password", "login", "profile"] ): if "password" in query_lower or "reset" in query_lower: return self.knowledge["account"]["password_reset"] if "update" in query_lower or "change" in query_lower: return self.knowledge["account"]["update_info"] if "delete" in query_lower: return self.knowledge["account"]["delete_account"] if any( word in query_lower for word in ["product", "warranty", "available", "spec"] ): if "warranty" in query_lower: return self.knowledge["products"]["warranty"] if "available" in query_lower or "stock" in query_lower: return self.knowledge["products"]["availability"] return self.knowledge["products"]["specifications"] return ( "I couldn't find specific information about that. " "Let me help you with a general response." ) class CustomerSupportAgent: """Customer support agent using an OpenAI-compatible LiteLLM proxy.""" def __init__( self, *, base_url: str, api_key: str, model: str, ) -> None: self.client = OpenAI(base_url=base_url, api_key=api_key) self.model = model self.knowledge_base = KnowledgeBase() self.conversation_history: list[dict] = [] self.system_prompt = """ You are a helpful and friendly customer support agent. Your role is to: 1. Assist customers with their questions and concerns 2. Use the available tools whenever they help answer the question 3. Be polite, professional, and empathetic 4. If you don't know something, admit it and offer to escalate to a human agent 5. Keep responses concise but helpful Always maintain a positive and solution-oriented attitude. Prefer calling tools for shipping, returns, order status, refunds, accounts, and products instead of inventing details. """.strip() def search_knowledge_base(self, query: str) -> str: return self.knowledge_base.search(query) def get_order_status(self, order_id: str) -> str: statuses = ["Processing", "Shipped", "Out for Delivery", "Delivered"] return f"Order {order_id} status: {random.choice(statuses)}" def process_refund(self, order_id: str, reason: str) -> str: return ( f"Refund request for order {order_id} has been submitted. " f"Reason: {reason}. You'll receive confirmation within 24 hours." ) def get_available_tools(self) -> list[dict]: return [ { "type": "function", "function": { "name": "search_knowledge_base", "description": ( "Search the knowledge base for shipping, returns, " "accounts, and products" ), "parameters": { "type": "object", "properties": { "query": { "type": "string", "description": "The customer's question or topic", } }, "required": ["query"], }, }, }, { "type": "function", "function": { "name": "get_order_status", "description": "Get the current status of a customer's order", "parameters": { "type": "object", "properties": { "order_id": { "type": "string", "description": "The order ID to check", } }, "required": ["order_id"], }, }, }, { "type": "function", "function": { "name": "process_refund", "description": "Process a refund request for a customer", "parameters": { "type": "object", "properties": { "order_id": { "type": "string", "description": "The order ID to refund", }, "reason": { "type": "string", "description": "Reason for the refund", }, }, "required": ["order_id", "reason"], }, }, }, ] def execute_tool_call(self, tool_call) -> str: function_name = tool_call.function.name arguments = json.loads(tool_call.function.arguments) if function_name == "search_knowledge_base": return self.search_knowledge_base(arguments["query"]) if function_name == "get_order_status": return self.get_order_status(arguments["order_id"]) if function_name == "process_refund": return self.process_refund(arguments["order_id"], arguments["reason"]) return f"Unknown tool: {function_name}" def chat(self, user_message: str) -> str: self.conversation_history.append({"role": "user", "content": user_message}) messages = [ {"role": "system", "content": self.system_prompt} ] + self.conversation_history try: response = self.client.chat.completions.create( model=self.model, messages=messages, tools=self.get_available_tools(), tool_choice="auto", temperature=0.4, max_tokens=700, ) assistant_message = response.choices[0].message # Allow a few tool rounds so one user turn can expand into several LLM calls. for _ in range(4): if not assistant_message.tool_calls: break self.conversation_history.append( { "role": "assistant", "content": assistant_message.content, "tool_calls": [ { "id": tool_call.id, "type": "function", "function": { "name": tool_call.function.name, "arguments": tool_call.function.arguments, }, } for tool_call in assistant_message.tool_calls ], } ) for tool_call in assistant_message.tool_calls: tool_result = self.execute_tool_call(tool_call) print(f" [tool] {tool_call.function.name} -> {tool_result}") self.conversation_history.append( { "role": "tool", "tool_call_id": tool_call.id, "content": tool_result, } ) messages = [ {"role": "system", "content": self.system_prompt} ] + self.conversation_history response = self.client.chat.completions.create( model=self.model, messages=messages, tools=self.get_available_tools(), tool_choice="auto", temperature=0.4, max_tokens=700, ) assistant_message = response.choices[0].message response_content = assistant_message.content or "" self.conversation_history.append( {"role": "assistant", "content": response_content} ) return response_content except Exception as exc: # noqa: BLE001 - surface proxy/LLM errors to the chat return f"I apologize, but I'm experiencing technical difficulties. Error: {exc}" def reset_conversation(self) -> None: self.conversation_history = [] DEMO_TURNS = [ "Hi! I need express shipping info for an order I'm about to place.", "Also, what is the status of my existing order ORD-10482?", "The item arrived damaged. What is your return policy and how do I start a return?", "Please process a refund for order ORD-10482 because the item arrived damaged.", "One last thing: how long is the product warranty, and how do I reset my account password?", ] def run_demo(agent: CustomerSupportAgent) -> int: print("Customer Support Agent (LiteLLM proxy demo)") print("=" * 60) print(f"Proxy model: {agent.model}") print(f"Turns: {len(DEMO_TURNS)}\n") for idx, user_input in enumerate(DEMO_TURNS, start=1): print(f"[{idx}/{len(DEMO_TURNS)}] You: {user_input}") response = agent.chat(user_input) print(f"Agent: {response}\n") print("Demo finished.") return 0 def run_interactive(agent: CustomerSupportAgent) -> int: print("Customer Support Agent") print("=" * 50) print("Type 'quit' to exit or 'reset' to start a new conversation.\n") while True: try: user_input = input("\nYou: ").strip() except (EOFError, KeyboardInterrupt): print( "\nAgent: Thank you for contacting customer support. Have a great day!" ) return 0 if user_input.lower() in {"quit", "exit", "bye"}: print( "\nAgent: Thank you for contacting customer support. Have a great day!" ) return 0 if user_input.lower() == "reset": agent.reset_conversation() print("\nAgent: Conversation reset. How can I help you today?") continue if not user_input: continue print(f"\nAgent: {agent.chat(user_input)}") def build_agent_from_env() -> CustomerSupportAgent: return CustomerSupportAgent( base_url=os.getenv("LITELLM_PROXY_BASE", DEFAULT_PROXY_BASE), api_key=os.getenv("LITELLM_PROXY_KEY", DEFAULT_PROXY_KEY), model=os.getenv("LITELLM_MODEL", DEFAULT_MODEL), ) def main(argv: list[str] | None = None) -> int: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument( "--interactive", action="store_true", help="Run an interactive REPL instead of the multi-step demo", ) args = parser.parse_args(argv) agent = build_agent_from_env() if args.interactive: return run_interactive(agent) return run_demo(agent) if __name__ == "__main__": sys.exit(main())Through LiteLLM Proxy, the application sends five customer messages to the model one by one: a question about delivery, an order status check, a question on return policy, a refund request, and a final message with two questions, one regarding warranty and password reset.
To prepare a response, the model can request a tool call: a local knowledge base search, an order status check, or a refund transaction. The application processes this tool call and sends the results back to the model through the proxy in the next request. Order checks and refunds are simulated in the app code.
The number of requests depends on which tools the model calls. For example, during an order status check, the first request allows the model to select the tool, while the second one returns the tool results so the model can generate a response for the customer. LiteLLM Proxy generates a unique trace for each request and routes it to Monium Traces. The application itself does not send any telemetry data.
-
Run the test app:
python3 agent_example.pyResult:
Customer Support Agent (LiteLLM proxy demo) ============================================================ Proxy model: deepseek-v4-flash Turns: 5 ... [1/5] You: Hi! I need express shipping info for an order I'm about to place. [tool] search_knowledge_base -> Express shipping takes 2-3 business days and costs $12.99 Agent: Great news! Here's the express shipping information: ... ... Demo finished.
Find and view traces in Monium
To find the traces generated by the test application:
-
Navigate
to Monium. -
In the left-hand panel, select
LLM Observability. -
Use the timeline selector at the top of the screen to choose the period when you ran the test application.
-
Click the
icon in the top-right corner of the query bar and enter the following filter:{project = "folder__<folder_ID>", service = "litellm-proxy"}Where
<folder_id>is the ID of the folder where you created the service accounts. -
Click Run query.
The search results will display the traces generated by your test application. To view all operations within a trace, click its row and select Show full trace.
The expanded trace will display these spans:
POST /v1/chat/completions: HTTP request to LiteLLM Proxy.POST /v1/chat/completions: LiteLLM Proxy access key verification.chat gpt://<folder_ID>/deepseek-v4-flash: Model call. This span contains the model name, input and output token counts, and message text.
If the model requests tool calls, the model span logs their names (search_knowledge_base, get_order_status, or process_refund) and arguments.
The application stores the conversation history and appends it to each new request, causing the context size to increase in subsequent traces.
For more information on viewing traces, see Viewing and analyzing LLM traces.
Troubleshooting
|
Issue |
Solution |
|
Connection error ( |
|
|
Yandex Cloud AI Studio returns HTTP code |
|
|
The test application runs successfully, but traces do not appear in Monium |
Make sure that:
|
|
No container network connection ( |
Run the If it returns After you make the changes, run |
|
Spans do not contain message text |
Check that the After you make the changes, run |
How to delete the resources you created
If you deployed this solution for testing purposes only, you can delete the resources after completing this tutorial to stop paying for them:
- Delete the LiteLLM Proxy VM.
- Delete the disk if it was not deleted along with the VM.
- Delete the static public IP address if you reserved one for the VM.
Delete the litellm-ai-studio-sa and litellm-monium-sa service accounts if you no longer need them.
Scope is an API key parameter that restricts the key's use to specific operations in addition to the service account's access permissions. For details, see API keys with scope and validity limits.
A span is a basic distributed tracing element representing a single operation in your system, e.g., HTTP request, database call, or function execution. For more information, see Span.