Yandex Cloud
Documentation
PresentationContact UsTry it for free
© 2026 Direct Cursus Technology L.L.C.
Tutorials
    • All tutorials
    • Differentiation of access permissions for user groups
    • Inviting a new user and assigning roles
    • Creating an L7 load balancer with a Smart Web Security profile through an Application Load Balancer ingress controller
    • Creating a distributed infrastructure with secure access
    • Centralized online publication and DDoS protection of applications
    • Basic SWS setup
    • Connecting SolidWall WAF to a web application
    • Emergency L7 DDoS protection in Application Load Balancer
    • Delivering logs from a VM instance to Cloud Logging
    • Writing load balancer logs to PostgreSQL
    • Secure storage of GitLab CI passwords as Yandex Lockbox secrets
    • Service account with an OS Login profile for VM management via Ansible
    • Transferring logs from Container Optimized Image to Cloud Logging
    • Adding an HTML page for SmartCaptcha
    • Configuring alerts and dashboards in Monitoring
    • Uploading audit logs to Splunk SIEM
    • Uploading audit logs to ArcSight SIEM
    • Server-side encryption for an Object Storage bucket
    • Encrypting secrets in Hashicorp Terraform
    • Managing KMS keys with Hashicorp Terraform
    • Auto Unseal in Hashicorp Vault
    • Transferring a Yandex MPP Analytics for PostgreSQL cluster's logs to Yandex Cloud Logging
    • Obtaining the information you need to request the Russian Ministry of Digital Development to whitelist a resource
    • Uploading objects into an Object Storage bucket using an ephemeral access key
    • Integrating LiteLLM Proxy with Yandex Cloud AI Studio and Monium Traces

In this article:

  • Required paid resources
  • Steps to follow
  • Getting started
  • Set up your environment
  • Create API keys
  • Deploy and configure the LiteLLM Proxy instance
  • Send requests to the model and check the traces
  • Create and run a test application
  • Find and view traces in Monium
  • Troubleshooting
  • How to delete the resources you created
  1. Security
  2. Integrating LiteLLM Proxy with Yandex Cloud AI Studio and Monium Traces

Integrating LiteLLM Proxy with Yandex Cloud AI Studio and Monium Traces

Written by
Yandex Cloud
Updated at November 17, 2026
View in Markdown
  • Required paid resources
  • Steps to follow
  • Getting started
    • Set up your environment
    • Create API keys
  • Deploy and configure the LiteLLM Proxy instance
  • Send requests to the model and check the traces
    • Create and run a test application
    • Find and view traces in Monium
    • Troubleshooting
  • How to delete the resources you created

LiteLLM Proxy is a proxy server that enables access to LLM providers through a single interface compatible with the OpenAI API.

LiteLLM Proxy simplifies multi-model workflows by providing centralized application access management, rate limiting, cost tracking, and trace generation for request analysis.

You can monitor your requests using Monium Traces, a distributed tracing system. A trace aggregates data about individual operations, such as their sequence, duration, and errors.

Traces help you:

  • Identify the exact step, service, or application component where an error or latency occurs.
  • Analyze prompts, responses, and token spend. For example, you can view the exact context provided to the model, generation time, and prompt/response token counts.

Integrating LiteLLM Proxy with the tracing system allows you to collect this data on the proxy side: you do not need to add telemetry code to your application.

In this tutorial, you will deploy LiteLLM Proxy and configure trace export to Monium Traces via OpenTelemetry v2. You will then run a test application simulating a conversation with the DeepSeek V4 Flash model in Yandex Cloud AI Studio through LiteLLM Proxy and analyze the captured traces.

Required paid resourcesRequired paid resources

The solution support costs include:

  • VM computing resources and a disk (see Yandex Compute Cloud pricing).
  • Public IP address and outgoing traffic (see Yandex Virtual Private Cloud pricing).
  • Text generation (see Yandex Cloud AI Studio pricing).
  • Trace logging to Monium Traces (see Yandex Monium pricing).

Steps to followSteps to follow

  1. Get your cloud ready.
  2. Deploy and configure the LiteLLM Proxy instance.
  3. Send requests to the model and check the traces.

If you deployed this solution only for testing, you can delete the created resources after completing the tutorial.

Getting startedGetting started

Sign up for Yandex Cloud and create a billing account:

  1. Navigate to the management console and log in to Yandex Cloud or create a new account.
  2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can create or select a folder for your infrastructure on the cloud page.

Set up your environmentSet up your environment

Create a VM from the Ubuntu 24.04 LTS public image.

Warning

When creating the VM, specify at least 4 GB of RAM.

Create API keysCreate API keys

Create two service accounts with API keys (one per service) so LiteLLM Proxy can access Yandex Cloud AI Studio and send traces to Monium Traces:

Management console
  1. Create a service account to enable LiteLLM Proxy to access Yandex Cloud AI Studio:

    1. In the management console, select the folder where you are going to create your infrastructure.
    2. Navigate to Identity and Access Management.
    3. Click Create service account.
    4. In the window that opens, enter the service account name: litellm-ai-studio-sa.
    5. In the Roles in the folder field, click Add role and select role ai.languageModels.user.
    6. Click Create.
  2. Create an API key for LiteLLM Proxy access to Yandex Cloud AI Studio:

    1. In the list of service accounts, select litellm-ai-studio-sa.
    2. At the top of the page, click and select Create new key → Create API key.
    3. In the window that opens, select a scope yc.ai.languageModels.execute and click Create.
    4. Save the secret part of the API key. You will not be able to view it again after closing the dialog window.
  3. Create a service account and API key to enable LiteLLM Proxy to access Yandex Monium. Specify the following properties:

    • Service account name: litellm-monium-sa.
    • Role: monium.traces.writer.
    • API key scope: yc.monium.traces.write.

Deploy and configure the LiteLLM Proxy instanceDeploy and configure the LiteLLM Proxy instance

To deploy LiteLLM Proxy on the VM you created, install Docker, prepare the configuration files, and start the container:

  1. Connect to the VM over SSH or via OS Login depending on the connection method you selected when creating your resources.

  2. Install and configure Docker:

    1. Install the required dependencies:

      sudo apt update
      sudo apt install ca-certificates curl gnupg
      
    2. Install the Docker GPG key:

      sudo install -m 0755 -d /etc/apt/keyrings
      sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
      sudo chmod a+r /etc/apt/keyrings/docker.asc
      
    3. Add the Docker repository to your apt package sources:

      echo \
        "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
        $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
        sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
      
    4. Install Docker:

      sudo apt update
      sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
      
    5. Add the current user to the docker group and open a new shell to apply the changes:

      sudo usermod -aG docker $USER
      newgrp docker
      
    6. Make sure Docker is installed and operational:

      docker run hello-world
      

      Result:

      Unable to find image 'hello-world:latest' locally
      latest: Pulling from library/hello-world
      4f55086f7dd0: Pull complete 
      d5e71e642bf5: Download complete 
      Digest: sha256:5e23090353324d887c48ad5e5c56d294eab81588df9605b07d1afe89********
      Status: Downloaded newer image for hello-world:latest
      
      Hello from Docker!
      This message shows that your installation appears to be working correctly.
      ...
      
  3. In your home directory, create a project directory for Docker Compose and navigate into it:

    mkdir ~/litellm-proxy && cd ~/litellm-proxy
    
  4. Create the required configuration files for the project:

    touch .env
    touch config.yaml
    touch docker-compose.yaml
    
  5. In a text editor, open the .env file:

    nano .env
    
  6. Add the following environment variables to the .env file:

    YANDEX_AI_STUDIO_API_KEY=<ai_studio_key_secret_part>
    MONIUM_API_KEY=<monium_key_secret_part>
    MONIUM_PROJECT=folder__<folder_ID>
    LITELLM_MASTER_KEY=sk-local-litellm
    

    Where:

    • YANDEX_AI_STUDIO_API_KEY: Secret part of the API key for the litellm-ai-studio-sa service account.

    • MONIUM_API_KEY: Secret part of the API key for the litellm-monium-sa service account.

    • MONIUM_PROJECT: Monium project ID. It consists of the folder__ prefix followed by your folder ID where the VM and service accounts were created. Here is an example: folder__b1gt6g8ht345********.

    • LITELLM_MASTER_KEY: Key that applications use to access LiteLLM Proxy. You define it in the .env file so the proxy can authenticate incoming requests. For this tutorial, keep the sk-local-litellm value. This value is used in the code of the agent_example.py test application you will create in the next step.

      Tip

      For production environments, run openssl rand -hex 32 to generate a random string and prepend it with sk-.

      If you updated LITELLM_MASTER_KEY in the .env file after starting the container, recreate the container by running docker compose up -d --force-recreate. If using a custom key, make sure to set the exact same value in the LITELLM_PROXY_KEY environment variable in your terminal before running the agent_example.py test application:

      export LITELLM_PROXY_KEY='<key_from_env_file>'
      

      The openai library inside the test application automatically attaches this key to the Authorization header using the Bearer scheme. The header value must match the key configured for the proxy in .env.

  7. In a text editor, open the config.yaml file:

    nano config.yaml
    
  8. Add the following configuration to the config.yaml file:

    # LiteLLM proxy: Yandex AI Studio (OpenAI-compatible) → traces to Yandex Monium.
    # Model URI: gpt://<folder_ID>/deepseek-v4-flash
    
    model_list:
      - model_name: deepseek-v4-flash
        litellm_params:
          model: openai/gpt://<folder_ID>/deepseek-v4-flash
          api_base: https://ai.api.cloud.yandex.net/v1
          api_key: os.environ/YANDEX_AI_STUDIO_API_KEY
    
    litellm_settings:
      drop_params: true
      set_verbose: false
      # Both this callback and LITELLM_OTEL_V2=true are required.
      callbacks:
        - otel
    
    general_settings:
      master_key: os.environ/LITELLM_MASTER_KEY
    

    Where model is the model ID in LiteLLM: the openai/ prefix and the Yandex Cloud AI Studio model URI containing your folder ID. Here is an example: openai/gpt://b1gt6g8ht345********/deepseek-v4-flash.

    The os.environ/ prefix indicates that the parameter value should be retrieved from the respective environment variable.

  9. In a text editor, open the docker-compose.yaml file:

    nano docker-compose.yaml
    
  10. Add the Docker Compose project configuration into the docker-compose.yaml file:

    services:
      litellm:
        image: docker.litellm.ai/berriai/litellm:main-latest
        container_name: litellm-proxy
        restart: unless-stopped
        # Host network: on this class of host net.ipv4.ip_forward=0 and bridge NAT drops connections.
        network_mode: host
        volumes:
          - ./config.yaml:/app/config.yaml:ro
        environment:
          - LITELLM_OTEL_V2=true
          - OTEL_EXPORTER=otlp_http
          - OTEL_ENDPOINT=https://ingest.monium.yandex.cloud/otlp
          - OTEL_SERVICE_NAME=litellm-proxy
          - OTEL_ENVIRONMENT_NAME=local
          - OTEL_HEADERS=Authorization=Api-Key ${MONIUM_API_KEY:?MONIUM_API_KEY},x-monium-project=${MONIUM_PROJECT:?MONIUM_PROJECT},x-monium-cluster=${MONIUM_CLUSTER:-default},x-monium-service=litellm-proxy
          - OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_only
          - YANDEX_AI_STUDIO_API_KEY=${YANDEX_AI_STUDIO_API_KEY:?YANDEX_AI_STUDIO_API_KEY}
          - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY:-sk-local-litellm}
        command:
          - --config
          - /app/config.yaml
          - --port
          - "4000"
          - --host
          - "0.0.0.0"
    

    Configuration parameters:

    • environment: Environment variables:

      • OTEL_EXPORTER=otlp_http: Configures trace delivery via OTLP over HTTP.
      • OTEL_ENDPOINT=https://ingest.monium.yandex.cloud/otlp: Sets the base endpoint address. LiteLLM Proxy appends /v1/traces to it and sends traces to https://ingest.monium.yandex.cloud/otlp/v1/traces.
      • x-monium-service=litellm-proxy: Name to assign to the service in the Monium UI.
      • OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_only: Enables capturing model request and response text inside spans.
    • network_mode: host: Runs the container in the host's network.

      This allows the container to access Yandex Cloud AI Studio and Yandex Monium even if IP packet forwarding is disabled on the host (net.ipv4.ip_forward = 0). LiteLLM Proxy listens for incoming connections on VM port 4000.

  11. Run the container:

    docker compose up -d
    

    Result:

    [+] up 27/27
    ✔ Image docker.litellm.ai/berriai/litellm:main-latest Pulled                                                      82.4s
    ✔ Container litellm-proxy                             Started                                                     43.1s
    
  12. To check that LiteLLM Proxy is ready to accept requests, run this command in the VM terminal:

    for _ in $(seq 1 30); do
      curl -sf http://127.0.0.1:4000/health/liveliness >/dev/null && echo "proxy is up" && break
      sleep 1
    done
    

    Result:

    proxy is up
    

Send requests to the model and check the tracesSend requests to the model and check the traces

To test trace delivery, create and run a test application on the same VM where LiteLLM Proxy is running. The application will route requests to the model through the proxy, which in turn will send the traces to Monium Traces. You can then find these traces in Monium to review the request details.

Create and run a test applicationCreate and run a test application

The Python-based test application simulates a customer conversation with a support agent. It uses the openai library to send requests to the model via LiteLLM Proxy. Install this library inside a dedicated virtual environment to isolate the application dependencies.

To create and run the application:

  1. Navigate to the ~/litellm-proxy directory you created when deploying LiteLLM Proxy on the VM:

    cd ~/litellm-proxy
    
  2. Install Python and the venv module for virtual environment management:

    sudo apt install python3 python3.12-venv
    
  3. Create and activate the litellm-venv virtual environment:

    python3 -m venv ~/litellm-venv
    source ~/litellm-venv/bin/activate
    
  4. Install the openai library:

    pip install 'openai==2.54.0'
    
  5. Create and open the agent_example.py test application file in a text editor:

    touch agent_example.py
    nano agent_example.py
    
  6. Add this code to the agent_example.py file:

    Application code
    """Customer support agent that talks to a local LiteLLM proxy only.
    
    No client-side telemetry: tracing is handled by the proxy (OTEL v2 → Monium).
    """
    
    from __future__ import annotations
    
    import argparse
    import json
    import os
    import random
    import sys
    
    from openai import OpenAI
    
    DEFAULT_PROXY_BASE = "http://127.0.0.1:4000/v1"
    DEFAULT_PROXY_KEY = "sk-local-litellm"
    DEFAULT_MODEL = "deepseek-v4-flash"
    
    
    class KnowledgeBase:
        """Simple knowledge base for customer support."""
    
        def __init__(self) -> None:
            self.knowledge = {
                "shipping": {
                    "standard": "Standard shipping takes 5-7 business days and costs $5.99",
                    "express": "Express shipping takes 2-3 business days and costs $12.99",
                    "overnight": "Overnight shipping delivers next business day and costs $24.99",
                },
                "returns": {
                    "policy": "You can return items within 30 days of purchase for a full refund",
                    "process": "To return an item, go to your account, find the order, and click 'Return Item'",
                    "condition": "Items must be in original condition with tags attached",
                },
                "account": {
                    "password_reset": "Click 'Forgot Password' on the login page and follow the email instructions",
                    "update_info": "You can update your account information in the 'My Account' section",
                    "delete_account": "Contact customer service to delete your account permanently",
                },
                "products": {
                    "warranty": "All products come with a 1-year manufacturer warranty",
                    "availability": "Check product availability on the product page",
                    "specifications": "Product specifications are listed in the product description",
                },
            }
    
        def search(self, query: str) -> str:
            query_lower = query.lower()
    
            if any(
                word in query_lower for word in ["ship", "delivery", "shipping", "deliver"]
            ):
                if "express" in query_lower or "fast" in query_lower:
                    return self.knowledge["shipping"]["express"]
                if "overnight" in query_lower or "next day" in query_lower:
                    return self.knowledge["shipping"]["overnight"]
                return self.knowledge["shipping"]["standard"]
    
            if any(word in query_lower for word in ["return", "refund", "exchange"]):
                if "how" in query_lower or "process" in query_lower:
                    return self.knowledge["returns"]["process"]
                if "condition" in query_lower:
                    return self.knowledge["returns"]["condition"]
                return self.knowledge["returns"]["policy"]
    
            if any(
                word in query_lower for word in ["account", "password", "login", "profile"]
            ):
                if "password" in query_lower or "reset" in query_lower:
                    return self.knowledge["account"]["password_reset"]
                if "update" in query_lower or "change" in query_lower:
                    return self.knowledge["account"]["update_info"]
                if "delete" in query_lower:
                    return self.knowledge["account"]["delete_account"]
    
            if any(
                word in query_lower for word in ["product", "warranty", "available", "spec"]
            ):
                if "warranty" in query_lower:
                    return self.knowledge["products"]["warranty"]
                if "available" in query_lower or "stock" in query_lower:
                    return self.knowledge["products"]["availability"]
                return self.knowledge["products"]["specifications"]
    
            return (
                "I couldn't find specific information about that. "
                "Let me help you with a general response."
            )
    
    
    class CustomerSupportAgent:
        """Customer support agent using an OpenAI-compatible LiteLLM proxy."""
    
        def __init__(
            self,
            *,
            base_url: str,
            api_key: str,
            model: str,
        ) -> None:
            self.client = OpenAI(base_url=base_url, api_key=api_key)
            self.model = model
            self.knowledge_base = KnowledgeBase()
            self.conversation_history: list[dict] = []
            self.system_prompt = """
    You are a helpful and friendly customer support agent. Your role is to:
    1. Assist customers with their questions and concerns
    2. Use the available tools whenever they help answer the question
    3. Be polite, professional, and empathetic
    4. If you don't know something, admit it and offer to escalate to a human agent
    5. Keep responses concise but helpful
    
    Always maintain a positive and solution-oriented attitude.
    Prefer calling tools for shipping, returns, order status, refunds, accounts, and products
    instead of inventing details.
    """.strip()
    
        def search_knowledge_base(self, query: str) -> str:
            return self.knowledge_base.search(query)
    
        def get_order_status(self, order_id: str) -> str:
            statuses = ["Processing", "Shipped", "Out for Delivery", "Delivered"]
            return f"Order {order_id} status: {random.choice(statuses)}"
    
        def process_refund(self, order_id: str, reason: str) -> str:
            return (
                f"Refund request for order {order_id} has been submitted. "
                f"Reason: {reason}. You'll receive confirmation within 24 hours."
            )
    
        def get_available_tools(self) -> list[dict]:
            return [
                {
                    "type": "function",
                    "function": {
                        "name": "search_knowledge_base",
                        "description": (
                            "Search the knowledge base for shipping, returns, "
                            "accounts, and products"
                        ),
                        "parameters": {
                            "type": "object",
                            "properties": {
                                "query": {
                                    "type": "string",
                                    "description": "The customer's question or topic",
                                }
                            },
                            "required": ["query"],
                        },
                    },
                },
                {
                    "type": "function",
                    "function": {
                        "name": "get_order_status",
                        "description": "Get the current status of a customer's order",
                        "parameters": {
                            "type": "object",
                            "properties": {
                                "order_id": {
                                    "type": "string",
                                    "description": "The order ID to check",
                                }
                            },
                            "required": ["order_id"],
                        },
                    },
                },
                {
                    "type": "function",
                    "function": {
                        "name": "process_refund",
                        "description": "Process a refund request for a customer",
                        "parameters": {
                            "type": "object",
                            "properties": {
                                "order_id": {
                                    "type": "string",
                                    "description": "The order ID to refund",
                                },
                                "reason": {
                                    "type": "string",
                                    "description": "Reason for the refund",
                                },
                            },
                            "required": ["order_id", "reason"],
                        },
                    },
                },
            ]
    
        def execute_tool_call(self, tool_call) -> str:
            function_name = tool_call.function.name
            arguments = json.loads(tool_call.function.arguments)
    
            if function_name == "search_knowledge_base":
                return self.search_knowledge_base(arguments["query"])
            if function_name == "get_order_status":
                return self.get_order_status(arguments["order_id"])
            if function_name == "process_refund":
                return self.process_refund(arguments["order_id"], arguments["reason"])
            return f"Unknown tool: {function_name}"
    
        def chat(self, user_message: str) -> str:
            self.conversation_history.append({"role": "user", "content": user_message})
            messages = [
                {"role": "system", "content": self.system_prompt}
            ] + self.conversation_history
    
            try:
                response = self.client.chat.completions.create(
                    model=self.model,
                    messages=messages,
                    tools=self.get_available_tools(),
                    tool_choice="auto",
                    temperature=0.4,
                    max_tokens=700,
                )
                assistant_message = response.choices[0].message
    
                # Allow a few tool rounds so one user turn can expand into several LLM calls.
                for _ in range(4):
                    if not assistant_message.tool_calls:
                        break
    
                    self.conversation_history.append(
                        {
                            "role": "assistant",
                            "content": assistant_message.content,
                            "tool_calls": [
                                {
                                    "id": tool_call.id,
                                    "type": "function",
                                    "function": {
                                        "name": tool_call.function.name,
                                        "arguments": tool_call.function.arguments,
                                    },
                                }
                                for tool_call in assistant_message.tool_calls
                            ],
                        }
                    )
    
                    for tool_call in assistant_message.tool_calls:
                        tool_result = self.execute_tool_call(tool_call)
                        print(f"  [tool] {tool_call.function.name} -> {tool_result}")
                        self.conversation_history.append(
                            {
                                "role": "tool",
                                "tool_call_id": tool_call.id,
                                "content": tool_result,
                            }
                        )
    
                    messages = [
                        {"role": "system", "content": self.system_prompt}
                    ] + self.conversation_history
                    response = self.client.chat.completions.create(
                        model=self.model,
                        messages=messages,
                        tools=self.get_available_tools(),
                        tool_choice="auto",
                        temperature=0.4,
                        max_tokens=700,
                    )
                    assistant_message = response.choices[0].message
    
                response_content = assistant_message.content or ""
                self.conversation_history.append(
                    {"role": "assistant", "content": response_content}
                )
                return response_content
    
            except Exception as exc:  # noqa: BLE001 - surface proxy/LLM errors to the chat
                return f"I apologize, but I'm experiencing technical difficulties. Error: {exc}"
    
        def reset_conversation(self) -> None:
            self.conversation_history = []
    
    
    DEMO_TURNS = [
        "Hi! I need express shipping info for an order I'm about to place.",
        "Also, what is the status of my existing order ORD-10482?",
        "The item arrived damaged. What is your return policy and how do I start a return?",
        "Please process a refund for order ORD-10482 because the item arrived damaged.",
        "One last thing: how long is the product warranty, and how do I reset my account password?",
    ]
    
    
    def run_demo(agent: CustomerSupportAgent) -> int:
        print("Customer Support Agent (LiteLLM proxy demo)")
        print("=" * 60)
        print(f"Proxy model: {agent.model}")
        print(f"Turns: {len(DEMO_TURNS)}\n")
    
        for idx, user_input in enumerate(DEMO_TURNS, start=1):
            print(f"[{idx}/{len(DEMO_TURNS)}] You: {user_input}")
            response = agent.chat(user_input)
            print(f"Agent: {response}\n")
    
        print("Demo finished.")
        return 0
    
    
    def run_interactive(agent: CustomerSupportAgent) -> int:
        print("Customer Support Agent")
        print("=" * 50)
        print("Type 'quit' to exit or 'reset' to start a new conversation.\n")
    
        while True:
            try:
                user_input = input("\nYou: ").strip()
            except (EOFError, KeyboardInterrupt):
                print(
                    "\nAgent: Thank you for contacting customer support. Have a great day!"
                )
                return 0
    
            if user_input.lower() in {"quit", "exit", "bye"}:
                print(
                    "\nAgent: Thank you for contacting customer support. Have a great day!"
                )
                return 0
            if user_input.lower() == "reset":
                agent.reset_conversation()
                print("\nAgent: Conversation reset. How can I help you today?")
                continue
            if not user_input:
                continue
    
            print(f"\nAgent: {agent.chat(user_input)}")
    
    
    def build_agent_from_env() -> CustomerSupportAgent:
        return CustomerSupportAgent(
            base_url=os.getenv("LITELLM_PROXY_BASE", DEFAULT_PROXY_BASE),
            api_key=os.getenv("LITELLM_PROXY_KEY", DEFAULT_PROXY_KEY),
            model=os.getenv("LITELLM_MODEL", DEFAULT_MODEL),
        )
    
    
    def main(argv: list[str] | None = None) -> int:
        parser = argparse.ArgumentParser(description=__doc__)
        parser.add_argument(
            "--interactive",
            action="store_true",
            help="Run an interactive REPL instead of the multi-step demo",
        )
        args = parser.parse_args(argv)
    
        agent = build_agent_from_env()
        if args.interactive:
            return run_interactive(agent)
        return run_demo(agent)
    
    
    if __name__ == "__main__":
        sys.exit(main())
    

    Through LiteLLM Proxy, the application sends five customer messages to the model one by one: a question about delivery, an order status check, a question on return policy, a refund request, and a final message with two questions, one regarding warranty and password reset.

    To prepare a response, the model can request a tool call: a local knowledge base search, an order status check, or a refund transaction. The application processes this tool call and sends the results back to the model through the proxy in the next request. Order checks and refunds are simulated in the app code.

    The number of requests depends on which tools the model calls. For example, during an order status check, the first request allows the model to select the tool, while the second one returns the tool results so the model can generate a response for the customer. LiteLLM Proxy generates a unique trace for each request and routes it to Monium Traces. The application itself does not send any telemetry data.

  7. Run the test app:

    python3 agent_example.py
    

    Result:

    Customer Support Agent (LiteLLM proxy demo)
    ============================================================
    Proxy model: deepseek-v4-flash
    Turns: 5
    ...
    [1/5] You: Hi! I need express shipping info for an order I'm about to place.
      [tool] search_knowledge_base -> Express shipping takes 2-3 business days and costs $12.99
    Agent: Great news! Here's the express shipping information:
    ...
    ...
    Demo finished.
    

Find and view traces in MoniumFind and view traces in Monium

To find the traces generated by the test application:

Monium UI
  1. Navigate to Monium.

  2. In the left-hand panel, select  LLM Observability.

  3. Use the timeline selector at the top of the screen to choose the period when you ran the test application.

  4. Click the icon in the top-right corner of the query bar and enter the following filter:

    {project = "folder__<folder_ID>", service = "litellm-proxy"}
    

    Where <folder_id> is the ID of the folder where you created the service accounts.

  5. Click Run query.

The search results will display the traces generated by your test application. To view all operations within a trace, click its row and select Show full trace.

The expanded trace will display these spans:

  • POST /v1/chat/completions: HTTP request to LiteLLM Proxy.
  • POST /v1/chat/completions: LiteLLM Proxy access key verification.
  • chat gpt://<folder_ID>/deepseek-v4-flash: Model call. This span contains the model name, input and output token counts, and message text.

If the model requests tool calls, the model span logs their names (search_knowledge_base, get_order_status, or process_refund) and arguments.

The application stores the conversation history and appends it to each new request, causing the context size to increase in subsequent traces.

For more information on viewing traces, see Viewing and analyzing LLM traces.

TroubleshootingTroubleshooting

Issue

Solution

Connection error (Error: Connection error)

  • During the final step of the LiteLLM Proxy deployment, wait for the proxy is up message.
  • Make sure VM port 4000 is available.

Yandex Cloud AI Studio returns HTTP code 401 or 403

  • Check that the YANDEX_AI_STUDIO_API_KEY variable contains only the key, without Bearer, Api-Key, or any extra characters.

  • Check config.yaml configurations:

    • model (the model ID in LiteLLM): openai/gpt://<folder_ID>/deepseek-v4-flash
    • api_base (address): https://ai.api.cloud.yandex.net/v1

The test application runs successfully, but traces do not appear in Monium

Make sure that:

  • The MONIUM_PROJECT value has this format: folder__<folder_ID> (with two underscores).
  • The Authorization header in docker-compose.yaml contains a literal space between Api-Key and the key (not %20).
  • The litellm-monium-sa service account holds the monium.traces.writer role for the folder defined in MONIUM_PROJECT.
  • The service account API key scope is set to yc.monium.traces.write.
  • The filter in the Monium UI is set to {project = "folder__<folder_id>", service = "litellm-proxy"} and the selected time range covers the exact period when you ran the test application.

No container network connection (Connection reset)

Run the sysctl net.ipv4.ip_forward command.

If it returns net.ipv4.ip_forward = 0, keep the network_mode: host line in docker-compose.yaml.

After you make the changes, run docker compose up -d --force-recreate.

Spans do not contain message text

Check that the docker-compose.yaml file includes the OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_only environment variable.

After you make the changes, run docker compose up -d --force-recreate.

How to delete the resources you createdHow to delete the resources you created

If you deployed this solution for testing purposes only, you can delete the resources after completing this tutorial to stop paying for them:

  1. Delete the LiteLLM Proxy VM.
  2. Delete the disk if it was not deleted along with the VM.
  3. Delete the static public IP address if you reserved one for the VM.

Delete the litellm-ai-studio-sa and litellm-monium-sa service accounts if you no longer need them.

Scope is an API key parameter that restricts the key's use to specific operations in addition to the service account's access permissions. For details, see API keys with scope and validity limits.

A span is a basic distributed tracing element representing a single operation in your system, e.g., HTTP request, database call, or function execution. For more information, see Span.

Was the article helpful?

Previous
Uploading objects into an Object Storage bucket using an ephemeral access key
Next
All tutorials
© 2026 Direct Cursus Technology L.L.C.