Running a vLLM library with the Gemma 3 language model on a Yandex Compute Cloud VM instance with a GPU

Using this tutorial, you will create a VM instance with a single GPU to run the Gemma 3 lightweight multimodal language model.

To run a language model:

1. Get your cloud ready.
2. Get access to the Gemma 3 model.
3. Create a VM with a GPU.
4. Run the language model.
5. Test the language model performance.

If you no longer need the resources you created, delete them.

Get your cloud readyGet your cloud ready

Sign up in Yandex Cloud and create a billing account:

1. Navigate to the management console and log in to Yandex Cloud or register a new account.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can navigate to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

Make sure the cloud has enough quotas for the total number of AMD EPYC™ 9474F with Gen2 GPUs, amount of RAM, number of vCPUs, and SSD size to create the VM. To do this, use Yandex Cloud Quota Manager.

Required paid resourcesRequired paid resources

The infrastructure support cost includes a fee for continuously running VMs and disks (see Yandex Compute Cloud pricing).

Get access to the Gemma 3 modelGet access to the Gemma 3 model

1. Sign up for Hugging Face.

2. Create an access token:

   1. After logging into your account, click your avatar → Settings → Access Tokens.
   2. Click + Create new token.
   3. Select the Read token type.
   4. Enter a token name.
   5. Click Create token.
   6. Copy the token value.

3. Request access to the Gemma-3-27b-it model:

   1. Go to the model page.
   2. Click Request access.
   3. Accept the license terms.
   4. Wait for access confirmation.

Create a VM with a GPUCreate a VM with a GPU

Run the language modelRun the language model

1. Connect to the VM over SSH.

2. Add the current user to the docker group:

   sudo groupadd docker
   sudo usermod -aG docker $USER
   newgrp docker
   

3. Fill in the variables:

   TOKEN=<HF_token>
   MODEL=google/gemma-3-27b-it
   MODEL_OPTS="--max-num-seqs 256 --max-model-len 16384 --gpu-memory-utilization 0.98 --max_num_batched_tokens 2048"
   

   Where HF_token is the Hugging Face access token.

4. Run this command:

   docker run  --runtime nvidia --gpus '"device=0"' \
   --name vllm-gema3-0 \
   -v ~/.cache/huggingface:/root/.cache/huggingface \
   --env "HUGGING_FACE_HUB_TOKEN=$TOKEN" \
   --env "HF_HUB_ENABLE_HF_TRANSFER=0" \
   --env "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True" \
   -p 8000:8000 \
   --ipc=host \
   --shm-size=32g \
   vllm/vllm-openai:latest \
   --model $MODEL $MODEL_OPTS
   

5. Wait for the server to start:

   INFO:     Started server process [1]
   INFO:     Waiting for application startup.
   INFO:     Application startup complete.
   

Test the language model performanceTest the language model performance

1. Connect to the VM over SSH in a new session.

2. Install the openai package:

   sudo apt update
   sudo apt install python3-pip
   pip install openai
   

3. Create a test_model.py script with the following contents:

   import openai
   client = openai.Client(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
   response = client.chat.completions.create(
      model="google/gemma-3-27b-it",
      messages=[
         {
               "role": "user",
               "content": [
                  {
                     "type": "image_url",
                     "image_url": {
                           "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
                     },
                  },
                  {"type": "text", "text": "Describe this image in one sentence."},
               ],
         }
      ],
      temperature=0.3,
      max_tokens=128,
   )
   print(response.choices[0].message.content)
   

4. Run the script:

   python3 test_model.py
   

   Model response example:

   Here's a one-sentence description of the image:
   
   The Statue of Liberty stands prominently on Liberty Island with the Manhattan skyline, including the Empire State Building, visible in the background across the water on a clear, sunny day.
   

How to delete the resources you createdHow to delete the resources you created

To stop paying for the resources you created, delete the VM instance you created in Compute Cloud.