Using LLMs to process recognition results

With the SpeechKit API v3, you can recognize audio and also process recognition results using large language models. This means you can automate processing of speech recognition results, e.g., instantly summarize the recognized audio, use the audio to add data to your CRM, or translate the conversation into English.

To use generative models in audio recognition, fill in the summarization section in the request configuration. You can specify only a text prompt for the model or provide the response structure as JSON. For more information on the response structure, see AI Studio documentation. To work with generative models in the SpeechKit API v3, you need the ai.languageModels.user role or higher.

{
...
  "summarization": {
    "modelUri": "gpt://<folder_ID>/<model_name>",
    "properties": [
      {
        "instruction": "Model prompt"
      }
    ]
  }
  ...
}

{
...
  "summarization": {
    "modelUri": "gpt://<folder_ID>/<model_name>",
    "properties": [
      {
        "instruction": "Model prompt requiring a structured output"
        "jsonObject": true
      }
    ]
  }
  ...
}

{
...
  "summarization": {
    "modelUri": "gpt://<folder_ID>/<model_name>",
    "properties": [
      {
        "instruction": "Model prompt requiring a structured output"
        "jsonSchema": {
          // Specified output schema 
          "schema": "<json_schema>"
        }
      }
    ]
  }
  ...
}

Where:

• modelUri: Model available in synchronous mode.
• instruction: Model prompt (Highlight the main points, Translate into English).
• jsonObject: If true, the model will return JSON output in arbitrary formatting.
• jsonSchema: Schema for strictly formatted JSON output.

The model will return its results in the summarization object:

{
...
  "result": {
    "summarization": {
      "results": [
        {
          "response": "LLM-generated text"
        },
        {
          "response": "LLM-generated JSON"
        }
      ],
      "contentUsage": {
        "inputTextTokens": 150,    // Input tokens
        "completionTokens": 80,    // Output tokens
        "totalTokens": 230         // Total consumption
      }
    }
  }
...
}

The contentUsage field contains token usage details. Calculated according to the AI Studio pricing policy, the cost of use depends on the selected model and total number of input and output tokens indicated in totalTokens.