Yandex Cloud
Search
Contact UsGet started
  • Blog
  • Pricing
  • Documentation
  • All Services
  • System Status
    • Featured
    • Infrastructure & Network
    • Data Platform
    • Containers
    • Developer tools
    • Serverless
    • Security
    • Monitoring & Resources
    • ML & AI
    • Business tools
  • All Solutions
    • By industry
    • By use case
    • Economics and Pricing
    • Security
    • Technical Support
    • Customer Stories
    • Cloud credits to scale your IT product
    • Gateway to Russia
    • Cloud for Startups
    • Education and Science
    • Yandex Cloud Partner program
  • Blog
  • Pricing
  • Documentation
© 2025 Direct Cursus Technology L.L.C.
Tutorials
    • All tutorials
    • URL shortener
    • Entering data into storage systems
    • Storing application runtime logs
    • Deploying a web application using the Java Servlet API
    • Developing a Slack bot
    • Developing a Telegram bot
    • Developing a custom integration in API Gateway
    • Developing CRUD APIs for movie services
    • Building a CI/CD pipeline in GitLab
    • Working with an API gateway via WebSocket
    • Creating an interactive serverless application using WebSocket
    • Automatically copying objects from one Object Storage bucket to another
    • Visualizing logs in Grafana using the Cloud Logging plugin
    • Canary release of a Cloud Functions function
    • Interactive debugging of Cloud Functions functions
    • Creating a Node.js function using TypeScript
    • Running a containerized app in Serverless Containers
    • Streaming Yandex Cloud Postbox events to Data Streams and analyzing them using DataLens
    • Using API Gateway to set up speech synthesis in SpeechKit
    • Connecting to YDB from a Cloud Functions function in Python
    • Connecting to a YDB database from a Cloud Functions function in Node.js
    • API Gateway protection with Smart Web Security
    • Deploying a web app with JWT authorization in API Gateway and authentication in Firebase
    • Automatic data upload to Yandex SpeechSense using Yandex Workflows
    • Configuring responses in Cloud Logging and Yandex Cloud Functions
    • Setting up Workflows integration with Tracker, YandexGPT, and Yandex Cloud Postbox
    • Developing functions in Functions Framework and deploying them to Yandex Serverless Containers

In this article:

  • Get your cloud ready
  • Required paid resources
  • Create service accounts
  • Create an API key for the service account
  • Create a space
  • Add the service account to the space
  • Create a connection
  • Create a project
  • Create an infrastructure
  • Create a secret
  • Create a data model
  • Create directories to store files and their metadata
  • Prepare the metadata
  • Upload your files
  • Check the result
  • Delete resources
  1. Serverless technologies
  2. Automatic data upload to Yandex SpeechSense using Yandex Workflows

Automatic data upload to Yandex SpeechSense using Yandex Workflows

Written by
Yandex Cloud
Updated at May 7, 2025
  • Get your cloud ready
    • Required paid resources
    • Create service accounts
    • Create an API key for the service account
    • Create a space
    • Add the service account to the space
    • Create a connection
    • Create a project
  • Create an infrastructure
  • Create a secret
  • Create a data model
  • Create directories to store files and their metadata
  • Prepare the metadata
  • Upload your files
  • Check the result
  • Delete resources

Note

Workflows is at the Preview stage. To gain access, submit a request in the management console.

You can configure automatic loading of dialog files and their metadata from the Object Storage bucket to SpeechSense space. Supported formats are:

  • MP3, WAV, and OggOpus: For audio files.
  • JSON: For chat conversations.

On the diagram:

  1. Trigger for Object Storage monitors for new JSON files with metadata as may appear in the selected bucket directory or any of its subdirectories.
  2. When new files appear in the directory, the trigger calls the workflow-call function, which starts the Workflows workflow.
  3. The workflow retrieves the contents of JSON metadata files and checks their syntax using the verify-file function.
  4. The workflow gets the SpeechSense connection parameters from the relevant Yandex Lockbox secret.
  5. The path to the audio or text file and its metadata are provided to the speechsense-upload upload function.
  6. speechsense-upload uploads the files and their metadata into the SpeechSense space.
  7. At runtime, the workflow accesses the database with metadata:
    1. Logging syntax errors in metadata files.
    2. Logging entry syntax errors in metadata files.
    3. Checking for duplicates: before calling the speechsense-upload function, the workflow checks whether a file has already been loaded into the SpeechSense space.
    4. Logging the speechsense-upload function errors.
    5. Logging the file metadata and unique identifier in SpeechSense space once the file is successfully uploaded.
  8. WebSQL provides access to the metadata DB. One database user is used for browsing, and another one for uploading files.

You can set up automatic data upload for several SpeechSense connections at once.

To automate uploading data to SpeechSense, follow these steps:

  1. Get your cloud ready.
  2. Create an infrastructure for uploading files.
  3. Create a Yandex Lockbox secret.
  4. Create a Managed Service for PostgreSQL cluster data model.
  5. In the Object Storage bucket, create directories to store files and their metadata.
  6. Prepare the metadata.
  7. Upload the files to the Object Storage bucket.
  8. Check the result.

If you no longer need the resources you created, delete them.

Get your cloud readyGet your cloud ready

Sign up in Yandex Cloud and create a billing account:

  1. Navigate to the management console and log in to Yandex Cloud or register a new account.
  2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can navigate to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

Required paid resourcesRequired paid resources

The cost of resources includes:

  • Fee for bucket data storage and data operations (see Yandex Object Storage pricing).
  • Cluster usage fee (see Yandex Managed Service for PostgreSQL pricing).
  • Fee for invoking functions (see Yandex Cloud Functions pricing).
  • Fee for storing the secret and requests to the secret (see Yandex Lockbox pricing).

Create service accountsCreate service accounts

Create two service accounts:

  • deploy-sa: This one will be used for creating the infrastructure.

  • speechsense-sa: This one will be used for invoking functions and starting the workflow.

Management console
Yandex Cloud CLI
API
  1. In the management console, select the appropriate folder.
  2. From the list of services, select Identity and Access Management.
  3. Click Create service account.
  4. Enter a name for the service account: deploy-sa.
  5. Click Add role and select the following roles: functions.admin, storage.editor, iam.editor, mdb.admin, and serverless.workflows.admin.
  6. Click Create.
  7. Repeat the above steps and create a service account named speechsense-sa with the following roles: storage.viewer, functions.functionInvoker, functions.mdbProxiesUser, lockbox.payloadViewer, and serverless.workflows.executor.

If you do not have the Yandex Cloud CLI yet, install and initialize it.

The folder specified when creating the CLI profile is used by default. To change the default folder, use the yc config set folder-id <folder_ID> command. You can specify a different folder using the --folder-name or --folder-id parameter.

  1. Create a service account named deploy-sa:

    yc iam service-account create deploy-sa
    

    Result:

    id: nfersamh4sjq********
    folder_id: b1gc1t4cb638********
    created_at: "2023-09-21T10:36:29.726397755Z"
    name: deploy-sa
    

    Save the id of the deploy-sa service account and the folder where you created it (folder_id).

    For more information about the yc iam service-account create command, see the CLI reference.

  2. Assign the functions.admin, storage.editor, iam.editor, mdb.admin, and serverless.workflows.admin roles for the folder to the deploy-sa service account by specifying the folder and service account IDs you previously saved:

    yc resource-manager folder add-access-binding <folder_ID> \
      --role <role> \
      --subject serviceAccount:<service_account_ID>
    

    The command only accepts one role at a time.

    For more information about the yc resource-manager folder add-access-binding command, see the CLI reference.

    If you will be creating a Yandex Lockbox secret through the Yandex Cloud CLI under the deploy-sa service account, also assign the lockbox.editor role to that account.

  3. Repeat the above steps and create a service account named speechsense-sa with the following roles: storage.viewer, functions.functionInvoker, functions.mdbProxiesUser, lockbox.payloadViewer, and serverless.workflows.executor.

To create a service account, use the create method for the ServiceAccount resource or the ServiceAccountService.Create gRPC API call.

To assign the functions.admin, storage.editor, iam.editor, mdb.admin, and serverless.workflows.admin roles to the deploy-sa service account, use the setAccessBindings method for the ServiceAccount resource or the ServiceAccountService.SetAccessBindings gRPC API call.

In the same way, assign the following roles to the speechsense-sa service account: storage.viewer, functions.functionInvoker, functions.mdbProxiesUser, lockbox.payloadViewer, and serverless.workflows.executor.

Create an API key for the service accountCreate an API key for the service account

Create an API key for the speechsense-sa service account.

Management console
Yandex Cloud CLI
API
  1. In the management console, select the folder containing the service account.

  2. From the list of services, select Identity and Access Management.

  3. In the left-hand panel, select Service accounts.

  4. Select the speechsense-sa service account.

  5. In the top panel, click Create new key and select Create API key.

  6. Click Create.

  7. Save the ID and secret key.

    Alert

    After you close this dialog, the key value will not be shown again.

Create an API key for the speechsense-sa service account and save the response to the api_key.yaml file:

yc iam api-key create \
  --service-account-name speechsense-sa \
  > api_key.yaml

As a result, you will get the api_key.yaml file with the API key value in the secret field:

api_key:
  id: ajeke74kbp5b********
  service_account_id: ajepg0mjt06s********
  created_at: "2019-04-09T08:41:27Z"
secret: AQVN1HHJReSrfo9jU3aopsXrJyfq_UHs********

Create an API key using the create REST API method for the ApiKey resource:

export SERVICEACCOUNT_ID=<service_account_ID>
export IAM_TOKEN=<token>
curl \
  --request POST \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer $IAM_TOKEN" \
  --data "{ \"serviceAccountId\": \"$SERVICEACCOUNT_ID\" }" \
  https://iam.api.cloud.yandex.net/iam/v1/apiKeys

Where:

  • SERVICEACCOUNT_ID: Service account ID.
  • IAM_TOKEN: IAM token

You can also create an API key using the ApiKeyService.Create gRPC API call.

Create a spaceCreate a space

SpeechSense interface
  1. Open the SpeechSense home page.
  2. Click Create space.
  3. Enter a name for the space.
  4. Click Create.

Add the service account to the spaceAdd the service account to the space

Add the speechsense-sa service account to the SpeechSense space.

SpeechSense interface
  1. Open the SpeechSense home page.
  2. Go to your new space.
  3. Click Add participant → Add from organization.
  4. Copy the ID of the speechsense-sa service account you created earlier and paste it to the search bar.
  5. Select the speechsense-sa service account and specify the Data editor role. This role will allow speechsense-sa to upload data to SpeechSense.
  6. Click Add.

Create a connectionCreate a connection

Depending on the type of files to be uploaded to SpeechSense, create a connection for an audio or chat.

Create a connection for an audioCreate a connection for an audio

SpeechSense interface
  1. Open the SpeechSense home page.

  2. Go to the space of your choice.

  3. In the top-right corner, click More → Create connection.

  4. Enter the connection name.

  5. Select the Two-channel audio data type.

  6. Under Agent, Customer:

    1. Specify the channels with the agent's voice and customer's voice.
    2. Specify agent and customer keys from the metadata file. This file contains the call information collected from CRM systems, PBXs, or other sources.

    By default, connections include keys with the name and ID of the agent and the customer. In the Name in the system field, enter a name for the key to display in SpeechSense.

    To specify additional metadata for the agent and the customer, click Add key.

  7. Under Shared metadata, provide the keys from the metadata file that are not related to the agent or the customer.

    By default, connections include keys with the date, direction, and language of the call. In the Name in the system field, enter a name for the key to display in SpeechSense.

    To specify additional metadata, click Add key.

  8. Click Create connection.

Create a connection for a chatCreate a connection for a chat

SpeechSense interface
  1. Open the SpeechSense home page.

  2. Go to the space of your choice.

  3. In the top-right corner, click More → Create connection.

  4. Enter the connection name.

  5. Select the Сhat data type.

  6. Under Agent, Customer, Bot, specify the keys from the metadata file. This file contains the dialog information collected from chats, CRM systems, or other sources.

    By default, connections include keys with agent's, customer's, and bot's names and IDs. In the Name in the system field, enter a name for the key to display in SpeechSense.

    To specify additional agent, customer, and bot metadata, click Add key.

  7. Under Shared metadata, provide the keys from the metadata file that are not related to the agent, customer, and bot.

    By default, keys with date, direction, and dialog language are added to the connection. In the Name in the system field, enter a name for the key to display in SpeechSense.

    To specify additional metadata, click Add key.

  8. Click Create connection.

Create a projectCreate a project

SpeechSense interface
  1. Open the SpeechSense home page.
  2. Go to the space of your choice.
  3. Click Create project.
  4. Enter a name for your project.
  5. Under Connection, click Add connection and select the connection you created earlier.
  6. Click Create project.

Create an infrastructureCreate an infrastructure

  1. Clone the yc-serverless-speechsense-workflows repository:

    git clone https://github.com/yandex-cloud-examples/yc-serverless-speechsense-workflows.git
    

    The repository contains a script that will create in the cloud the infrastructure needed to upload files to SpeechSense:

    • Object Storage bucket.
    • Managed Service for PostgreSQL cluster.
    • Cloud Functions functions.
    • Trigger to invoke the Cloud Functions function.
    • Workflows workflow.
    • Connections to a Managed Service for PostgreSQL cluster database.
  2. For the script to run successfully, configure the Yandex Cloud CLI authentication under the deploy-sa service account:

    Yandex Cloud CLI
    1. Create an authorized key for the deploy-sa service account and save that key to the file:

      yc iam key create --output <key_file_path> --service-account-name deploy-sa
      

      Where --output is the path to the file for saving the authorized key in JSON format.

      Result:

      id: aje4lue48687********
      service_account_id: ajeb9l33h6m********
      created_at: "2024-08-01T11:58:52.313177213Z"
      key_algorithm: RSA_2048
      

      For more information about the yc iam key create command, see the CLI reference.

    2. Create a profile to execute operations under the deploy-sa service account:

      yc config profile create <profile_name>
      
    3. Specify the deploy-sa service account’s authorized key in the profile configuration:

      yc config set service-account-key <key_file_path>
      
  3. Navigate to the repository directory and run the script:

    Bash
    cd deploy && bash deploy.sh
    

    In the command line, enter the folder ID, speechsense-sa as the name of the service account that will call functions and run the workflow, and bucket name.

    The script execution time is about 10-15 minutes.

Create a secretCreate a secret

Management console
Yandex Cloud CLI
API
  1. In the management console, select the folder where you want to create a secret.

  2. In the list of services, select Lockbox.

  3. Click Create secret.

  4. In the Name field, specify the secret's name: speechsense-secret.

  5. Under Secret data:

    1. Select the Custom secret type.

    2. Add the service account's API key:

      • In the Key field, put: speechsense_api_key.
      • In the Value field, specify the value of the speechsense-sa service account's API key you created earlier.
    3. Click Add key/value and add the SpeechSense connection ID:

      • In the Key field, put: speechsense_connection_id.
      • In the Value field, specify the ID of the connection you created earlier.
    4. Click Add key/value and add the format for the dialog files that will be uploaded to SpeechSense:

      • In the Key field, put: speechsense_file_format.
      • In the Value field, specify the file format. Acceptable values: mp3, wav, ogg, and text.
  6. Click Create.

To create a secret, run this command:

yc lockbox secret create --name speechsense-secret \
  --payload "[{'key': 'speechsense_api_key', 'text_value': '<API_key>'},{'key': 'speechsense_connection_id', 'text_value': '<connection_ID>'}, {'key': 'speechsense_file_format', 'text_value': '<file_format>'}]"

Result:

id: e6q2ad0j9b55********
folder_id: b1gktjk2rg49********
created_at: "2021-11-08T19:23:00.383Z"
name: speechsense-secret
status: ACTIVE
current_version:
  id: g6q4fn3b6okj********
  secret_id: e6e2ei4u9b55********
  created_at: "2023-03-21T19:23:00.383Z"
  status: ACTIVE
  payload_entry_keys:
    - speechsense_api_key
    - speechsense_connection_id
    - speechsense_file_format

To create a secret, use the create REST API method for the Secret resource or the SecretService.Create gRPC API call.

Create a data modelCreate a data model

  1. Open the pg_metadata.sql file in the repository directory.

  2. Under the section for inserting data into the public.source_system table, specify the values of the parameters:

    insert into public.source_system(source_system_id, lockbox_secret_id, source_system_desc)
    values ('<origin_ID>', '<secret_ID>', '<source_description>');
    

    Where:

    • source_system_id: Data source ID to use in metadata. Specify any unique string value, e.g., 000001.
    • lockbox_secret_id: ID of the Yandex Lockbox secret you created earlier, e.g., e6qigo0vbci2********.
    • source_system_desc: Data source description. For example: Uploading telephony data.
  3. Copy the pg_metadata.sql file contents and run the resulting query with WebSQL.

    Management console
    1. Go to the folder page and select Managed Service for PostgreSQL.
    2. Click the name of the cluster you created earlier. By default, it is speechsense-upload-metadata.
    3. Select the WebSQL tab.
    4. Click the connection name that ends with -uploader.
    5. On the WebSQL home page, click the database name: uploader.
    6. Paste the query to the editor and click Run.

Create directories to store files and their metadataCreate directories to store files and their metadata

In the Object Storage bucket you created earlier, create two directories as follows:

  • client_data: For dialog files.
  • client_metadata: For metadata files.

Make sure the directories are not nested one inside the other.

Management console
Yandex Cloud CLI
AWS CLI
API

To create a directory:

  1. In the management console, select the folder the bucket is in.
  2. From the list of services, select Object Storage.
  3. Select the bucket you need.
  4. Click Create folder and specify the directory name.
  5. Click Create.

To create a directory, run this command:

yc storage s3api put-object \
  --bucket <bucket_name> \
  --key <directory_name>/

Result:

etag: '"d41d8cd98f00b204e9800998********"'
request_id: ba96231*********

If you do not have the AWS CLI yet, install and configure it.

To create a directory, run this command:

aws s3api put-object \
  --endpoint-url=https://storage.yandexcloud.net \
  --bucket <bucket_name>
  --key <directory_name>/

Result:

{
"ETag": "\"d41d8cd98f00b204e9800998********\""
}

To create a directory, use the upload S3 API method.

Prepare the metadataPrepare the metadata

To upload a file to SpeechSense, first prepare the file metadata in JSON format. For example:

{
  "source_system_id": "000001",
  "bucket_folder": "bucket://client_data",
  "metadata": [
    {
      "id": "my_audio.ogg",
      "operator_id": "42",
      "operator_name": "James Doe",
      "client_id": "327142",
      "client_name": "Jane Smith",
      "date": "2024-08-30 19:32:11",
      "direction_outgoing": "0",
      "language": "RU",
      "file_name": "my_audio.ogg"
    }
  ]
}

Where:

  • Parameters the speechsense-upload function uses to upload files:

    • source_system_id: Data source ID as specified in the public.source_system table.

    • bucket_folder: Path to the directory you created earlier for audio or text files.

    • file_name: Name of the file to upload.

  • Parameters that have to be be provided to SpeechSense:

    • id: File ID, unique across the SpeechSense space.

    • operator_id: Agent ID.

    • operator_name: Agent name.

    • client_id: Customer ID.

    • client_name: Customer name.

    • date: Call date and time, in YYYY-MM-DD HH24:MI:SS format.

    • direction_outgoing: Call direction, 0 for outgoing call, 1 for incoming call.

    • language: Dialog language, RU for the Russian language.

  • Additional parameters for analytics in SpeechSense:

    • cpn_region_id: Region ID.

    • cpn_region_name: Region name.

    Here, cpn_region_id and cpn_region_name are used as examples only. Additional parameters may vary.

Metadata can contain a single entry or an array of entries.

Warning

Do not transmit more than 100 records at the same time: this may lead to a data loss because the Cloud Functions function has an execution timeout.

If there are more than 100 files to upload, split their metadata into separate JSON files of 100 entries each.

Upload your filesUpload your files

Import the dialog files into the client_data directory and the JSON metadata files, into the client_metadata directory.

Warning

Do not import any dialog files into the client_metadata directory or its subdirectories. New files appearing in the directory will start the workflow, but the dialog files will not pass the syntax check that the metadata files do, so the workflow will fail.

If there are more than 1,000 files, do not use the management console for uploading.

Management console
Yandex Cloud CLI
AWS CLI
API

To upload your files:

  1. In the management console, select Object Storage from the list of services and go to the bucket you want to upload your files to.
  2. In the left-hand panel, select Objects.
  3. Click the directory name to go to that directory.
  4. Within the directory you need, click Upload on the top panel.
  5. In the window that opens, select the files and click Open.
  6. The management console will display all the files you selected for uploading and prompt you to select a storage class for each of them. The bucket configuration determines the default storage class.
  7. Click Upload.
  8. Refresh the page.

In the management console, the information about the number of objects and storage space used in the bucket is updated with a few minutes' delay.

To upload a file to a directory, run this command:

yc storage s3api put-object \
  --body <path_to_file_to_upload> \
  --bucket <bucket_name> \
  --key <directory_name>/<file_name>

Where:

  • --body: Path to the file you need to upload.
  • --bucket: Bucket name.
  • --key: Path to the file in the directory.

Result:

etag: '"d41d8cd98f00b204e980099********"'
request_id: 3f2705f********

To upload a single file, run this command:

aws --endpoint-url=https://storage.yandexcloud.net/ \
  s3 cp <path_to_file_to_upload> s3://<bucket_name>/<directory_name>/<file_name>

To upload all files from a local directory with its nested directories, use the following command:

aws --endpoint-url=https://storage.yandexcloud.net/ \
  s3 cp --recursive <path_to_directory_with_files_to_upload>/ s3://<bucket_name>/<directory_name>/

Result:

upload: <file_name> to <path_to_file_in_bucket>

To upload a file, use the upload S3 API method.

Tip

To process your audio recordings faster, use OggOpus files. To convert files in other formats to OggOpus, use the FFmpeg or SoX utilities.

Check the resultCheck the result

To check on the workflow:

Management console
  1. In the management console, select Serverless Integrations from the list of services.
  2. In the left-hand panel, select Workflows.
  3. Click the workflow name. By default, it is wf-speechsense-upload.
  4. Navigate to the Executions tab.
  5. Make sure the workflow status is Complete.

To check that the files were successfully uploaded into SpeechSense:

Management console
  1. Open the WebSQL home page.

  2. Under Connections, select the speechsense-upload-metadata connection and the uploader database.

  3. Select the public schema.

  4. In the Tables group, select a table:

    • talk: To view metadata uploaded into SpeechSense. If a dialog file is uploaded, its metadata must include the talk_id ID.
    • errors: To view errors if files could not be uploaded.

Delete resourcesDelete resources

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them.

  1. Delete objects from the Object Storage bucket and the bucket itself.

  2. Delete the Managed Service for PostgreSQL cluster.

  3. Delete the trigger invoking the function in Cloud Functions.

  4. Delete the Cloud Functions functions.

  5. Delete the connection to the Managed Service for PostgreSQL cluster database:

    1. In the management console, go to the folder you want to delete the connection in.
    2. Select Cloud Functions.
    3. In the left-hand panel, select Database connections.
    4. In the speechsense-upload-metadata-connection connection row, click and select Delete.
    5. In the window that opens, click Delete.
  6. Delete the Workflows workflow:

    1. In the management console, go to the folder you want to delete the workflow in.
    2. Select Serverless Integrations.
    3. In the left-hand panel, select Workflows.
    4. Click next to wf-speechsense-upload and select Delete.
    5. In the window that opens, click Delete.

Was the article helpful?

Previous
Deploying a web app with JWT authorization in API Gateway and authentication in Firebase
Next
Configuring responses in Cloud Logging and Yandex Cloud Functions
© 2025 Direct Cursus Technology L.L.C.