Automatic data upload to Yandex SpeechSense using Yandex Workflows

You can configure automatic loading of dialog files and their metadata from the Object Storage bucket to SpeechSense space. Supported formats are:

• MP3, WAV, and OggOpus: For audio files.
• JSON: For chat conversations.

On the diagram:

1. Trigger for Object Storage monitors for new JSON files with metadata as may appear in the selected bucket directory or any of its subdirectories.
2. When new files appear in the directory, the trigger calls the workflow-call function, which starts the Workflows workflow.
3. The workflow retrieves the contents of JSON metadata files and checks their syntax using the verify-file function.
4. The workflow gets the SpeechSense connection parameters from the relevant Yandex Lockbox secret.
5. The path to the audio or text file and its metadata are provided to the speechsense-upload upload function.
6. speechsense-upload uploads the files and their metadata into the SpeechSense space.
7. At runtime, the workflow accesses the database with metadata:
   1. Logging syntax errors in metadata files.
   2. Logging entry syntax errors in metadata files.
   3. Checking for duplicates: before calling the speechsense-upload function, the workflow checks whether a file has already been loaded into the SpeechSense space.
   4. Logging the speechsense-upload function errors.
   5. Logging the file metadata and unique identifier in SpeechSense space once the file is successfully uploaded.
8. WebSQL provides access to the metadata DB. One database user is used for browsing, and another one for uploading files.

You can set up automatic data upload for several SpeechSense connections at once.

To automate uploading data to SpeechSense, follow these steps:

1. Get your cloud ready.
2. Create an infrastructure for uploading files.
3. Create a Yandex Lockbox secret.
4. Create a Managed Service for PostgreSQL cluster data model.
5. In the Object Storage bucket, create directories to store files and their metadata.
6. Prepare the metadata.
7. Upload the files to the Object Storage bucket.
8. Check the result.

If you no longer need the resources you created, delete them.

Get your cloud readyGet your cloud ready

Sign up for Yandex Cloud and create a billing account:

1. Go to the management console and log in to Yandex Cloud or create an account if you do not have one yet.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one.

If you have an active billing account, you can go to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

Required paid resourcesRequired paid resources

The cost of resources includes:

• Fee for bucket data storage and data operations (see Yandex Object Storage pricing).
• Cluster usage fee (see Yandex Managed Service for PostgreSQL pricing).
• Fee for invoking functions (see Yandex Cloud Functions pricing).
• Fee for storing the secret and requests to the secret (see Yandex Lockbox pricing).

Create service accountsCreate service accounts

Create two service accounts:

• deploy-sa: This one will be used for creating the infrastructure.

• speechsense-sa: This one will be used for invoking functions and starting the workflow.

Create an API key for the service accountCreate an API key for the service account

Create an API key for the speechsense-sa service account.

yc iam api-key create \
  --service-account-name speechsense-sa \
  > api_key.yaml

api_key:
  id: ajeke74kbp5b********
  service_account_id: ajepg0mjt06s********
  created_at: "2019-04-09T08:41:27Z"
secret: AQVN1HHJReSrfo9jU3aopsXrJyfq_UHs********

export SERVICEACCOUNT_ID=<service_account_ID>
export IAM_TOKEN=<token>
curl \
  --request POST \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer $IAM_TOKEN" \
  --data "{ \"serviceAccountId\": \"$SERVICEACCOUNT_ID\" }" \
  https://iam.api.cloud.yandex.net/iam/v1/apiKeys

Create a spaceCreate a space

Add the service account to the spaceAdd the service account to the space

Add the speechsense-sa service account to the SpeechSense space.

Create a connectionCreate a connection

Depending on the type of files to be uploaded to SpeechSense, create a connection for an audio or chat.

Create a connection for an audioCreate a connection for an audio

Create a connection for a chatCreate a connection for a chat

Create a projectCreate a project

Create an infrastructureCreate an infrastructure

1. Clone the yc-serverless-speechsense-workflows repository:

   git clone https://github.com/yandex-cloud-examples/yc-serverless-speechsense-workflows.git
   

   The repository contains a script that will create in the cloud the infrastructure needed to upload files to SpeechSense:

   • Object Storage bucket.
   • Managed Service for PostgreSQL cluster.
   • Cloud Functions functions.
   • Trigger to invoke the Cloud Functions function.
   • Workflows workflow.
   • Connections to a Managed Service for PostgreSQL cluster database.

2. For the script to run successfully, configure the Yandex Cloud CLI authentication under the deploy-sa service account:

   Yandex Cloud CLI

   1. Create an authorized key for the deploy-sa service account and save that key to the file:

      yc iam key create --output <key_file_path> --service-account-name deploy-sa
      

      Where --output is the path to the file for saving the authorized key in JSON format.

      Result:

      id: aje4lue48687********
      service_account_id: ajeb9l33h6m********
      created_at: "2024-08-01T11:58:52.313177213Z"
      key_algorithm: RSA_2048
      

      For more information about the yc iam key create command, see the CLI reference.

   2. Create a profile to execute operations under the deploy-sa service account:

      yc config profile create <profile_name>
      

   3. Specify the deploy-sa service account’s authorized key in the profile configuration:

      yc config set service-account-key <key_file_path>
      

3. Navigate to the repository directory and run the script:

   Bash

   cd deploy && bash deploy.sh
   

   In the command line, enter the folder ID, speechsense-sa as the name of the service account that will call functions and run the workflow, and bucket name.

   The script execution time is about 10-15 minutes.

Create a secretCreate a secret

yc lockbox secret create --name speechsense-secret \
  --payload "[{'key': 'speechsense_api_key', 'text_value': '<API_key>'},{'key': 'speechsense_connection_id', 'text_value': '<connection_ID>'}, {'key': 'speechsense_file_format', 'text_value': '<file_format>'}]"

id: e6q2ad0j9b55********
folder_id: b1gktjk2rg49********
created_at: "2021-11-08T19:23:00.383Z"
name: speechsense-secret
status: ACTIVE
current_version:
  id: g6q4fn3b6okj********
  secret_id: e6e2ei4u9b55********
  created_at: "2023-03-21T19:23:00.383Z"
  status: ACTIVE
  payload_entry_keys:
    - speechsense_api_key
    - speechsense_connection_id
    - speechsense_file_format

Create a data modelCreate a data model

1. Open the pg_metadata.sql file in the repository directory.

2. Under the section for inserting data into the public.source_system table, specify the values of the parameters:

   insert into public.source_system(source_system_id, lockbox_secret_id, source_system_desc)
   values ('<origin_ID>', '<secret_ID>', '<source_description>');
   

   Where:

   • source_system_id: Data source ID to use in metadata. Specify any unique string value, e.g., 000001.
   • lockbox_secret_id: ID of the Yandex Lockbox secret you created earlier, e.g., e6qigo0vbci2********.
   • source_system_desc: Data source description. For example: Uploading telephony data.

3. Copy the pg_metadata.sql file contents and run the resulting query with WebSQL.

   Management console

   1. Go to the folder page and select Managed Service for PostgreSQL.
   2. Click the name of the cluster you created earlier. By default, it is speechsense-upload-metadata.
   3. Select the WebSQL tab.
   4. Click the connection name that ends with -uploader.
   5. On the WebSQL home page, click the database name: uploader.
   6. Paste the query to the editor and click Run.

Create directories to store files and their metadataCreate directories to store files and their metadata

In the Object Storage bucket you created earlier, create two directories as follows:

• client_data: For dialog files.
• client_metadata: For metadata files.

Make sure the directories are not nested one inside the other.

yc storage s3api put-object \
  --bucket <bucket_name> \
  --key <directory_name>/

etag: '"d41d8cd98f00b204e9800998********"'
request_id: ba96231*********

aws s3api put-object \
  --endpoint-url=https://storage.yandexcloud.net \
  --bucket <bucket_name>
  --key <directory_name>/

{
"ETag": "\"d41d8cd98f00b204e9800998********\""
}

Prepare the metadataPrepare the metadata

To upload a file to SpeechSense, first prepare the file metadata in JSON format. For example:

{
  "source_system_id": "000001",
  "bucket_folder": "bucket://client_data",
  "metadata": [
    {
      "id": "my_audio.ogg",
      "operator_id": "42",
      "operator_name": "James Doe",
      "client_id": "327142",
      "client_name": "Jane Smith",
      "date": "2024-08-30 19:32:11",
      "direction_outgoing": "0",
      "language": "RU",
      "file_name": "my_audio.ogg"
    }
  ]
}

Where:

• Parameters the speechsense-upload function uses to upload files:

  • source_system_id: Data source ID as specified in the public.source_system table.

  • bucket_folder: Path to the directory you created earlier for audio or text files.

  • file_name: Name of the file to upload.

• Parameters that have to be be provided to SpeechSense:

  • id: File ID, unique across the SpeechSense space.

  • operator_id: Agent ID.

  • operator_name: Agent name.

  • client_id: Customer ID.

  • client_name: Customer name.

  • date: Call date and time, in YYYY-MM-DD HH24:MI:SS format.

  • direction_outgoing: Call direction, 0 for outgoing call, 1 for incoming call.

  • language: Dialog language, RU for the Russian language.

• Additional parameters for analytics in SpeechSense:

  • cpn_region_id: Region ID.

  • cpn_region_name: Region name.

  Here, cpn_region_id and cpn_region_name are used as examples only. Additional parameters may vary.

Metadata can contain a single entry or an array of entries.

Upload your filesUpload your files

Import the dialog files into the client_data directory and the JSON metadata files, into the client_metadata directory.

If there are more than 1,000 files, do not use the management console for uploading.

yc storage s3api put-object \
  --body <path_to_file_to_upload> \
  --bucket <bucket_name> \
  --key <directory_name>/<file_name>

etag: '"d41d8cd98f00b204e980099********"'
request_id: 3f2705f********

aws --endpoint-url=https://storage.yandexcloud.net/ \
  s3 cp <path_to_file_to_upload> s3://<bucket_name>/<directory_name>/<file_name>

aws --endpoint-url=https://storage.yandexcloud.net/ \
  s3 cp --recursive <path_to_directory_with_files_to_upload>/ s3://<bucket_name>/<directory_name>/

upload: <file_name> to <path_to_file_in_bucket>

Check the resultCheck the result

To check on the workflow:

To check that the files were successfully uploaded into SpeechSense:

Delete resourcesDelete resources

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them.

1. Delete objects from the Object Storage bucket and the bucket itself.

2. Delete the Managed Service for PostgreSQL cluster.

3. Delete the trigger invoking the function in Cloud Functions.

4. Delete the Cloud Functions functions.

5. Delete the connection to the Managed Service for PostgreSQL cluster database:

   1. In the management console, go to the folder you want to delete the connection in.
   2. Select Cloud Functions.
   3. In the left-hand panel, select Database connections.
   4. In the speechsense-upload-metadata-connection connection row, click and select Delete.
   5. In the window that opens, click Delete.

6. Delete the Workflows workflow:

   1. In the management console, go to the folder you want to delete the workflow in.
   2. Select Serverless Integrations.
   3. In the left-hand panel, select Workflows.
   4. Click next to wf-speechsense-upload and select Delete.
   5. In the window that opens, click Delete.