Developing a Telegram bot for text recognition in images, audio synthesis and recognition

In this tutorial, you will create a Telegram bot that can:

• Synthesize speech from a message text and recognize speech in voice messages using the Yandex SpeechKit Python SDK.
• Recognize text in images using Yandex Vision OCR.

Authentication in Yandex Cloud services is performed under a service account using an IAM token. The IAM token is contained in the handler context of the function which manages user conversation with the bot.

The Yandex API Gateway will receive requests from your bot and forward them to Yandex Cloud Functions for processing.

To create a bot:

1. Get your cloud ready.
2. Set up resources.
3. Register your Telegram bot.
4. Create a function.
5. Create an API gateway.
6. Link the function and the bot.
7. Test the bot.

If you no longer need the resources you created, delete them.

Getting startedGetting started

Sign up for Yandex Cloud and create a billing account:

1. Navigate to the management console and log in to Yandex Cloud or create a new account.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can navigate to the cloud page to create or select a folder for your infrastructure.

Learn more about clouds and folders here.

Required paid resourcesRequired paid resources

The cost of Telegram bot support includes:

• Fee for using SpeechKit (see SpeechKit pricing).
• Fee for using Vision OCR (see Vision OCR pricing).
• Fee for function invocation count, computing resources allocated to run the function, and outbound traffic (see Cloud Functions pricing).
• Fee for the number of requests to the API gateway and outbound traffic (see API Gateway pricing).

Set up resourcesSet up resources

1. Create a service account named recognizer-bot-sa and assign it the ai.editor and functions.editor roles for your folder.

2. Download the archive with the FFmpeg package for the SpeechKit Python SDK to work correctly in the function execution environment.

3. Create a ZIP archive with the function code:

   1. Create a file named index.py and paste the code below to it.

      index.py

      import logging
      import requests
      import telebot
      import json
      import os
      import base64
      from speechkit import model_repository, configure_credentials, creds
      from speechkit.stt import AudioProcessingType
      
      
      # Service endpoints and authentication credentials
      
      API_TOKEN = os.environ['TELEGRAM_TOKEN']
      vision_url = 'https://ocr.api.cloud.yandex.net/ocr/v1/recognizeText'
      folder_id = ""
      iam_token = ''
      path = os.environ.get("PATH")
      os.environ["PATH"] = path + ':/function/code'
      
      
      logger = telebot.logger
      telebot.logger.setLevel(logging.INFO)
      bot = telebot.TeleBot(API_TOKEN, threaded=False)
      
      # Getting the folder ID
      
      def get_folder_id(iam_token, version_id):
      
          headers = {'Authorization': f'Bearer {iam_token}'}
          function_id_req = requests.get(f'https://serverless-functions.api.cloud.yandex.net/functions/v1/versions/{version_id}',
                                         headers=headers)
          function_id_data = function_id_req.json()
          function_id = function_id_data['functionId']
          folder_id_req = requests.get(f'https://serverless-functions.api.cloud.yandex.net/functions/v1/functions/{function_id}',
                                       headers=headers)
          folder_id_data = folder_id_req.json()
          folder_id = folder_id_data['folderId']
          return folder_id
      
      def process_event(event):
      
          request_body_dict = json.loads(event['body'])
          update = telebot.types.Update.de_json(request_body_dict)
      
          bot.process_new_updates([update])
      
      def handler(event, context):
          global iam_token, folder_id
          iam_token = context.token["access_token"]
          version_id = context.function_version
          folder_id = get_folder_id(iam_token, version_id)
      
          # Authenticating in SpeechKit with an IAM token
          configure_credentials(
              yandex_credentials=creds.YandexCredentials(
                  iam_token=iam_token
              )
          )
      
          process_event(event)
          return {
              'statusCode': 200
          }
      
      # Command and message handlers
      
      @bot.message_handler(commands=['help', 'start'])
      def send_welcome(message):
          bot.reply_to(message,
                       "The bot can do the following:\n*Recognize text from images.\n* Generate voice messages from text.\n* Convert voice messages to text.")
      
      @bot.message_handler(func=lambda message: True, content_types=['text'])
      def echo_message(message):
          export_path = '/tmp/audio.ogg'
          synthesize(message.text, export_path)
          voice = open(export_path, 'rb')
          bot.send_voice(message.chat.id, voice)
      
      @bot.message_handler(func=lambda message: True, content_types=['voice'])
      def echo_audio(message):
          file_id = message.voice.file_id
          file_info = bot.get_file(file_id)
          downloaded_file = bot.download_file(file_info.file_path)
          response_text = audio_analyze(downloaded_file)
          bot.reply_to(message, response_text)
      
      @bot.message_handler(func=lambda message: True, content_types=['photo'])
      def echo_photo(message):
          file_id = message.photo[-1].file_id
          file_info = bot.get_file(file_id)
          downloaded_file = bot.download_file(file_info.file_path)
          image_data = base64.b64encode(downloaded_file).decode('utf-8')
          response_text = image_analyze(vision_url, iam_token, folder_id, image_data)
          bot.reply_to(message, response_text)
      
      # Image recognition
      
      def image_analyze(vision_url, iam_token, folder_id, image_data):
          response = requests.post(vision_url, headers={'Authorization': 'Bearer '+iam_token, 'x-folder-id': folder_id}, json={
              "mimeType": "image",
              "languageCodes": ["en", "ru"],
              "model": "page",
              "content": image_data
              })
          blocks = response.json()['result']['textAnnotation']['blocks']
          text = ''
          for block in blocks:
              for line in block['lines']:
                  for word in line['words']:
                      text += word['text'] + ' '
                  text += '\n'
          return text
      
      # Speech recognition
      
      def audio_analyze(audio_data):
          model = model_repository.recognition_model()
      
          # Recognition settings
          model.model = 'general'
          model.language = 'ru-RU'
          model.audio_processing_type = AudioProcessingType.Full
      
          try:
              result = model.transcribe(audio_data)
              speech_text = [res.normalized_text for res in result]
              return ' '.join(speech_text)
          except:
              return 'Cannot recognize message'
      
      # Speech synthesis
      
      def synthesize(folder_id, iam_token, text):
          model = model_repository.synthesis_model()
      
          # Synthesis settings
          model.voice = 'kirill'
      
          result = model.synthesize(text, raw_format=False)
          result.export(export_path, 'ogg')
      

   2. Create a file named requirements.txt. In this file, specify a library to use for the bot and the Python SDK library.

      telebot
      yandex-speechkit
      

   3. Add the index.py and requirements.txt files and the ffmpeg and ffprobe binary files from the FFMpeg utility into the index.zip ZIP archive.

   4. Create an Object Storage bucket and upload the created ZIP archive into it.

Register your Telegram botRegister your Telegram bot

Register your bot in Telegram and get a token.

1. Start BotFather and send it the following command:

   /newbot
   

2. In the name field, enter a name for the new bot. This is the name users will see when chatting with the bot.

3. In the username field, enter a username for the new bot. You can use it to find the bot in Telegram. The username must end with ...Bot or ..._bot.

   Once done, you will get a token. Save it, as you will need it later.

Create a functionCreate a function

Create a function to process user actions in the chat.

Create an API gatewayCreate an API gateway

The Telegram server will notify your bot of new messages using a webhook. The API gateway will receive requests on the bot side and forward them to the for-recognizer-bot function for processing.

Configure a link between the function and the Telegram botConfigure a link between the function and the Telegram bot

Install a webhook for your Telegram bot:

curl --request POST \
  --url https://api.telegram.org/bot<bot_token>/setWebhook \
  --header 'content-type: application/json' \ 
  --data '{"url": "<API_gateway_domain>/for-recognizer-bot-function"}'

Where:

• <bot_token>: Telegram bot token.
• <API_gateway_domain>: recognizer-bot-api-gw API gateway's service domain.

Result:

{"ok":true,"result":true,"description":"Webhook was set"}

Test the botTest the bot

Chat with the bot:

1. Open Telegram and search for the bot by the specified username.

2. Send /start to the chat.

   The bot should respond with:

   The bot can do the following:
   
   * Recognize text from images.
   * Generate voice messages from text.
   * Convert voice messages to text.
   

3. Send a text message to the chat. The bot will respond with a voice message synthesized from your text.

4. Send a voice message to the chat. The bot will respond with a message containing the text recognized from your speech.

5. Send an image with text to the chat. The bot will respond with a message containing the recognized text.

   Note

   The image must meet these requirements.

How to delete the resources you createdHow to delete the resources you created

To stop paying for the resources you created:

• Delete the API Gateway.
• Delete the function in Cloud Functions.