Developing a Telegram bot for text recognition in images, audio synthesis and recognition

In this tutorial, you will create a bot for Telegram that can:

• Synthesize speech from a message text using the Yandex SpeechKit API v1.
• Recognize speech in voice messages and convert it into text using the Yandex SpeechKit synchronous recognition API.
• Recognize text in images using Yandex Vision OCR.

The Yandex API Gateway API gateway will receive requests from the bot and forward them to the Yandex Cloud Functions function for processing.

To create a bot:

1. Prepare your cloud.
2. Create resources.
3. Register the Telegram bot.
4. Create a function.
5. Create an API gateway.
6. Link the function and the bot.
7. Test the bot.

If you no longer need the resources you created, delete them.

Getting startedGetting started

Sign up for Yandex Cloud and create a billing account:

1. Go to the management console and log in to Yandex Cloud or create an account if you do not have one yet.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one.

If you have an active billing account, you can go to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

Required paid resourcesRequired paid resources

The cost of Telegram bot support includes:

• Fee for using SpeechKit (see SpeechKit pricing).
• Fee for using Vision OCR (see Vision OCR pricing).
• Fee for the number of function calls, computing resources allocated to executing the function, and outgoing traffic (see Cloud Functions pricing).
• Fee for the number of requests to the API gateway and outgoing traffic (see API Gateway pricing).

Create resourcesCreate resources

1. Create a service account named recognizer-bot-sa and assign it the ai.editor and functions.editor roles for your folder.

2. Prepare a ZIP archive with the function code:

   1. Create a file named index.py and paste the code below to it.

      index.py

      import logging
      import requests
      import telebot
      import json
      import os
      import base64
      
      # Service endpoints and authentication data
      
      API_TOKEN = os.environ['TELEGRAM_TOKEN']
      vision_url = 'https://ocr.api.cloud.yandex.net/ocr/v1/recognizeText'
      speechkit_url = 'https://stt.api.cloud.yandex.net/speech/v1/stt:recognize'
      speechkit_synthesis_url = 'https://tts.api.cloud.yandex.net/speech/v1/tts:synthesize'
      folder_id = ""
      iam_token = ''
      
      logger = telebot.logger
      telebot.logger.setLevel(logging.INFO)
      bot = telebot.TeleBot(API_TOKEN, threaded=False)
      
      # Getting the folder ID
      
      def get_folder_id(iam_token, version_id):
      
          headers = {'Authorization': f'Bearer {iam_token}'}
          function_id_req = requests.get(f'https://serverless-functions.api.cloud.yandex.net/functions/v1/versions/{version_id}',
                                         headers=headers)
          function_id_data = function_id_req.json()
          function_id = function_id_data['functionId']
          folder_id_req = requests.get(f'https://serverless-functions.api.cloud.yandex.net/functions/v1/functions/{function_id}',
                                       headers=headers)
          folder_id_data = folder_id_req.json()
          folder_id = folder_id_data['folderId']
          return folder_id
      
      def process_event(event):
      
          request_body_dict = json.loads(event['body'])
          update = telebot.types.Update.de_json(request_body_dict)
      
          bot.process_new_updates([update])
      
      def handler(event, context):
          global iam_token, folder_id
          iam_token = context.token["access_token"]
          version_id = context.function_version
          folder_id = get_folder_id(iam_token, version_id)
          process_event(event)
          return {
              'statusCode': 200
          }
      
      # Command and message listeners
      
      @bot.message_handler(commands=['help', 'start'])
      def send_welcome(message):
          bot.reply_to(message,
                       "The bot can do the following:\n*Recognize text from images.\n* Generate voice messages from text.\n* Convert voice messages to text.")
      
      @bot.message_handler(func=lambda message: True, content_types=['text'])
      def echo_message(message):
          global iam_token, folder_id
          with open('/tmp/audio.ogg', "wb") as f:
              for audio_content in synthesize(folder_id, iam_token, message.text):
                  f.write(audio_content)
          voice = open('/tmp/audio.ogg', 'rb')
          bot.send_voice(message.chat.id, voice)
      
      @bot.message_handler(func=lambda message: True, content_types=['voice'])
      def echo_audio(message):
          file_id = message.voice.file_id
          file_info = bot.get_file(file_id)
          downloaded_file = bot.download_file(file_info.file_path)
          response_text = audio_analyze(speechkit_url, iam_token, folder_id, downloaded_file)
          bot.reply_to(message, response_text)
      
      @bot.message_handler(func=lambda message: True, content_types=['photo'])
      def echo_photo(message):
          file_id = message.photo[-1].file_id
          file_info = bot.get_file(file_id)
          downloaded_file = bot.download_file(file_info.file_path)
          image_data = base64.b64encode(downloaded_file).decode('utf-8')
          response_text = image_analyze(vision_url, iam_token, folder_id, image_data)
          bot.reply_to(message, response_text)
      
      # Image recognition
      
      def image_analyze(vision_url, iam_token, folder_id, image_data):
          response = requests.post(vision_url, headers={'Authorization': 'Bearer '+iam_token, 'x-folder-id': folder_id}, json={
              "mimeType": "image",
              "languageCodes": ["en", "ru"],
              "model": "page",
              "content": image_data
              })
          blocks = response.json()['result']['textAnnotation']['blocks']
          text = ''
          for block in blocks:
              for line in block['lines']:
                  for word in line['words']:
                      text += word['text'] + ' '
                  text += '\n'
          return text
      
      # Speech recognition
      
      def audio_analyze(speechkit_url, iam_token, folder_id, audio_data):
          headers = {'Authorization': f'Bearer {iam_token}'}
          params = {
              "topic": "general",
              "folderId": f"{folder_id}",
              "lang": "ru-RU"}
      
          audio_request = requests.post(speechkit_url, params=params, headers=headers, data=audio_data)
          responseData = audio_request.json()
          response = 'error'
          if responseData.get("error_code") is None:
              response = (responseData.get("result"))
          return response
      
      # Speech synthesis
      
      def synthesize(folder_id, iam_token, text):
         headers = {
             'Authorization': 'Bearer ' + iam_token,
         }
      
         data = {
             'text': text,
             'lang': 'ru-RU',
             'voice': 'filipp',
             'folderId': folder_id
         }
      
         with requests.post(speechkit_synthesis_url, headers=headers, data=data, stream=True) as resp:
             if resp.status_code != 200:
                 raise RuntimeError("Invalid response received: code: %d, message: %s" % (resp.status_code, resp.text))
      
             for chunk in resp.iter_content(chunk_size=None):
                 yield chunk
      

   2. Create a file named requirements.txt and specify in it the library to work with the bot.

      telebot
      

   3. Add both files to the index.zip archive.

Register the Telegram botRegister the Telegram bot

Register the bot in Telegram and get a token.

1. Run the BotFather and send it the following command:

   /newbot
   

2. In the name field, specify the name of the bot you are creating. This is the name users will see when communicating with the bot.

3. In the username field, specify the username of the bot you are creating. You can use the username to search for the bot in Telegram. The username must end with ...Bot or ..._bot.

   As a result, you will get a token. Save it. You will need it later.

Create a functionCreate a function

Create a function to process user actions in the chat.

Create an API gatewayCreate an API gateway

The Telegram server will notify your bot of new messages using a webhook. The API gateway will accept requests on the bot side and redirect them to the for-recognizer-bot function for processing.

Configure a link between the function and the Telegram botConfigure a link between the function and the Telegram bot

Install a webhook for your Telegram bot:

curl --request POST \
  --url https://api.telegram.org/bot<bot_token>/setWebhook \
  --header 'content-type: application/json' \ 
  --data '{"url": "<API_gateway_domain>/for-recognizer-bot-function"}'

Where:

• <bot_token>: Telegram bot token.
• <API_gateway_domain>: recognizer-bot-api-gw API gateway's service domain.

Result:

{"ok":true,"result":true,"description":"Webhook was set"}

Test the botTest the bot

Talk to the bot:

1. Open Telegram and search for the bot by the specified username.

2. Send the /start message to the chat.

   The bot must respond with:

   The bot can do the following:
   
   * Recognize text from images.
   * Generate voice messages from text.
   * Convert voice messages to text.
   

3. Send a text message to the chat. The bot will respond with a voice message synthesized from your text.

4. Send a voice message to the chat. The bot will respond with a message containing the text recognized from your speech.

5. Send an image with text to the chat. The bot will respond with a message containing the recognized text.

   Note

   The image must meet the requirements.

How to delete the resources you createdHow to delete the resources you created

To stop paying for the resources you created:

• Delete the API Gateway API gateway.
• Delete the Cloud Functions function.