Developing a Telegram bot for text recognition in images, audio synthesis and recognition

In this tutorial, you will create a bot for Telegram that can:

• Synthesize speech from a message text using the Yandex SpeechKit API v1.
• Recognize speech in voice messages and convert it into text using the Yandex SpeechKit synchronous recognition API.
• Recognize text in images using Yandex Vision OCR.

The Yandex API Gateway API gateway will receive requests from the bot and forward them to the Yandex Cloud Functions function for processing.

To create a bot:

1. Get your cloud ready.
2. Prepare the resources.
3. Register your Telegram bot.
4. Create a function.
5. Create an API gateway.
6. Link the function and the bot.
7. Test the bot.

If you no longer need the resources you created, delete them.

Getting startedGetting started

Sign up in Yandex Cloud and create a billing account:

1. Navigate to the management console and log in to Yandex Cloud or register a new account.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can navigate to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

Required paid resourcesRequired paid resources

The cost of Telegram bot support includes:

• Fee for using SpeechKit (see SpeechKit pricing).
• Fee for using Vision OCR (see Vision OCR pricing).
• Fee for the number of function calls, computing resources allocated to executing the function, and outgoing traffic (see Cloud Functions pricing).
• Fee for the number of requests to the API gateway and outgoing traffic (see API Gateway pricing).

Prepare the resourcesPrepare the resources

1. Create a service account named recognizer-bot-sa and assign it the ai.editor and functions.editor roles for your folder.

2. Prepare a ZIP archive with the function code:

   1. Create a file named index.py and paste the code below to it.

      index.py

      import logging
      import requests
      import telebot
      import json
      import os
      import base64
      
      # Service endpoints and authentication data
      
      API_TOKEN = os.environ['TELEGRAM_TOKEN']
      vision_url = 'https://ocr.api.cloud.yandex.net/ocr/v1/recognizeText'
      speechkit_url = 'https://stt.api.cloud.yandex.net/speech/v1/stt:recognize'
      speechkit_synthesis_url = 'https://tts.api.cloud.yandex.net/speech/v1/tts:synthesize'
      folder_id = ""
      iam_token = ''
      
      logger = telebot.logger
      telebot.logger.setLevel(logging.INFO)
      bot = telebot.TeleBot(API_TOKEN, threaded=False)
      
      # Getting the folder ID
      
      def get_folder_id(iam_token, version_id):
      
          headers = {'Authorization': f'Bearer {iam_token}'}
          function_id_req = requests.get(f'https://serverless-functions.api.cloud.yandex.net/functions/v1/versions/{version_id}',
                                         headers=headers)
          function_id_data = function_id_req.json()
          function_id = function_id_data['functionId']
          folder_id_req = requests.get(f'https://serverless-functions.api.cloud.yandex.net/functions/v1/functions/{function_id}',
                                       headers=headers)
          folder_id_data = folder_id_req.json()
          folder_id = folder_id_data['folderId']
          return folder_id
      
      def process_event(event):
      
          request_body_dict = json.loads(event['body'])
          update = telebot.types.Update.de_json(request_body_dict)
      
          bot.process_new_updates([update])
      
      def handler(event, context):
          global iam_token, folder_id
          iam_token = context.token["access_token"]
          version_id = context.function_version
          folder_id = get_folder_id(iam_token, version_id)
          process_event(event)
          return {
              'statusCode': 200
          }
      
      # Command and message listeners
      
      @bot.message_handler(commands=['help', 'start'])
      def send_welcome(message):
          bot.reply_to(message,
                       "The bot can do the following:\n*Recognize text from images.\n* Generate voice messages from text.\n* Convert voice messages to text.")
      
      @bot.message_handler(func=lambda message: True, content_types=['text'])
      def echo_message(message):
          global iam_token, folder_id
          with open('/tmp/audio.ogg', "wb") as f:
              for audio_content in synthesize(folder_id, iam_token, message.text):
                  f.write(audio_content)
          voice = open('/tmp/audio.ogg', 'rb')
          bot.send_voice(message.chat.id, voice)
      
      @bot.message_handler(func=lambda message: True, content_types=['voice'])
      def echo_audio(message):
          file_id = message.voice.file_id
          file_info = bot.get_file(file_id)
          downloaded_file = bot.download_file(file_info.file_path)
          response_text = audio_analyze(speechkit_url, iam_token, folder_id, downloaded_file)
          bot.reply_to(message, response_text)
      
      @bot.message_handler(func=lambda message: True, content_types=['photo'])
      def echo_photo(message):
          file_id = message.photo[-1].file_id
          file_info = bot.get_file(file_id)
          downloaded_file = bot.download_file(file_info.file_path)
          image_data = base64.b64encode(downloaded_file).decode('utf-8')
          response_text = image_analyze(vision_url, iam_token, folder_id, image_data)
          bot.reply_to(message, response_text)
      
      # Image recognition
      
      def image_analyze(vision_url, iam_token, folder_id, image_data):
          response = requests.post(vision_url, headers={'Authorization': 'Bearer '+iam_token, 'x-folder-id': folder_id}, json={
              "mimeType": "image",
              "languageCodes": ["en", "ru"],
              "model": "page",
              "content": image_data
              })
          blocks = response.json()['result']['textAnnotation']['blocks']
          text = ''
          for block in blocks:
              for line in block['lines']:
                  for word in line['words']:
                      text += word['text'] + ' '
                  text += '\n'
          return text
      
      # Speech recognition
      
      def audio_analyze(speechkit_url, iam_token, folder_id, audio_data):
          headers = {'Authorization': f'Bearer {iam_token}'}
          params = {
              "topic": "general",
              "folderId": f"{folder_id}",
              "lang": "ru-RU"}
      
          audio_request = requests.post(speechkit_url, params=params, headers=headers, data=audio_data)
          responseData = audio_request.json()
          response = 'error'
          if responseData.get("error_code") is None:
              response = (responseData.get("result"))
          return response
      
      # Speech synthesis
      
      def synthesize(folder_id, iam_token, text):
         headers = {
             'Authorization': 'Bearer ' + iam_token,
         }
      
         data = {
             'text': text,
             'lang': 'ru-RU',
             'voice': 'filipp',
             'folderId': folder_id
         }
      
         with requests.post(speechkit_synthesis_url, headers=headers, data=data, stream=True) as resp:
             if resp.status_code != 200:
                 raise RuntimeError("Invalid response received: code: %d, message: %s" % (resp.status_code, resp.text))
      
             for chunk in resp.iter_content(chunk_size=None):
                 yield chunk
      

   2. Create a file named requirements.txt and specify in it the library to work with the bot.

      telebot
      

   3. Add both files to the index.zip archive.

Register your Telegram botRegister your Telegram bot

Register your bot in Telegram and get a token.

1. Run BotFather and send to it the following command:

   /newbot
   

2. In the name field, enter a name for the new bot. This is the name the bot users will see.

3. In the username field, enter a username for the new bot. You can use it to locate the bot in Telegram. The username must end with ...Bot or ..._bot.

   As a result, you will get a token. Save it. You will need it later.

Create a functionCreate a function

Create a function to process user actions in the chat.

Create an API gatewayCreate an API gateway

The Telegram server will notify your bot of new messages using a webhook. The API gateway will accept requests on the bot side and redirect them to the for-recognizer-bot function for processing.

Configure a link between the function and the Telegram botConfigure a link between the function and the Telegram bot

Install a webhook for your Telegram bot:

curl --request POST \
  --url https://api.telegram.org/bot<bot_token>/setWebhook \
  --header 'content-type: application/json' \ 
  --data '{"url": "<API_gateway_domain>/for-recognizer-bot-function"}'

Where:

• <bot_token>: Telegram bot token.
• <API_gateway_domain>: recognizer-bot-api-gw API gateway's service domain.

Result:

{"ok":true,"result":true,"description":"Webhook was set"}

Test the botTest the bot

Talk to the bot:

1. Open Telegram and search for the bot by the specified username.

2. Send the /start message to the chat.

   The bot must respond with:

   The bot can do the following:
   
   * Recognize text from images.
   * Generate voice messages from text.
   * Convert voice messages to text.
   

3. Send a text message to the chat. The bot will respond with a voice message synthesized from your text.

4. Send a voice message to the chat. The bot will respond with a message containing the text recognized from your speech.

5. Send an image with text to the chat. The bot will respond with a message containing the recognized text.

   Note

   The image must meet the requirements.

How to delete the resources you createdHow to delete the resources you created

To stop paying for the resources you created:

• Delete the API Gateway API gateway.
• Delete the Cloud Functions function.