Streaming Yandex Cloud Postbox events to Yandex Data Streams and analyzing them using Yandex DataLens

1. In the management console, select the appropriate folder.
2. At the top of the screen, go to the Service accounts tab.
3. Click Create service account.
4. Enter a name for the service account: yds-functions.
5. Click Add role and select the yds.editor and functions.functionInvoker roles.
6. Click Create.

1. In the management console, select the appropriate folder.
2. Select Managed Service for YDB.
3. Click Create a database.
4. Specify the DB Name, postbox-events-ydb.
5. Under Database type, select Serverless.
6. Click Create a database.

Wait for the DB to start. While being created, your database will have the Provisioning status. Once it is ready for use, its status will change to Running.

1. On the Databases page, select the postbox-events-ydb DB.

2. To open the DB root directory, go to the Navigation tab.

3. To create a DB query, click New SQL query in the upper right corner. The Query page opens.

4. In the Query field, enter:

   CREATE TABLE postbox_events
   (
       saved_datetime Datetime NOT NULL,
       eventid String NOT NULL,
       eventtype String,
       mail_timestamp Timestamp,
       mail_messageid String,
       mail_ch_from String,
       mail_ch_to String, 
       mail_ch_messageid String,
       mail_ch_subject String,
       delivery_timestamp Timestamp,
       delivery_time_ms Uint64,
       delivery_recipients String,
       bounce_bounceType String,
       bounce_bounceSubType String,
       bounce_bouncedRecipients String,
       bounce_timestamp Timestamp,
       -- message Json,
       PRIMARY KEY (saved_datetime, eventid)
   )
   

5. Click Run.

   After you run the query, the postbox_events table will be created.

1. In the management console, select Data Streams.
2. Click Create stream.
3. Specify the postbox-events-ydb database.
4. Enter a name for the data stream: postbox-events-stream.
5. Select the Actual usage pricing plan.
6. Click Create.

Wait for the stream to start. Once the stream is ready for use, its status will change from Creating to Active.

1. In the management console, select the folder containing the address and your domain zone.

2. Select Cloud DNS.

3. Select your domain zone.

4. Click Create record.

5. In the Name field, specify the name generated when creating the address, omitting the domain. For example, postbox._domainkey..

   Note

   For other DNS services, you may need to copy the entire record. The final record must look like this: postbox._domainkey.example.com..

6. In the Type list, select TXT.

7. Copy the contents of the Value field from the Signature verification section and paste it into the Data field. Note that the record value must be enclosed in quotes, for example:

   "v=DKIM1;h=sha256;k=rsa;p=M1B...aCA8"
   

8. Click Create.

9. Go to Cloud Postbox.

10. Select the address you created.

11. Click Verify address. If the record is correct, the verification status on the address page will change to Success.

DNS server responses are cached, so delays may occur when updating a resource record.

1. In the management console, select Cloud Postbox.

2. In the left-hand panel, select Configurations.

3. Click Create configuration.

4. Enter the configuration name: postbox-events-config.

5. Under Subscriptions, click Add:

   1. Enter a name for the subscription: postbox-events-subscribe.
   2. Select the postbox-events-stream data stream.
   3. Toggle the Enabled option on to activate the subscription.

6. Click Create configuration.

Download the postbox-events.zip archive to your computer. The archive contains the index.py and requirements.txt files with the function code.

1. Go to the management console.

2. Select the folder hosting your DB and go to Managed Service for YDB.

3. Select the DB you want to get the endpoint and path for.

   • The DB endpoint is specified under Connection in the first part of the Endpoint field value (preceding /?database=):

     For example, this is the endpoint of a DB in serverless mode: grpcs://ydb.serverless.yandexcloud.net:2135.

   • The DB path is specified under Connection in the second part of the Endpoint field value (following /?database=).

     Example of a DB path: /ru-central1/b1gia87mbaomkfvs6rgl/etnudu2n9ri35luqe4h1.

1. In the management console, select Cloud Functions.

2. Create a function:

   1. Click Create function.
   2. In the Name field, enter a name for the function, e.g., postbox-events-function.
   3. Click Create.

3. Create a function version:

   1. In the Editor window that opens, select Python 3.12.

   2. Disable the Add files with code examples option.

   3. Click Continue.

   4. In the Method field, select ZIP archive.

   5. Click Attach file and select the postbox-events.zip archive.

   6. In the Entry point field, specify index.handler.

   7. Under Parameters, specify:

      • Timeout, sec: 10.

      • Memory: 128 MB.

      • Service account: yds-functions.

      • Environment variables:

        Key	Description	Sample value
        YDB_DATABASE	DB path	/ru-central1/b1go123e9vjq********/etnu15kr22********
        YDB_ENDPOINT	DB endpoint	grpcs://ydb.serverless.yandexcloud.net:2135
        YDB_TABLE	Table name	postbox_events

   8. Click Save changes.

1. In the management console, select Cloud Functions.
2. In the left-hand panel, select Triggers.
3. Click Create trigger.
4. In the Name field, enter a name for the trigger, e.g., postbox-events-trigger.
5. In the Type field, select Data Streams.
6. In the Launched resource field, select Function.
7. Under Data Streams settings, select the postbox-events-stream data stream and the yds-functions service account.
8. Under Function settings, select postbox-events-function and the yds-functions service account.
9. Click Create trigger.

1. Go to the DataLens home page.

2. Click Create connection.

3. Select the YDB connector.

4. Specify the connection parameters:

   • Cloud and folder: Select the folder you are using to complete this tutorial.
   • Service account: yds-functions.
   • Database: postbox-events-ydb.
   • Cache TTL in seconds: Default.
   • SQL query access level: Allow subqueries from datasets and queries from charts.

5. Click Create connection.

6. Enter a connection name, e.g., postbox-events-connection, and click Create.

1. Go to the DataLens home page.
2. Click Create dataset.
3. In the left-hand panel, click Add.
4. Select a postbox-events-connection connection.
5. In the left-hand menu, select the postbox-events table and drag it to the right.
6. Click Save.
7. In the window that opens, specify postbox-events-dataset and click Create.

1. Go to the DataLens home page.

2. Click Create chart.

3. In the left-hand panel, click Select dataset and select postbox-events-dataset.

4. Select Bar chart as the chart type.

5. Drag the dimensions as follows:

   • delivery_timestamp to the X section: Dates will be plotted on the X axis.
   • eventid to the Y section: Number of events will be plotted on the Y axis.
   • eventtype to the Colors section: This enables using a different color for each event type.

6. Click Save.

7. In the window that opens, specify the chart name, Events by day, and click Save.

1. Go to the DataLens home page.

2. Click Create chart.

3. In the left-hand panel, click Select dataset and select postbox-events-dataset.

4. Select Table as the chart type.

5. Drag the following dimensions to the Columns section:

   • saved_datetime: Date and time of saving the event.
   • eventid: Event type.
   • mail_ch_to: Address of the recipient.
   • delivery_timestamp: Email delivery date and time.
   • delivery_time_ms: Delivery time in milliseconds.
   • mail_ch_subject: Email subject.
   • bounce_bounceType: Delivery error type.
   • bounce_bounceSubType: Non-delivery message.

6. Drag the delivery_timestamp dimension to the Sorting section to sort the table by delivery time.

7. Click Save.

8. In the window that opens, specify the chart name, Event list, and click Save.

1. Go to the DataLens home page.

2. Click Create dashboard.

3. Add the Events by day, List of events, and other charts you created earlier to your dashboard:

   1. In the bottom panel, click Chart.
   2. In the Chart field of the window that opens, click Select and select Events by day.
   3. Click Add.
   4. Repeat these steps to add the other charts.

4. Change the size and position of your charts.

5. Click Save.

6. In the window that opens, enter a name for the dashboard: postbox-events-dashboard.

7. Click Create.