Streaming Yandex Cloud Postbox events to Yandex Data Streams and analyzing them using Yandex DataLens

1. In the management console, select the relevant folder.
2. In the list of services, select Identity and Access Management.
3. Click Create service account.
4. Name the service account: yds-functions.
5. Click Add role and select the yds.editor and functions.functionInvoker roles.
6. Click Create.

1. In the management console, select the relevant folder.
2. Select Managed Service for YDB.
3. Click Create a database.
4. Specify the DB Name: postbox-events-ydb.
5. Under Database type, select Serverless.
6. Click Create a database.

Wait for the database to start. While being created, your database will have the Provisioning status. Once it is ready for use, its status will change to Running.

1. On the Databases page, select the postbox-events-ydb DB.

2. To open the DB root directory, navigate to the Navigation tab.

3. To create a DB query, click New SQL query in the top-right corner. The Query page will open.

4. In the Query field, enter:

   CREATE TABLE postbox_events
   (
       saved_datetime Datetime NOT NULL,
       eventid String NOT NULL,
       eventtype String,
       mail_timestamp Timestamp,
       mail_messageid String,
       mail_ch_from String,
       mail_ch_to String, 
       mail_ch_messageid String,
       mail_ch_subject String,
       delivery_timestamp Timestamp,
       delivery_time_ms Uint64,
       delivery_recipients String,
       bounce_bounceType String,
       bounce_bounceSubType String,
       bounce_bouncedRecipients String,
       bounce_timestamp Timestamp,
       -- message Json,
       PRIMARY KEY (saved_datetime, eventid)
   )
   

5. Click Run.

   After you run the query, the postbox_events table will appear in your database.

1. In the management console, select Data Streams.
2. Click Create stream.
3. Specify the postbox-events-ydb database.
4. Name the data stream: postbox-events-stream.
5. Select the Actual usage pricing plan.
6. Click Create.

Wait for the stream to start. Once the stream is ready for use, its status will change from Creating to Active.

1. In the management console, select the folder containing the address and your domain zone.

2. Select Cloud DNS.

3. Select your domain zone.

4. Click Create record.

5. In the Name field, specify the name portion generated when creating the address, omitting the domain in <selector>._domainkey format, e.g., postbox._domainkey.

   Note

   For other DNS services, you may need to copy the entire record. The final record must look like this: <selector>._domainkey.<domain>., e.g., postbox._domainkey.example.com..

6. In the Type list, select TXT.

7. Copy the contents of the Value field from the Signature verification section and paste it into the Data field. Note that the record value must be enclosed in quotes, for example:

   "v=DKIM1;h=sha256;k=rsa;p=M1B...aCA8"
   

8. Click Create.

9. Go to Cloud Postbox.

10. Select the address you created.

11. Click Verify address. If the record is correct, the verification status on the address page will change to Success.

DNS server responses are cached, so delays may occur when updating a resource record.

1. In the management console, select Cloud Postbox.

2. In the left-hand panel, select Configurations.

3. Click Create configuration.

4. Name the configuration: postbox-events-config.

5. Under Subscriptions, click Add:

   1. Enter the subscription name: postbox-events-subscribe.
   2. Select the postbox-events-stream data stream.
   3. Toggle the Enabled option on to activate the subscription.

6. Click Create configuration.

Download the postbox-events.zip archive to your computer. The archive contains the index.py and requirements.txt files with the function code.

1. Navigate to the management console.

2. Select the folder with your DB and go to Managed Service for YDB.

3. Select the DB for which you need to get the endpoint and path.

   • The DB endpoint is specified under Connection in the first part of the Endpoint field value (preceding /?database=):

     For example, the endpoint of a serverless DB is grpcs://ydb.serverless.yandexcloud.net:2135.

   • The DB path is specified under Connection in the second part of the Endpoint field value (following /?database=).

     Here is an example of a DB path: /ru-central1/b1gia87mbaomkfvs6rgl/etnudu2n9ri35luqe4h1.

1. In the management console, select Cloud Functions.

2. Create a function:

   1. Click Create function.
   2. In the Name field, enter a name for the function, e.g., postbox-events-function.
   3. Click Create.

3. Create a function version:

   1. In the Editor window that opens, select Python 3.12.

   2. Disable Add files with code examples.

   3. Click Continue.

   4. In the Method field, select ZIP archive.

   5. Click Attach file and select postbox-events.zip.

   6. In the Entry point field, specify index.handler.

   7. Under Parameters, specify:

      • Timeout: 10

      • Memory: 128 MB

      • Service account: yds-functions

      • Environment variables:

        Key	Description	Value (example)
        YDB_DATABASE	DB path	/ru-central1/b1go123e9vjq********/etnu15kr22********
        YDB_ENDPOINT	DB endpoint	grpcs://ydb.serverless.yandexcloud.net:2135
        YDB_TABLE	Table name	postbox_events

   8. Click Save changes.

1. In the management console, select Cloud Functions.
2. In the left-hand panel, select Triggers.
3. Click Create trigger.
4. In the Name field, enter a name for the trigger, e.g., postbox-events-trigger.
5. In the Type field, select Data Streams.
6. In the Launched resource field, select Function.
7. Under Data Streams settings, select the data stream named postbox-events-stream and the yds-functions service account.
8. Under Function settings, select the function named postbox-events-function and the yds-functions service account.
9. Click Create trigger.

1. Go to the DataLens home page.

2. Click Create connection.

3. Select the YDB connector.

4. Configure the connection as follows:

   • Cloud and folder: Select the folder you are using to complete this tutorial.
   • Service account: yds-functions.
   • Database: postbox-events-ydb.
   • Cache TTL in seconds: Default.
   • SQL query access level: Allow subqueries from datasets and queries from charts.

5. Click Create connection.

6. Enter a connection name, e.g., postbox-events-connection, and click Create.

1. Go to the DataLens home page.
2. Click Create dataset.
3. In the left-hand panel, click Add.
4. Select postbox-events-connection.
5. In the left-hand menu, select the postbox-events table and drag it to the right.
6. Click Save.
7. In the window that opens, specify postbox-events-dataset and click Create.

1. Go to the DataLens home page.

2. Click Create chart.

3. In the left-hand panel, click Select dataset and select postbox-events-dataset.

4. Select Column chart as the chart type.

5. Drag the dimensions as follows:

   • delivery_timestamp to the X section: Dates will be plotted along the X axis.
   • eventid to the Y section: Number of events will be plotted on the Y axis.
   • eventtype to the Colors section: This enables using a different color for each event type.

6. Click Save.

7. In the window that opens, specify the chart name, Events by day, and click Save.

1. Go to the DataLens home page.

2. Click Create chart.

3. In the left-hand panel, click Select dataset and select postbox-events-dataset.

4. Select Table as the chart type.

5. Drag the following dimensions to the Columns section:

   • saved_datetime: Date and time of saving the event.
   • eventid: Event type.
   • mail_ch_to: Recipient's email address.
   • delivery_timestamp: Email delivery date and time.
   • delivery_time_ms: Delivery time in milliseconds.
   • mail_ch_subject: Email subject.
   • bounce_bounceType: Delivery error type.
   • bounce_bounceSubType: Non-delivery report.

6. Drag the delivery_timestamp dimension to the Sorting section to sort the table by delivery time.

7. Click Save.

8. In the window that opens, specify the chart name, Event list, and click Save.

1. Go to the DataLens home page.

2. Click Create dashboard.

3. Add Events by day, Event list, and other charts you created earlier to your dashboard:

   1. In the bottom panel, click Chart.
   2. In the Chart field of the window that opens, click Select and select Events by day.
   3. Click Add.
   4. Repeat these steps to add other charts.

4. Resize and move your charts as needed.

5. Click Save.

6. In the window that opens, name the dashboard: postbox-events-dashboard.

7. Click Create.