Configuring an SMTP server to send e-mail notifications

Use a directed acyclic graph (DAG) to configure your SMTP server to send e-mail notifications. Data for connecting to the DB is stored in Yandex Lockbox and automatically inserted into the graph.

To configure an SMTP server to send e-mail notifications, follow these steps:

1. Set up your infrastructure.
2. Create a Yandex Lockbox secret.
3. Prepare the DAG file and run the graph.
4. Check the result.

If you no longer need the resources you created, delete them.

Getting startedGetting started

Sign up for Yandex Cloud and create a billing account:

1. Navigate to the management console and log in to Yandex Cloud or create a new account.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can create or select a folder for your infrastructure on the cloud page.

Learn more about clouds and folders here.

Required paid resourcesRequired paid resources

• Managed Service for Apache Airflow™ cluster: computing resources of cluster components (see Managed Service for Apache Airflow™ pricing).
• Yandex Object Storage bucket: use of storage, data operations (see Object Storage pricing).
• Yandex Lockbox secret: number of stored secret versions and requests to them (see Yandex Lockbox pricing).

Set up your infrastructureSet up your infrastructure

1. Create a service account named airflow-sa with the following roles:

   • managed-airflow.integrationProvider
   • lockbox.payloadViewer

   There is no need to assign the lockbox.payloadViewer role for the whole folder. It is enough to assign it for a specific Yandex Lockbox secret once you create it.

2. Create an Object Storage bucket in any configuration.

3. Edit the ACL of the new bucket to give the READ permission to the airflow-sa service account.

4. Create a Managed Service for Apache Airflow™ cluster with the following parameters:

   • Service account: airflow-sa.
   • Bucket name: Name of the new bucket.
   • Use Lockbox Secret Backend: Make sure to enable this option.

   Under Airflow configuration, set up additional Apache Airflow™ properties to configure the SMTP server:

   Key	Value
   smtp.smtp_host	smtp.yandex.ru
   smtp.smtp_port	465
   smtp.smtp_starttls	False
   smtp.smtp_ssl	True
   smtp.smtp_mail_from	user@yandex.ru

   Fill in the fields manually or import the settings from a configuration file (see a configuration file example).

Create a Yandex Lockbox secretCreate a Yandex Lockbox secret

Create a Yandex Lockbox secret named airflow/connections/smtp_default, containing json.dumps(conn), where conn is a connection to an SMTP server.

conn = {
         "conn_type": "smtp",
         "login": "user@yandex.ru",
         "password": "<password_for_external_apps>"
}

The airflow/connections/smtp_default secret will store the data for connecting to the SMTP server.

Prepare the DAG file and run the graphPrepare the DAG file and run the graph

1. Create a local file named test_smtp_notification.py and paste the following script to it:

   from airflow.decorators import dag, task
   
   default_args = {
    "email": ["user@yandex.ru"],
    "email_on_failure": True,
    "email_on_retry": True
   }
   
   
   @dag(schedule=None, default_args=default_args)
   def test_smtp_notification():
       @task
       def failing_task():
           raise RuntimeError("some test error")
   
       failing_task()
   
   
   test_smtp_notification()
   

2. Upload the test_smtp_notification.py DAG file to the bucket you created earlier. This will automatically create a graph with the same name in the Apache Airflow™ web interface.

3. Open the Apache Airflow™ web interface.

4. Make sure a new graph named test_smtp_notification has appeared in the DAGs section.

   It may take a few minutes to upload a DAG file from the bucket.

5. To run the graph, click in the line with its name.

Check the resultCheck the result

To check the result in the Apache Airflow™ web interface:

1. In the DAGs section, open the test_smtp_notification graph.
2. Go to the Graph section.
3. Select failing_task.
4. Go to Logs.
5. Make sure the logs do not contain the some test error line. This means the query was successful.

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

1. Service account.
2. Object Storage bucket.
3. Managed Service for Apache Airflow™ cluster.
4. Yandex Lockbox secret.