Configuring an SMTP server to send e-mail notifications

Use a directed acyclic graph (DAG) to configure your SMTP server to send email notifications. Data for connecting to the DB is stored in Yandex Lockbox and automatically substituted into the graph.

To configure an SMTP server to send e-mail notifications, follow these steps:

1. Set up your infrastructure.
2. Create a Yandex Lockbox secret.
3. Prepare the DAG file and run the graph.
4. Check the result.

If you no longer need the resources you created, delete them.

Set up your infrastructureSet up your infrastructure

1. Create a service account named airflow-sa with the following roles:

   • managed-airflow.integrationProvider
   • lockbox.payloadViewer

   You do not have to assign the lockbox.payloadViewer role for the whole folder. It is enough to assign it for a specific Yandex Lockbox secret once you create it.

2. Create an Object Storage bucket in any configuration.

3. Edit the ACL of the new bucket to give the READ permission to the airflow-sa service account.

4. Create a Managed Service for Apache Airflow™ cluster with the following parameters:

   • Service account: airflow-sa.
   • Bucket name: Name of the new bucket.
   • Use Lockbox Secret Backend: Make sure to enable this option.

   Under Airflow configuration, set up additional Apache Airflow™ properties to configure the SMTP server:

   Key	Value
   smtp.smtp_host	smtp.yandex.ru
   smtp.smtp_port	465
   smtp.smtp_starttls	False
   smtp.smtp_ssl	True
   smtp.smtp_mail_from	user@yandex.ru

   Fill in the fields manually or import the settings from a configuration file (see configuration file example).

Required paid resourcesRequired paid resources

The support cost includes:

• Managed Service for Apache Airflow™ cluster fee: Using computing resources allocated to hosts and disk space (see Managed Service for Apache Airflow™ pricing).
• Fee for using public IP addresses if public access is enabled for cluster hosts (see Virtual Private Cloud pricing).
• Object Storage bucket fee: Storing data and performing operations with it (see Object Storage pricing).
• Yandex Lockbox fee: Storing secrets (see Yandex Lockbox pricing).

Create a Yandex Lockbox secretCreate a Yandex Lockbox secret

Create a Yandex Lockbox secret named airflow/connections/smtp_default, containing json.dumps(conn), where conn is a connection to an SMTP server.

conn = {
         "conn_type": "smtp",
         "login": "user@yandex.ru",
         "password": "<password_for_external_apps>"
}

The airflow/connections/smtp_default secret will store the data to connect to the SMTP server.

Prepare the DAG file and run the graphPrepare the DAG file and run the graph

1. Create a local file named test_smtp_notification.py and copy the following script to it:

   from airflow.decorators import dag, task
   
   default_args = {
    "email": ["user@yandex.ru"],
    "email_on_failure": True,
    "email_on_retry": True
   }
   
   
   @dag(schedule=None, default_args=default_args)
   def test_smtp_notification():
       @task
       def failing_task():
           raise RuntimeError("some test error")
   
       failing_task()
   
   
   test_smtp_notification()
   

2. Upload the test_smtp_notification.py DAG file to the bucket you created earlier. This will automatically create a graph with the same name in the Apache Airflow™ web interface.

3. Open the Apache Airflow™ web interface.

4. Make sure a new graph named test_smtp_notification has appeared in the DAGs section.

   It may take a few minutes to upload a DAG file from the bucket.

5. To run the graph, click in the line with its name.

Check the resultCheck the result

To check the result in the Apache Airflow™ web interface:

1. In the DAGs section, open the test_smtp_notification graph.
2. Go to the Graph section.
3. Select the failing_task job.
4. Go to Logs.
5. Make sure the logs do not contain the some test error string. This means the query was successful.

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

1. Service account.
2. Object Storage bucket.
3. Managed Service for Apache Airflow™ cluster.
4. Yandex Lockbox secret.