Configuring an SMTP server to send e-mail notifications

Use a directed acyclic graph (DAG) to configure your SMTP server to send e-mail notifications. In this setup, the SMTP server parameters are stored in the Apache Airflow™ configuration, while the sender's credentials are stored in a Yandex Lockbox secret.

This guide features Yandex Mail (smtp.yandex.ru) as the SMTP server. To verify notification delivery, a DAG file is used in which the task is intentionally made to fail. If an error occurs, a notification is sent to the email address specified in the DAG file.

Before sending notifications, prepare the sender's mailbox:

• Allow access to the mailbox for email clients.
• Create a password for external applications.

To configure the SMTP server for sending email notifications:

1. Set up your infrastructure.
2. Create a Yandex Lockbox secret.
3. Prepare the DAG file and run the graph.
4. Check the result.

If you no longer need the resources you created, delete them.

Getting startedGetting started

Sign up for Yandex Cloud and create a billing account:

1. Navigate to the management console and log in to Yandex Cloud or create a new account.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can create or select a folder for your infrastructure on the cloud page.

Learn more about clouds and folders here.

Required paid resourcesRequired paid resources

• Managed Service for Apache Airflow™ cluster: computing resources of cluster components (see Managed Service for Apache Airflow™ pricing).
• Yandex Object Storage bucket: use of storage, data operations (see Object Storage pricing).
• Yandex Lockbox secret: number of stored secret versions and requests to them (see Yandex Lockbox pricing).

Set up your infrastructureSet up your infrastructure

1. Create a service account named airflow-sa with the following roles:

   • managed-airflow.integrationProvider
   • lockbox.payloadViewer

   Note

   There is no need to assign the lockbox.payloadViewer role for the whole folder. It is enough to assign it for a specific Yandex Lockbox secret once you create it.

2. Create an Object Storage bucket in any configuration.

3. Edit the ACL of the new bucket to give the READ permission to the airflow-sa service account.

4. Create a Managed Service for Apache Airflow™ cluster with the following parameters:

   • Service account: airflow-sa.
   • Bucket name: Name of the new bucket.
   • Use Lockbox Secret Backend: Make sure to enable this option.

   Under Airflow configuration, set up additional Apache Airflow™ properties to configure the SMTP server:

   Key	Value
   smtp.smtp_host	smtp.yandex.ru
   smtp.smtp_port	465
   smtp.smtp_starttls	False
   smtp.smtp_ssl	True
   smtp.smtp_mail_from	<sender_email_address>

   Fill in the fields manually or upload a configuration from a file. To upload a configuration from a file, click Load from file and select the Text uploading method. Copy the following configuration into the Content field:

   SMTP server configuration

   [smtp]
   smtp_host = smtp.yandex.ru
   smtp_port = 465
   smtp_starttls = False
   smtp_ssl = True
   smtp_mail_from = <sender_email_address>
   

   Read more about the SMTP server configuration parameters in the relevant Apache Airflow™ guide.

5. Configure a NAT gateway for the Managed Service for Apache Airflow™ cluster's subnets.

Create a Yandex Lockbox secretCreate a Yandex Lockbox secret

Create a Yandex Lockbox secret with the following settings:

• Name: airflow/connections/smtp_default.

• Secret type: Custom.

• Key: conn.

• Value: Select Text and specify the following contents:

  {
    "conn_type": "smtp",
    "login": "<sender_email_address>",
    "password": "<password_for_external_apps>"
  }
  

Prepare the DAG file and run the graphPrepare the DAG file and run the graph

1. Create a local file named test_smtp_notification.py and paste the following script to it:

   from airflow.decorators import dag, task
   
   default_args = {
    "email": ["<recipient_email_address>"],
    "email_on_failure": True,
    "email_on_retry": True
   }
   
   
   @dag(schedule=None, default_args=default_args)
   def test_smtp_notification():
       @task
       def failing_task():
           raise RuntimeError("some test error")
   
       failing_task()
   
   
   test_smtp_notification()
   

2. Upload the test_smtp_notification.py DAG file to the bucket you created earlier. This will automatically create a graph with the same name in the Apache Airflow™ web interface.

3. Open the Apache Airflow™ web interface.

4. Make sure a new graph named test_smtp_notification has appeared in the DAGs section.

   It may take a few minutes to upload a DAG file from the bucket.

5. To run the graph, click in the line with its name.

6. Wait for the task to fail. A notification will be sent to the specified email address.

   Warning

   If the SMTP server parameters specified in the Apache Airflow™ configuration are incorrect, notifications will not be sent.

Check the resultCheck the result

To check the result in the Apache Airflow™ 2.11 web interface:

1. In the DAGs section, open the test_smtp_notification graph.
2. Select the failing_task.
3. On the Task Instance Duration chart, select a task launch.
4. Navigate to the Logs tab.
5. Make sure the logs contain a Sent an alert email to ['<recipient_email_address>'] line.

To check the result in the Apache Airflow™ 3.1 web interface:

1. In the DAGs section, open the test_smtp_notification graph.
2. Navigate to the Tasks tab and select the failing_task.
3. Navigate to the Task Instances tab and select a task launch.
4. Navigate to the Logs tab.
5. Make sure the logs contain a Sent an alert email to ['<recipient_email_address>'] line.

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

1. Service account.
2. Object Storage bucket.
3. Managed Service for Apache Airflow™ cluster.
4. Yandex Lockbox secret.