Running a PySpark job in Yandex Managed Service for YTsaurus

In a YTsaurus cluster, you can run Spark and PySpark jobs using the spark-submit script. It saves the calculation results to HDFS. For more information about spark-submit, see the Spark documentation. These jobs are executed in a Apache Spark™ cluster that runs on the computing resources of the YTsaurus cluster, using the SPYT tool.

The example below shows how to run an application in Python.

To run a PySpark job:

1. Prepare the infrastructure.
2. Prepare your VM.
3. Prepare your test data.
4. Create and run a PySpark job.

If you no longer need the resources you created, delete them.

Required paid resourcesRequired paid resources

• The YTsaurus cluster, which includes the use of computing resources of cluster components and storage size (see YTsaurus pricing).
• VM instance: use of computing resources, storage, public IP address, and OS (see Compute Cloud pricing).

Getting startedGetting started

Set up your infrastructure:

Prepare your VMPrepare your VM

1. Connect to a virtual machine over SSH.

2. If you do not have the Yandex Cloud CLI installed yet, install and initialize it.

3. Install the dependencies:

   sudo apt update && sudo apt install -y python3 python3-pip python3-venv
   

4. Create and activate a virtual environment to run a Python script:

   python3 -m venv .venv/yt312 && \
   source .venv/yt312/bin/activate
   

5. Install Java:

   sudo apt install -y openjdk-11-jdk
   

6. Install the YTsaurus CLI:

   pip install ytsaurus-client
   

7. Create an IAM token:

   yc iam create-token
   

8. Get the service account ID:

   yc iam whoami
   

9. Set the environment variables:

   export YT_PROXY=http://hp.<YTsaurus_cluster_ID>.ytsaurus.mdb.yandexcloud.net:32100 ; \
   export YT_TOKEN=<IAM_token> ; \
   export YT_USER=<service_account_ID>
   

10. Get the SPYR version:

    yt list //home/spark/conf/releases
    

    Result:

    2.7.0
    

11. Install the ytsaurus-spyt package of the same version as SPYT:

    pip install ytsaurus-spyt==2.7.0
    

12. Get the Spark version from the name of the tgz file:

    yt find //home/spark/distrib --name *tgz
    

    Result:

    //home/spark/distrib/3/5/6/spark-3.5.6-bin-hadoop3.tgz
    

13. Install pyspark of the same version as the spark distribution:

    pip install pyspark==3.5.6
    

14. Activate the SPYT configuration:

    source spyt-env
    

    Result:

    SPYT environment has been successfully activated
    

15. Check the value of the $SPARK_CONF_DIR variable:

    echo $SPARK_CONF_DIR
    

    Result:

    /home/<home_directory>/.venv/yt312/lib/python3.12/site-packages/spyt/conf
    

Prepare your test dataPrepare your test data

Run these commands on the VM:

1. Create a table named table1 in YTsaurus:

   yt create table //tmp/table1 \
     --attributes '{schema = [{name = id; type = int64}; {name = text; type = string}]}'
   

2. Write the data to the table:

   echo '{ "id": 0, "text": "Hello" } { "id": 1, "text": "World!" }' | yt write-table //tmp/table1 --format json
   

Create and run a PySpark jobCreate and run a PySpark job

Perform these actions on the VM:

1. Create the spark-job.py file with the PySpark job.

   import spyt
   from pyspark import SparkConf
   
   conf = SparkConf()
   conf.set("spark.app.name","Show table")
   with spyt.direct_spark_session("http-proxies-lb.yt.svc.cluster.local", conf) as spark:
        spark.read.yt('//tmp/table1').show()
   

2. Run the job:

   spark-submit --master "ytsaurus://${YT_PROXY}" \
     --deploy-mode cluster \
     --num-executors 1 \
     --conf spark.hadoop.yt.proxyNetworkName=external \
     --queue research \
     ./spark-job.py
   

   Result:

   ...
   25/12/09 23:29:49 INFO YTsaurusClusterApplication: Operation: fdff4aa2-d11767f6-e03e8-6affdd7c, State: running
   25/12/09 23:30:23 INFO YTsaurusClusterApplication: Operation: fdff4aa2-d11767f6-e03e8-6affdd7c, State: completed
   

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them: