
There are different ways to submit your application on a cluster but the most common is to use the spark-submit.
spark-submit
spark-submitis a command-line tool provided by Apache Spark for submitting Spark applications to a cluster. It is used to launch applications on a standalone Spark cluster, a Hadoop YARN cluster, or a Mesos cluster.
The spark-submit tool takes a JAR file or a Python file as input along with the application’s configuration options and submits the application to the cluster. The configuration options can be used to set various parameters for the application, such as the number of executor cores, the amount of memory allocated to each executor, and the number of executors.
The general structure of the spark-submit command is as follows:
./bin/spark-submit [options] <application-jar> [application-arguments]
spark-submit: The name of the command-line tool for submitting Spark applications.[options]: Optional command-line options that configure the behavior of the spark-submit tool and the Spark application being submitted.<application-jar>: The path to the JAR file containing the Spark application code. This JAR file must be created beforehand using a build tool. (A JAR (Java Archive) file is a package file format used to aggregate Java class files, associated metadata, and resources (such as images, sound files, and other supporting files) into a single file. It is a standard format used for distributing Java applications and libraries.)[application-arguments]: Optional command-line arguments that are passed to the Spark application's main method.
spark-submit options
some of the common options, configurations, and specific options to use with Scala and Python.
./bin/spark-submit --help
Submit Scala Application
./bin/spark-submit \
--master yarn \
--class org.nppntraining.WordCountExample \
user-data-analysis.jar
Submit PySpark Application
./bin/spark-submit \
--master yarn \
WordCountExample.py
Commonly used options
Here are some examples of commonly used options:
Using
--masteroption, you specify what cluster manager to use to run your application. Spark currently supports Yarn, Mesos, Kubernetes, Stand-alone, and local.--class <main-class>: Specifies the fully qualified name of the main class for the Spark application. This option is required for Java or Scala applications, but not needed for pySpark.
./bin/spark-submit \
--master yarn \
--class org.apache.spark.examples.SparkPi \
/home/npntraining/apache-spark-2.4.0/jars/spark-examples_versionxx.jar 80
Cluster Managers
Using --master option, you specify what cluster manager to use to run your application. Spark currently supports Yarn, Mesos, Kubernetes, Stand-alone, and local.
Deployment Modes
Using --deploy-mode, you specify where to run the Spark application driver program. Spark supports cluster and client deployment modes.
Cluster: In cluster mode, the driver runs on one of the worker nodes, and this node shows as a driver on the Spark Web UI of your application. cluster mode is used to run production jobs.
Client: In client mode, the driver runs locally where you are submitting your application from. in client mode, only the driver runs locally and all other executors run on different nodes on the cluster.
Driver program: This is your bundled jar application that drives the entire application. The Driver talks to an instance of Spark Master to submit your job to the cluster.
Driver and Executor Resources (Cores & Memory)
While submitting an application, you can also specify how much memory and cores you wanted to give for driver and executors.
--executor-memory <memory>: Specifies the amount of memory to allocate to each executor in the Spark application. The memory value can be specified in units such asg(gigabytes) orm(megabytes).--executor-coresNumber of CPU cores to use for the executor process.--total-executor-coresThe total number of executor cores to use.--driver-memoyMemory to be used by the Spark driver.--driver-coresCPU cores to be used by the Spark driver.--num-executorsThe total number of executors to use.
./bin/spark-submit \
--master yarn \
--deploy-mode cluster \
--driver-memory 8g \
--executor-memory 16g \
--executor-cores 2 \
--class org.apache.spark.examples.SparkPi \
/home/npntraining/apache-spark-2.4.0/jars/spark-examples_versionxx.jar 80
Other Options
--files: Use the comma-separated files you wanted to use. Usually, these can be files from your resource folder. Using this option, Spark submits all these files to the cluster.
--verbose: Displays the verbose information. For example, writes all configurations the spark application uses to the log file.
Note: Files specified with --files are uploaded to the cluster.
Example: Below example submits the application to yarn cluster manager by using cluster deployment mode and with 16g driver memory, 32g, and 4 cores for each executor.
./bin/spark-submit \
--verbose
--master yarn \
--deploy-mode cluster \
--driver-memory 16g \
--executor-memory 32g \
--executor-cores 4 \
--files /path/log4j.properties,/path/file2.conf,/path/file3.json
--class org.apache.spark.examples.SparkPi \
/home/npntraining/apache-spark-2.4.0/jars/spark-examples_versionxx.jar 80
Spark Submit Configurations
Spark submit supports several configurations using --config, these configurations are used to specify Application configurations, shuffle parameters, runtime configurations.
| Configuration Key | Description |
| spark.sql.shuffle.partitions | Number of partitions to create for wider shuffle transformations (joins and aggregations). |
| spark.executor.memoryOverhead | The amount of additional memory to be allocated per executor process in cluster mode, it is typically memory for JVM overheads. (Not supported for PySpark) |
| spark.serializer | org.apache.spark.serializer.<br>JavaSerializer (default) |
org.apache.spark.serializer.KryoSerializer |
| spark.sql.files.maxPartitionBytes | The maximum number of bytes to be used for every partition when reading files. Default 128MB. |
| spark.dynamicAllocation.enabled | Specifies whether to dynamically increase or decrease the number of executors based on the workload. Default true. |
| spark.dynamicAllocation
.minExecutors | A minimum number of executors to use when dynamic allocation is enabled. |
| spark.dynamicAllocation
.maxExecutors | A maximum number of executors to use when dynamic allocation is enabled. |
| spark.executor.extraJavaOptions | Specify JVM options (see example below) |
./bin/spark2-submit \
--master yarn \
--deploy-mode cluster \
--conf "spark.sql.shuffle.partitions=20000" \
--conf "spark.executor.memoryOverhead=5244" \
--conf "spark.memory.fraction=0.8" \
--conf "spark.memory.storageFraction=0.2" \
--conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" \
--conf "spark.sql.files.maxPartitionBytes=168435456" \
--conf "spark.dynamicAllocation.minExecutors=1" \
--conf "spark.dynamicAllocation.maxExecutors=200" \
--conf "spark.dynamicAllocation.enabled=true" \
--conf "spark.executor.extraJavaOptions=-XX:+PrintGCDetails -XX:+PrintGCTimeStamps" \
--files /path/log4j.properties,/path/file2.conf,/path/file3.json \
--class org.apache.spark.examples.SparkPi \
/home/npntraining/apache-spark-2.4.0/jars/spark-examples_versionxx.jar 80
Reference: http://spark.apache.org/docs/latest/submitting-applications.html
If you like my work connect with me
I share tips, tricks and insights on #softwareengineering, #dataengineering #cloud #ml on LinkedIn.
Do you want to connect with me I have started mentoring others for career and interviews at 𝐭𝐨𝐩𝐦𝐚𝐭𝐞.𝐢𝐨/𝐧𝐚𝐯𝐞𝐞𝐧𝐩𝐧
About the author
I am a Tech Enthusiast having 13+ years of experience in 𝐈𝐓 as a 𝐂𝐨𝐧𝐬𝐮𝐥𝐭𝐚𝐧𝐭, 𝐂𝐨𝐫𝐩𝐨𝐫𝐚𝐭𝐞 𝐓𝐫𝐚𝐢𝐧𝐞𝐫, 𝐌𝐞𝐧𝐭𝐨𝐫, with 12+ years in training and mentoring in 𝐒𝐨𝐟𝐭𝐰𝐚𝐫𝐞 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐓𝐞𝐬𝐭 𝐀𝐮𝐭𝐨𝐦𝐚𝐭𝐢𝐨𝐧 𝐚𝐧𝐝 𝐃𝐚𝐭𝐚 𝐒𝐜𝐢𝐞𝐧𝐜𝐞. I have 𝒕𝒓𝒂𝒊𝒏𝒆𝒅 𝒎𝒐𝒓𝒆 𝒕𝒉𝒂𝒏 10,000+ 𝑰𝑻 𝑷𝒓𝒐𝒇𝒆𝒔𝒔𝒊𝒐𝒏𝒂𝒍𝒔 and 𝒄𝒐𝒏𝒅𝒖𝒄𝒕𝒆𝒅 𝒎𝒐𝒓𝒆 𝒕𝒉𝒂𝒏 500+ 𝒕𝒓𝒂𝒊𝒏𝒊𝒏𝒈 𝒔𝒆𝒔𝒔𝒊𝒐𝒏𝒔 in the areas of 𝐒𝐨𝐟𝐭𝐰𝐚𝐫𝐞 𝐃𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭, 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐂𝐥𝐨𝐮𝐝, 𝐃𝐚𝐭𝐚 𝐀𝐧𝐚𝐥𝐲𝐬𝐢𝐬, 𝐃𝐚𝐭𝐚 𝐕𝐢𝐬𝐮𝐚𝐥𝐢𝐳𝐚𝐭𝐢𝐨𝐧𝐬, 𝐀𝐫𝐭𝐢𝐟𝐢𝐜𝐢𝐚𝐥 𝐈𝐧𝐭𝐞𝐥𝐥𝐢𝐠𝐞𝐧𝐜𝐞 𝐚𝐧𝐝 𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠. I am interested in 𝐰𝐫𝐢𝐭𝐢𝐧𝐠 𝐛𝐥𝐨𝐠𝐬, 𝐬𝐡𝐚𝐫𝐢𝐧𝐠 𝐭𝐞𝐜𝐡𝐧𝐢𝐜𝐚𝐥 𝐤𝐧𝐨𝐰𝐥𝐞𝐝𝐠𝐞, 𝐬𝐨𝐥𝐯𝐢𝐧𝐠 𝐭𝐞𝐜𝐡𝐧𝐢𝐜𝐚𝐥 𝐢𝐬𝐬𝐮𝐞𝐬, 𝐫𝐞𝐚𝐝𝐢𝐧𝐠 𝐚𝐧𝐝 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠 new subjects.
Comments
Join the discussionNo comments yet. Be the first to comment.


