How to choose number of executors required for our cluster ?

Let's consider it is a "20 Node Cluster"

Each Node (30 Cores , 128GB RAM

• For good throughput let's assign 5 CORES per EXECUTOR

  --executor-cores = 5

• Should leave 1 core for Background activity (Hadoop/Yarn daemons) Number of cores available = 30 -1 = 29

• Total available of cores in cluster = 29*20 =580

• Number of executors in cluster
  \= total cores per cluster / no.of cores per executors
  \= 580/5 = 116

• Should leave 1 executor for Yarn (Application Manager)
  Now available executors in cluster = 116 - 1 = 115

• Number of executors per node = 116/20 = 6 (approx)

• Memory per executor = 128GB /6 = 21 GB
  7% of 21 GB approximately 1.5 GB will be allocated to heap overhead.

• Now actual --executor-memory = 21GB - 2GB = 19 GB per executor.

  So, recommended
  116 Executors (19 GB Memory, 5 cores) for "20 Node Cluster".