Configure High Availability for the Spark Master so that the master does not become the single point of failure. By using ZooKeeper to provide leader election and some state storage, you can launch multiple masters in your cluster that are connected to the same ZooKeeper instance. Zookeeper elects one master to be the “leader” and the others remain in standby mode. If the leader goes down, Zookeeper elects another master, recovers the old master’s state, and resumes scheduling.
spark-env.shwith the appropriate ZooKeeper information for the cluster:
Restart Spark Master and Spark HistoryServer services:
On the master node, restart the Spark slaves as the map user: