Configure Livy with Spark Modes

This topic describes how to configure Livy with different Spark modes.

Use these steps to configure Livy:
  1. Modify the livy.conf file (/opt/mapr/livy/livy-<version>/conf/livy.conf):
    1. If Spark jobs run in local mode, set the livy.spark.master property:
      ...
      # What spark master Livy sessions should use.
      livy.spark.master = local[*]
      ...
    2. If Spark jobs run in YARN mode, set the livy.spark.master and livy.spark.deployMode properties (client or cluster). For example:
      ...
      # What spark master Livy sessions should use.
      livy.spark.master = yarn
      # What spark deploy mode Livy sessions should use.
      livy.spark.deployMode = client
      ...
      or
      ...
      # What spark master Livy sessions should use.
      livy.spark.master = yarn
      # What spark deploy mode Livy sessions should use.
      livy.spark.deployMode = cluster
      ...
    3. If Spark jobs run in Standalone mode, set the livy.spark.master and livy.spar.deployMode properties (client or cluster). For example:
      …
      # What spark master Livy sessions should use. 
      livy.spark.master = spark://node:7077 
      # What spark deploy mode Livy sessions should use. 
      livy.spark.deployMode = client 
      …
      or
      …
      # What spark master Livy sessions should use. 
      livy.spark.master = spark://node:7077 
      # What spark deploy mode Livy sessions should use. 
      livy.spark.deployMode = cluster 
      … 
    4. If Spark jobs run in Mesos mode, set the livy.spark.master property. For example:
      # What spark master Livy sessions should use. 
      livy.spark.master = mesos://<mesos-master-node-ip>:5050  
  2. To you want to use impersonation with Livy, set livy.impersonation.enabled to true in livy.conf. For example:
    # If livy should impersonate the requesting users when creating a new session. 
    livy.impersonation.enabled = true 
  3. To apply the needed changes, restart the Livy service:
    maprcli node services -name livy -action restart -nodes <livy node>