Creating the Custom Resources for Spark Applications

When you create a custom CR or modify a sample CR, specify values for the following properties in the CR file for your Spark application:
Property Type Description
name string Specified in the metadata section of the CR. This is the name of the Spark job. The name cannot be longer than 63 characters.
sparkConf object Specifies the settings for the Spark application. For more information, see Settings for Spark Configuration.
hadoopConf object Specifies the settings for Hadoop configuration used in the Spark application. Refer to Specifying Hadoop Configuration for the properties to set in this object.
type string Specifies the type of Spark job. Use one of the following values:
  • Scala
  • Java
  • R
  • Python
mode string Specifies the deployment mode. The value should be cluster. The client mode is not supported while using the operator, and non-operator workflows are currently not supported.
image string Specifies the image to use for the pod container. The default value is spark-2.4.4:<TIMESTAMP>.
imagePullPolicy string Specifies when to pull in the latest image. Use one of the following values:
  • Always
  • IfNotPresent
  • Never
The default value is Always.
mainClass string Specifies the main class of the application.
mainApplicationFile string Specifies the path to the JAR file for the Spark application.
restartPolicy string Specifies when to restart Spark pods. The value can be:
  • Never
  • OnFailure
  • Always

The default value is Never. For more information, see Configuring Automatic Application Restart and Failure Handling.

arguments string Specifies the argument to run the Spark job.
imagePullSecrets string Specifies the name of the secret that contains access information for the image repository.
volumes object Specifies the name of the temporary Kubernetes volume used by the driver and executor.
driver object Specifies the resources for the Spark driver. For more information, see Settings for the Spark Driver and Executors.
executor object Specifies the resources for the Spark executor. For more information, see Settings for the Spark Driver and Executors.
sparkVersion string Specifies the version of Spark. The value must be 2.4.4.
pythonVersion string Specifies the version of Python. This is required only if type is Python. Use one of the following values:
  • 2
  • 3