Monitoring the Spark Application in the Compute Space

The CSpace Terminal pod, which is a utility pod for interacting with CSpace applications using kubectl, contains sparkctl, which can be used to monitor and track the Spark jobs that were submitted. You can use sparkctl to create, list, check the status of, get logs for, and delete Spark applications in the CSpace namespace. For more information about using sparkctl, see sparkctl.

Commands in the CSpace Terminal run as the terminal service account rather than an individual Spark user. For security purposes, the terminal service account has limited permissions. Unless you specify the CSpace in an -n <cspace> argument to sparkctl, you will not be able to see any running jobs. For example, this command shows the status of Spark jobs in the finance CSpace namespace.
$ sparkctl list -n finance
+---------------+------------+----------------+-----------------+
|     NAME      |   STATE    | SUBMISSION AGE | TERMINATION AGE |
+---------------+------------+----------------+-----------------+
| spark-pi-job  | COMPLETED  | 1m             | 9s              |
+---------------+------------+----------------+-----------------+