job table

Retrieves detailed information about the jobs running on the cluster including histograms and line charts for job metrics.

A Hadoop job sets the rules that the JobTracker service uses to break an input data set into discrete tasks and assign those tasks to individual nodes. Use the job table command to retrieve job metrics about the jobs running on your cluster. The job metric data includes the number of task attempts for jobs on the cluster, job duration, job computing resource use (CPU and memory), and job data throughput (both records and bytes per second). The metrics data can be formatted for histogram display or line chart display. In order to issue the job table command, the mapr-metrics package must be installed on all the nodes where webserver and jobtracker are configured to run.

Syntax

CLI
N/A
REST
http[s]://<host>:<port>/rest/job/table?output=terse&filter=string&chart=chart_type&columns=list_of_columns&scale=scale_type<parameters> 

Parameters

Parameter

Description

filter

Filters results to match the value of a specified string.

chart

Chart type to use: line for a line chart, bar for a histogram.

columns

Comma-separated list of column names to return.

bincount

Number of histogram bins.

scale

Scale to use for the histogram. Specify linear for a linear scale and log for a logarithmic scale.

Table 1. Column Names

Parameter

Description

Notes

jmadavg

Job Average Map Attempt Duration

jradavg

Job Average Reduce Attempt Duration

jtadavg

Job Average Task Duration

Filter Only

jcmtct

Job Complete Map Task Count

Filter Only

jcrtct

Job Complete Reduce Task Count

Filter Only

jctct

Job Complete Task Count

Filter Only

jccpu

Job Cumulative CPU

jcmem

Job Cumulative Physical Memory

jcpu

Job Current CPU

Filter Only

jmem

Job Current Memory

Filter Only

jfmtact

Job Failed Map Task Attempt Count

jfmtct

Job Failed Map Task Count

jfrtact

Job Failed Reduce Task Attempt Count

jfrtct

Job Failed Reduce Task Count

jftact

Job Failed Task Attempt Count

Filter Only

jftct

Job Failed Task Count

Filter Only

jmibps

Job Map Input Bytes Rate

Per-second throughput rate

jmirps

Job Map Input Records Rate

Per-second throughput rate

jmobps

Job Map Output Bytes Rate

Per-second throughput rate

jmorps

Job Map Output Records Rate

Per-second throughput rate

jmtact

Job Mask Task Attempt Count

Per-second throughput rate

jmtct

Job Map Task Count

jmadmax

Job Maximum Map Attempt Duration

jradmax

Job Maximum Reduce Attempt Duration

jtadmax

Job Maximum Task Duration

Filter Only

jribps

Job Reduce Input Bytes Rate

jrirps

Job Reduce Input Records Rate

jrobps

Job Reduce Output Bytes Rate

jrorps

Job Reduce Output Records Rate

jrsbps

Job Reduce Shuffle Bytes Rate

jrtact

Job Reduce Task Attempt Count

jrtct

Job Reduce Task Count

jrumtct

Job Running Map Task Count

Filter Only

jrurtct

Job Running Reduce Task Count

Filter Only

jrutct

Job Running Task Count

Filter Only

jtact

Job Task Attempt Count

Filter Only

jtct

Job Total Task Count

Filter Only

jd

Job Duration

Histogram Only

jid

Job ID

Filter Only

jn

Job Name

Filter Only

ju

Job User

Filter Only

js

Job Status

Filter Only

jmcmem

Job Map Cumulative Memory Bytes

Histogram Only

jrcmem

Job Reduce Cumulative Memory Bytes

Histogram Only

jpri

Job Priority

Filter Only

jmpro

Job Map Progress

Filter Only

jrpro

Job Reduce Progress

Filter Only

jmtst

Map Tasks Start Time

jmtft

Map Tasks Finish Time

jrtst

Reduce Tasks Start Time

jrtft

Reduce Tasks Finish Time

jsbt

Job Submit Time

Filter Only

jst

Job Start Time

Filter Only

jft

Job Finish Time

Filter Only

jrrsw

Job Reduce Reserve Slot Wait

jmrsw

Job Map Reserve Slot Wait

jdlmt

Job Data-local Map Tasks

jolmt

Job Non-local Map Tasks

jrlmt

Job Rack-local Map Tasks

jrtd

Job Reduce Tasks Duration

jmtd

Job Map Tasks Duration

jmmbr

Job MapR-FS Map Bytes Read

jrmbr

Job MapR-FS Reduce Bytes Read

jtmbr

Job MapR-FS Total Bytes Read

jmmbw

Job MapR-FS Map Bytes Written

jrmbw

Job MapR-FS Reduce Bytes Written

jtmbw

Job MapR-FS Total Bytes Written

jmfbw

Job Map File Bytes Written

jrfbw

Job Reduce File Bytes Written

jtfbw

Job Total File Bytes Written

jmir

Cumulative Job Map Input Records

jrir

Cumulative Job Reduce Input Records

jcir

Cumulative Job Combine Input Records

jmor

Cumulative Job Map Output Records

jror

Cumulative Job Reduce Output Records

jcor

Job Combine Output Records

jrsb

Cumulative Job Reduce Shuffle Bytes

jmsr

Job Map Spilled Records

jrsr

Job Reduce Spilled Records

jtsr

Job Total Spilled Records

jmob

Cumulative Job Map Output Bytes

jmib

Cumulative Job Map Input Bytes

jmcpu

Job Map CPU

jrcpu

Job Reduce CPU

jmsrb

Job Map Split Raw Bytes

jrsrb

Job Reduce Split Raw Bytes

jsrb

Job Split Raw Bytes

jrig

Job Reduce Input Groups

jvmb

Virtual Memory Bytes

jmvmb

Job Map Virtual Memory Bytes

jrvmb

Job Reduce Virtual Memory Bytes

jgct

Job Total GC Time

jmgct

Job Map GC Time

jrgct

Job Reduce GC Time

Examples

Retrieve a Histogram:

REST
https://centos-n1.user.prv:8443/rest/job/table?columns=jn,jid,jd&filter=[jn==*select*]and[jst>1404246318000]
CURL
curl -d @json https://r1n1.sj.us:8443/rest/job/table 

REST Example Output

{"total":4,"status":"OK","data":[["select disks from loadtest(Stage-1)","job_201406121453_0017","4088"],["select id from usertable(Stage-1)","job_201406121453_0016","1"],["select disks from loadtest(Stage-1)","job_201406121453_0015","3865"],["select disks from loadtest(Stage-1)","job_201406121453_0014","1481"]],"columns":["jn","jid","jd"]}
CURL Example Request and Response

In the curl example above, the json file contains a URL-encoded version of the information in the Request section below.

Request

GENERAL_PARAMS:
{
        [chart: "bar"|"line"],
        columns: <comma-sep list of column terse names>,
        [filter: "[<terse_field>{operator}<value>]and[...]",]
        [output: terse,]
        [start: int,]
        [limit: int]
}

REQUEST_PARAMS_HISTOGRAM:
{
        chart:bar
        columns:jd
        filter: <anything>
}

REQUEST_PARAMS_LINE:
{
        chart:line,
        columns:jmem,
        filter:  NOT PARSED, UNUSED IN BACKEND
}

REQUEST_PARAMS_GRID:
{
        columns:jid,jn,js,jd
        filter:<any real filter expression>
        output:terse,
        start:0,
        limit:50
}

Response

RESPONSE_SUCCESS_HISTOGRAM:
{
        "status" : "OK",
        "total" : 15,
        "columns" : ["jd"],
        "binlabels" : ["0-5s","5-10s","10-30s","30-60s","60-90s","90s-2m","2m-5m","5m-10m","10m-30m","30m-1h","1h-2h","2h-6h","6h-12h","12h-24h",">24h"],
        "binranges" : [
                [0,5000],
                [5000,10000],
                [10000,30000],
                [30000,60000],
                [60000,90000],
                [90000,120000],
                [120000,300000],
                [300000,600000],
                [600000,1800000],
                [1800000,3600000],
                [3600000,7200000],
                [7200000,21600000],
                [21600000,43200000],
                [43200000,86400000],
                [86400000]
        ],
        "data" : [33,919,1,133,9820,972,39,2,44,80,11,93,31,0,0]
}

RESPONSE_SUCCESS_GRID:
{
        "status": "OK",
        "total"       : 50,
        "columns" : ["jid","jn","ju","jg","js","jcpu","jccpu","jmem","jcmem","jpri","jmpro","jrpro","jsbt","jst","jft","jd","jmtct","jfmtact","jrtct","jmtact","jrtact","jtact","jfrtact","jftact","jfmtct","jfrtct","jftct","jtct","jrumtct","jrurtct","jrutct","jctct","jcmtct","jcrtct","jmirps","jmorps","jmibps","jmobps","jrirps","jrorps","jribps","jrobps","jtadavg","jmadavg","jmadmax","jtadmax","jradavg","jradmax"],
        "data"        : [
                ["job_201210216041_7311","Billboard Top 10","heman","jobberwockies","PREP",69,9106628,857124,181087410,"LOW",30,48,1309992275580,1316685654403,1324183149687,7497495284,72489,25227,6171223,95464,6171184,6266648,-38,25189,13115,-4,13111,6243712,5329,4,6243712,6225268,54045,6171223,403,128,570,137,172,957,490,179,246335,367645,1758151,1758151,125024,514028],
                ["job_201129309372_8897","Super Big","srivas","jobberwockies","KILLED",59,3125830,2895159,230270693,"LOW",91,1,1313111819653,1323504739893,1326859602015,3354862122,8705980,3774739,7691269,12515000,16273631,28788631,8196156,11970895,2706470,4365698,7072168,16397249,215570,35,16397249,9109476,5783940,3325536,707,509,345,463,429,93,88,752,406336,553455,3392429,3392429,259216,511285],
                ["job_201165490737_7144","Trending Human Interaction","mickey","jobberwockies","PREP",100,1304791,504092,728635524,"VERY_LOW",57,90,1301684627596,1331548331890,1331592957521,44625631,7389503,3770494,5433308,11011495,15048822,26060317,9769362,13539856,2544349,4315172,6859521,12822811,21932,327,12822811,5941031,4823222,1117809,739,654,561,426,925,23,420,597,292024,470314,1854688,1854688,113733,566672],
                ["job_201152533959_6159","Star Search","darth","fbi","FAILED",82,7151113,2839682,490527441,"NORMAL",51,61,1305367042224,1325920952001,1327496965896,1576013895,8939964,2041524,4965024,10795895,8786681,19582576,3924842,5966366,1130482,3422544,4553026,13904988,833761,2,13904988,8518199,6975721,1542478,665,916,10,34,393,901,608,916,186814,331708,2500504,2500504,41920,251453]
        ]
}

RESPONSE_SUCCESS_LINE:
{
        "status" : "OK",
        "total" : 22,
        "columns" : ["jcmem"],
        "data" : [
                [1329891055016,0],
                [1329891060016,8],
                [1329891065016,16],
                [1329891070016,1024],
                [1329891075016,2310],
                [1329891080016,3243],
                [1329891085016,4345],
                [1329891090016,7345],
                [1329891095016,7657],
                [1329891100016,8758],
                [1329891105016,9466],
                [1329891110016,10345],
                [1329891115016,235030],
                [1329891120016,235897],
                [1329891125016,287290],
                [1329891130016,298390],
                [1329891135016,301355],
                [1329891140016,302984],
                [1329891145016,303985],
                [1329891150016,304403],
                [1329891155016,503030],
                [1329891160016,983038]
        ]
}