public class MRPigStatsUtil extends PigStatsUtil
| Modifier and Type | Field and Description |
|---|---|
static String |
FS_COUNTER_GROUP |
static String |
TASK_COUNTER_GROUP |
FILE_BYTES_READ, FILE_BYTES_WRITTEN, HDFS_BYTES_READ, HDFS_BYTES_WRITTEN, MAP_INPUT_RECORDS, MAP_OUTPUT_RECORDS, MULTI_INPUTS_COUNTER_GROUP, MULTI_INPUTS_RECORD_COUNTER, MULTI_STORE_COUNTER_GROUP, MULTI_STORE_RECORD_COUNTER, REDUCE_INPUT_RECORDS, REDUCE_OUTPUT_RECORDS| Constructor and Description |
|---|
MRPigStatsUtil() |
| Modifier and Type | Method and Description |
|---|---|
static void |
accumulateStats(org.apache.hadoop.mapred.jobcontrol.JobControl jc)
Updates the statistics after a patch of jobs is done
|
static JobStats |
addJobStats(org.apache.hadoop.mapred.jobcontrol.Job job)
Add stats for a new Job, which doesn't yet need to be completed.
|
static MRJobStats |
addNativeJobStats(PigStats ps,
NativeMapReduceOper mr,
boolean success) |
static MRJobStats |
addNativeJobStats(PigStats ps,
NativeMapReduceOper mr,
boolean success,
Exception e) |
static void |
displayStatistics()
Logs the statistics in the Pig log file at INFO level
|
static long |
getMultiStoreCount(org.apache.hadoop.mapred.jobcontrol.Job job,
org.apache.hadoop.mapred.JobClient jobClient,
String counterName)
Returns the count for the given counter name in the counter group
'MultiStoreCounters'
|
static void |
setBackendException(org.apache.hadoop.mapred.jobcontrol.Job job,
Exception e) |
static void |
startCollection(PigContext pc,
org.apache.hadoop.mapred.JobClient client,
JobControlCompiler jcc,
MROperPlan plan)
Starts collecting statistics for the given MR plan
|
static void |
stopCollection(boolean display)
Stops collecting statistics for a MR plan
|
static void |
updateJobMroMap(Map<org.apache.hadoop.mapred.jobcontrol.Job,MapReduceOper> jobMroMap)
Updates the
PigStats.JobGraph of the PigStats. |
getEmptyPigStats, getMultiInputsCounterName, getMultiStoreCounterName, getPigStats, getShortName, isTempFile, setErrorCode, setErrorMessage, setErrorThrowable, setStatsMappublic static final String TASK_COUNTER_GROUP
public static final String FS_COUNTER_GROUP
public static long getMultiStoreCount(org.apache.hadoop.mapred.jobcontrol.Job job,
org.apache.hadoop.mapred.JobClient jobClient,
String counterName)
job - the MR jobjobClient - the Hadoop job clientcounterName - the counter namepublic static void startCollection(PigContext pc, org.apache.hadoop.mapred.JobClient client, JobControlCompiler jcc, MROperPlan plan)
pc - the Pig contextclient - the Hadoop job clientjcc - the job compilerplan - the MR planpublic static void stopCollection(boolean display)
display - if true, log collected statistics in the Pig log
file at INFO levelpublic static JobStats addJobStats(org.apache.hadoop.mapred.jobcontrol.Job job)
job - the job being runpublic static void displayStatistics()
public static void updateJobMroMap(Map<org.apache.hadoop.mapred.jobcontrol.Job,MapReduceOper> jobMroMap)
PigStats.JobGraph of the PigStats. The initial
PigStats.JobGraph is created without job ids using MROperPlan,
before any job is submitted for execution. The PigStats.JobGraph then
is updated with job ids after jobs are executed.jobMroMap - the map that maps Jobs to MapReduceOperspublic static void accumulateStats(org.apache.hadoop.mapred.jobcontrol.JobControl jc)
jc - the job control@InterfaceAudience.Private public static void setBackendException(org.apache.hadoop.mapred.jobcontrol.Job job, Exception e)
public static MRJobStats addNativeJobStats(PigStats ps, NativeMapReduceOper mr, boolean success)
public static MRJobStats addNativeJobStats(PigStats ps, NativeMapReduceOper mr, boolean success, Exception e)
Copyright © 2007-2017 The Apache Software Foundation