org.apache.hadoop.mapred

クラス FileOutputCommitter

java.lang.Object

org.apache.hadoop.mapreduce.OutputCommitter

org.apache.hadoop.mapred.OutputCommitter

org.apache.hadoop.mapred.FileOutputCommitter

@InterfaceAudience.Public
 @InterfaceStability.Stable
public class FileOutputCommitter
extends OutputCommitter

An OutputCommitter that commits files specified in job output directory i.e. ${mapreduce.output.fileoutputformat.outputdir}.

フィールドのサマリー

フィールド

修飾子とタイプ	フィールドと説明
static org.apache.commons.logging.Log	LOG
static String	SUCCEEDED_FILE_NAME
static String	TEMP_DIR_NAME Temporary directory name

コンストラクタのサマリー

コンストラクタ

コンストラクタと説明
FileOutputCommitter()

メソッドのサマリー

修飾子とタイプ	メソッドと説明
void	abortJob(JobContext context, int runState) For aborting an unsuccessful job's output.
void	abortTask(TaskAttemptContext context) Discard the task output.
void	cleanupJob(JobContext context) 非推奨です。
void	commitJob(JobContext context) For committing job's output after successful job completion.
void	commitTask(TaskAttemptContext context) To promote the task's temporary output to final output location.
Path	getWorkPath(TaskAttemptContext context, Path outputPath)
boolean	isCommitJobRepeatable(JobContext context) Returns true if an in-progress job commit can be retried.
boolean	isRecoverySupported() 非推奨です。
boolean	isRecoverySupported(JobContext context) Is task output recovery supported for restarting jobs?
boolean	needsTaskCommit(TaskAttemptContext context) Check whether task needs a commit.
void	recoverTask(TaskAttemptContext context) Recover the task output.
void	setupJob(JobContext context) For the framework to setup the job output during initialization.
void	setupTask(TaskAttemptContext context) Sets up output for the task.

クラスから継承されたメソッド org.apache.hadoop.mapred.OutputCommitter

abortJob, abortTask, cleanupJob, commitJob, commitTask, isCommitJobRepeatable, isRecoverySupported, needsTaskCommit, recoverTask, setupJob, setupTask

クラスから継承されたメソッド java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

フィールドの詳細

LOG

public static final org.apache.commons.logging.Log LOG

TEMP_DIR_NAME

public static final String TEMP_DIR_NAME

Temporary directory name

関連項目:

定数フィールド値

SUCCEEDED_FILE_NAME

public static final String SUCCEEDED_FILE_NAME

関連項目:

定数フィールド値

コンストラクタの詳細

FileOutputCommitter

public FileOutputCommitter()

メソッドの詳細

getWorkPath

public Path getWorkPath(TaskAttemptContext context,
                        Path outputPath)
                 throws IOException

例外:

IOException

setupJob

public void setupJob(JobContext context)
              throws IOException

クラスからコピーされた説明: OutputCommitter

For the framework to setup the job output during initialization. This is called from the application master process for the entire job. This will be called multiple times, once per job attempt.

定義:

setupJob クラス内 OutputCommitter

パラメータ:

context - Context of the job whose output is being written.

例外:

IOException - if temporary output could not be created

commitJob

public void commitJob(JobContext context)
               throws IOException

クラスからコピーされた説明: OutputCommitter

For committing job's output after successful job completion. Note that this is invoked for jobs with final runstate as SUCCESSFUL. This is called from the application master process for the entire job. This is guaranteed to only be called once. If it throws an exception the entire job will fail.

オーバーライド:

commitJob クラス内 OutputCommitter

パラメータ:

context - Context of the job whose output is being written.

例外:

IOException

cleanupJob

@Deprecated
public void cleanupJob(JobContext context)
                            throws IOException

非推奨です。

クラスからコピーされた説明: OutputCommitter

For cleaning up the job's output after job completion. This is called from the application master process for the entire job. This may be called multiple times.

オーバーライド:

cleanupJob クラス内 OutputCommitter

パラメータ:

context - Context of the job whose output is being written.

例外:

IOException

abortJob

public void abortJob(JobContext context,
                     int runState)
              throws IOException

クラスからコピーされた説明: OutputCommitter

For aborting an unsuccessful job's output. Note that this is invoked for jobs with final runstate as JobStatus.FAILED or JobStatus.KILLED. This is called from the application master process for the entire job. This may be called multiple times.

オーバーライド:

abortJob クラス内 OutputCommitter

パラメータ:

context - Context of the job whose output is being written.

runState - final runstate of the job

例外:

IOException

setupTask

public void setupTask(TaskAttemptContext context)
               throws IOException

クラスからコピーされた説明: OutputCommitter

Sets up output for the task. This is called from each individual task's process that will output to HDFS, and it is called just for that task. This may be called multiple times for the same task, but for different task attempts.

定義:

setupTask クラス内 OutputCommitter

パラメータ:

context - Context of the task whose output is being written.

例外:

IOException

commitTask

public void commitTask(TaskAttemptContext context)
                throws IOException

クラスからコピーされた説明: OutputCommitter

To promote the task's temporary output to final output location. If OutputCommitter.needsTaskCommit(TaskAttemptContext) returns true and this task is the task that the AM determines finished first, this method is called to commit an individual task's output. This is to mark that tasks output as complete, as OutputCommitter.commitJob(JobContext) will also be called later on if the entire job finished successfully. This is called from a task's process. This may be called multiple times for the same task, but different task attempts. It should be very rare for this to be called multiple times and requires odd networking failures to make this happen. In the future the Hadoop framework may eliminate this race.

定義:

commitTask クラス内 OutputCommitter

パラメータ:

context - Context of the task whose output is being written.

例外:

IOException - if commit is not

abortTask

public void abortTask(TaskAttemptContext context)
               throws IOException

クラスからコピーされた説明: OutputCommitter

Discard the task output. This is called from a task's process to clean up a single task's output that can not yet been committed. This may be called multiple times for the same task, but for different task attempts.

定義:

abortTask クラス内 OutputCommitter

例外:

IOException

needsTaskCommit

public boolean needsTaskCommit(TaskAttemptContext context)
                        throws IOException

クラスからコピーされた説明: OutputCommitter

Check whether task needs a commit. This is called from each individual task's process that will output to HDFS, and it is called just for that task.

定義:

needsTaskCommit クラス内 OutputCommitter

戻り値:

true/false

例外:

IOException

isRecoverySupported

@Deprecated
public boolean isRecoverySupported()

非推奨です。

クラスからコピーされた説明: OutputCommitter

This method implements the new interface by calling the old method. Note that the input types are different between the new and old apis and this is a bridge between the two.

オーバーライド:

isRecoverySupported クラス内 OutputCommitter

戻り値:

true if task output recovery is supported, false otherwise

関連項目:

OutputCommitter.recoverTask(TaskAttemptContext)

isCommitJobRepeatable

public boolean isCommitJobRepeatable(JobContext context)
                              throws IOException

クラスからコピーされた説明: OutputCommitter

Returns true if an in-progress job commit can be retried. If the MR AM is re-run then it will check this value to determine if it can retry an in-progress commit that was started by a previous version. Note that in rare scenarios, the previous AM version might still be running at that time, due to system anomalies. Hence if this method returns true then the retry commit operation should be able to run concurrently with the previous operation. If repeatable job commit is supported, job restart can tolerate previous AM failures during job commit. By default, it is not supported. Extended classes (like: FileOutputCommitter) should explicitly override it if provide support.

オーバーライド:

isCommitJobRepeatable クラス内 OutputCommitter

パラメータ:

context - Context of the job whose output is being written.

戻り値:

true repeatable job commit is supported, false otherwise

例外:

IOException

isRecoverySupported

public boolean isRecoverySupported(JobContext context)
                            throws IOException

クラスからコピーされた説明: OutputCommitter

Is task output recovery supported for restarting jobs? If task output recovery is supported, job restart can be done more efficiently.

オーバーライド:

isRecoverySupported クラス内 OutputCommitter

パラメータ:

context - Context of the job whose output is being written.

戻り値:

true if task output recovery is supported, false otherwise

例外:

IOException

関連項目:

OutputCommitter.recoverTask(TaskAttemptContext)

recoverTask

public void recoverTask(TaskAttemptContext context)
                 throws IOException

クラスからコピーされた説明: OutputCommitter

Recover the task output. The retry-count for the job will be passed via the MRConstants.APPLICATION_ATTEMPT_ID key in JobContext.getConfiguration() for the OutputCommitter. This is called from the application master process, but it is called individually for each task. If an exception is thrown the task will be attempted again.

オーバーライド:

recoverTask クラス内 OutputCommitter

パラメータ:

context - Context of the task whose output is being recovered

例外:

IOException