org.apache.hadoop.tools

クラス DistCp

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.tools.DistCp

すべての実装されたインタフェース:

Configurable, Tool

@InterfaceAudience.Public
 @InterfaceStability.Evolving
public class DistCp
extends Configured
implements Tool

DistCp is the main driver-class for DistCpV2. For command-line use, DistCp::main() orchestrates the parsing of command-line parameters and the launch of the DistCp job. For programmatic use, a DistCp object can be constructed by specifying options (in a DistCpOptions object), and DistCp::execute() may be used to launch the copy-job. DistCp may alternatively be sub-classed to fine-tune behaviour.

コンストラクタのサマリー

コンストラクタ

コンストラクタと説明
DistCp(Configuration configuration, org.apache.hadoop.tools.DistCpOptions inputOptions) Public Constructor.

メソッドのサマリー

修飾子とタイプ	メソッドと説明
Job	createAndSubmitJob() Create and submit the mapreduce job.
protected Path	createInputFileListing(Job job) Create input listing by invoking an appropriate copy listing implementation.
Job	execute() Implements the core-execution.
protected Path	getFileListingPath() Get default name of the copy listing file.
static void	main(String[] argv) Main function of the DistCp program.
int	run(String[] argv) Implementation of Tool::run().
void	waitForJobCompletion(Job job) Wait for the given job to complete.

クラスから継承されたメソッド org.apache.hadoop.conf.Configured

getConf, setConf

クラスから継承されたメソッド java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

インタフェースから継承されたメソッド org.apache.hadoop.conf.Configurable

getConf, setConf

コンストラクタの詳細

DistCp

public DistCp(Configuration configuration,
              org.apache.hadoop.tools.DistCpOptions inputOptions)
       throws Exception

Public Constructor. Creates DistCp object with specified input-parameters. (E.g. source-paths, target-location, etc.)

パラメータ:

inputOptions - Options (indicating source-paths, target-location.)

configuration - The Hadoop configuration against which the Copy-mapper must run.

例外:

Exception

メソッドの詳細

run

public int run(String[] argv)

Implementation of Tool::run(). Orchestrates the copy of source file(s) to target location, by: 1. Creating a list of files to be copied to target. 2. Launching a Map-only job to copy the files. (Delegates to execute().)

定義:

run インタフェース内 Tool

パラメータ:

argv - List of arguments passed to DistCp, from the ToolRunner.

戻り値:

On success, it returns 0. Else, -1.

execute

public Job execute()
            throws Exception

Implements the core-execution. Creates the file-list for copy, and launches the Hadoop-job, to do the copy.

戻り値:

Job handle

例外:

Exception

createAndSubmitJob

public Job createAndSubmitJob()
                       throws Exception

Create and submit the mapreduce job.

戻り値:

The mapreduce job object that has been submitted

例外:

Exception

waitForJobCompletion

public void waitForJobCompletion(Job job)
                          throws Exception

Wait for the given job to complete.

パラメータ:

job - the given mapreduce job that has already been submitted

例外:

Exception

createInputFileListing

protected Path createInputFileListing(Job job)
                               throws IOException

Create input listing by invoking an appropriate copy listing implementation. Also add delegation tokens for each path to job's credential store

パラメータ:

job - - Handle to job

戻り値:

Returns the path where the copy listing is created

例外:

IOException - - If any

getFileListingPath

protected Path getFileListingPath()
                           throws IOException

Get default name of the copy listing file. Use the meta folder to create the copy listing file

戻り値:

- Path where the copy listing file has to be saved

例外:

IOException - - Exception if any

main

public static void main(String[] argv)

Main function of the DistCp program. Parses the input arguments (via OptionsParser), and invokes the DistCp::run() method, via the ToolRunner.

パラメータ:

argv - Command-line arguments sent to DistCp.