org.apache.hadoop.mapreduce.lib.input

クラス NLineInputFormat

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<K,V>

org.apache.hadoop.mapreduce.lib.input.FileInputFormat<LongWritable,Text>

org.apache.hadoop.mapreduce.lib.input.NLineInputFormat

@InterfaceAudience.Public
 @InterfaceStability.Stable
public class NLineInputFormat
extends FileInputFormat<LongWritable,Text>

NLineInputFormat which splits N lines of input as one split. In many "pleasantly" parallel applications, each process/mapper processes the same input file (s), but with computations are controlled by different parameters.(Referred to as "parameter sweeps"). One way to achieve this, is to specify a set of parameters (one set per line) as input in a control file (which is the input path to the map-reduce application, where as the input dataset is specified via a config variable in JobConf.). The NLineInputFormat can be used in such applications, that splits the input file such that by default, one line is fed as a value to one map task, and key is the offset. i.e. (k,v) is (LongWritable, Text). The location hints will span the whole mapred cluster.

フィールドのサマリー

フィールド

修飾子とタイプ	フィールドと説明
static String	LINES_PER_MAP

クラスから継承されたフィールド org.apache.hadoop.mapreduce.lib.input.FileInputFormat

DEFAULT_LIST_STATUS_NUM_THREADS, INPUT_DIR, INPUT_DIR_RECURSIVE, LIST_STATUS_NUM_THREADS, NUM_INPUT_FILES, PATHFILTER_CLASS, SPLIT_MAXSIZE, SPLIT_MINSIZE

コンストラクタのサマリー

コンストラクタ

コンストラクタと説明
NLineInputFormat()

メソッドのサマリー

修飾子とタイプ	メソッドと説明
protected static FileSplit	createFileSplit(Path fileName, long begin, long length) NLineInputFormat uses LineRecordReader, which always reads (and consumes) at least one character out of its upper split boundary.
RecordReader<LongWritable,Text>	createRecordReader(InputSplit genericSplit, TaskAttemptContext context) Create a record reader for a given split.
static int	getNumLinesPerSplit(JobContext job) Get the number of lines per split
List<InputSplit>	getSplits(JobContext job) Logically splits the set of input files for the job, splits N lines of the input as one split.
static List<FileSplit>	getSplitsForFile(FileStatus status, Configuration conf, int numLinesPerSplit)
static void	setNumLinesPerSplit(Job job, int numLines) Set the number of lines per split

クラスから継承されたメソッド org.apache.hadoop.mapreduce.lib.input.FileInputFormat

addInputPath, addInputPathRecursively, addInputPaths, computeSplitSize, getBlockIndex, getFormatMinSplitSize, getInputDirRecursive, getInputPathFilter, getInputPaths, getMaxSplitSize, getMinSplitSize, isSplitable, listStatus, makeSplit, makeSplit, setInputDirRecursive, setInputPathFilter, setInputPaths, setInputPaths, setMaxInputSplitSize, setMinInputSplitSize

クラスから継承されたメソッド java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

フィールドの詳細

LINES_PER_MAP

public static final String LINES_PER_MAP

関連項目:

定数フィールド値

コンストラクタの詳細

NLineInputFormat

public NLineInputFormat()

メソッドの詳細

createRecordReader

public RecordReader<LongWritable,Text> createRecordReader(InputSplit genericSplit,
                                                          TaskAttemptContext context)
                                                   throws IOException

クラスからコピーされた説明: InputFormat

Create a record reader for a given split. The framework will call RecordReader.initialize(InputSplit, TaskAttemptContext) before the split is used.

定義:

createRecordReader クラス内 InputFormat<LongWritable,Text>

パラメータ:

genericSplit - the split to be read

context - the information about the task

戻り値:

a new record reader

例外:

IOException

getSplits

public List<InputSplit> getSplits(JobContext job)
                           throws IOException

Logically splits the set of input files for the job, splits N lines of the input as one split.

オーバーライド:

getSplits クラス内 FileInputFormat<LongWritable,Text>

パラメータ:

job - the job context

戻り値:

an array of InputSplits for the job.

例外:

IOException

関連項目:

FileInputFormat.getSplits(JobContext)

getSplitsForFile

public static List<FileSplit> getSplitsForFile(FileStatus status,
                                               Configuration conf,
                                               int numLinesPerSplit)
                                        throws IOException

例外:

IOException

createFileSplit

protected static FileSplit createFileSplit(Path fileName,
                                           long begin,
                                           long length)

NLineInputFormat uses LineRecordReader, which always reads (and consumes) at least one character out of its upper split boundary. So to make sure that each mapper gets N lines, we move back the upper split limits of each split by one character here.

パラメータ:

fileName - Path of file

begin - the position of the first byte in the file to process

length - number of bytes in InputSplit

戻り値:

FileSplit

setNumLinesPerSplit

public static void setNumLinesPerSplit(Job job,
                                       int numLines)

Set the number of lines per split

パラメータ:

job - the job to modify

numLines - the number of lines per split

getNumLinesPerSplit

public static int getNumLinesPerSplit(JobContext job)

Get the number of lines per split

パラメータ:

job - the job

戻り値:

the number of lines per split