org.apache.hadoop.mapreduce.lib.db

クラス DataDrivenDBInputFormat<T extends DBWritable>

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<LongWritable,T>

org.apache.hadoop.mapreduce.lib.db.DBInputFormat<T>

org.apache.hadoop.mapreduce.lib.db.DataDrivenDBInputFormat<T>

すべての実装されたインタフェース:

Configurable

直系の既知のサブクラス:

OracleDataDrivenDBInputFormat

@InterfaceAudience.Public
 @InterfaceStability.Evolving
public class DataDrivenDBInputFormat<T extends DBWritable>
extends DBInputFormat<T>
implements Configurable

A InputFormat that reads input data from an SQL table. Operates like DBInputFormat, but instead of using LIMIT and OFFSET to demarcate splits, it tries to generate WHERE clauses which separate the data into roughly equivalent shards.

フィールドのサマリー

フィールド

修飾子とタイプ	フィールドと説明
static String	SUBSTITUTE_TOKEN If users are providing their own query, the following string is expected to appear in the WHERE clause, which will be substituted with a pair of conditions on the input to allow input splits to parallelise the import.

クラスから継承されたフィールド org.apache.hadoop.mapreduce.lib.db.DBInputFormat

conditions, connection, dbConf, dbProductName, fieldNames, tableName

コンストラクタのサマリー

コンストラクタ

コンストラクタと説明
DataDrivenDBInputFormat()

メソッドのサマリー

修飾子とタイプ	メソッドと説明
protected RecordReader<LongWritable,T>	createDBRecordReader(org.apache.hadoop.mapreduce.lib.db.DBInputFormat.DBInputSplit split, Configuration conf)
protected String	getBoundingValsQuery()
List<InputSplit>	getSplits(JobContext job) Logically split the set of input files for the job.
protected DBSplitter	getSplitter(int sqlDataType)
static void	setBoundingQuery(Configuration conf, String query) Set the user-defined bounding query to use with a user-defined query.
static void	setInput(Job job, Class<? extends DBWritable> inputClass, String inputQuery, String inputBoundingQuery) setInput() takes a custom query and a separate "bounding query" to use instead of the custom "count query" used by DBInputFormat.
static void	setInput(Job job, Class<? extends DBWritable> inputClass, String tableName, String conditions, String splitBy, String... fieldNames) Note that the "orderBy" column is called the "splitBy" in this version.

クラスから継承されたメソッド org.apache.hadoop.mapreduce.lib.db.DBInputFormat

closeConnection, createConnection, createRecordReader, getConf, getConnection, getCountQuery, getDBConf, getDBProductName, setConf

クラスから継承されたメソッド java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

インタフェースから継承されたメソッド org.apache.hadoop.conf.Configurable

getConf, setConf

フィールドの詳細

SUBSTITUTE_TOKEN

public static final String SUBSTITUTE_TOKEN

If users are providing their own query, the following string is expected to appear in the WHERE clause, which will be substituted with a pair of conditions on the input to allow input splits to parallelise the import.

関連項目:

定数フィールド値

コンストラクタの詳細

DataDrivenDBInputFormat

public DataDrivenDBInputFormat()

メソッドの詳細

getSplitter

protected DBSplitter getSplitter(int sqlDataType)

戻り値:

the DBSplitter implementation to use to divide the table/query into InputSplits.

getSplits

public List<InputSplit> getSplits(JobContext job)
                           throws IOException

Logically split the set of input files for the job.

Each InputSplit is then assigned to an individual Mapper for processing.

Note: The split is a logical split of the inputs and the input files are not physically split into chunks. For e.g. a split could be <input-file-path, start, offset> tuple. The InputFormat also creates the RecordReader to read the InputSplit.

オーバーライド:

getSplits クラス内 DBInputFormat<T extends DBWritable>

パラメータ:

job - job configuration.

戻り値:

an array of InputSplits for the job.

例外:

IOException

getBoundingValsQuery

protected String getBoundingValsQuery()

戻り値:

a query which returns the minimum and maximum values for the order-by column. The min value should be in the first column, and the max value should be in the second column of the results.

setBoundingQuery

public static void setBoundingQuery(Configuration conf,
                                    String query)

Set the user-defined bounding query to use with a user-defined query. This *must* include the substring "$CONDITIONS" (DataDrivenDBInputFormat.SUBSTITUTE_TOKEN) inside the WHERE clause, so that DataDrivenDBInputFormat knows where to insert split clauses. e.g., "SELECT foo FROM mytable WHERE $CONDITIONS" This will be expanded to something like: SELECT foo FROM mytable WHERE (id > 100) AND (id < 250) inside each split.

createDBRecordReader

protected RecordReader<LongWritable,T> createDBRecordReader(org.apache.hadoop.mapreduce.lib.db.DBInputFormat.DBInputSplit split,
                                                            Configuration conf)
                                                     throws IOException

オーバーライド:

createDBRecordReader クラス内 DBInputFormat<T extends DBWritable>

例外:

IOException

setInput

public static void setInput(Job job,
                            Class<? extends DBWritable> inputClass,
                            String tableName,
                            String conditions,
                            String splitBy,
                            String... fieldNames)

Note that the "orderBy" column is called the "splitBy" in this version. We reuse the same field, but it's not strictly ordering it -- just partitioning the results.

setInput

public static void setInput(Job job,
                            Class<? extends DBWritable> inputClass,
                            String inputQuery,
                            String inputBoundingQuery)

setInput() takes a custom query and a separate "bounding query" to use instead of the custom "count query" used by DBInputFormat.