org.apache.hadoop.fs.azure

クラス NativeAzureFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.azure.NativeAzureFileSystem

すべての実装されたインタフェース:

Closeable, AutoCloseable, Configurable

@InterfaceAudience.Public
 @InterfaceStability.Stable
public class NativeAzureFileSystem
extends FileSystem

A FileSystem for reading and writing files stored on Windows Azure. This implementation is blob-based and stores files on Azure in their native form so they can be read by other Azure tools.

フィールドのサマリー

フィールド

修飾子とタイプ	フィールドと説明
static String	APPEND_SUPPORT_ENABLE_PROPERTY_NAME
static String	AZURE_DELETE_THREADS The configuration property to set number of threads to be used for delete operation.
static String	AZURE_RENAME_THREADS The configuration property to set number of threads to be used for rename operation.
static int	DEFAULT_AZURE_DELETE_THREADS The default number of threads to be used for delete operation.
static int	DEFAULT_AZURE_RENAME_THREADS The default number of threads to be used for rename operation.
static org.slf4j.Logger	LOG
static String	SKIP_AZURE_METRICS_PROPERTY_NAME

クラスから継承されたフィールド org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX

コンストラクタのサマリー

コンストラクタ

コンストラクタと説明
NativeAzureFileSystem()
NativeAzureFileSystem(org.apache.hadoop.fs.azure.NativeFileSystemStore store)

メソッドのサマリー

修飾子とタイプ	メソッドと説明
org.apache.hadoop.fs.azure.SelfRenewingLease	acquireLease(Path path) Get a self-renewing lease on the specified file.
FSDataOutputStream	append(Path f, int bufferSize, Progressable progress) This optional operation is not yet supported.
protected void	checkPath(Path path) Check that a Path belongs to this FileSystem.
void	close() No more filesystem operations are needed.
FSDataOutputStream	create(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) Create an FSDataOutputStream at the indicated Path with write-progress reporting.
FSDataOutputStream	createNonRecursive(Path f, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) Opens an FSDataOutputStream at the indicated Path with write-progress reporting.
FSDataOutputStream	createNonRecursive(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) Opens an FSDataOutputStream at the indicated Path with write-progress reporting.
FSDataOutputStream	createNonRecursive(Path f, FsPermission permission, EnumSet<CreateFlag> flags, int bufferSize, short replication, long blockSize, Progressable progress) Opens an FSDataOutputStream at the indicated Path with write-progress reporting.
boolean	delete(Path path) 非推奨です。
boolean	delete(Path f, boolean recursive) Delete a file.
boolean	delete(Path f, boolean recursive, boolean skipParentFolderLastModifidedTimeUpdate) Delete the specified file or folder.
void	deleteFilesWithDanglingTempData(Path root) Looks under the given root path for any blob that are left "dangling", meaning that they are place-holder blobs that we created while we upload the data to a temporary blob, but for some reason we crashed in the middle of the upload and left them there.
protected void	finalize()
BlockLocation[]	getFileBlockLocations(FileStatus file, long start, long len) Return an array containing hostnames, offset and size of portions of the given file.
FileStatus	getFileStatus(Path f) Return a file status object that represents the path.
AzureFileSystemInstrumentation	getInstrumentation() Gets the metrics source for this file system.
String	getScheme() Return the protocol scheme for the FileSystem.
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore	getStore() For unit test purposes, retrieves the AzureNativeFileSystemStore store backing this file system.
org.apache.hadoop.fs.azure.AzureFileSystemThreadPoolExecutor	getThreadPoolExecutor(int threadCount, String threadNamePrefix, String operation, String key, String config)
URI	getUri() Returns a URI whose scheme and authority identify this FileSystem.
Path	getWorkingDirectory() Get the current working directory for the given file system
void	initialize(URI uri, Configuration conf) Called after a new FileSystem instance is constructed.
FileStatus[]	listStatus(Path f) Retrieve the status of a given path if it is a file, or of all the contained files if it is a directory.
Path	makeAbsolute(Path path) Get the absolute version of the path (fully qualified).
boolean	mkdirs(Path f, FsPermission permission) Make the given file and all non-existent parents into directories.
boolean	mkdirs(Path f, FsPermission permission, boolean noUmask)
static String	newMetricsSourceName() Creates a new metrics source name that's unique within this process.
FSDataInputStream	open(Path f, int bufferSize) Opens an FSDataInputStream at the indicated Path.
String	pathToKey(Path path) Convert the path to a key.
void	recoverFilesWithDanglingTempData(Path root, Path destination) Looks under the given root path for any blob that are left "dangling", meaning that they are place-holder blobs that we created while we upload the data to a temporary blob, but for some reason we crashed in the middle of the upload and left them there.
boolean	rename(Path src, Path dst) Renames Path src to Path dst.
void	setOwner(Path p, String username, String groupname) Set owner of a path (i.e. a file or a directory).
void	setPermission(Path p, FsPermission permission) Set permission of a path.
void	setWorkingDirectory(Path newDir) Set the working directory to the given directory.

クラスから継承されたメソッド org.apache.hadoop.fs.FileSystem

append, append, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, clearStatistics, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createNewFile, createSnapshot, createSnapshot, createSymlink, deleteOnExit, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAclStatus, getAllStatistics, getAllStoragePolicies, getBlockSize, getCanonicalUri, getContentSummary, getDefaultBlockSize, getDefaultBlockSize, getDefaultPort, getDefaultReplication, getDefaultReplication, getDefaultUri, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getHomeDirectory, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getQuotaUsage, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getStoragePolicy, getStorageStatistics, getTrashRoot, getTrashRoots, getUsed, getUsed, getXAttr, getXAttrs, getXAttrs, globStatus, globStatus, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, listStatusIterator, listXAttrs, makeQualified, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, setAcl, setDefaultUri, setDefaultUri, setReplication, setStoragePolicy, setTimes, setVerifyChecksum, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, supportsSymlinks, truncate, unsetStoragePolicy

クラスから継承されたメソッド org.apache.hadoop.conf.Configured

getConf, setConf

クラスから継承されたメソッド java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

フィールドの詳細

LOG

public static final org.slf4j.Logger LOG

SKIP_AZURE_METRICS_PROPERTY_NAME

public static final String SKIP_AZURE_METRICS_PROPERTY_NAME

関連項目:

定数フィールド値

APPEND_SUPPORT_ENABLE_PROPERTY_NAME

public static final String APPEND_SUPPORT_ENABLE_PROPERTY_NAME

関連項目:

定数フィールド値

AZURE_RENAME_THREADS

public static final String AZURE_RENAME_THREADS

The configuration property to set number of threads to be used for rename operation.

関連項目:

定数フィールド値

DEFAULT_AZURE_RENAME_THREADS

public static final int DEFAULT_AZURE_RENAME_THREADS

The default number of threads to be used for rename operation.

関連項目:

定数フィールド値

AZURE_DELETE_THREADS

public static final String AZURE_DELETE_THREADS

The configuration property to set number of threads to be used for delete operation.

関連項目:

定数フィールド値

DEFAULT_AZURE_DELETE_THREADS

public static final int DEFAULT_AZURE_DELETE_THREADS

The default number of threads to be used for delete operation.

関連項目:

定数フィールド値

コンストラクタの詳細

NativeAzureFileSystem

public NativeAzureFileSystem()

NativeAzureFileSystem

public NativeAzureFileSystem(org.apache.hadoop.fs.azure.NativeFileSystemStore store)

メソッドの詳細

getScheme

public String getScheme()

クラスからコピーされた説明: FileSystem

Return the protocol scheme for the FileSystem.

This implementation throws an UnsupportedOperationException.

オーバーライド:

getScheme クラス内 FileSystem

戻り値:

the protocol scheme for the FileSystem.

newMetricsSourceName

public static String newMetricsSourceName()

Creates a new metrics source name that's unique within this process.

checkPath

protected void checkPath(Path path)

クラスからコピーされた説明: FileSystem

Check that a Path belongs to this FileSystem.

オーバーライド:

checkPath クラス内 FileSystem

パラメータ:

path - to check

initialize

public void initialize(URI uri,
                       Configuration conf)
                throws IOException,
                       IllegalArgumentException

クラスからコピーされた説明: FileSystem

Called after a new FileSystem instance is constructed.

オーバーライド:

initialize クラス内 FileSystem

パラメータ:

uri - a uri whose authority section names the host, port, etc. for this FileSystem

conf - the configuration

例外:

IOException

IllegalArgumentException

pathToKey

public String pathToKey(Path path)

Convert the path to a key. By convention, any leading or trailing slash is removed, except for the special case of a single slash.

makeAbsolute

public Path makeAbsolute(Path path)

Get the absolute version of the path (fully qualified). This is public for testing purposes.

パラメータ:

path -

戻り値:

fully qualified path

getStore

public org.apache.hadoop.fs.azure.AzureNativeFileSystemStore getStore()

For unit test purposes, retrieves the AzureNativeFileSystemStore store backing this file system.

戻り値:

The store object.

getInstrumentation

public AzureFileSystemInstrumentation getInstrumentation()

Gets the metrics source for this file system. This is mainly here for unit testing purposes.

戻り値:

the metrics source.

append

public FSDataOutputStream append(Path f,
                                 int bufferSize,
                                 Progressable progress)
                          throws IOException

This optional operation is not yet supported.

定義:

append クラス内 FileSystem

パラメータ:

f - the existing file to be appended.

bufferSize - the size of the buffer to be used.

progress - for reporting progress if it is not null.

例外:

IOException

create

public FSDataOutputStream create(Path f,
                                 FsPermission permission,
                                 boolean overwrite,
                                 int bufferSize,
                                 short replication,
                                 long blockSize,
                                 Progressable progress)
                          throws IOException

クラスからコピーされた説明: FileSystem

Create an FSDataOutputStream at the indicated Path with write-progress reporting.

定義:

create クラス内 FileSystem

パラメータ:

f - the file name to open

permission - file permission

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

bufferSize - the size of the buffer to be used.

replication - required block replication for the file.

blockSize - block size

progress - the progress reporter

例外:

IOException

関連項目:

FileSystem.setPermission(Path, FsPermission)

acquireLease

public org.apache.hadoop.fs.azure.SelfRenewingLease acquireLease(Path path)
                                                          throws AzureException

Get a self-renewing lease on the specified file.

例外:

AzureException

createNonRecursive

public FSDataOutputStream createNonRecursive(Path f,
                                             FsPermission permission,
                                             boolean overwrite,
                                             int bufferSize,
                                             short replication,
                                             long blockSize,
                                             Progressable progress)
                                      throws IOException

クラスからコピーされた説明: FileSystem

Opens an FSDataOutputStream at the indicated Path with write-progress reporting. Same as create(), except fails if parent directory doesn't already exist.

オーバーライド:

createNonRecursive クラス内 FileSystem

パラメータ:

f - the file name to open

permission - file permission

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

bufferSize - the size of the buffer to be used.

replication - required block replication for the file.

blockSize - block size

progress - the progress reporter

例外:

IOException

関連項目:

FileSystem.setPermission(Path, FsPermission)

createNonRecursive

public FSDataOutputStream createNonRecursive(Path f,
                                             FsPermission permission,
                                             EnumSet<CreateFlag> flags,
                                             int bufferSize,
                                             short replication,
                                             long blockSize,
                                             Progressable progress)
                                      throws IOException

クラスからコピーされた説明: FileSystem

Opens an FSDataOutputStream at the indicated Path with write-progress reporting. Same as create(), except fails if parent directory doesn't already exist.

オーバーライド:

createNonRecursive クラス内 FileSystem

パラメータ:

f - the file name to open

permission - file permission

flags - CreateFlags to use for this stream.

bufferSize - the size of the buffer to be used.

replication - required block replication for the file.

blockSize - block size

progress - the progress reporter

例外:

IOException

関連項目:

FileSystem.setPermission(Path, FsPermission)

createNonRecursive

public FSDataOutputStream createNonRecursive(Path f,
                                             boolean overwrite,
                                             int bufferSize,
                                             short replication,
                                             long blockSize,
                                             Progressable progress)
                                      throws IOException

クラスからコピーされた説明: FileSystem

Opens an FSDataOutputStream at the indicated Path with write-progress reporting. Same as create(), except fails if parent directory doesn't already exist.

オーバーライド:

createNonRecursive クラス内 FileSystem

パラメータ:

f - the file name to open

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

bufferSize - the size of the buffer to be used.

replication - required block replication for the file.

blockSize - block size

progress - the progress reporter

例外:

IOException

関連項目:

FileSystem.setPermission(Path, FsPermission)

delete

@Deprecated
public boolean delete(Path path)
                           throws IOException

非推奨です。

クラスからコピーされた説明: FileSystem

Delete a file

オーバーライド:

delete クラス内 FileSystem

例外:

IOException

delete

public boolean delete(Path f,
                      boolean recursive)
               throws IOException

クラスからコピーされた説明: FileSystem

Delete a file.

定義:

delete クラス内 FileSystem

パラメータ:

f - the path to delete.

recursive - if path is a directory and set to true, the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false.

戻り値:

true if delete is successful else false.

例外:

IOException

delete

public boolean delete(Path f,
                      boolean recursive,
                      boolean skipParentFolderLastModifidedTimeUpdate)
               throws IOException

Delete the specified file or folder. The parameter skipParentFolderLastModifidedTimeUpdate is used in the case of atomic folder rename redo. In that case, there is a lease on the parent folder, so (without reworking the code) modifying the parent folder update time will fail because of a conflict with the lease. Since we are going to delete the folder soon anyway so accurate modified time is not necessary, it's easier to just skip the modified time update.

パラメータ:

f -

recursive -

skipParentFolderLastModifidedTimeUpdate - If true, don't update the folder last modified time.

戻り値:

true if and only if the file is deleted

例外:

IOException

getThreadPoolExecutor

public org.apache.hadoop.fs.azure.AzureFileSystemThreadPoolExecutor getThreadPoolExecutor(int threadCount,
                                                                                          String threadNamePrefix,
                                                                                          String operation,
                                                                                          String key,
                                                                                          String config)

getFileStatus

public FileStatus getFileStatus(Path f)
                         throws FileNotFoundException,
                                IOException

クラスからコピーされた説明: FileSystem

Return a file status object that represents the path.

定義:

getFileStatus クラス内 FileSystem

パラメータ:

f - The path we want information from

戻り値:

a FileStatus object

例外:

FileNotFoundException - when the path does not exist; IOException see specific implementation

IOException

getUri

public URI getUri()

クラスからコピーされた説明: FileSystem

Returns a URI whose scheme and authority identify this FileSystem.

定義:

getUri クラス内 FileSystem

listStatus

public FileStatus[] listStatus(Path f)
                        throws FileNotFoundException,
                               IOException

Retrieve the status of a given path if it is a file, or of all the contained files if it is a directory.

定義:

listStatus クラス内 FileSystem

パラメータ:

f - given path

戻り値:

the statuses of the files/directories in the given patch

例外:

FileNotFoundException - when the path does not exist; IOException see specific implementation

IOException

mkdirs

public boolean mkdirs(Path f,
                      FsPermission permission)
               throws IOException

クラスからコピーされた説明: FileSystem

Make the given file and all non-existent parents into directories. Has the semantics of Unix 'mkdir -p'. Existence of the directory hierarchy is not an error.

定義:

mkdirs クラス内 FileSystem

パラメータ:

f - path to create

permission - to apply to f

例外:

IOException

mkdirs

public boolean mkdirs(Path f,
                      FsPermission permission,
                      boolean noUmask)
               throws IOException

例外:

IOException

open

public FSDataInputStream open(Path f,
                              int bufferSize)
                       throws FileNotFoundException,
                              IOException

クラスからコピーされた説明: FileSystem

Opens an FSDataInputStream at the indicated Path.

定義:

open クラス内 FileSystem

パラメータ:

f - the file name to open

bufferSize - the size of the buffer to be used.

例外:

FileNotFoundException

IOException

rename

public boolean rename(Path src,
                      Path dst)
               throws FileNotFoundException,
                      IOException

クラスからコピーされた説明: FileSystem

Renames Path src to Path dst. Can take place on local fs or remote DFS.

定義:

rename クラス内 FileSystem

パラメータ:

src - path to be renamed

dst - new path after rename

戻り値:

true if rename is successful

例外:

IOException - on failure

FileNotFoundException

getFileBlockLocations

public BlockLocation[] getFileBlockLocations(FileStatus file,
                                             long start,
                                             long len)
                                      throws IOException

Return an array containing hostnames, offset and size of portions of the given file. For WASB we'll just lie and give fake hosts to make sure we get many splits in MR jobs.

オーバーライド:

getFileBlockLocations クラス内 FileSystem

パラメータ:

file - FilesStatus to get data from

start - offset into the given file

len - length for which to get locations for

例外:

IOException

setWorkingDirectory

public void setWorkingDirectory(Path newDir)

Set the working directory to the given directory.

定義:

setWorkingDirectory クラス内 FileSystem

パラメータ:

newDir - Path of new working directory

getWorkingDirectory

public Path getWorkingDirectory()

クラスからコピーされた説明: FileSystem

Get the current working directory for the given file system

定義:

getWorkingDirectory クラス内 FileSystem

戻り値:

the directory pathname

setPermission

public void setPermission(Path p,
                          FsPermission permission)
                   throws FileNotFoundException,
                          IOException

クラスからコピーされた説明: FileSystem

Set permission of a path.

オーバーライド:

setPermission クラス内 FileSystem

パラメータ:

p - The path

permission - permission

例外:

FileNotFoundException

IOException

setOwner

public void setOwner(Path p,
                     String username,
                     String groupname)
              throws IOException

クラスからコピーされた説明: FileSystem

Set owner of a path (i.e. a file or a directory). The parameters username and groupname cannot both be null.

オーバーライド:

setOwner クラス内 FileSystem

パラメータ:

p - The path

username - If it is null, the original username remains unchanged.

groupname - If it is null, the original groupname remains unchanged.

例外:

IOException

close

public void close()
           throws IOException

クラスからコピーされた説明: FileSystem

No more filesystem operations are needed. Will release any held locks.

定義:

close インタフェース内 Closeable

定義:

close インタフェース内 AutoCloseable

オーバーライド:

close クラス内 FileSystem

例外:

IOException

recoverFilesWithDanglingTempData

public void recoverFilesWithDanglingTempData(Path root,
                                             Path destination)
                                      throws IOException

Looks under the given root path for any blob that are left "dangling", meaning that they are place-holder blobs that we created while we upload the data to a temporary blob, but for some reason we crashed in the middle of the upload and left them there. If any are found, we move them to the destination given.

パラメータ:

root - The root path to consider.

destination - The destination path to move any recovered files to.

例外:

IOException

deleteFilesWithDanglingTempData

public void deleteFilesWithDanglingTempData(Path root)
                                     throws IOException

Looks under the given root path for any blob that are left "dangling", meaning that they are place-holder blobs that we created while we upload the data to a temporary blob, but for some reason we crashed in the middle of the upload and left them there. If any are found, we delete them.

パラメータ:

root - The root path to consider.

例外:

IOException

finalize

protected void finalize()
                 throws Throwable

オーバーライド:

finalize クラス内 Object

例外:

Throwable