Examples with ThrottledInputStream org.apache.hadoop.tools.util.ThrottledInputStream used on opensource projects

Example 1 with ThrottledInputStream

use of org.apache.hadoop.tools.util.ThrottledInputStream in project hadoop by apache.

the class RetriableFileCopyCommand method copyBytes.

@VisibleForTesting
long copyBytes(CopyListingFileStatus source2, long sourceOffset, OutputStream outStream, int bufferSize, Mapper.Context context) throws IOException {
    Path source = source2.getPath();
    byte[] buf = new byte[bufferSize];
    ThrottledInputStream inStream = null;
    long totalBytesRead = 0;
    try {
        inStream = getInputStream(source, context.getConfiguration());
        int bytesRead = readBytes(inStream, buf, sourceOffset);
        while (bytesRead >= 0) {
            totalBytesRead += bytesRead;
            if (action == FileAction.APPEND) {
                sourceOffset += bytesRead;
            }
            outStream.write(buf, 0, bytesRead);
            updateContextStatus(totalBytesRead, context, source2);
            bytesRead = readBytes(inStream, buf, sourceOffset);
        }
        outStream.close();
        outStream = null;
    } finally {
        IOUtils.cleanup(LOG, outStream, inStream);
    }
    return totalBytesRead;
}

Example 2 with ThrottledInputStream

use of org.apache.hadoop.tools.util.ThrottledInputStream in project hadoop by apache.

the class RetriableFileCopyCommand method getInputStream.

private static ThrottledInputStream getInputStream(Path path, Configuration conf) throws IOException {
    try {
        FileSystem fs = path.getFileSystem(conf);
        float bandwidthMB = conf.getFloat(DistCpConstants.CONF_LABEL_BANDWIDTH_MB, DistCpConstants.DEFAULT_BANDWIDTH_MB);
        FSDataInputStream in = fs.open(path);
        return new ThrottledInputStream(in, bandwidthMB * 1024 * 1024);
    } catch (IOException e) {
        throw new CopyReadException(e);
    }
}