Examples with FSDataInputStream org.apache.hadoop.fs.FSDataInputStream used on opensource projects

Example 6 with FSDataInputStream

use of org.apache.hadoop.fs.FSDataInputStream in project hadoop by apache.

the class SFTPFileSystem method open.

@Override
public FSDataInputStream open(Path f, int bufferSize) throws IOException {
    ChannelSftp channel = connect();
    Path workDir;
    try {
        workDir = new Path(channel.pwd());
    } catch (SftpException e) {
        throw new IOException(e);
    }
    Path absolute = makeAbsolute(workDir, f);
    FileStatus fileStat = getFileStatus(channel, absolute);
    if (fileStat.isDirectory()) {
        disconnect(channel);
        throw new IOException(String.format(E_PATH_DIR, f));
    }
    InputStream is;
    try {
        // the path could be a symbolic link, so get the real path
        absolute = new Path("/", channel.realpath(absolute.toUri().getPath()));
        is = channel.get(absolute.toUri().getPath());
    } catch (SftpException e) {
        throw new IOException(e);
    }
    FSDataInputStream fis = new FSDataInputStream(new SFTPInputStream(is, channel, statistics));
    return fis;
}

Example 7 with FSDataInputStream

use of org.apache.hadoop.fs.FSDataInputStream in project hadoop by apache.

the class AbstractContractSeekTest method testReadNullBuffer.

@Test
public void testReadNullBuffer() throws Throwable {
    describe("try to read a null buffer ");
    assumeSupportsPositionedReadable();
    try (FSDataInputStream in = getFileSystem().open(smallSeekFile)) {
        // Null buffer
        int r = in.read(0, null, 0, 16);
        fail("Expected an exception from a read into a null buffer, got " + r);
    } catch (IllegalArgumentException e) {
    // expected
    }
}

Example 8 with FSDataInputStream

use of org.apache.hadoop.fs.FSDataInputStream in project hadoop by apache.

the class AbstractContractSeekTest method testRandomSeeks.

/**
   * Lifted from TestLocalFileSystem:
   * Regression test for HADOOP-9307: BufferedFSInputStream returning
   * wrong results after certain sequences of seeks and reads.
   */
@Test
public void testRandomSeeks() throws Throwable {
    int limit = getContract().getLimit(TEST_RANDOM_SEEK_COUNT, DEFAULT_RANDOM_SEEK_COUNT);
    describe("Testing " + limit + " random seeks");
    int filesize = 10 * 1024;
    byte[] buf = dataset(filesize, 0, 255);
    Path randomSeekFile = path("testrandomseeks.bin");
    createFile(getFileSystem(), randomSeekFile, false, buf);
    Random r = new Random();
    // Record the sequence of seeks and reads which trigger a failure.
    int[] seeks = new int[10];
    int[] reads = new int[10];
    try (FSDataInputStream stm = getFileSystem().open(randomSeekFile)) {
        for (int i = 0; i < limit; i++) {
            int seekOff = r.nextInt(buf.length);
            int toRead = r.nextInt(Math.min(buf.length - seekOff, 32000));
            seeks[i % seeks.length] = seekOff;
            reads[i % reads.length] = toRead;
            verifyRead(stm, buf, seekOff, toRead);
        }
    } catch (AssertionError afe) {
        StringBuilder sb = new StringBuilder();
        sb.append("Sequence of actions:\n");
        for (int j = 0; j < seeks.length; j++) {
            sb.append("seek @ ").append(seeks[j]).append("  ").append("read ").append(reads[j]).append("\n");
        }
        LOG.error(sb.toString());
        throw afe;
    }
}

Example 9 with FSDataInputStream

use of org.apache.hadoop.fs.FSDataInputStream in project hadoop by apache.

the class ContractTestUtils method readDataset.

/**
   * Read the file and convert to a byte dataset.
   * This implements readfully internally, so that it will read
   * in the file without ever having to seek()
   * @param fs filesystem
   * @param path path to read from
   * @param len length of data to read
   * @return the bytes
   * @throws IOException IO problems
   */
public static byte[] readDataset(FileSystem fs, Path path, int len) throws IOException {
    byte[] dest = new byte[len];
    int offset = 0;
    int nread = 0;
    try (FSDataInputStream in = fs.open(path)) {
        while (nread < len) {
            int nbytes = in.read(dest, offset + nread, len - nread);
            if (nbytes < 0) {
                throw new EOFException("End of file reached before reading fully.");
            }
            nread += nbytes;
        }
    }
    return dest;
}

Example 10 with FSDataInputStream

use of org.apache.hadoop.fs.FSDataInputStream in project hadoop by apache.

the class TestCopy method testInterruptedCreate.

@Test
public void testInterruptedCreate() throws Exception {
    whenFsCreate().thenThrow(new InterruptedIOException());
    when(mockFs.getFileStatus(eq(tmpPath))).thenReturn(fileStat);
    FSDataInputStream in = mock(FSDataInputStream.class);
    tryCopyStream(in, false);
    verify(mockFs).delete(eq(tmpPath), anyBoolean());
    verify(mockFs, never()).rename(any(Path.class), any(Path.class));
    verify(mockFs, never()).delete(eq(path), anyBoolean());
    verify(mockFs, never()).close();
}