Examples with FileChecksum org.apache.hadoop.fs.FileChecksum used on opensource projects

Example 26 with FileChecksum

use of org.apache.hadoop.fs.FileChecksum in project hadoop by apache.

the class TestFileChecksum method testStripedAndReplicatedFileChecksum.

@Test(timeout = 90000)
public void testStripedAndReplicatedFileChecksum() throws Exception {
    prepareTestFiles(fileSize, new String[] { stripedFile1, replicatedFile });
    FileChecksum stripedFileChecksum1 = getFileChecksum(stripedFile1, 10, false);
    FileChecksum replicatedFileChecksum = getFileChecksum(replicatedFile, 10, false);
    Assert.assertFalse(stripedFileChecksum1.equals(replicatedFileChecksum));
}

Example 27 with FileChecksum

use of org.apache.hadoop.fs.FileChecksum in project hive by apache.

the class Hive method addInsertNonDirectoryInformation.

private static void addInsertNonDirectoryInformation(Path p, FileSystem fileSystem, InsertEventRequestData insertData) throws IOException {
    insertData.addToFilesAdded(p.toString());
    FileChecksum cksum = fileSystem.getFileChecksum(p);
    // File checksum is not implemented for local filesystem (RawLocalFileSystem)
    if (cksum != null) {
        String checksumString = StringUtils.byteToHexString(cksum.getBytes(), 0, cksum.getLength());
        insertData.addToFilesAddedChecksum(checksumString);
    } else {
        // Add an empty checksum string for filesystems that don't generate one
        insertData.addToFilesAddedChecksum("");
    }
}

Example 28 with FileChecksum

use of org.apache.hadoop.fs.FileChecksum in project hive by apache.

the class ReplChangeManager method checksumFor.

// Get checksum of a file
public static String checksumFor(Path path, FileSystem fs) throws IOException {
    // TODO: fs checksum only available on hdfs, need to
    // find a solution for other fs (eg, local fs, s3, etc)
    String checksumString = null;
    FileChecksum checksum = fs.getFileChecksum(path);
    if (checksum != null) {
        checksumString = StringUtils.byteToHexString(checksum.getBytes(), 0, checksum.getLength());
    }
    return checksumString;
}