Examples with PureJavaCrc32 org.apache.hadoop.util.PureJavaCrc32 used on opensource projects

Example 1 with PureJavaCrc32

use of org.apache.hadoop.util.PureJavaCrc32 in project hadoop by apache.

the class TestShuffleHandler method createIndexFile.

private static void createIndexFile(File indexFile, Configuration conf) throws IOException {
    if (indexFile.exists()) {
        System.out.println("Deleting existing file");
        indexFile.delete();
    }
    indexFile.createNewFile();
    FSDataOutputStream output = FileSystem.getLocal(conf).getRaw().append(new Path(indexFile.getAbsolutePath()));
    Checksum crc = new PureJavaCrc32();
    crc.reset();
    CheckedOutputStream chk = new CheckedOutputStream(output, crc);
    String msg = "Writing new index file. This file will be used only " + "for the testing.";
    chk.write(Arrays.copyOf(msg.getBytes(), MapTask.MAP_OUTPUT_INDEX_RECORD_LENGTH));
    output.writeLong(chk.getChecksum().getValue());
    output.close();
}

Example 2 with PureJavaCrc32

use of org.apache.hadoop.util.PureJavaCrc32 in project hadoop by apache.

the class GenSort method outputRecords.

public static void outputRecords(OutputStream out, boolean useAscii, Unsigned16 firstRecordNumber, Unsigned16 recordsToGenerate, Unsigned16 checksum) throws IOException {
    byte[] row = new byte[100];
    Unsigned16 recordNumber = new Unsigned16(firstRecordNumber);
    Unsigned16 lastRecordNumber = new Unsigned16(firstRecordNumber);
    Checksum crc = new PureJavaCrc32();
    Unsigned16 tmp = new Unsigned16();
    lastRecordNumber.add(recordsToGenerate);
    Unsigned16 ONE = new Unsigned16(1);
    Unsigned16 rand = Random16.skipAhead(firstRecordNumber);
    while (!recordNumber.equals(lastRecordNumber)) {
        Random16.nextRand(rand);
        if (useAscii) {
            generateAsciiRecord(row, rand, recordNumber);
        } else {
            generateRecord(row, rand, recordNumber);
        }
        if (checksum != null) {
            crc.reset();
            crc.update(row, 0, row.length);
            tmp.set(crc.getValue());
            checksum.add(tmp);
        }
        recordNumber.add(ONE);
        out.write(row);
    }
}

Example 3 with PureJavaCrc32

use of org.apache.hadoop.util.PureJavaCrc32 in project tez by apache.

the class TestShuffleHandler method createIndexFile.

private static void createIndexFile(File indexFile, Configuration conf) throws IOException {
    if (indexFile.exists()) {
        System.out.println("Deleting existing file");
        indexFile.delete();
    }
    indexFile.createNewFile();
    FSDataOutputStream output = FileSystem.getLocal(conf).getRaw().append(new Path(indexFile.getAbsolutePath()));
    Checksum crc = new PureJavaCrc32();
    crc.reset();
    CheckedOutputStream chk = new CheckedOutputStream(output, crc);
    String msg = "Writing new index file. This file will be used only " + "for the testing.";
    chk.write(Arrays.copyOf(msg.getBytes(), MapTask.MAP_OUTPUT_INDEX_RECORD_LENGTH));
    output.writeLong(chk.getChecksum().getValue());
    output.close();
}