Examples with HdfsBlockLocation org.apache.hadoop.fs.HdfsBlockLocation used on opensource projects

Example 6 with HdfsBlockLocation

use of org.apache.hadoop.fs.HdfsBlockLocation in project hadoop by apache.

the class TestFsDatasetCache method testPageRounder.

@Test(timeout = 600000)
public void testPageRounder() throws Exception {
    // Write a small file
    Path fileName = new Path("/testPageRounder");
    // This should be smaller than the page size
    final int smallBlocks = 512;
    assertTrue("Page size should be greater than smallBlocks!", PAGE_SIZE > smallBlocks);
    final int numBlocks = 5;
    final int fileLen = smallBlocks * numBlocks;
    FSDataOutputStream out = fs.create(fileName, false, 4096, (short) 1, smallBlocks);
    out.write(new byte[fileLen]);
    out.close();
    HdfsBlockLocation[] locs = (HdfsBlockLocation[]) fs.getFileBlockLocations(fileName, 0, fileLen);
    // Cache the file and check the sizes match the page size
    setHeartbeatResponse(cacheBlocks(locs));
    DFSTestUtil.verifyExpectedCacheUsage(PAGE_SIZE * numBlocks, numBlocks, fsd);
    // Uncache and check that it decrements by the page size too
    setHeartbeatResponse(uncacheBlocks(locs));
    DFSTestUtil.verifyExpectedCacheUsage(0, 0, fsd);
}

Example 7 with HdfsBlockLocation

use of org.apache.hadoop.fs.HdfsBlockLocation in project hadoop by apache.

the class DFSClient method getBlockLocations.

/**
   * Get block location info about file
   *
   * getBlockLocations() returns a list of hostnames that store
   * data for a specific file region.  It returns a set of hostnames
   * for every block within the indicated region.
   *
   * This function is very useful when writing code that considers
   * data-placement when performing operations.  For example, the
   * MapReduce system tries to schedule tasks on the same machines
   * as the data-block the task processes.
   */
public BlockLocation[] getBlockLocations(String src, long start, long length) throws IOException {
    checkOpen();
    try (TraceScope ignored = newPathTraceScope("getBlockLocations", src)) {
        LocatedBlocks blocks = getLocatedBlocks(src, start, length);
        BlockLocation[] locations = DFSUtilClient.locatedBlocks2Locations(blocks);
        HdfsBlockLocation[] hdfsLocations = new HdfsBlockLocation[locations.length];
        for (int i = 0; i < locations.length; i++) {
            hdfsLocations[i] = new HdfsBlockLocation(locations[i], blocks.get(i));
        }
        return hdfsLocations;
    }
}