Examples with BlockStoragePolicy org.apache.hadoop.hdfs.protocol.BlockStoragePolicy used on opensource projects

Search in sources :

Example 26 with BlockStoragePolicy

use of org.apache.hadoop.hdfs.protocol.BlockStoragePolicy in project hadoop by apache.

the class TestStoragePolicySummary method testMultipleWarmsInDifferentOrder.

@Test
public void testMultipleWarmsInDifferentOrder() {
    BlockStoragePolicySuite bsps = BlockStoragePolicySuite.createDefaultSuite();
    StoragePolicySummary sts = new StoragePolicySummary(bsps.getAllPolicies());
    BlockStoragePolicy warm = bsps.getPolicy("WARM");
    //DISK:1,ARCHIVE:1
    sts.add(new StorageType[] { StorageType.DISK, StorageType.ARCHIVE }, warm);
    sts.add(new StorageType[] { StorageType.ARCHIVE, StorageType.DISK }, warm);
    //DISK:2,ARCHIVE:1
    sts.add(new StorageType[] { StorageType.ARCHIVE, StorageType.DISK, StorageType.DISK }, warm);
    sts.add(new StorageType[] { StorageType.DISK, StorageType.ARCHIVE, StorageType.DISK }, warm);
    sts.add(new StorageType[] { StorageType.DISK, StorageType.DISK, StorageType.ARCHIVE }, warm);
    //DISK:1,ARCHIVE:2
    sts.add(new StorageType[] { StorageType.DISK, StorageType.ARCHIVE, StorageType.ARCHIVE }, warm);
    sts.add(new StorageType[] { StorageType.ARCHIVE, StorageType.DISK, StorageType.ARCHIVE }, warm);
    sts.add(new StorageType[] { StorageType.ARCHIVE, StorageType.ARCHIVE, StorageType.DISK }, warm);
    //DISK:2,ARCHIVE:2
    sts.add(new StorageType[] { StorageType.ARCHIVE, StorageType.ARCHIVE, StorageType.DISK, StorageType.DISK }, warm);
    Map<String, Long> actualOutput = convertToStringMap(sts);
    Assert.assertEquals(4, actualOutput.size());
    Map<String, Long> expectedOutput = new HashMap<>();
    expectedOutput.put("WARM|DISK:1,ARCHIVE:1(WARM)", 2l);
    expectedOutput.put("WARM|DISK:2,ARCHIVE:1", 3l);
    expectedOutput.put("WARM|DISK:1,ARCHIVE:2(WARM)", 3l);
    expectedOutput.put("WARM|DISK:2,ARCHIVE:2", 1l);
    Assert.assertEquals(expectedOutput, actualOutput);
}
Also used : BlockStoragePolicySuite(org.apache.hadoop.hdfs.server.blockmanagement.BlockStoragePolicySuite) LinkedHashMap(java.util.LinkedHashMap) HashMap(java.util.HashMap) BlockStoragePolicy(org.apache.hadoop.hdfs.protocol.BlockStoragePolicy) Test(org.junit.Test)

Example 27 with BlockStoragePolicy

use of org.apache.hadoop.hdfs.protocol.BlockStoragePolicy in project hadoop by apache.

the class TestStoragePolicySummary method testMultipleHots.

@Test
public void testMultipleHots() {
    BlockStoragePolicySuite bsps = BlockStoragePolicySuite.createDefaultSuite();
    StoragePolicySummary sts = new StoragePolicySummary(bsps.getAllPolicies());
    BlockStoragePolicy hot = bsps.getPolicy("HOT");
    sts.add(new StorageType[] { StorageType.DISK }, hot);
    sts.add(new StorageType[] { StorageType.DISK, StorageType.DISK }, hot);
    sts.add(new StorageType[] { StorageType.DISK, StorageType.DISK, StorageType.DISK }, hot);
    sts.add(new StorageType[] { StorageType.DISK, StorageType.DISK, StorageType.DISK, StorageType.DISK }, hot);
    Map<String, Long> actualOutput = convertToStringMap(sts);
    Assert.assertEquals(4, actualOutput.size());
    Map<String, Long> expectedOutput = new HashMap<>();
    expectedOutput.put("HOT|DISK:1(HOT)", 1l);
    expectedOutput.put("HOT|DISK:2(HOT)", 1l);
    expectedOutput.put("HOT|DISK:3(HOT)", 1l);
    expectedOutput.put("HOT|DISK:4(HOT)", 1l);
    Assert.assertEquals(expectedOutput, actualOutput);
}
Also used : BlockStoragePolicySuite(org.apache.hadoop.hdfs.server.blockmanagement.BlockStoragePolicySuite) LinkedHashMap(java.util.LinkedHashMap) HashMap(java.util.HashMap) BlockStoragePolicy(org.apache.hadoop.hdfs.protocol.BlockStoragePolicy) Test(org.junit.Test)

Example 28 with BlockStoragePolicy

use of org.apache.hadoop.hdfs.protocol.BlockStoragePolicy in project hadoop by apache.

the class TestStoragePolicySummary method testMultipleHotsWithDifferentCounts.

@Test
public void testMultipleHotsWithDifferentCounts() {
    BlockStoragePolicySuite bsps = BlockStoragePolicySuite.createDefaultSuite();
    StoragePolicySummary sts = new StoragePolicySummary(bsps.getAllPolicies());
    BlockStoragePolicy hot = bsps.getPolicy("HOT");
    sts.add(new StorageType[] { StorageType.DISK }, hot);
    sts.add(new StorageType[] { StorageType.DISK, StorageType.DISK }, hot);
    sts.add(new StorageType[] { StorageType.DISK, StorageType.DISK }, hot);
    sts.add(new StorageType[] { StorageType.DISK, StorageType.DISK, StorageType.DISK }, hot);
    sts.add(new StorageType[] { StorageType.DISK, StorageType.DISK, StorageType.DISK }, hot);
    sts.add(new StorageType[] { StorageType.DISK, StorageType.DISK, StorageType.DISK, StorageType.DISK }, hot);
    Map<String, Long> actualOutput = convertToStringMap(sts);
    Assert.assertEquals(4, actualOutput.size());
    Map<String, Long> expectedOutput = new HashMap<>();
    expectedOutput.put("HOT|DISK:1(HOT)", 1l);
    expectedOutput.put("HOT|DISK:2(HOT)", 2l);
    expectedOutput.put("HOT|DISK:3(HOT)", 2l);
    expectedOutput.put("HOT|DISK:4(HOT)", 1l);
    Assert.assertEquals(expectedOutput, actualOutput);
}
Also used : BlockStoragePolicySuite(org.apache.hadoop.hdfs.server.blockmanagement.BlockStoragePolicySuite) LinkedHashMap(java.util.LinkedHashMap) HashMap(java.util.HashMap) BlockStoragePolicy(org.apache.hadoop.hdfs.protocol.BlockStoragePolicy) Test(org.junit.Test)

Example 29 with BlockStoragePolicy

use of org.apache.hadoop.hdfs.protocol.BlockStoragePolicy in project hadoop by apache.

the class TestStoragePolicySummary method testSortInDescendingOrder.

@Test
public void testSortInDescendingOrder() {
    BlockStoragePolicySuite bsps = BlockStoragePolicySuite.createDefaultSuite();
    StoragePolicySummary sts = new StoragePolicySummary(bsps.getAllPolicies());
    BlockStoragePolicy hot = bsps.getPolicy("HOT");
    BlockStoragePolicy warm = bsps.getPolicy("WARM");
    BlockStoragePolicy cold = bsps.getPolicy("COLD");
    //DISK:3
    sts.add(new StorageType[] { StorageType.DISK, StorageType.DISK, StorageType.DISK }, hot);
    sts.add(new StorageType[] { StorageType.DISK, StorageType.DISK, StorageType.DISK }, hot);
    //DISK:1,ARCHIVE:2
    sts.add(new StorageType[] { StorageType.DISK, StorageType.ARCHIVE, StorageType.ARCHIVE }, warm);
    sts.add(new StorageType[] { StorageType.ARCHIVE, StorageType.DISK, StorageType.ARCHIVE }, warm);
    sts.add(new StorageType[] { StorageType.ARCHIVE, StorageType.ARCHIVE, StorageType.DISK }, warm);
    //ARCHIVE:3
    sts.add(new StorageType[] { StorageType.ARCHIVE, StorageType.ARCHIVE, StorageType.ARCHIVE }, cold);
    sts.add(new StorageType[] { StorageType.ARCHIVE, StorageType.ARCHIVE, StorageType.ARCHIVE }, cold);
    sts.add(new StorageType[] { StorageType.ARCHIVE, StorageType.ARCHIVE, StorageType.ARCHIVE }, cold);
    sts.add(new StorageType[] { StorageType.ARCHIVE, StorageType.ARCHIVE, StorageType.ARCHIVE }, cold);
    Map<String, Long> actualOutput = convertToStringMap(sts);
    Assert.assertEquals(3, actualOutput.size());
    Map<String, Long> expectedOutput = new LinkedHashMap<>();
    expectedOutput.put("COLD|ARCHIVE:3(COLD)", 4l);
    expectedOutput.put("WARM|DISK:1,ARCHIVE:2(WARM)", 3l);
    expectedOutput.put("HOT|DISK:3(HOT)", 2l);
    Assert.assertEquals(expectedOutput.toString(), actualOutput.toString());
}
Also used : BlockStoragePolicySuite(org.apache.hadoop.hdfs.server.blockmanagement.BlockStoragePolicySuite) BlockStoragePolicy(org.apache.hadoop.hdfs.protocol.BlockStoragePolicy) LinkedHashMap(java.util.LinkedHashMap) Test(org.junit.Test)

Example 30 with BlockStoragePolicy

use of org.apache.hadoop.hdfs.protocol.BlockStoragePolicy in project hadoop by apache.

the class TestStoragePolicySummary method testDifferentSpecifiedPolicies.

@Test
public void testDifferentSpecifiedPolicies() {
    BlockStoragePolicySuite bsps = BlockStoragePolicySuite.createDefaultSuite();
    StoragePolicySummary sts = new StoragePolicySummary(bsps.getAllPolicies());
    BlockStoragePolicy hot = bsps.getPolicy("HOT");
    BlockStoragePolicy warm = bsps.getPolicy("WARM");
    BlockStoragePolicy cold = bsps.getPolicy("COLD");
    //DISK:3
    sts.add(new StorageType[] { StorageType.DISK, StorageType.DISK, StorageType.DISK }, hot);
    sts.add(new StorageType[] { StorageType.DISK, StorageType.DISK, StorageType.DISK }, hot);
    sts.add(new StorageType[] { StorageType.DISK, StorageType.DISK, StorageType.DISK }, warm);
    sts.add(new StorageType[] { StorageType.DISK, StorageType.DISK, StorageType.DISK }, cold);
    //DISK:1,ARCHIVE:2
    sts.add(new StorageType[] { StorageType.DISK, StorageType.ARCHIVE, StorageType.ARCHIVE }, hot);
    sts.add(new StorageType[] { StorageType.ARCHIVE, StorageType.DISK, StorageType.ARCHIVE }, warm);
    sts.add(new StorageType[] { StorageType.ARCHIVE, StorageType.ARCHIVE, StorageType.DISK }, cold);
    sts.add(new StorageType[] { StorageType.ARCHIVE, StorageType.ARCHIVE, StorageType.DISK }, cold);
    //ARCHIVE:3
    sts.add(new StorageType[] { StorageType.ARCHIVE, StorageType.ARCHIVE, StorageType.ARCHIVE }, hot);
    sts.add(new StorageType[] { StorageType.ARCHIVE, StorageType.ARCHIVE, StorageType.ARCHIVE }, hot);
    sts.add(new StorageType[] { StorageType.ARCHIVE, StorageType.ARCHIVE, StorageType.ARCHIVE }, warm);
    sts.add(new StorageType[] { StorageType.ARCHIVE, StorageType.ARCHIVE, StorageType.ARCHIVE }, cold);
    Map<String, Long> actualOutput = convertToStringMap(sts);
    Assert.assertEquals(9, actualOutput.size());
    Map<String, Long> expectedOutput = new HashMap<>();
    expectedOutput.put("HOT|DISK:3(HOT)", 2l);
    expectedOutput.put("COLD|DISK:1,ARCHIVE:2(WARM)", 2l);
    expectedOutput.put("HOT|ARCHIVE:3(COLD)", 2l);
    expectedOutput.put("WARM|DISK:3(HOT)", 1l);
    expectedOutput.put("COLD|DISK:3(HOT)", 1l);
    expectedOutput.put("WARM|ARCHIVE:3(COLD)", 1l);
    expectedOutput.put("WARM|DISK:1,ARCHIVE:2(WARM)", 1l);
    expectedOutput.put("COLD|ARCHIVE:3(COLD)", 1l);
    expectedOutput.put("HOT|DISK:1,ARCHIVE:2(WARM)", 1l);
    Assert.assertEquals(expectedOutput, actualOutput);
}
Also used : BlockStoragePolicySuite(org.apache.hadoop.hdfs.server.blockmanagement.BlockStoragePolicySuite) LinkedHashMap(java.util.LinkedHashMap) HashMap(java.util.HashMap) BlockStoragePolicy(org.apache.hadoop.hdfs.protocol.BlockStoragePolicy) Test(org.junit.Test)

Aggregations

BlockStoragePolicy (org.apache.hadoop.hdfs.protocol.BlockStoragePolicy)48 Test (org.junit.Test)19 BlockStoragePolicySuite (org.apache.hadoop.hdfs.server.blockmanagement.BlockStoragePolicySuite)15 Path (org.apache.hadoop.fs.Path)12 StorageType (org.apache.hadoop.fs.StorageType)12 IOException (java.io.IOException)8 FileNotFoundException (java.io.FileNotFoundException)6 ArrayList (java.util.ArrayList)6 HashMap (java.util.HashMap)5 LinkedHashMap (java.util.LinkedHashMap)5 BlockInfo (org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo)5 DistributedFileSystem (org.apache.hadoop.hdfs.DistributedFileSystem)4 HadoopIllegalArgumentException (org.apache.hadoop.HadoopIllegalArgumentException)3 BlockStoragePolicySpi (org.apache.hadoop.fs.BlockStoragePolicySpi)3 HdfsFileStatus (org.apache.hadoop.hdfs.protocol.HdfsFileStatus)3 LocatedBlock (org.apache.hadoop.hdfs.protocol.LocatedBlock)3 QuotaCounts (org.apache.hadoop.hdfs.server.namenode.QuotaCounts)3 Field (java.lang.reflect.Field)2 List (java.util.List)2 FileSystem (org.apache.hadoop.fs.FileSystem)2
About Me
UseOf

Java Tutorials :

Simple Java RabbitMQ Example with Spring
Simple Springboot JPA Eclipeslink Example
Simple SpringBoot JPA Hibernate Example
Jackson JSON @JsonIgnore and @JsonIgnoreProperties Examples
Java Infinite recursion (StackOverflowError) Jackson solutions
Simple Java Jackson Serialize Objects to JSON and convert them back To Object
ElasticSearch 5 - Upload files using Java API in binary field Example
Java JAXB marshall unmarshall Examples from String and Stream
Java 8 forEach List and Map examples
ElasticSearch 5 Java API SearchRequestBuilder examples
Java ElasticSearch 5 examples with node index and mapping - running local
Convert String to int or Integer Java 8
Java 9 in action - JShell - Ubuntu
Java - removing invalid characters from a file name
Hello world!? or Hello Tutorial