Examples with StorageGroup org.smartdata.hdfs.action.move.StorageGroup used on opensource projects

Example 1 with StorageGroup

use of org.smartdata.hdfs.action.move.StorageGroup in project SSM by Intel-bigdata.

the class DatanodeStorageReportProcTask method run.

@Override
public void run() {
    try {
        reset();
        final List<DatanodeStorageReport> reports = getDNStorageReports();
        for (DatanodeStorageReport r : reports) {
            // TODO: store data abstracted from reports to MetaStore
            final DDatanode dn = new DDatanode(r.getDatanodeInfo(), maxConcurrentMovesPerNode);
            for (String t : CompatibilityHelperLoader.getHelper().getMovableTypes()) {
                final Source source = dn.addSource(t);
                final long maxRemaining = getMaxRemaining(r, t);
                final StorageGroup target = maxRemaining > 0L ? dn.addTarget(t) : null;
                storages.add(source, target);
            }
        }
    } catch (IOException e) {
        LOG.error("Process datanode report error", e);
    }
}

Example 2 with StorageGroup

use of org.smartdata.hdfs.action.move.StorageGroup in project SSM by Intel-bigdata.

the class TestMoveFileAction method createPlan.

private FileMovePlan createPlan(String dir, String storageType) throws Exception {
    URI namenode = cluster.getURI();
    FileMovePlan plan = new FileMovePlan(namenode, dir);
    // Schedule move in the same node
    for (LocatedBlock lb : getLocatedBlocks(dfsClient, dir, plan)) {
        ExtendedBlock block = lb.getBlock();
        for (DatanodeInfo datanodeInfo : lb.getLocations()) {
            StorageGroup source = new StorageGroup(datanodeInfo, StorageType.DISK.toString());
            StorageGroup target = new StorageGroup(datanodeInfo, storageType);
            addPlan(plan, source, target, block.getBlockId());
        }
    }
    return plan;
}

Example 3 with StorageGroup

use of org.smartdata.hdfs.action.move.StorageGroup in project SSM by Intel-bigdata.

the class DDatanode method put.

private static <G extends StorageGroup> void put(String storageType, G g, Map<String, G> map) {
    final StorageGroup existing = map.put(storageType, g);
    Preconditions.checkState(existing == null);
}

Example 4 with StorageGroup

use of org.smartdata.hdfs.action.move.StorageGroup in project SSM by Intel-bigdata.

the class DDatanode method addTarget.

public StorageGroup addTarget(String storageType) {
    final StorageGroup g = new StorageGroup(this.datanode, storageType);
    put(storageType, g, targetMap);
    return g;
}

Example 5 with StorageGroup

use of org.smartdata.hdfs.action.move.StorageGroup in project SSM by Intel-bigdata.

the class MovePlanMaker method chooseTarget.

boolean chooseTarget(DBlock db, Source source, List<String> targetTypes, Matcher matcher) {
    final NetworkTopology cluster = this.networkTopology;
    for (String t : targetTypes) {
        final List<StorageGroup> targets = storages.getTargetStorages(t);
        Collections.shuffle(targets);
        for (StorageGroup target : targets) {
            if (matcher.match(cluster, source.getDatanodeInfo(), target.getDatanodeInfo())) {
                addPlan(source, target, db.getBlock().getBlockId());
                return true;
            }
        }
    }
    return false;
}