Examples with Source org.smartdata.hdfs.action.move.Source used on opensource projects

Example 1 with Source

use of org.smartdata.hdfs.action.move.Source in project SSM by Intel-bigdata.

the class DDatanode method addSource.

public Source addSource(String storageType) {
    final Source s = new Source(storageType, this.getDatanodeInfo());
    put(storageType, s, sourceMap);
    return s;
}

Example 2 with Source

use of org.smartdata.hdfs.action.move.Source in project SSM by Intel-bigdata.

the class DatanodeStorageReportProcTask method run.

@Override
public void run() {
    try {
        reset();
        final List<DatanodeStorageReport> reports = getDNStorageReports();
        for (DatanodeStorageReport r : reports) {
            // TODO: store data abstracted from reports to MetaStore
            final DDatanode dn = new DDatanode(r.getDatanodeInfo(), maxConcurrentMovesPerNode);
            for (String t : CompatibilityHelperLoader.getHelper().getMovableTypes()) {
                final Source source = dn.addSource(t);
                final long maxRemaining = getMaxRemaining(r, t);
                final StorageGroup target = maxRemaining > 0L ? dn.addTarget(t) : null;
                storages.add(source, target);
            }
        }
    } catch (IOException e) {
        LOG.error("Process datanode report error", e);
    }
}

Example 3 with Source

use of org.smartdata.hdfs.action.move.Source in project SSM by Intel-bigdata.

the class MovePlanMaker method scheduleMoveBlock.

/**
 * TODO: consider the case that fails to move some blocks, i.e., scheduleMoveReplica fails.
 */
void scheduleMoveBlock(StorageTypeDiff diff, LocatedBlock lb, HdfsFileStatus status) {
    final List<MLocation> locations = MLocation.toLocations(lb);
    if (!CompatibilityHelperLoader.getHelper().isLocatedStripedBlock(lb)) {
        // Shuffle replica locations to make storage medium in balance.
        // E.g., if three replicas are under ALL_SSD policy and ONE_SSD is the target policy,
        // with shuffling locations, two randomly picked replicas will be moved to DISK.
        Collections.shuffle(locations);
    }
    // EC block case is considered.
    final DBlock db = CompatibilityHelperLoader.getHelper().newDBlock(lb, status);
    for (MLocation ml : locations) {
        StorageGroup source = storages.getSource(ml);
        if (source != null) {
            db.addLocation(source);
        }
    }
    for (int index = 0; index < diff.existing.size(); index++) {
        String t = diff.existing.get(index);
        Iterator<MLocation> iter = locations.iterator();
        while (iter.hasNext()) {
            MLocation ml = iter.next();
            final Source source = storages.getSource(ml);
            // in diff's existing list. If so, try to schedule the moving.
            if (ml.getStorageType() == t && source != null) {
                // The corresponding storage type in diff's expected list is used.
                if (scheduleMoveReplica(db, source, Arrays.asList(diff.expected.get(index)))) {
                    // If the replica is successfully scheduled to move.
                    // No need to consider it any more.
                    iter.remove();
                    // Tackle the next storage type in diff existing list.
                    break;
                }
            }
        }
    }
}