Examples with StatefulStorageException com.twitter.heron.spi.statefulstorage.StatefulStorageException used on opensource projects

Example 1 with StatefulStorageException

use of com.twitter.heron.spi.statefulstorage.StatefulStorageException in project incubator-heron by apache.

the class DlogStorage method dispose.

@Override
public void dispose(String topologyName, String oldestCheckpointId, boolean deleteAll) throws StatefulStorageException {
    // Currently dlog doesn't support recursive deletion. so we have to fetch all the checkpoints
    // and delete individual checkpoints.
    // TODO (sijie): replace the logic here once distributedlog supports recursive deletion.
    String topologyCheckpointRoot = getTopologyCheckpointRoot(topologyName);
    URI topologyUri = URI.create(checkpointNamespaceUriStr + topologyCheckpointRoot);
    // get checkpoints
    Namespace topologyNs = null;
    Iterator<String> checkpoints;
    try {
        topologyNs = initializeNamespace(topologyUri);
        checkpoints = topologyNs.getLogs();
    } catch (IOException ioe) {
        throw new StatefulStorageException("Failed to open topology namespace", ioe);
    } finally {
        if (null != topologyNs) {
            topologyNs.close();
        }
    }
    while (checkpoints.hasNext()) {
        String checkpointId = checkpoints.next();
        if (deleteAll || checkpointId.compareTo(oldestCheckpointId) < 0) {
            URI checkpointUri = URI.create(checkpointNamespaceUriStr + topologyCheckpointRoot + "/" + checkpointId);
            try {
                deleteCheckpoint(checkpointUri);
            } catch (IOException e) {
                throw new StatefulStorageException("Failed to remove checkpoint " + checkpointId + " for topology " + topologyName, e);
            }
        }
    }
}

Example 2 with StatefulStorageException

use of com.twitter.heron.spi.statefulstorage.StatefulStorageException in project incubator-heron by apache.

the class LocalFileSystemStorage method dispose.

@Override
public void dispose(String topologyName, String oldestCheckpointPreserved, boolean deleteAll) throws StatefulStorageException {
    String topologyCheckpointRoot = getTopologyCheckpointRoot(topologyName);
    if (deleteAll) {
        // Clean all checkpoint states
        FileUtils.deleteDir(topologyCheckpointRoot);
        if (FileUtils.isDirectoryExists(topologyCheckpointRoot)) {
            throw new StatefulStorageException("Failed to delete " + topologyCheckpointRoot);
        }
    } else {
        String[] names = new File(topologyCheckpointRoot).list();
        for (String name : names) {
            if (name.compareTo(oldestCheckpointPreserved) < 0) {
                FileUtils.deleteDir(new File(topologyCheckpointRoot, name), true);
            }
        }
        // Do a double check. Now all checkpoints with smaller checkpoint id should be cleaned
        names = new File(topologyCheckpointRoot).list();
        for (String name : names) {
            if (name.compareTo(oldestCheckpointPreserved) < 0) {
                throw new StatefulStorageException("Failed to delete " + name);
            }
        }
    }
}

Example 3 with StatefulStorageException

use of com.twitter.heron.spi.statefulstorage.StatefulStorageException in project incubator-heron by apache.

the class LocalFileSystemStorage method restore.

@Override
public Checkpoint restore(String topologyName, String checkpointId, PhysicalPlans.Instance instanceInfo) throws StatefulStorageException {
    String path = getCheckpointPath(topologyName, checkpointId, instanceInfo.getInfo().getComponentName(), instanceInfo.getInfo().getTaskId());
    byte[] res = FileUtils.readFromFile(path);
    if (res.length != 0) {
        // Try to parse the protobuf
        CheckpointManager.InstanceStateCheckpoint state;
        try {
            state = CheckpointManager.InstanceStateCheckpoint.parseFrom(res);
        } catch (InvalidProtocolBufferException e) {
            throw new StatefulStorageException("Failed to parse the data", e);
        }
        return new Checkpoint(topologyName, instanceInfo, state);
    } else {
        throw new StatefulStorageException("Failed to parse the data");
    }
}

Example 4 with StatefulStorageException

use of com.twitter.heron.spi.statefulstorage.StatefulStorageException in project incubator-heron by apache.

the class HDFSStorage method store.

@Override
public void store(Checkpoint checkpoint) throws StatefulStorageException {
    Path path = new Path(getCheckpointPath(checkpoint.getTopologyName(), checkpoint.getCheckpointId(), checkpoint.getComponent(), checkpoint.getTaskId()));
    // We need to ensure the existence of directories structure,
    // since it is not guaranteed that FileSystem.create(..) always creates parents' dirs.
    String checkpointDir = getCheckpointDir(checkpoint.getTopologyName(), checkpoint.getCheckpointId(), checkpoint.getComponent());
    createDir(checkpointDir);
    FSDataOutputStream out = null;
    try {
        out = fileSystem.create(path);
        checkpoint.getCheckpoint().writeTo(out);
    } catch (IOException e) {
        throw new StatefulStorageException("Failed to persist", e);
    } finally {
        SysUtils.closeIgnoringExceptions(out);
    }
}

Example 5 with StatefulStorageException

use of com.twitter.heron.spi.statefulstorage.StatefulStorageException in project incubator-heron by apache.

the class HDFSStorage method init.

@Override
public void init(Map<String, Object> conf) throws StatefulStorageException {
    LOG.info("Initializing... Config: " + conf.toString());
    LOG.info("Class path: " + System.getProperty("java.class.path"));
    checkpointRootPath = (String) conf.get(ROOT_PATH_KEY);
    // Notice, we pass the config folder via classpath
    // So hadoop will automatically search config files from classpath
    Configuration hadoopConfig = new Configuration();
    try {
        fileSystem = FileSystem.get(hadoopConfig);
        LOG.info("Hadoop FileSystem URI: " + fileSystem.getUri() + " ; Home Dir: " + fileSystem.getHomeDirectory());
    } catch (IOException e) {
        throw new StatefulStorageException("Failed to get hadoop file system", e);
    }
}