Search in sources :

Example 1 with ReadableBlobMeta

use of backtype.storm.generated.ReadableBlobMeta in project jstorm by alibaba.

the class LocalFsBlobStore method getBlobMeta.

@Override
public ReadableBlobMeta getBlobMeta(String key) throws KeyNotFoundException {
    validateKey(key);
    if (!checkForBlobOrDownload(key)) {
        checkForBlobUpdate(key);
    }
    SettableBlobMeta meta = getStoredBlobMeta(key);
    ReadableBlobMeta rbm = new ReadableBlobMeta();
    rbm.set_settable(meta);
    try {
        LocalFsBlobStoreFile pf = fbs.read(DATA_PREFIX + key);
        rbm.set_version(pf.getModTime());
    } catch (IOException e) {
        throw new RuntimeException(e);
    }
    return rbm;
}
Also used : ReadableBlobMeta(backtype.storm.generated.ReadableBlobMeta) SettableBlobMeta(backtype.storm.generated.SettableBlobMeta)

Example 2 with ReadableBlobMeta

use of backtype.storm.generated.ReadableBlobMeta in project jstorm by alibaba.

the class BlobStoreUtils method downloadMissingBlob.

// Download missing blobs from potential nimbodes
public static boolean downloadMissingBlob(Map conf, BlobStore blobStore, String key, Set<NimbusInfo> nimbusInfos) throws TTransportException {
    NimbusClient client;
    ReadableBlobMeta rbm;
    ClientBlobStore remoteBlobStore;
    InputStreamWithMeta in;
    boolean isSuccess = false;
    LOG.debug("Download blob NimbusInfos {}", nimbusInfos);
    for (NimbusInfo nimbusInfo : nimbusInfos) {
        if (isSuccess) {
            break;
        }
        try {
            client = new NimbusClient(conf, nimbusInfo.getHost(), nimbusInfo.getPort(), null);
            rbm = client.getClient().getBlobMeta(key);
            remoteBlobStore = new NimbusBlobStore();
            remoteBlobStore.setClient(conf, client);
            in = remoteBlobStore.getBlob(key);
            blobStore.createBlob(key, in, rbm.get_settable());
            // if key already exists while creating the blob else update it
            Iterator<String> keyIterator = blobStore.listKeys();
            while (keyIterator.hasNext()) {
                if (keyIterator.next().equals(key)) {
                    LOG.debug("Success creating key, {}", key);
                    isSuccess = true;
                    break;
                }
            }
        } catch (IOException exception) {
            throw new RuntimeException(exception);
        } catch (KeyAlreadyExistsException kae) {
            LOG.info("KeyAlreadyExistsException Key: {} {}", key, kae);
        } catch (KeyNotFoundException knf) {
            // Catching and logging KeyNotFoundException because, if
            // there is a subsequent update and delete, the non-leader
            // nimbodes might throw an exception.
            LOG.info("KeyNotFoundException Key: {} {}", key, knf);
        } catch (Exception exp) {
            // Logging an exception while client is connecting
            LOG.error("Exception ", exp);
        }
    }
    if (!isSuccess) {
        LOG.error("Could not download blob with key " + key);
    }
    return isSuccess;
}
Also used : ReadableBlobMeta(backtype.storm.generated.ReadableBlobMeta) NimbusClient(backtype.storm.utils.NimbusClient) IOException(java.io.IOException) KeyAlreadyExistsException(backtype.storm.generated.KeyAlreadyExistsException) KeyNotFoundException(backtype.storm.generated.KeyNotFoundException) TTransportException(org.apache.thrift.transport.TTransportException) TException(org.apache.thrift.TException) IOException(java.io.IOException) KeyAlreadyExistsException(backtype.storm.generated.KeyAlreadyExistsException) FileExistsException(org.apache.commons.io.FileExistsException) NimbusInfo(backtype.storm.nimbus.NimbusInfo) KeyNotFoundException(backtype.storm.generated.KeyNotFoundException)

Aggregations

ReadableBlobMeta (backtype.storm.generated.ReadableBlobMeta)2 KeyAlreadyExistsException (backtype.storm.generated.KeyAlreadyExistsException)1 KeyNotFoundException (backtype.storm.generated.KeyNotFoundException)1 SettableBlobMeta (backtype.storm.generated.SettableBlobMeta)1 NimbusInfo (backtype.storm.nimbus.NimbusInfo)1 NimbusClient (backtype.storm.utils.NimbusClient)1 IOException (java.io.IOException)1 FileExistsException (org.apache.commons.io.FileExistsException)1 TException (org.apache.thrift.TException)1 TTransportException (org.apache.thrift.transport.TTransportException)1