Search in sources :

Example 1 with BlobSynchronizer

use of org.apache.storm.blobstore.BlobSynchronizer in project storm by apache.

the class Nimbus method blobSync.

private void blobSync() throws Exception {
    if ("distributed".equals(conf.get(Config.STORM_CLUSTER_MODE))) {
        if (!isLeader()) {
            IStormClusterState state = stormClusterState;
            NimbusInfo nimbusInfo = nimbusHostPortInfo;
            BlobStore store = blobStore;
            Set<String> allKeys = new HashSet<>();
            for (Iterator<String> it = store.listKeys(); it.hasNext(); ) {
                allKeys.add(it.next());
            }
            Set<String> zkKeys = new HashSet<>(state.blobstore(() -> {
                try {
                    this.blobSync();
                } catch (Exception e) {
                    throw new RuntimeException(e);
                }
            }));
            LOG.debug("blob-sync blob-store-keys {} zookeeper-keys {}", allKeys, zkKeys);
            BlobSynchronizer sync = new BlobSynchronizer(store, conf);
            sync.setNimbusInfo(nimbusInfo);
            sync.setBlobStoreKeySet(allKeys);
            sync.setZookeeperKeySet(zkKeys);
            sync.syncBlobs();
        }
    //else not leader (NOOP)
    }
//else local (NOOP)
}
Also used : BlobSynchronizer(org.apache.storm.blobstore.BlobSynchronizer) IStormClusterState(org.apache.storm.cluster.IStormClusterState) BlobStore(org.apache.storm.blobstore.BlobStore) LocalFsBlobStore(org.apache.storm.blobstore.LocalFsBlobStore) AuthorizationException(org.apache.storm.generated.AuthorizationException) NotAliveException(org.apache.storm.generated.NotAliveException) InterruptedIOException(java.io.InterruptedIOException) TException(org.apache.thrift.TException) IOException(java.io.IOException) AlreadyAliveException(org.apache.storm.generated.AlreadyAliveException) KeyAlreadyExistsException(org.apache.storm.generated.KeyAlreadyExistsException) KeyNotFoundException(org.apache.storm.generated.KeyNotFoundException) InvalidTopologyException(org.apache.storm.generated.InvalidTopologyException) BindException(java.net.BindException) NimbusInfo(org.apache.storm.nimbus.NimbusInfo) HashSet(java.util.HashSet)

Aggregations

IOException (java.io.IOException)1 InterruptedIOException (java.io.InterruptedIOException)1 BindException (java.net.BindException)1 HashSet (java.util.HashSet)1 BlobStore (org.apache.storm.blobstore.BlobStore)1 BlobSynchronizer (org.apache.storm.blobstore.BlobSynchronizer)1 LocalFsBlobStore (org.apache.storm.blobstore.LocalFsBlobStore)1 IStormClusterState (org.apache.storm.cluster.IStormClusterState)1 AlreadyAliveException (org.apache.storm.generated.AlreadyAliveException)1 AuthorizationException (org.apache.storm.generated.AuthorizationException)1 InvalidTopologyException (org.apache.storm.generated.InvalidTopologyException)1 KeyAlreadyExistsException (org.apache.storm.generated.KeyAlreadyExistsException)1 KeyNotFoundException (org.apache.storm.generated.KeyNotFoundException)1 NotAliveException (org.apache.storm.generated.NotAliveException)1 NimbusInfo (org.apache.storm.nimbus.NimbusInfo)1 TException (org.apache.thrift.TException)1