Search in sources :

Example 1 with PlainBlobMetadata

use of org.elasticsearch.common.blobstore.support.PlainBlobMetadata in project elasticsearch by elastic.

the class AzureStorageServiceImpl method listBlobsByPrefix.

@Override
public Map<String, BlobMetaData> listBlobsByPrefix(String account, LocationMode mode, String container, String keyPath, String prefix) throws URISyntaxException, StorageException {
    // NOTE: this should be here: if (prefix == null) prefix = "";
    // however, this is really inefficient since deleteBlobsByPrefix enumerates everything and
    // then does a prefix match on the result; it should just call listBlobsByPrefix with the prefix!
    logger.debug("listing container [{}], keyPath [{}], prefix [{}]", container, keyPath, prefix);
    MapBuilder<String, BlobMetaData> blobsBuilder = MapBuilder.newMapBuilder();
    CloudBlobClient client = this.getSelectedClient(account, mode);
    CloudBlobContainer blobContainer = client.getContainerReference(container);
    SocketAccess.doPrivilegedVoidException(() -> {
        if (blobContainer.exists()) {
            for (ListBlobItem blobItem : blobContainer.listBlobs(keyPath + (prefix == null ? "" : prefix))) {
                URI uri = blobItem.getUri();
                logger.trace("blob url [{}]", uri);
                // uri.getPath is of the form /container/keyPath.* and we want to strip off the /container/
                // this requires 1 + container.length() + 1, with each 1 corresponding to one of the /
                String blobPath = uri.getPath().substring(1 + container.length() + 1);
                CloudBlockBlob blob = blobContainer.getBlockBlobReference(blobPath);
                // fetch the blob attributes from Azure (getBlockBlobReference does not do this)
                // this is needed to retrieve the blob length (among other metadata) from Azure Storage
                blob.downloadAttributes();
                BlobProperties properties = blob.getProperties();
                String name = blobPath.substring(keyPath.length());
                logger.trace("blob url [{}], name [{}], size [{}]", uri, name, properties.getLength());
                blobsBuilder.put(name, new PlainBlobMetaData(name, properties.getLength()));
            }
        }
    });
    return blobsBuilder.immutableMap();
}
Also used : CloudBlobClient(com.microsoft.azure.storage.blob.CloudBlobClient) ListBlobItem(com.microsoft.azure.storage.blob.ListBlobItem) PlainBlobMetaData(org.elasticsearch.common.blobstore.support.PlainBlobMetaData) BlobMetaData(org.elasticsearch.common.blobstore.BlobMetaData) PlainBlobMetaData(org.elasticsearch.common.blobstore.support.PlainBlobMetaData) BlobProperties(com.microsoft.azure.storage.blob.BlobProperties) CloudBlobContainer(com.microsoft.azure.storage.blob.CloudBlobContainer) CloudBlockBlob(com.microsoft.azure.storage.blob.CloudBlockBlob) URI(java.net.URI)

Example 2 with PlainBlobMetadata

use of org.elasticsearch.common.blobstore.support.PlainBlobMetadata in project elasticsearch by elastic.

the class AzureStorageServiceMock method listBlobsByPrefix.

@Override
public Map<String, BlobMetaData> listBlobsByPrefix(String account, LocationMode mode, String container, String keyPath, String prefix) {
    MapBuilder<String, BlobMetaData> blobsBuilder = MapBuilder.newMapBuilder();
    blobs.forEach((String blobName, ByteArrayOutputStream bos) -> {
        final String checkBlob;
        if (keyPath != null && !keyPath.isEmpty()) {
            // strip off key path from the beginning of the blob name
            checkBlob = blobName.replace(keyPath, "");
        } else {
            checkBlob = blobName;
        }
        if (prefix == null || startsWithIgnoreCase(checkBlob, prefix)) {
            blobsBuilder.put(blobName, new PlainBlobMetaData(checkBlob, bos.size()));
        }
    });
    return blobsBuilder.immutableMap();
}
Also used : PlainBlobMetaData(org.elasticsearch.common.blobstore.support.PlainBlobMetaData) PlainBlobMetaData(org.elasticsearch.common.blobstore.support.PlainBlobMetaData) BlobMetaData(org.elasticsearch.common.blobstore.BlobMetaData) ByteArrayOutputStream(java.io.ByteArrayOutputStream)

Example 3 with PlainBlobMetadata

use of org.elasticsearch.common.blobstore.support.PlainBlobMetadata in project elasticsearch by elastic.

the class HdfsBlobContainer method listBlobsByPrefix.

@Override
public Map<String, BlobMetaData> listBlobsByPrefix(@Nullable final String prefix) throws IOException {
    FileStatus[] files = store.execute(fileContext -> (fileContext.util().listStatus(path, path -> prefix == null || path.getName().startsWith(prefix))));
    Map<String, BlobMetaData> map = new LinkedHashMap<String, BlobMetaData>();
    for (FileStatus file : files) {
        map.put(file.getPath().getName(), new PlainBlobMetaData(file.getPath().getName(), file.getLen()));
    }
    return Collections.unmodifiableMap(map);
}
Also used : FileStatus(org.apache.hadoop.fs.FileStatus) PlainBlobMetaData(org.elasticsearch.common.blobstore.support.PlainBlobMetaData) PlainBlobMetaData(org.elasticsearch.common.blobstore.support.PlainBlobMetaData) BlobMetaData(org.elasticsearch.common.blobstore.BlobMetaData) LinkedHashMap(java.util.LinkedHashMap)

Example 4 with PlainBlobMetadata

use of org.elasticsearch.common.blobstore.support.PlainBlobMetadata in project elasticsearch by elastic.

the class FsBlobContainer method listBlobsByPrefix.

@Override
public Map<String, BlobMetaData> listBlobsByPrefix(String blobNamePrefix) throws IOException {
    // If we get duplicate files we should just take the last entry
    Map<String, BlobMetaData> builder = new HashMap<>();
    blobNamePrefix = blobNamePrefix == null ? "" : blobNamePrefix;
    try (DirectoryStream<Path> stream = Files.newDirectoryStream(path, blobNamePrefix + "*")) {
        for (Path file : stream) {
            final BasicFileAttributes attrs = Files.readAttributes(file, BasicFileAttributes.class);
            if (attrs.isRegularFile()) {
                builder.put(file.getFileName().toString(), new PlainBlobMetaData(file.getFileName().toString(), attrs.size()));
            }
        }
    }
    return unmodifiableMap(builder);
}
Also used : BlobPath(org.elasticsearch.common.blobstore.BlobPath) Path(java.nio.file.Path) PlainBlobMetaData(org.elasticsearch.common.blobstore.support.PlainBlobMetaData) HashMap(java.util.HashMap) PlainBlobMetaData(org.elasticsearch.common.blobstore.support.PlainBlobMetaData) BlobMetaData(org.elasticsearch.common.blobstore.BlobMetaData) BasicFileAttributes(java.nio.file.attribute.BasicFileAttributes)

Aggregations

BlobMetaData (org.elasticsearch.common.blobstore.BlobMetaData)4 PlainBlobMetaData (org.elasticsearch.common.blobstore.support.PlainBlobMetaData)4 BlobProperties (com.microsoft.azure.storage.blob.BlobProperties)1 CloudBlobClient (com.microsoft.azure.storage.blob.CloudBlobClient)1 CloudBlobContainer (com.microsoft.azure.storage.blob.CloudBlobContainer)1 CloudBlockBlob (com.microsoft.azure.storage.blob.CloudBlockBlob)1 ListBlobItem (com.microsoft.azure.storage.blob.ListBlobItem)1 ByteArrayOutputStream (java.io.ByteArrayOutputStream)1 URI (java.net.URI)1 Path (java.nio.file.Path)1 BasicFileAttributes (java.nio.file.attribute.BasicFileAttributes)1 HashMap (java.util.HashMap)1 LinkedHashMap (java.util.LinkedHashMap)1 FileStatus (org.apache.hadoop.fs.FileStatus)1 BlobPath (org.elasticsearch.common.blobstore.BlobPath)1