Search in sources :

Example 1 with DirectoryDto

use of io.datarouter.storage.node.op.raw.read.DirectoryDto in project datarouter by hotpads.

the class S3BucketHandler method index.

@Handler(defaultHandler = true)
public Mav index(@Param(P_client) String client, @Param(P_bucket) String bucket, @Param(P_prefix) OptionalString prefix, @Param(P_after) OptionalString after, @Param(P_offset) OptionalInteger offset, @Param(P_limit) OptionalInteger limit, @Param(P_currentDirectory) OptionalBoolean currentDirectory, @Param(P_delimiter) OptionalString delimiter) {
    var form = new HtmlForm().withMethod("get");
    form.addTextField().withDisplay("Client").withName(P_client).withPlaceholder("theClientName").withValue(client);
    form.addTextField().withDisplay("Bucket").withName(P_bucket).withPlaceholder("the.bucket.name").withValue(bucket);
    form.addTextField().withDisplay("Prefix").withName(P_prefix).withValue(prefix.orElse(""));
    form.addTextField().withDisplay("After").withName(P_after).withValue(after.orElse(""));
    form.addTextField().withDisplay("Offset").withName(P_offset).withValue(offset.orElse(0) + "");
    form.addTextField().withDisplay("Limit").withName(P_limit).withValue(limit.orElse(100) + "");
    form.addTextField().withDisplay("Delimiter").withName(P_delimiter).withValue(delimiter.orElse(""));
    form.addCheckboxField().withDisplay("currentDirectory").withName(P_currentDirectory).withChecked(currentDirectory.orElse(false));
    form.addButton().withDisplay("Submit").withValue("");
    var htmlForm = Bootstrap4FormHtml.render(form).withClass("card card-body bg-light");
    ClientId clientId = clients.getClientId(client);
    DatarouterS3Client s3Client = s3ClientManager.getClient(clientId);
    List<DirectoryDto> objects = s3Client.scanSubdirectories(bucket, prefix.orElse(null), after.orElse(null), delimiter.orElse(null), limit.orElse(100), currentDirectory.orElse(false)).list();
    int sizePadding = sizePadding(objects);
    ContainerTag<?> table = new J2HtmlTable<DirectoryDto>().withClasses("sortable table table-sm table-striped my-4 border").withHtmlColumn("Key", object -> {
        String name = object.name;
        if (object.isDirectory) {
            return td(makePrefixLink(client, bucket, name, "/"));
        }
        return td(name);
    }).withHtmlColumn("Directory", object -> {
        boolean isDirectory = object.isDirectory;
        if (isDirectory) {
            String href = new URIBuilder().setPath(request.getContextPath() + paths.datarouter.clients.awsS3.countObjects.toSlashedString()).addParameter(P_client, client).addParameter(P_bucket, bucket).addParameter(P_prefix, object.name).toString();
            return td(a("true, view count").withHref(href));
        }
        return td(String.valueOf(isDirectory));
    }).withHtmlColumn("Size", object -> {
        String commas = NumberFormatter.addCommas(object.size);
        String padded = StringTool.pad(commas, ' ', sizePadding);
        String escaped = padded.replaceAll(" ", "&nbsp;");
        return td(rawHtml(escaped));
    }).withColumn("Last Modified", object -> object.lastModified).withColumn("Storage Class", object -> object.storageClass).build(objects);
    ContainerTag<?> tableWrapper = table.withStyle("font-family:monospace; font-size:.9em;");
    var content = div(htmlForm, h4(bucket), tableWrapper).withClass("container-fluid my-4");
    return pageFactory.startBuilder(request).withTitle("S3 Bucket").withRequires(DatarouterWebRequireJsV2.SORTTABLE).withContent(content).buildMav();
}
Also used : Scanner(io.datarouter.scanner.Scanner) DatarouterS3Client(io.datarouter.aws.s3.DatarouterS3Client) LoggerFactory(org.slf4j.LoggerFactory) TagCreator.h4(j2html.TagCreator.h4) OptionalString(io.datarouter.web.handler.types.optional.OptionalString) AtomicReference(java.util.concurrent.atomic.AtomicReference) S3ClientManager(io.datarouter.aws.s3.client.S3ClientManager) Inject(javax.inject.Inject) NumberFormatter(io.datarouter.util.number.NumberFormatter) DatarouterWebRequireJsV2(io.datarouter.web.requirejs.DatarouterWebRequireJsV2) ClientId(io.datarouter.storage.client.ClientId) Param(io.datarouter.web.handler.types.Param) Bootstrap4FormHtml(io.datarouter.web.html.j2html.bootstrap4.Bootstrap4FormHtml) OptionalInteger(io.datarouter.web.handler.types.optional.OptionalInteger) J2HtmlTable(io.datarouter.web.html.j2html.J2HtmlTable) DatarouterClients(io.datarouter.storage.client.DatarouterClients) TagCreator.rawHtml(j2html.TagCreator.rawHtml) Logger(org.slf4j.Logger) URIBuilder(org.apache.http.client.utils.URIBuilder) DatarouterAwsS3Paths(io.datarouter.aws.s3.config.DatarouterAwsS3Paths) Mav(io.datarouter.web.handler.mav.Mav) StringTool(io.datarouter.util.string.StringTool) TagCreator.a(j2html.TagCreator.a) ContainerTag(j2html.tags.ContainerTag) OptionalBoolean(io.datarouter.web.handler.types.optional.OptionalBoolean) AtomicLong(java.util.concurrent.atomic.AtomicLong) List(java.util.List) DirectoryDto(io.datarouter.storage.node.op.raw.read.DirectoryDto) HtmlForm(io.datarouter.web.html.form.HtmlForm) TagCreator.td(j2html.TagCreator.td) BaseHandler(io.datarouter.web.handler.BaseHandler) Bootstrap4PageFactory(io.datarouter.web.html.j2html.bootstrap4.Bootstrap4PageFactory) Comparator(java.util.Comparator) TagCreator.div(j2html.TagCreator.div) OptionalString(io.datarouter.web.handler.types.optional.OptionalString) URIBuilder(org.apache.http.client.utils.URIBuilder) DirectoryDto(io.datarouter.storage.node.op.raw.read.DirectoryDto) HtmlForm(io.datarouter.web.html.form.HtmlForm) DatarouterS3Client(io.datarouter.aws.s3.DatarouterS3Client) ClientId(io.datarouter.storage.client.ClientId) BaseHandler(io.datarouter.web.handler.BaseHandler)

Example 2 with DirectoryDto

use of io.datarouter.storage.node.op.raw.read.DirectoryDto in project datarouter by hotpads.

the class BaseDatarouterS3Client method scanSubdirectories.

@Override
public Scanner<DirectoryDto> scanSubdirectories(String bucket, String prefix, String startAfter, String delimiter, int pageSize, boolean currentDirectory) {
    ListObjectsV2Request request = ListObjectsV2Request.builder().bucket(bucket).prefix(prefix).startAfter(startAfter).delimiter(delimiter).maxKeys(pageSize).build();
    ListObjectsV2Iterable pages;
    try (var $ = TracerTool.startSpan("S3 listObjectsV2Paginator", TraceSpanGroupType.CLOUD_STORAGE)) {
        pages = getS3ClientForBucket(bucket).listObjectsV2Paginator(request);
    }
    return Scanner.of(pages).map(res -> {
        Scanner<DirectoryDto> objects = Scanner.of(res.contents()).map(object -> new DirectoryDto(object.key(), false, object.size(), object.lastModified(), object.storageClass().name()));
        Scanner<DirectoryDto> prefixes = Scanner.of(res.commonPrefixes()).map(commonPrefix -> new DirectoryDto(commonPrefix.prefix(), true, 0L, null, null));
        return Scanner.concat(objects, prefixes);
    }).concat(Function.identity());
}
Also used : Scanner(io.datarouter.scanner.Scanner) AmazonS3ClientBuilder(com.amazonaws.services.s3.AmazonS3ClientBuilder) GetBucketLocationRequest(software.amazon.awssdk.services.s3.model.GetBucketLocationRequest) ApacheHttpClient(software.amazon.awssdk.http.apache.ApacheHttpClient) CompletedMultipartUpload(software.amazon.awssdk.services.s3.model.CompletedMultipartUpload) URL(java.net.URL) LoggerFactory(org.slf4j.LoggerFactory) CreateMultipartUploadResponse(software.amazon.awssdk.services.s3.model.CreateMultipartUploadResponse) TransferProgress(com.amazonaws.services.s3.transfer.TransferProgress) S3Exception(software.amazon.awssdk.services.s3.model.S3Exception) ResponseTransformer(software.amazon.awssdk.core.sync.ResponseTransformer) ReaderTool(io.datarouter.util.io.ReaderTool) HeadBucketResponse(software.amazon.awssdk.services.s3.model.HeadBucketResponse) Matcher(java.util.regex.Matcher) ObjectMetadata(com.amazonaws.services.s3.model.ObjectMetadata) ListBucketsResponse(software.amazon.awssdk.services.s3.model.ListBucketsResponse) Duration(java.time.Duration) Map(java.util.Map) GetObjectRequest(software.amazon.awssdk.services.s3.model.GetObjectRequest) PutObjectRequest(software.amazon.awssdk.services.s3.model.PutObjectRequest) ResponseInputStream(software.amazon.awssdk.core.ResponseInputStream) ThreadTool(io.datarouter.util.concurrent.ThreadTool) Path(java.nio.file.Path) TracerTool(io.datarouter.instrumentation.trace.TracerTool) TransferManagerBuilder(com.amazonaws.services.s3.transfer.TransferManagerBuilder) ObjectCannedACL(software.amazon.awssdk.services.s3.model.ObjectCannedACL) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) Upload(com.amazonaws.services.s3.transfer.Upload) NoSuchBucketException(software.amazon.awssdk.services.s3.model.NoSuchBucketException) HeadBucketRequest(software.amazon.awssdk.services.s3.model.HeadBucketRequest) ResponseBytes(software.amazon.awssdk.core.ResponseBytes) CompleteMultipartUploadRequest(software.amazon.awssdk.services.s3.model.CompleteMultipartUploadRequest) HeadObjectRequest(software.amazon.awssdk.services.s3.model.HeadObjectRequest) ListObjectsV2Request(software.amazon.awssdk.services.s3.model.ListObjectsV2Request) Instant(java.time.Instant) Transfer(com.amazonaws.services.s3.transfer.Transfer) Collectors(java.util.stream.Collectors) Serializable(java.io.Serializable) List(java.util.List) DirectoryDto(io.datarouter.storage.node.op.raw.read.DirectoryDto) HeadObjectResponse(software.amazon.awssdk.services.s3.model.HeadObjectResponse) TraceSpanGroupType(io.datarouter.instrumentation.trace.TraceSpanGroupType) ListObjectsV2Iterable(software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable) Optional(java.util.Optional) Pattern(java.util.regex.Pattern) Require(io.datarouter.util.Require) UploadPartResponse(software.amazon.awssdk.services.s3.model.UploadPartResponse) Bucket(software.amazon.awssdk.services.s3.model.Bucket) TransferManager(com.amazonaws.services.s3.transfer.TransferManager) NoSuchKeyException(software.amazon.awssdk.services.s3.model.NoSuchKeyException) ByteArrayOutputStream(java.io.ByteArrayOutputStream) S3Object(software.amazon.awssdk.services.s3.model.S3Object) CopyObjectRequest(software.amazon.awssdk.services.s3.model.CopyObjectRequest) Function(java.util.function.Function) ListObjectsV2Response(software.amazon.awssdk.services.s3.model.ListObjectsV2Response) GetObjectResponse(software.amazon.awssdk.services.s3.model.GetObjectResponse) ArrayList(java.util.ArrayList) Regions(com.amazonaws.regions.Regions) Download(com.amazonaws.services.s3.transfer.Download) NumberFormatter(io.datarouter.util.number.NumberFormatter) S3Presigner(software.amazon.awssdk.services.s3.presigner.S3Presigner) GetBucketLocationResponse(software.amazon.awssdk.services.s3.model.GetBucketLocationResponse) OutputStreamWriter(java.io.OutputStreamWriter) Region(software.amazon.awssdk.regions.Region) OutputStream(java.io.OutputStream) S3ContentType(io.datarouter.aws.s3.S3Headers.S3ContentType) UploadPartRequest(software.amazon.awssdk.services.s3.model.UploadPartRequest) Logger(org.slf4j.Logger) Files(java.nio.file.Files) ContentType(io.datarouter.aws.s3.S3Headers.ContentType) BufferedWriter(java.io.BufferedWriter) S3Client(software.amazon.awssdk.services.s3.S3Client) GetObjectPresignRequest(software.amazon.awssdk.services.s3.presigner.model.GetObjectPresignRequest) IOException(java.io.IOException) CompletedPart(software.amazon.awssdk.services.s3.model.CompletedPart) CommonPrefix(software.amazon.awssdk.services.s3.model.CommonPrefix) InputStreamReader(java.io.InputStreamReader) DeleteObjectRequest(software.amazon.awssdk.services.s3.model.DeleteObjectRequest) Headers(com.amazonaws.services.s3.Headers) RequestBody(software.amazon.awssdk.core.sync.RequestBody) BufferedReader(java.io.BufferedReader) CreateMultipartUploadRequest(software.amazon.awssdk.services.s3.model.CreateMultipartUploadRequest) Comparator(java.util.Comparator) InputStream(java.io.InputStream) ByteUnitTool(io.datarouter.bytes.ByteUnitTool) Scanner(io.datarouter.scanner.Scanner) DirectoryDto(io.datarouter.storage.node.op.raw.read.DirectoryDto) ListObjectsV2Request(software.amazon.awssdk.services.s3.model.ListObjectsV2Request) ListObjectsV2Iterable(software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable)

Aggregations

Scanner (io.datarouter.scanner.Scanner)2 DirectoryDto (io.datarouter.storage.node.op.raw.read.DirectoryDto)2 NumberFormatter (io.datarouter.util.number.NumberFormatter)2 Comparator (java.util.Comparator)2 List (java.util.List)2 Regions (com.amazonaws.regions.Regions)1 AmazonS3ClientBuilder (com.amazonaws.services.s3.AmazonS3ClientBuilder)1 Headers (com.amazonaws.services.s3.Headers)1 ObjectMetadata (com.amazonaws.services.s3.model.ObjectMetadata)1 Download (com.amazonaws.services.s3.transfer.Download)1 Transfer (com.amazonaws.services.s3.transfer.Transfer)1 TransferManager (com.amazonaws.services.s3.transfer.TransferManager)1 TransferManagerBuilder (com.amazonaws.services.s3.transfer.TransferManagerBuilder)1 TransferProgress (com.amazonaws.services.s3.transfer.TransferProgress)1 Upload (com.amazonaws.services.s3.transfer.Upload)1 DatarouterS3Client (io.datarouter.aws.s3.DatarouterS3Client)1 ContentType (io.datarouter.aws.s3.S3Headers.ContentType)1 S3ContentType (io.datarouter.aws.s3.S3Headers.S3ContentType)1 S3ClientManager (io.datarouter.aws.s3.client.S3ClientManager)1 DatarouterAwsS3Paths (io.datarouter.aws.s3.config.DatarouterAwsS3Paths)1