Search in sources :

Example 1 with ClientId

use of io.datarouter.storage.client.ClientId in project datarouter by hotpads.

the class S3BucketHandler method countObjects.

@Handler
public Mav countObjects(@Param(P_client) String client, @Param(P_bucket) String bucket, @Param(P_prefix) OptionalString prefix) {
    ClientId clientId = clients.getClientId(client);
    DatarouterS3Client s3Client = s3ClientManager.getClient(clientId);
    var count = new AtomicLong();
    var size = new AtomicLong();
    var message = new AtomicReference<String>();
    s3Client.scanObjects(bucket, prefix.orElse("")).each($ -> count.incrementAndGet()).each(obj -> size.addAndGet(obj.size())).sample(10_000, true).map(obj -> String.format("client=%s, bucket=%s, prefix=%s, count=%s, size=%s, through=%s", client, bucket, prefix.orElse(null), NumberFormatter.addCommas(count.get()), NumberFormatter.addCommas(size.get()), obj.key())).each(message::set).forEach(logger::warn);
    return pageFactory.message(request, message.get());
}
Also used : Scanner(io.datarouter.scanner.Scanner) DatarouterS3Client(io.datarouter.aws.s3.DatarouterS3Client) LoggerFactory(org.slf4j.LoggerFactory) TagCreator.h4(j2html.TagCreator.h4) OptionalString(io.datarouter.web.handler.types.optional.OptionalString) AtomicReference(java.util.concurrent.atomic.AtomicReference) S3ClientManager(io.datarouter.aws.s3.client.S3ClientManager) Inject(javax.inject.Inject) NumberFormatter(io.datarouter.util.number.NumberFormatter) DatarouterWebRequireJsV2(io.datarouter.web.requirejs.DatarouterWebRequireJsV2) ClientId(io.datarouter.storage.client.ClientId) Param(io.datarouter.web.handler.types.Param) Bootstrap4FormHtml(io.datarouter.web.html.j2html.bootstrap4.Bootstrap4FormHtml) OptionalInteger(io.datarouter.web.handler.types.optional.OptionalInteger) J2HtmlTable(io.datarouter.web.html.j2html.J2HtmlTable) DatarouterClients(io.datarouter.storage.client.DatarouterClients) TagCreator.rawHtml(j2html.TagCreator.rawHtml) Logger(org.slf4j.Logger) URIBuilder(org.apache.http.client.utils.URIBuilder) DatarouterAwsS3Paths(io.datarouter.aws.s3.config.DatarouterAwsS3Paths) Mav(io.datarouter.web.handler.mav.Mav) StringTool(io.datarouter.util.string.StringTool) TagCreator.a(j2html.TagCreator.a) ContainerTag(j2html.tags.ContainerTag) OptionalBoolean(io.datarouter.web.handler.types.optional.OptionalBoolean) AtomicLong(java.util.concurrent.atomic.AtomicLong) List(java.util.List) DirectoryDto(io.datarouter.storage.node.op.raw.read.DirectoryDto) HtmlForm(io.datarouter.web.html.form.HtmlForm) TagCreator.td(j2html.TagCreator.td) BaseHandler(io.datarouter.web.handler.BaseHandler) Bootstrap4PageFactory(io.datarouter.web.html.j2html.bootstrap4.Bootstrap4PageFactory) Comparator(java.util.Comparator) TagCreator.div(j2html.TagCreator.div) AtomicLong(java.util.concurrent.atomic.AtomicLong) DatarouterS3Client(io.datarouter.aws.s3.DatarouterS3Client) ClientId(io.datarouter.storage.client.ClientId) AtomicReference(java.util.concurrent.atomic.AtomicReference) BaseHandler(io.datarouter.web.handler.BaseHandler)

Example 2 with ClientId

use of io.datarouter.storage.client.ClientId in project datarouter by hotpads.

the class S3BucketHandler method index.

@Handler(defaultHandler = true)
public Mav index(@Param(P_client) String client, @Param(P_bucket) String bucket, @Param(P_prefix) OptionalString prefix, @Param(P_after) OptionalString after, @Param(P_offset) OptionalInteger offset, @Param(P_limit) OptionalInteger limit, @Param(P_currentDirectory) OptionalBoolean currentDirectory, @Param(P_delimiter) OptionalString delimiter) {
    var form = new HtmlForm().withMethod("get");
    form.addTextField().withDisplay("Client").withName(P_client).withPlaceholder("theClientName").withValue(client);
    form.addTextField().withDisplay("Bucket").withName(P_bucket).withPlaceholder("the.bucket.name").withValue(bucket);
    form.addTextField().withDisplay("Prefix").withName(P_prefix).withValue(prefix.orElse(""));
    form.addTextField().withDisplay("After").withName(P_after).withValue(after.orElse(""));
    form.addTextField().withDisplay("Offset").withName(P_offset).withValue(offset.orElse(0) + "");
    form.addTextField().withDisplay("Limit").withName(P_limit).withValue(limit.orElse(100) + "");
    form.addTextField().withDisplay("Delimiter").withName(P_delimiter).withValue(delimiter.orElse(""));
    form.addCheckboxField().withDisplay("currentDirectory").withName(P_currentDirectory).withChecked(currentDirectory.orElse(false));
    form.addButton().withDisplay("Submit").withValue("");
    var htmlForm = Bootstrap4FormHtml.render(form).withClass("card card-body bg-light");
    ClientId clientId = clients.getClientId(client);
    DatarouterS3Client s3Client = s3ClientManager.getClient(clientId);
    List<DirectoryDto> objects = s3Client.scanSubdirectories(bucket, prefix.orElse(null), after.orElse(null), delimiter.orElse(null), limit.orElse(100), currentDirectory.orElse(false)).list();
    int sizePadding = sizePadding(objects);
    ContainerTag<?> table = new J2HtmlTable<DirectoryDto>().withClasses("sortable table table-sm table-striped my-4 border").withHtmlColumn("Key", object -> {
        String name = object.name;
        if (object.isDirectory) {
            return td(makePrefixLink(client, bucket, name, "/"));
        }
        return td(name);
    }).withHtmlColumn("Directory", object -> {
        boolean isDirectory = object.isDirectory;
        if (isDirectory) {
            String href = new URIBuilder().setPath(request.getContextPath() + paths.datarouter.clients.awsS3.countObjects.toSlashedString()).addParameter(P_client, client).addParameter(P_bucket, bucket).addParameter(P_prefix, object.name).toString();
            return td(a("true, view count").withHref(href));
        }
        return td(String.valueOf(isDirectory));
    }).withHtmlColumn("Size", object -> {
        String commas = NumberFormatter.addCommas(object.size);
        String padded = StringTool.pad(commas, ' ', sizePadding);
        String escaped = padded.replaceAll(" ", "&nbsp;");
        return td(rawHtml(escaped));
    }).withColumn("Last Modified", object -> object.lastModified).withColumn("Storage Class", object -> object.storageClass).build(objects);
    ContainerTag<?> tableWrapper = table.withStyle("font-family:monospace; font-size:.9em;");
    var content = div(htmlForm, h4(bucket), tableWrapper).withClass("container-fluid my-4");
    return pageFactory.startBuilder(request).withTitle("S3 Bucket").withRequires(DatarouterWebRequireJsV2.SORTTABLE).withContent(content).buildMav();
}
Also used : Scanner(io.datarouter.scanner.Scanner) DatarouterS3Client(io.datarouter.aws.s3.DatarouterS3Client) LoggerFactory(org.slf4j.LoggerFactory) TagCreator.h4(j2html.TagCreator.h4) OptionalString(io.datarouter.web.handler.types.optional.OptionalString) AtomicReference(java.util.concurrent.atomic.AtomicReference) S3ClientManager(io.datarouter.aws.s3.client.S3ClientManager) Inject(javax.inject.Inject) NumberFormatter(io.datarouter.util.number.NumberFormatter) DatarouterWebRequireJsV2(io.datarouter.web.requirejs.DatarouterWebRequireJsV2) ClientId(io.datarouter.storage.client.ClientId) Param(io.datarouter.web.handler.types.Param) Bootstrap4FormHtml(io.datarouter.web.html.j2html.bootstrap4.Bootstrap4FormHtml) OptionalInteger(io.datarouter.web.handler.types.optional.OptionalInteger) J2HtmlTable(io.datarouter.web.html.j2html.J2HtmlTable) DatarouterClients(io.datarouter.storage.client.DatarouterClients) TagCreator.rawHtml(j2html.TagCreator.rawHtml) Logger(org.slf4j.Logger) URIBuilder(org.apache.http.client.utils.URIBuilder) DatarouterAwsS3Paths(io.datarouter.aws.s3.config.DatarouterAwsS3Paths) Mav(io.datarouter.web.handler.mav.Mav) StringTool(io.datarouter.util.string.StringTool) TagCreator.a(j2html.TagCreator.a) ContainerTag(j2html.tags.ContainerTag) OptionalBoolean(io.datarouter.web.handler.types.optional.OptionalBoolean) AtomicLong(java.util.concurrent.atomic.AtomicLong) List(java.util.List) DirectoryDto(io.datarouter.storage.node.op.raw.read.DirectoryDto) HtmlForm(io.datarouter.web.html.form.HtmlForm) TagCreator.td(j2html.TagCreator.td) BaseHandler(io.datarouter.web.handler.BaseHandler) Bootstrap4PageFactory(io.datarouter.web.html.j2html.bootstrap4.Bootstrap4PageFactory) Comparator(java.util.Comparator) TagCreator.div(j2html.TagCreator.div) OptionalString(io.datarouter.web.handler.types.optional.OptionalString) URIBuilder(org.apache.http.client.utils.URIBuilder) DirectoryDto(io.datarouter.storage.node.op.raw.read.DirectoryDto) HtmlForm(io.datarouter.web.html.form.HtmlForm) DatarouterS3Client(io.datarouter.aws.s3.DatarouterS3Client) ClientId(io.datarouter.storage.client.ClientId) BaseHandler(io.datarouter.web.handler.BaseHandler)

Example 3 with ClientId

use of io.datarouter.storage.client.ClientId in project datarouter by hotpads.

the class AuroraInstancesHandler method inspectClientUrl.

@Handler(defaultHandler = true)
public Mav inspectClientUrl() {
    List<DnsHostEntryDto> otherReaderInstances = new ArrayList<>();
    List<ClientId> clientsMissingOtherInstances = new ArrayList<>();
    for (ClientId primaryClientId : dnsService.getPrimaryClientIds()) {
        DnsHostEntryDto otherEntry = dnsService.getOtherReader(primaryClientId.getName());
        if (otherEntry == null) {
            clientsMissingOtherInstances.add(primaryClientId);
        } else {
            otherReaderInstances.add(otherEntry);
        }
    }
    List<DomContent> fragments = new ArrayList<>();
    fragments.add(makeAuroraClientsTable("Aurora Clients", dnsService.checkClientEndpoint().getLeft()));
    if (otherReaderInstances.size() != 0) {
        fragments.add(makeAuroraClientsTable("Aurora Other Instances", otherReaderInstances));
    }
    if (clientsMissingOtherInstances.size() != 0) {
        fragments.add(makeForm(clientsMissingOtherInstances));
    }
    ContainerTag<?> content = div(each(fragments.stream())).withClass("container my-4");
    return pageFactory.startBuilder(request).withTitle("Aurora Clients").withContent(content).buildMav();
}
Also used : ArrayList(java.util.ArrayList) ClientId(io.datarouter.storage.client.ClientId) DomContent(j2html.tags.DomContent) DnsHostEntryDto(io.datarouter.aws.rds.service.AuroraDnsService.DnsHostEntryDto) BaseHandler(io.datarouter.web.handler.BaseHandler)

Example 4 with ClientId

use of io.datarouter.storage.client.ClientId in project datarouter by hotpads.

the class AwsMemcachedWebInspector method getDetails.

@Override
protected Pair<Integer, DivTag> getDetails(ClientId clientId) {
    MemcachedClientMode mode = options.getClientMode(clientId.getName());
    Pair<Integer, DivTag> nodeCountByNodeTag = new Pair<>();
    if (mode == MemcachedClientMode.DYNAMIC) {
        List<AwsMemcachedNodeEndpointDto> nodeEndpointDtos = Scanner.of(getSpyClient(clientId).getAllNodeEndPoints()).map(nodeEndPoint -> new AwsMemcachedNodeEndpointDto(nodeEndPoint.getHostName(), nodeEndPoint.getIpAddress(), nodeEndPoint.getPort())).list();
        var table = new J2HtmlTable<AwsMemcachedNodeEndpointDto>().withClasses("sortable table table-sm table-striped my-4 border").withColumn("HostName", dto -> dto.hostName).withColumn("IpAddress", dto -> dto.ipAddress).withColumn("Port", dto -> dto.port).build(nodeEndpointDtos);
        DivTag divTable = div(table).withClass("container-fluid my-4").withStyle("padding-left: 0px");
        nodeCountByNodeTag.setLeft(nodeEndpointDtos.size());
        nodeCountByNodeTag.setRight(divTable);
    } else {
        List<LiTag> socketAddresses = Scanner.of(getSpyClient(clientId).getAvailableServers()).map(Object::toString).map(TagCreator::li).list();
        DivTag div = div(ul(socketAddresses.toArray(new ContainerTag[0])));
        nodeCountByNodeTag.setLeft(socketAddresses.size());
        nodeCountByNodeTag.setRight(div);
    }
    return nodeCountByNodeTag;
}
Also used : Scanner(io.datarouter.scanner.Scanner) MemcachedClientMode(io.datarouter.aws.memcached.client.MemcachedClientMode) DivTag(j2html.tags.specialized.DivTag) LiTag(j2html.tags.specialized.LiTag) MemcachedWebInspector(io.datarouter.client.memcached.web.MemcachedWebInspector) ContainerTag(j2html.tags.ContainerTag) Inject(javax.inject.Inject) List(java.util.List) ClientId(io.datarouter.storage.client.ClientId) Pair(io.datarouter.util.tuple.Pair) TagCreator.div(j2html.TagCreator.div) AwsMemcachedOptions(io.datarouter.aws.memcached.client.options.AwsMemcachedOptions) J2HtmlTable(io.datarouter.web.html.j2html.J2HtmlTable) TagCreator(j2html.TagCreator) TagCreator.ul(j2html.TagCreator.ul) DivTag(j2html.tags.specialized.DivTag) LiTag(j2html.tags.specialized.LiTag) MemcachedClientMode(io.datarouter.aws.memcached.client.MemcachedClientMode) Pair(io.datarouter.util.tuple.Pair)

Example 5 with ClientId

use of io.datarouter.storage.client.ClientId in project datarouter by hotpads.

the class SessionExecutor method run.

public <T> T run(BaseMysqlOp<T> parallelTxnOp, String traceName) throws SessionExecutorPleaseRetryException {
    ClientId clientId = parallelTxnOp.getClientId();
    TxnClientManager clientManager = (TxnClientManager) datarouterClients.getClientManager(clientId);
    try {
        startTrace(traceName);
        clientManager.reserveConnection(clientId);
        return innerRun(clientId, clientManager, parallelTxnOp);
    } finally {
        finishTrace(traceName);
    }
}
Also used : TxnClientManager(io.datarouter.client.mysql.TxnClientManager) ClientId(io.datarouter.storage.client.ClientId)

Aggregations

ClientId (io.datarouter.storage.client.ClientId)21 List (java.util.List)10 Inject (javax.inject.Inject)9 Scanner (io.datarouter.scanner.Scanner)7 BaseHandler (io.datarouter.web.handler.BaseHandler)7 TagCreator.div (j2html.TagCreator.div)6 ContainerTag (j2html.tags.ContainerTag)6 ArrayList (java.util.ArrayList)6 Optional (java.util.Optional)6 DatarouterChangelogDtoBuilder (io.datarouter.instrumentation.changelog.ChangelogRecorder.DatarouterChangelogDtoBuilder)4 DatarouterClients (io.datarouter.storage.client.DatarouterClients)4 PhysicalNode (io.datarouter.storage.node.type.physical.PhysicalNode)4 Mav (io.datarouter.web.handler.mav.Mav)4 J2HtmlTable (io.datarouter.web.html.j2html.J2HtmlTable)4 Bootstrap4PageFactory (io.datarouter.web.html.j2html.bootstrap4.Bootstrap4PageFactory)4 DatarouterWebRequireJsV2 (io.datarouter.web.requirejs.DatarouterWebRequireJsV2)4 Collections (java.util.Collections)4 Map (java.util.Map)4 Logger (org.slf4j.Logger)4 LoggerFactory (org.slf4j.LoggerFactory)4