Search in sources :

Example 1 with RDFParserSuppliers

use of org.wikidata.query.rdf.tool.rdf.RDFParserSuppliers in project wikidata-query-rdf by wikimedia.

the class StreamingUpdate method build.

static StreamingUpdaterConsumer build(StreamingUpdateOptions options, MetricRegistry metrics) {
    RDFChunkDeserializer deser = new RDFChunkDeserializer(new RDFParserSuppliers(RDFParserRegistry.getInstance()));
    KafkaStreamConsumer consumer = KafkaStreamConsumer.build(options.brokers(), options.topic(), options.partition(), options.consumerGroup(), options.batchSize(), deser, parseInitialOffset(options), KafkaStreamConsumerMetricsListener.forRegistry(metrics), options.bufferedInputMessages(), buildFilter(StreamingUpdateOptions.entityFilterPattern(options)));
    HttpClient httpClient = buildHttpClient(getHttpProxyHost(), getHttpProxyPort());
    Retryer<ContentResponse> retryer = buildHttpClientRetryer();
    Duration rdfClientTimeout = RdfRepositoryUpdater.getRdfClientTimeout();
    RdfClient rdfClient = new RdfClient(httpClient, StreamingUpdateOptions.sparqlUri(options), retryer, rdfClientTimeout);
    UrisScheme uris = UrisSchemeFactory.getURISystem();
    return new StreamingUpdaterConsumer(consumer, new RdfRepositoryUpdater(rdfClient, uris), metrics, options.inconsistenciesWarningThreshold());
}
Also used : RDFParserSuppliers(org.wikidata.query.rdf.tool.rdf.RDFParserSuppliers) ContentResponse(org.eclipse.jetty.client.api.ContentResponse) UrisScheme(org.wikidata.query.rdf.common.uri.UrisScheme) HttpClient(org.eclipse.jetty.client.HttpClient) HttpClientUtils.buildHttpClient(org.wikidata.query.rdf.tool.HttpClientUtils.buildHttpClient) RdfRepositoryUpdater(org.wikidata.query.rdf.tool.rdf.RdfRepositoryUpdater) Duration(java.time.Duration) RdfClient(org.wikidata.query.rdf.tool.rdf.client.RdfClient) RDFChunkDeserializer(org.wikidata.query.rdf.updater.RDFChunkDeserializer)

Aggregations

Duration (java.time.Duration)1 HttpClient (org.eclipse.jetty.client.HttpClient)1 ContentResponse (org.eclipse.jetty.client.api.ContentResponse)1 UrisScheme (org.wikidata.query.rdf.common.uri.UrisScheme)1 HttpClientUtils.buildHttpClient (org.wikidata.query.rdf.tool.HttpClientUtils.buildHttpClient)1 RDFParserSuppliers (org.wikidata.query.rdf.tool.rdf.RDFParserSuppliers)1 RdfRepositoryUpdater (org.wikidata.query.rdf.tool.rdf.RdfRepositoryUpdater)1 RdfClient (org.wikidata.query.rdf.tool.rdf.client.RdfClient)1 RDFChunkDeserializer (org.wikidata.query.rdf.updater.RDFChunkDeserializer)1