Search in sources :

Example 1 with ClientCertConfig

use of com.couchbase.connector.config.common.ClientCertConfig in project couchbase-elasticsearch-connector by couchbase.

the class ElasticsearchConfig method from.

static ImmutableElasticsearchConfig from(ConfigTable config) {
    config.expectOnly("hosts", "username", "pathToPassword", "secureConnection", "clientCertificate", "aws", "bulkRequestLimits", "docStructure", "typeDefaults", "type", "rejectionLog");
    final boolean secureConnection = config.getBoolean("secureConnection").orElse(false);
    final AwsConfig aws = AwsConfig.from(config.getTableOrEmpty("aws"));
    final ClientCertConfig clientCert = ClientCertConfig.from(config.getTableOrEmpty("clientCertificate"), "elasticsearch.clientCertificate");
    // Standard ES HTTP port is 9200. Amazon Elasticsearch Service listens on ports 80 & 443.
    final int defaultPort = aws.region().isEmpty() ? 9200 : secureConnection ? 443 : 80;
    final ImmutableElasticsearchConfig.Builder builder = ImmutableElasticsearchConfig.builder().secureConnection(secureConnection).hosts(config.getRequiredStrings("hosts").stream().map(h -> createHttpHost(h, defaultPort, secureConnection)).collect(toList())).username(config.getString("username").orElse("")).password(readPassword(config, "elasticsearch", "pathToPassword")).bulkRequest(BulkRequestConfig.from(config.getTableOrEmpty("bulkRequestLimits"))).aws(aws).clientCert(clientCert).docStructure(DocStructureConfig.from(config.getTableOrEmpty("docStructure")));
    final ConfigTable typeDefaults = config.getTableOrEmpty("typeDefaults");
    typeDefaults.expectOnly("typeName", "index", "pipeline", "ignore", "ignoreDeletes", "matchOnQualifiedKey");
    final TypeConfig defaultTypeConfig = ImmutableTypeConfig.builder().index(typeDefaults.getString("index").orElse(null)).type(typeDefaults.getString("typeName").orElse("_doc")).pipeline(typeDefaults.getString("pipeline").orElse(null)).ignore(typeDefaults.getBoolean("ignore").orElse(false)).ignoreDeletes(typeDefaults.getBoolean("ignoreDeletes").orElse(false)).matchOnQualifiedKey(typeDefaults.getBoolean("matchOnQualifiedKey").orElse(false)).matcher(s -> null).build();
    ImmutableList.Builder<TypeConfig> typeConfigs = ImmutableList.builder();
    ConfigArray types = config.getArrayOrEmpty("type");
    for (int i = 0; i < types.size(); i++) {
        typeConfigs.add(TypeConfig.from(types.getTable(i), types.inputPositionOf(i), defaultTypeConfig));
    }
    builder.types(typeConfigs.build());
    builder.rejectLog(RejectLogConfig.from(config.getTableOrEmpty("rejectionLog"), defaultTypeConfig.type()));
    return builder.build();
}
Also used : ConfigTable(com.couchbase.connector.config.toml.ConfigTable) ConfigArray(com.couchbase.connector.config.toml.ConfigArray) Collectors.toList(java.util.stream.Collectors.toList) ImmutableList(com.google.common.collect.ImmutableList) Value(org.immutables.value.Value) ConfigHelper.readPassword(com.couchbase.connector.config.ConfigHelper.readPassword) ConfigHelper.createHttpHost(com.couchbase.connector.config.ConfigHelper.createHttpHost) ClientCertConfig(com.couchbase.connector.config.common.ClientCertConfig) ConfigException(com.couchbase.connector.config.ConfigException) HttpHost(org.apache.http.HttpHost) ClientCertConfig(com.couchbase.connector.config.common.ClientCertConfig) ImmutableList(com.google.common.collect.ImmutableList) ConfigArray(com.couchbase.connector.config.toml.ConfigArray) ConfigTable(com.couchbase.connector.config.toml.ConfigTable)

Example 2 with ClientCertConfig

use of com.couchbase.connector.config.common.ClientCertConfig in project couchbase-elasticsearch-connector by couchbase.

the class ElasticsearchHelper method newElasticsearchClient.

public static RestHighLevelClient newElasticsearchClient(List<HttpHost> hosts, String username, String password, boolean secureConnection, Supplier<KeyStore> trustStore, ClientCertConfig clientCert, AwsConfig aws, TimeValue bulkRequestTimeout) throws Exception {
    final int connectTimeoutMillis = (int) SECONDS.toMillis(5);
    final int socketTimeoutMillis = (int) Math.max(SECONDS.toMillis(60), toMillis(bulkRequestTimeout) + SECONDS.toMillis(3));
    LOGGER.info("Elasticsearch client connect timeout = {}ms; socket timeout={}ms", connectTimeoutMillis, socketTimeoutMillis);
    final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
    credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(username, password));
    final SSLContext sslContext = !secureConnection ? null : newSslContext(trustStore.get(), clientCert);
    final RestClientBuilder builder = RestClient.builder(Iterables.toArray(hosts, HttpHost.class)).setHttpClientConfigCallback(httpClientBuilder -> {
        httpClientBuilder.setSSLContext(sslContext);
        if (!clientCert.use()) {
            httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
        }
        awsSigner(aws).ifPresent(httpClientBuilder::addInterceptorLast);
        return httpClientBuilder;
    }).setRequestConfigCallback(requestConfigBuilder -> requestConfigBuilder.setConnectTimeout(connectTimeoutMillis).setSocketTimeout(socketTimeoutMillis)).setFailureListener(new RestClient.FailureListener() {

        @Override
        public void onFailure(Node host) {
            Metrics.elasticsearchHostOffline().increment();
        }
    });
    return new RestHighLevelClient(builder);
}
Also used : RestClient(org.elasticsearch.client.RestClient) Iterables(com.google.common.collect.Iterables) SSLContext(javax.net.ssl.SSLContext) Arrays(java.util.Arrays) RestClientBuilder(org.elasticsearch.client.RestClientBuilder) DefaultAWSCredentialsProviderChain(com.amazonaws.auth.DefaultAWSCredentialsProviderChain) LoggerFactory(org.slf4j.LoggerFactory) Node(org.elasticsearch.client.Node) Supplier(java.util.function.Supplier) BasicCredentialsProvider(org.apache.http.impl.client.BasicCredentialsProvider) Matcher(java.util.regex.Matcher) AWSRequestSigningApacheInterceptor(com.amazonaws.http.AWSRequestSigningApacheInterceptor) SSLContexts(org.apache.http.ssl.SSLContexts) TimeValue(org.elasticsearch.common.unit.TimeValue) AWS4Signer(com.amazonaws.auth.AWS4Signer) TrustStoreConfig(com.couchbase.connector.config.common.TrustStoreConfig) HttpRequestInterceptor(org.apache.http.HttpRequestInterceptor) Logger(org.slf4j.Logger) Iterator(java.util.Iterator) BulkItemResponse(org.elasticsearch.action.bulk.BulkItemResponse) KeyStore(java.security.KeyStore) Version(com.couchbase.client.dcp.util.Version) MILLISECONDS(java.util.concurrent.TimeUnit.MILLISECONDS) RestHighLevelClient(org.elasticsearch.client.RestHighLevelClient) SSLContextBuilder(org.apache.http.ssl.SSLContextBuilder) UsernamePasswordCredentials(org.apache.http.auth.UsernamePasswordCredentials) List(java.util.List) ElasticsearchConfig(com.couchbase.connector.config.es.ElasticsearchConfig) AuthScope(org.apache.http.auth.AuthScope) ClientCertConfig(com.couchbase.connector.config.common.ClientCertConfig) ThrowableHelper(com.couchbase.connector.util.ThrowableHelper) Optional(java.util.Optional) ConnectionClosedException(org.apache.http.ConnectionClosedException) Pattern(java.util.regex.Pattern) CredentialsProvider(org.apache.http.client.CredentialsProvider) StringUtils.substringBefore(org.apache.commons.lang3.StringUtils.substringBefore) HttpHost(org.apache.http.HttpHost) SECONDS(java.util.concurrent.TimeUnit.SECONDS) AwsConfig(com.couchbase.connector.config.es.AwsConfig) MoreBackoffPolicies.truncatedExponentialBackoff(com.couchbase.connector.elasticsearch.io.MoreBackoffPolicies.truncatedExponentialBackoff) BasicCredentialsProvider(org.apache.http.impl.client.BasicCredentialsProvider) Node(org.elasticsearch.client.Node) RestClient(org.elasticsearch.client.RestClient) RestClientBuilder(org.elasticsearch.client.RestClientBuilder) BasicCredentialsProvider(org.apache.http.impl.client.BasicCredentialsProvider) CredentialsProvider(org.apache.http.client.CredentialsProvider) SSLContext(javax.net.ssl.SSLContext) RestHighLevelClient(org.elasticsearch.client.RestHighLevelClient) UsernamePasswordCredentials(org.apache.http.auth.UsernamePasswordCredentials)

Aggregations

ClientCertConfig (com.couchbase.connector.config.common.ClientCertConfig)2 HttpHost (org.apache.http.HttpHost)2 AWS4Signer (com.amazonaws.auth.AWS4Signer)1 DefaultAWSCredentialsProviderChain (com.amazonaws.auth.DefaultAWSCredentialsProviderChain)1 AWSRequestSigningApacheInterceptor (com.amazonaws.http.AWSRequestSigningApacheInterceptor)1 Version (com.couchbase.client.dcp.util.Version)1 ConfigException (com.couchbase.connector.config.ConfigException)1 ConfigHelper.createHttpHost (com.couchbase.connector.config.ConfigHelper.createHttpHost)1 ConfigHelper.readPassword (com.couchbase.connector.config.ConfigHelper.readPassword)1 TrustStoreConfig (com.couchbase.connector.config.common.TrustStoreConfig)1 AwsConfig (com.couchbase.connector.config.es.AwsConfig)1 ElasticsearchConfig (com.couchbase.connector.config.es.ElasticsearchConfig)1 ConfigArray (com.couchbase.connector.config.toml.ConfigArray)1 ConfigTable (com.couchbase.connector.config.toml.ConfigTable)1 MoreBackoffPolicies.truncatedExponentialBackoff (com.couchbase.connector.elasticsearch.io.MoreBackoffPolicies.truncatedExponentialBackoff)1 ThrowableHelper (com.couchbase.connector.util.ThrowableHelper)1 ImmutableList (com.google.common.collect.ImmutableList)1 Iterables (com.google.common.collect.Iterables)1 KeyStore (java.security.KeyStore)1 Arrays (java.util.Arrays)1