Search in sources :

Example 1 with Prime

use of com.datastax.oss.simulacron.common.stubbing.Prime in project dsbulk by datastax.

the class JsonEndToEndSimulacronIT method load_errors.

@Test
void load_errors() throws Exception {
    primeIpByCountryTable(simulacron);
    Map<String, Object> params = new HashMap<>();
    params.put("country_name", "Sweden");
    RequestPrime prime1 = createParameterizedQuery(INSERT_INTO_IP_BY_COUNTRY, params, new SuccessResult(emptyList(), new LinkedHashMap<>()));
    simulacron.prime(new Prime(prime1));
    // recoverable errors only
    params.put("country_name", "France");
    prime1 = createParameterizedQuery(INSERT_INTO_IP_BY_COUNTRY, params, new ReadTimeoutResult(LOCAL_ONE, 1, 0, false));
    simulacron.prime(new Prime(prime1));
    params.put("country_name", "Gregistan");
    prime1 = createParameterizedQuery(INSERT_INTO_IP_BY_COUNTRY, params, new WriteTimeoutResult(LOCAL_ONE, 0, 0, WriteType.BATCH_LOG));
    simulacron.prime(new Prime(prime1));
    params.put("country_name", "Andybaijan");
    prime1 = createParameterizedQuery(INSERT_INTO_IP_BY_COUNTRY, params, new WriteFailureResult(ONE, 0, 0, emptyMap(), WriteType.BATCH));
    simulacron.prime(new Prime(prime1));
    params = new HashMap<>();
    params.put("country_name", "United States");
    prime1 = createParameterizedQuery(INSERT_INTO_IP_BY_COUNTRY, params, new FunctionFailureResult("keyspace", "function", emptyList(), "bad function call"));
    simulacron.prime(new Prime(prime1));
    String[] args = { "load", "-c", "json", "--connector.json.url", StringUtils.quoteJson(JsonUtils.JSON_RECORDS_ERROR), "--driver.advanced.retry-policy.max-retries", "1", "--schema.keyspace", "ks1", "--schema.query", INSERT_INTO_IP_BY_COUNTRY, "--schema.mapping", IP_BY_COUNTRY_MAPPING_NAMED };
    ExitStatus status = new DataStaxBulkLoader(addCommonSettings(args)).run();
    assertStatus(status, STATUS_COMPLETED_WITH_ERRORS);
    // There are 24 rows of data, but two extra queries due to the retry for the write timeout and
    // the unavailable.
    validateQueryCount(simulacron, 26, "INSERT INTO ip_by_country", LOCAL_ONE);
    validateNumberOfBadRecords(4);
    validateExceptionsLog(4, "Source:", "load-errors.log");
}
Also used : Prime(com.datastax.oss.simulacron.common.stubbing.Prime) RequestPrime(com.datastax.oss.simulacron.common.cluster.RequestPrime) ReadTimeoutResult(com.datastax.oss.simulacron.common.result.ReadTimeoutResult) HashMap(java.util.HashMap) LinkedHashMap(java.util.LinkedHashMap) FunctionFailureResult(com.datastax.oss.simulacron.common.result.FunctionFailureResult) SuccessResult(com.datastax.oss.simulacron.common.result.SuccessResult) DataStaxBulkLoader(com.datastax.oss.dsbulk.runner.DataStaxBulkLoader) LinkedHashMap(java.util.LinkedHashMap) ExitStatus(com.datastax.oss.dsbulk.runner.ExitStatus) WriteFailureResult(com.datastax.oss.simulacron.common.result.WriteFailureResult) RequestPrime(com.datastax.oss.simulacron.common.cluster.RequestPrime) WriteTimeoutResult(com.datastax.oss.simulacron.common.result.WriteTimeoutResult) Test(org.junit.jupiter.api.Test) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest)

Example 2 with Prime

use of com.datastax.oss.simulacron.common.stubbing.Prime in project dsbulk by datastax.

the class JsonEndToEndSimulacronIT method unload_failure_during_read_multi_thread.

@Test
void unload_failure_during_read_multi_thread() {
    primeIpByCountryTable(simulacron);
    RequestPrime prime = createQueryWithError(SELECT_FROM_IP_BY_COUNTRY, new SyntaxErrorResult("Invalid table", 0L, false));
    simulacron.prime(new Prime(prime));
    String[] args = { "unload", "-c", "json", "--connector.json.url", StringUtils.quoteJson(unloadDir), "--connector.json.maxConcurrentFiles", "4", "--schema.keyspace", "ks1", "--schema.query", SELECT_FROM_IP_BY_COUNTRY, "--schema.mapping", IP_BY_COUNTRY_MAPPING_NAMED };
    ExitStatus status = new DataStaxBulkLoader(addCommonSettings(args)).run();
    assertStatus(status, STATUS_ABORTED_FATAL_ERROR);
    validateQueryCount(simulacron, 0, SELECT_FROM_IP_BY_COUNTRY, LOCAL_ONE);
    validatePrepare(simulacron, SELECT_FROM_IP_BY_COUNTRY);
}
Also used : ExitStatus(com.datastax.oss.dsbulk.runner.ExitStatus) Prime(com.datastax.oss.simulacron.common.stubbing.Prime) RequestPrime(com.datastax.oss.simulacron.common.cluster.RequestPrime) RequestPrime(com.datastax.oss.simulacron.common.cluster.RequestPrime) SyntaxErrorResult(com.datastax.oss.simulacron.common.result.SyntaxErrorResult) DataStaxBulkLoader(com.datastax.oss.dsbulk.runner.DataStaxBulkLoader) Test(org.junit.jupiter.api.Test) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest)

Example 3 with Prime

use of com.datastax.oss.simulacron.common.stubbing.Prime in project dsbulk by datastax.

the class JsonEndToEndSimulacronIT method full_load_crlf.

@Test
void full_load_crlf() {
    primeIpByCountryTable(simulacron);
    RequestPrime insert = createSimpleParameterizedQuery(INSERT_INTO_IP_BY_COUNTRY);
    simulacron.prime(new Prime(insert));
    String[] args = { "load", "-c", "json", "--connector.json.url", StringUtils.quoteJson(JsonUtils.JSON_RECORDS_CRLF), "--schema.keyspace", "ks1", "--schema.query", INSERT_INTO_IP_BY_COUNTRY, "--schema.mapping", IP_BY_COUNTRY_MAPPING_NAMED };
    ExitStatus status = new DataStaxBulkLoader(addCommonSettings(args)).run();
    assertStatus(status, STATUS_OK);
    validateQueryCount(simulacron, 24, "INSERT INTO ip_by_country", LOCAL_ONE);
}
Also used : ExitStatus(com.datastax.oss.dsbulk.runner.ExitStatus) Prime(com.datastax.oss.simulacron.common.stubbing.Prime) RequestPrime(com.datastax.oss.simulacron.common.cluster.RequestPrime) RequestPrime(com.datastax.oss.simulacron.common.cluster.RequestPrime) DataStaxBulkLoader(com.datastax.oss.dsbulk.runner.DataStaxBulkLoader) Test(org.junit.jupiter.api.Test) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest)

Example 4 with Prime

use of com.datastax.oss.simulacron.common.stubbing.Prime in project dsbulk by datastax.

the class JsonEndToEndSimulacronIT method partial_load.

@Test
void partial_load() throws Exception {
    primeIpByCountryTable(simulacron);
    RequestPrime insert = createSimpleParameterizedQuery(INSERT_INTO_IP_BY_COUNTRY);
    simulacron.prime(new Prime(insert));
    String[] args = { "load", "-c", "json", "--connector.json.url", StringUtils.quoteJson(JsonUtils.JSON_RECORDS_PARTIAL_BAD), "--schema.keyspace", "ks1", "--schema.query", INSERT_INTO_IP_BY_COUNTRY, "--schema.mapping", IP_BY_COUNTRY_MAPPING_NAMED, "--schema.allowMissingFields", "true" };
    ExitStatus status = new DataStaxBulkLoader(addCommonSettings(args)).run();
    assertStatus(status, STATUS_COMPLETED_WITH_ERRORS);
    validateQueryCount(simulacron, 21, "INSERT INTO ip_by_country", LOCAL_ONE);
    validateNumberOfBadRecords(2);
    validateExceptionsLog(2, "Source:", "mapping-errors.log");
}
Also used : ExitStatus(com.datastax.oss.dsbulk.runner.ExitStatus) Prime(com.datastax.oss.simulacron.common.stubbing.Prime) RequestPrime(com.datastax.oss.simulacron.common.cluster.RequestPrime) RequestPrime(com.datastax.oss.simulacron.common.cluster.RequestPrime) DataStaxBulkLoader(com.datastax.oss.dsbulk.runner.DataStaxBulkLoader) Test(org.junit.jupiter.api.Test) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest)

Example 5 with Prime

use of com.datastax.oss.simulacron.common.stubbing.Prime in project dsbulk by datastax.

the class JsonEndToEndSimulacronIT method full_load.

@Test
void full_load() {
    primeIpByCountryTable(simulacron);
    RequestPrime insert = createSimpleParameterizedQuery(INSERT_INTO_IP_BY_COUNTRY);
    simulacron.prime(new Prime(insert));
    String[] args = { "load", "-c", "json", "--connector.json.url", StringUtils.quoteJson(JsonUtils.JSON_RECORDS_UNIQUE), "--schema.keyspace", "ks1", "--schema.query", INSERT_INTO_IP_BY_COUNTRY, "--schema.mapping", IP_BY_COUNTRY_MAPPING_NAMED };
    ExitStatus status = new DataStaxBulkLoader(addCommonSettings(args)).run();
    assertStatus(status, STATUS_OK);
    assertThat(logs.getAllMessagesAsString()).contains("Records: total: 24, successful: 24, failed: 0").contains("Batches: total: 24, size: 1.00 mean, 1 min, 1 max").contains("Writes: total: 24, successful: 24, failed: 0");
    validateQueryCount(simulacron, 24, "INSERT INTO ip_by_country", LOCAL_ONE);
}
Also used : ExitStatus(com.datastax.oss.dsbulk.runner.ExitStatus) Prime(com.datastax.oss.simulacron.common.stubbing.Prime) RequestPrime(com.datastax.oss.simulacron.common.cluster.RequestPrime) RequestPrime(com.datastax.oss.simulacron.common.cluster.RequestPrime) DataStaxBulkLoader(com.datastax.oss.dsbulk.runner.DataStaxBulkLoader) Test(org.junit.jupiter.api.Test) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest)

Aggregations

RequestPrime (com.datastax.oss.simulacron.common.cluster.RequestPrime)32 Prime (com.datastax.oss.simulacron.common.stubbing.Prime)32 DataStaxBulkLoader (com.datastax.oss.dsbulk.runner.DataStaxBulkLoader)30 ExitStatus (com.datastax.oss.dsbulk.runner.ExitStatus)30 ParameterizedTest (org.junit.jupiter.params.ParameterizedTest)30 Test (org.junit.jupiter.api.Test)28 SyntaxErrorResult (com.datastax.oss.simulacron.common.result.SyntaxErrorResult)4 SuccessResult (com.datastax.oss.simulacron.common.result.SuccessResult)3 HashMap (java.util.HashMap)3 LinkedHashMap (java.util.LinkedHashMap)3 Record (com.datastax.oss.dsbulk.connectors.api.Record)2 Query (com.datastax.oss.simulacron.common.request.Query)2 ErrorResult (com.datastax.oss.simulacron.common.result.ErrorResult)2 FunctionFailureResult (com.datastax.oss.simulacron.common.result.FunctionFailureResult)2 ReadTimeoutResult (com.datastax.oss.simulacron.common.result.ReadTimeoutResult)2 ServerErrorResult (com.datastax.oss.simulacron.common.result.ServerErrorResult)2 WriteFailureResult (com.datastax.oss.simulacron.common.result.WriteFailureResult)2 WriteTimeoutResult (com.datastax.oss.simulacron.common.result.WriteTimeoutResult)2 Config (com.typesafe.config.Config)2 NonNull (edu.umd.cs.findbugs.annotations.NonNull)2