Search in sources :

Example 1 with DefaultIndexedField

use of com.datastax.oss.dsbulk.connectors.api.DefaultIndexedField in project dsbulk by datastax.

the class CSVConnectorTest method should_honor_ignoreLeadingWhitespacesInQuotes_and_ignoreTrailingWhitespacesInQuotes2.

@Test
void should_honor_ignoreLeadingWhitespacesInQuotes_and_ignoreTrailingWhitespacesInQuotes2() throws Exception {
    Path file = Files.createTempFile("test", ".csv");
    Files.write(file, Collections.singleton("\" foo \""));
    CSVConnector connector = new CSVConnector();
    Config settings = TestConfigUtils.createTestConfig("dsbulk.connector.csv", "url", StringUtils.quoteJson(file), "ignoreLeadingWhitespacesInQuotes", true, "ignoreTrailingWhitespacesInQuotes", true, "header", false);
    connector.configure(settings, true, true);
    connector.init();
    List<Record> records = Flux.merge(connector.read()).collectList().block();
    assertThat(records).hasSize(1);
    assertThat(records.get(0).getFieldValue(new DefaultIndexedField(0))).isEqualTo("foo");
    connector.close();
}
Also used : Path(java.nio.file.Path) DefaultIndexedField(com.datastax.oss.dsbulk.connectors.api.DefaultIndexedField) Config(com.typesafe.config.Config) DefaultRecord(com.datastax.oss.dsbulk.connectors.api.DefaultRecord) Record(com.datastax.oss.dsbulk.connectors.api.Record) ErrorRecord(com.datastax.oss.dsbulk.connectors.api.ErrorRecord) Test(org.junit.jupiter.api.Test) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest)

Example 2 with DefaultIndexedField

use of com.datastax.oss.dsbulk.connectors.api.DefaultIndexedField in project dsbulk by datastax.

the class CSVConnectorTest method should_honor_nullValue_when_reading.

@ParameterizedTest
@MethodSource
void should_honor_nullValue_when_reading(String nullValue, String expected) throws Exception {
    Path file = Files.createTempFile("test", ".csv");
    Files.write(file, Collections.singleton(","));
    CSVConnector connector = new CSVConnector();
    Config settings = TestConfigUtils.createTestConfig("dsbulk.connector.csv", "url", StringUtils.quoteJson(file), "nullValue", StringUtils.quoteJson(nullValue), "header", false);
    connector.configure(settings, true, true);
    connector.init();
    List<Record> records = Flux.merge(connector.read()).collectList().block();
    assertThat(records).hasSize(1);
    assertThat(records.get(0).getFieldValue(new DefaultIndexedField(0))).isEqualTo(expected);
    connector.close();
}
Also used : Path(java.nio.file.Path) DefaultIndexedField(com.datastax.oss.dsbulk.connectors.api.DefaultIndexedField) Config(com.typesafe.config.Config) DefaultRecord(com.datastax.oss.dsbulk.connectors.api.DefaultRecord) Record(com.datastax.oss.dsbulk.connectors.api.Record) ErrorRecord(com.datastax.oss.dsbulk.connectors.api.ErrorRecord) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest) MethodSource(org.junit.jupiter.params.provider.MethodSource)

Example 3 with DefaultIndexedField

use of com.datastax.oss.dsbulk.connectors.api.DefaultIndexedField in project dsbulk by datastax.

the class CSVConnectorTest method should_honor_ignoreLeadingWhitespaces_and_ignoreTrailingWhitespaces_when_reading2.

@Test
void should_honor_ignoreLeadingWhitespaces_and_ignoreTrailingWhitespaces_when_reading2() throws Exception {
    Path file = Files.createTempFile("test", ".csv");
    Files.write(file, Collections.singleton(" foo "));
    CSVConnector connector = new CSVConnector();
    Config settings = TestConfigUtils.createTestConfig("dsbulk.connector.csv", "url", StringUtils.quoteJson(file), "ignoreLeadingWhitespaces", true, "ignoreTrailingWhitespaces", true, "header", false);
    connector.configure(settings, true, true);
    connector.init();
    List<Record> records = Flux.merge(connector.read()).collectList().block();
    assertThat(records).hasSize(1);
    assertThat(records.get(0).getFieldValue(new DefaultIndexedField(0))).isEqualTo("foo");
    connector.close();
}
Also used : Path(java.nio.file.Path) DefaultIndexedField(com.datastax.oss.dsbulk.connectors.api.DefaultIndexedField) Config(com.typesafe.config.Config) DefaultRecord(com.datastax.oss.dsbulk.connectors.api.DefaultRecord) Record(com.datastax.oss.dsbulk.connectors.api.Record) ErrorRecord(com.datastax.oss.dsbulk.connectors.api.ErrorRecord) Test(org.junit.jupiter.api.Test) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest)

Example 4 with DefaultIndexedField

use of com.datastax.oss.dsbulk.connectors.api.DefaultIndexedField in project dsbulk by datastax.

the class CSVConnectorTest method should_honor_ignoreLeadingWhitespacesInQuotes_and_ignoreTrailingWhitespacesInQuotes.

@Test
void should_honor_ignoreLeadingWhitespacesInQuotes_and_ignoreTrailingWhitespacesInQuotes() throws Exception {
    Path file = Files.createTempFile("test", ".csv");
    Files.write(file, Collections.singleton("\" foo \""));
    CSVConnector connector = new CSVConnector();
    Config settings = TestConfigUtils.createTestConfig("dsbulk.connector.csv", "url", StringUtils.quoteJson(file), "ignoreLeadingWhitespacesInQuotes", false, "ignoreTrailingWhitespacesInQuotes", false, "header", false);
    connector.configure(settings, true, true);
    connector.init();
    List<Record> records = Flux.merge(connector.read()).collectList().block();
    assertThat(records).hasSize(1);
    assertThat(records.get(0).getFieldValue(new DefaultIndexedField(0))).isEqualTo(" foo ");
    connector.close();
}
Also used : Path(java.nio.file.Path) DefaultIndexedField(com.datastax.oss.dsbulk.connectors.api.DefaultIndexedField) Config(com.typesafe.config.Config) DefaultRecord(com.datastax.oss.dsbulk.connectors.api.DefaultRecord) Record(com.datastax.oss.dsbulk.connectors.api.Record) ErrorRecord(com.datastax.oss.dsbulk.connectors.api.ErrorRecord) Test(org.junit.jupiter.api.Test) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest)

Example 5 with DefaultIndexedField

use of com.datastax.oss.dsbulk.connectors.api.DefaultIndexedField in project dsbulk by datastax.

the class CSVConnectorTest method should_honor_emptyValue_when_reading.

@ParameterizedTest
@MethodSource
void should_honor_emptyValue_when_reading(String quote, String emptyValue, String expected) throws Exception {
    Path file = Files.createTempFile("test", ".csv");
    Files.write(file, Collections.singleton(Strings.repeat(quote, 2)));
    CSVConnector connector = new CSVConnector();
    Config settings = TestConfigUtils.createTestConfig("dsbulk.connector.csv", "url", StringUtils.quoteJson(file), "quote", StringUtils.quoteJson(quote), "emptyValue", StringUtils.quoteJson(emptyValue), "header", false);
    connector.configure(settings, true, true);
    connector.init();
    List<Record> records = Flux.merge(connector.read()).collectList().block();
    assertThat(records).hasSize(1);
    assertThat(records.get(0).getFieldValue(new DefaultIndexedField(0))).isEqualTo(expected);
    connector.close();
}
Also used : Path(java.nio.file.Path) DefaultIndexedField(com.datastax.oss.dsbulk.connectors.api.DefaultIndexedField) Config(com.typesafe.config.Config) DefaultRecord(com.datastax.oss.dsbulk.connectors.api.DefaultRecord) Record(com.datastax.oss.dsbulk.connectors.api.Record) ErrorRecord(com.datastax.oss.dsbulk.connectors.api.ErrorRecord) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest) MethodSource(org.junit.jupiter.params.provider.MethodSource)

Aggregations

DefaultIndexedField (com.datastax.oss.dsbulk.connectors.api.DefaultIndexedField)10 DefaultRecord (com.datastax.oss.dsbulk.connectors.api.DefaultRecord)9 ErrorRecord (com.datastax.oss.dsbulk.connectors.api.ErrorRecord)7 Record (com.datastax.oss.dsbulk.connectors.api.Record)7 Config (com.typesafe.config.Config)7 ParameterizedTest (org.junit.jupiter.params.ParameterizedTest)7 Path (java.nio.file.Path)6 Test (org.junit.jupiter.api.Test)5 DefaultMappedField (com.datastax.oss.dsbulk.connectors.api.DefaultMappedField)3 MethodSource (org.junit.jupiter.params.provider.MethodSource)2 CqlIdentifier (com.datastax.oss.driver.api.core.CqlIdentifier)1 ImmutableSetMultimap (com.datastax.oss.driver.shaded.guava.common.collect.ImmutableSetMultimap)1 SetMultimap (com.datastax.oss.driver.shaded.guava.common.collect.SetMultimap)1 Field (com.datastax.oss.dsbulk.connectors.api.Field)1 CQLWord (com.datastax.oss.dsbulk.mapping.CQLWord)1 ArgumentMatchers.anyString (org.mockito.ArgumentMatchers.anyString)1