Search in sources :

Example 1 with RowFormatter

use of com.datastax.oss.dsbulk.format.row.RowFormatter in project dsbulk by datastax.

the class LogManagerTest method should_stop_when_sample_size_is_met_and_percentage_exceeded.

@Test
void should_stop_when_sample_size_is_met_and_percentage_exceeded() throws Exception {
    Path outputDir = Files.createTempDirectory("test");
    LogManager logManager = new LogManager(session, outputDir, ErrorThreshold.forRatio(0.01f, 100), ErrorThreshold.forAbsoluteValue(0), true, statementFormatter, EXTENDED, rowFormatter);
    logManager.init();
    Flux<ReadResult> stmts = Flux.just(failedReadResult1);
    try {
        stmts.repeat(101).transform(logManager.newTotalItemsCounter()).transform(logManager.newFailedReadsHandler()).blockLast();
        fail("Expecting TooManyErrorsException to be thrown");
    } catch (TooManyErrorsException e) {
        assertThat(e).hasMessage("Too many errors, the maximum allowed is 1%.");
        Assertions.assertThat(((RatioErrorThreshold) e.getThreshold()).getMaxErrorRatio()).isEqualTo(0.01f);
    }
    logManager.close();
    Path errors = logManager.getOperationDirectory().resolve("unload-errors.log");
    assertThat(errors.toFile()).exists();
    assertThat(FileUtils.listAllFilesInDirectory(logManager.getOperationDirectory())).containsOnly(errors);
    List<String> lines = Files.readAllLines(errors, UTF_8);
    assertThat(lines.stream().filter(l -> l.contains("BulkExecutionException")).count()).isEqualTo(100);
}
Also used : Path(java.nio.file.Path) ReadResult(com.datastax.oss.dsbulk.executor.api.result.ReadResult) BeforeEach(org.junit.jupiter.api.BeforeEach) MappedBoundStatementPrinter(com.datastax.oss.dsbulk.workflow.commons.format.statement.MappedBoundStatementPrinter) DefaultWriteResult(com.datastax.oss.dsbulk.executor.api.result.DefaultWriteResult) EXTENDED(com.datastax.oss.dsbulk.format.statement.StatementFormatVerbosity.EXTENDED) SimpleStatement(com.datastax.oss.driver.api.core.cql.SimpleStatement) DefaultErrorRecord(com.datastax.oss.dsbulk.connectors.api.DefaultErrorRecord) ByteBuffer(java.nio.ByteBuffer) UnmappableStatement(com.datastax.oss.dsbulk.workflow.commons.statement.UnmappableStatement) AsyncResultSet(com.datastax.oss.driver.api.core.cql.AsyncResultSet) ExtendWith(org.junit.jupiter.api.extension.ExtendWith) Assertions(org.assertj.core.api.Assertions) MappedBoundStatement(com.datastax.oss.dsbulk.workflow.commons.statement.MappedBoundStatement) URI(java.net.URI) DefaultRecord(com.datastax.oss.dsbulk.connectors.api.DefaultRecord) Path(java.nio.file.Path) Record(com.datastax.oss.dsbulk.connectors.api.Record) LogInterceptor(com.datastax.oss.dsbulk.tests.logging.LogInterceptor) DriverUtils.mockSession(com.datastax.oss.dsbulk.tests.driver.DriverUtils.mockSession) StatementFormatter(com.datastax.oss.dsbulk.format.statement.StatementFormatter) RowFormatter(com.datastax.oss.dsbulk.format.row.RowFormatter) TestAssertions.assertThat(com.datastax.oss.dsbulk.tests.assertions.TestAssertions.assertThat) TooManyErrorsException(com.datastax.oss.dsbulk.workflow.api.error.TooManyErrorsException) BoundStatement(com.datastax.oss.driver.api.core.cql.BoundStatement) ImmutableList(com.datastax.oss.driver.shaded.guava.common.collect.ImmutableList) Test(org.junit.jupiter.api.Test) List(java.util.List) BatchableStatement(com.datastax.oss.driver.api.core.cql.BatchableStatement) Lists(org.assertj.core.util.Lists) WARN(org.slf4j.event.Level.WARN) Mockito.mock(org.mockito.Mockito.mock) Fail.fail(org.assertj.core.api.Fail.fail) RatioErrorThreshold(com.datastax.oss.dsbulk.workflow.api.error.RatioErrorThreshold) DefaultReadResult(com.datastax.oss.dsbulk.executor.api.result.DefaultReadResult) ErrorThreshold(com.datastax.oss.dsbulk.workflow.api.error.ErrorThreshold) CqlSession(com.datastax.oss.driver.api.core.CqlSession) Row(com.datastax.oss.driver.api.core.cql.Row) DefaultBatchType(com.datastax.oss.driver.api.core.cql.DefaultBatchType) AbsoluteErrorThreshold(com.datastax.oss.dsbulk.workflow.api.error.AbsoluteErrorThreshold) FileUtils(com.datastax.oss.dsbulk.tests.utils.FileUtils) Files(java.nio.file.Files) DriverUtils.mockBoundStatement(com.datastax.oss.dsbulk.tests.driver.DriverUtils.mockBoundStatement) UTF_8(java.nio.charset.StandardCharsets.UTF_8) DriverTimeoutException(com.datastax.oss.driver.api.core.DriverTimeoutException) DriverUtils.mockRow(com.datastax.oss.dsbulk.tests.driver.DriverUtils.mockRow) Mockito.when(org.mockito.Mockito.when) BatchStatement(com.datastax.oss.driver.api.core.cql.BatchStatement) LogCapture(com.datastax.oss.dsbulk.tests.logging.LogCapture) ExecutionInfo(com.datastax.oss.driver.api.core.cql.ExecutionInfo) Flux(reactor.core.publisher.Flux) WriteResult(com.datastax.oss.dsbulk.executor.api.result.WriteResult) BulkExecutionException(com.datastax.oss.dsbulk.executor.api.exception.BulkExecutionException) DriverExecutionException(com.datastax.oss.driver.api.core.DriverExecutionException) LogInterceptingExtension(com.datastax.oss.dsbulk.tests.logging.LogInterceptingExtension) Statement(com.datastax.oss.driver.api.core.cql.Statement) TooManyErrorsException(com.datastax.oss.dsbulk.workflow.api.error.TooManyErrorsException) ReadResult(com.datastax.oss.dsbulk.executor.api.result.ReadResult) DefaultReadResult(com.datastax.oss.dsbulk.executor.api.result.DefaultReadResult) Test(org.junit.jupiter.api.Test)

Example 2 with RowFormatter

use of com.datastax.oss.dsbulk.format.row.RowFormatter in project dsbulk by datastax.

the class LogSettings method newLogManager.

public LogManager newLogManager(CqlSession session, boolean trackPositions) {
    StatementFormatter statementFormatter = StatementFormatter.builder().withMaxQueryStringLength(maxQueryStringLength).withMaxBoundValueLength(maxBoundValueLength).withMaxBoundValues(maxBoundValues).withMaxInnerStatements(maxInnerStatements).addStatementPrinters(new MappedBoundStatementPrinter()).build();
    RowFormatter rowFormatter = new RowFormatter(maxResultSetValueLength, maxResultSetValues);
    return new LogManager(session, operationDirectory, errorThreshold, queryWarningsThreshold, trackPositions, statementFormatter, level, rowFormatter);
}
Also used : RowFormatter(com.datastax.oss.dsbulk.format.row.RowFormatter) StatementFormatter(com.datastax.oss.dsbulk.format.statement.StatementFormatter) MappedBoundStatementPrinter(com.datastax.oss.dsbulk.workflow.commons.format.statement.MappedBoundStatementPrinter) LogManager(com.datastax.oss.dsbulk.workflow.commons.log.LogManager)

Aggregations

RowFormatter (com.datastax.oss.dsbulk.format.row.RowFormatter)2 StatementFormatter (com.datastax.oss.dsbulk.format.statement.StatementFormatter)2 MappedBoundStatementPrinter (com.datastax.oss.dsbulk.workflow.commons.format.statement.MappedBoundStatementPrinter)2 CqlSession (com.datastax.oss.driver.api.core.CqlSession)1 DriverExecutionException (com.datastax.oss.driver.api.core.DriverExecutionException)1 DriverTimeoutException (com.datastax.oss.driver.api.core.DriverTimeoutException)1 AsyncResultSet (com.datastax.oss.driver.api.core.cql.AsyncResultSet)1 BatchStatement (com.datastax.oss.driver.api.core.cql.BatchStatement)1 BatchableStatement (com.datastax.oss.driver.api.core.cql.BatchableStatement)1 BoundStatement (com.datastax.oss.driver.api.core.cql.BoundStatement)1 DefaultBatchType (com.datastax.oss.driver.api.core.cql.DefaultBatchType)1 ExecutionInfo (com.datastax.oss.driver.api.core.cql.ExecutionInfo)1 Row (com.datastax.oss.driver.api.core.cql.Row)1 SimpleStatement (com.datastax.oss.driver.api.core.cql.SimpleStatement)1 Statement (com.datastax.oss.driver.api.core.cql.Statement)1 ImmutableList (com.datastax.oss.driver.shaded.guava.common.collect.ImmutableList)1 DefaultErrorRecord (com.datastax.oss.dsbulk.connectors.api.DefaultErrorRecord)1 DefaultRecord (com.datastax.oss.dsbulk.connectors.api.DefaultRecord)1 Record (com.datastax.oss.dsbulk.connectors.api.Record)1 BulkExecutionException (com.datastax.oss.dsbulk.executor.api.exception.BulkExecutionException)1