Search in sources :

Example 1 with Reference

use of io.crate.metadata.Reference in project crate by crate.

the class OrderedLuceneBatchIteratorBenchmark method createLuceneBatchIterator.

@Setup
public void createLuceneBatchIterator() throws Exception {
    IndexWriter iw = new IndexWriter(new RAMDirectory(), new IndexWriterConfig(new StandardAnalyzer()));
    dummyShardId = new ShardId("dummy", 1);
    columnName = "x";
    for (int i = 0; i < 10_000_000; i++) {
        Document doc = new Document();
        doc.add(new NumericDocValuesField(columnName, i));
        iw.addDocument(doc);
    }
    iw.commit();
    iw.forceMerge(1, true);
    indexSearcher = new IndexSearcher(DirectoryReader.open(iw, true));
    collectorContext = new CollectorContext(mock(IndexFieldDataService.class), new CollectorFieldsVisitor(0));
    fieldTypeLookup = column -> {
        IntegerFieldMapper.IntegerFieldType integerFieldType = new IntegerFieldMapper.IntegerFieldType();
        integerFieldType.setNames(new MappedFieldType.Names(column));
        return integerFieldType;
    };
    reference = new Reference(new ReferenceIdent(new TableIdent(null, "dummyTable"), columnName), RowGranularity.DOC, DataTypes.INTEGER);
    orderBy = new OrderBy(Collections.singletonList(reference), reverseFlags, nullsFirst);
}
Also used : OrderBy(io.crate.analyze.OrderBy) Reference(io.crate.metadata.Reference) TableIdent(io.crate.metadata.TableIdent) Document(org.apache.lucene.document.Document) IntegerFieldMapper(org.elasticsearch.index.mapper.core.IntegerFieldMapper) RAMDirectory(org.apache.lucene.store.RAMDirectory) ReferenceIdent(io.crate.metadata.ReferenceIdent) ShardId(org.elasticsearch.index.shard.ShardId) NumericDocValuesField(org.apache.lucene.document.NumericDocValuesField) IndexWriter(org.apache.lucene.index.IndexWriter) StandardAnalyzer(org.apache.lucene.analysis.standard.StandardAnalyzer) MappedFieldType(org.elasticsearch.index.mapper.MappedFieldType) CollectorContext(io.crate.operation.reference.doc.lucene.CollectorContext) IndexWriterConfig(org.apache.lucene.index.IndexWriterConfig)

Example 2 with Reference

use of io.crate.metadata.Reference in project crate by crate.

the class BulkShardProcessorTest method testNonEsRejectedExceptionDoesNotResultInRetryButAborts.

@Test
public void testNonEsRejectedExceptionDoesNotResultInRetryButAborts() throws Throwable {
    expectedException.expect(RuntimeException.class);
    expectedException.expectMessage("a random exception");
    final AtomicReference<ActionListener<ShardResponse>> ref = new AtomicReference<>();
    BulkRequestExecutor<ShardUpsertRequest> transportShardBulkAction = (request, listener) -> ref.set(listener);
    BulkRetryCoordinator bulkRetryCoordinator = new BulkRetryCoordinator(threadPool);
    BulkRetryCoordinatorPool coordinatorPool = mock(BulkRetryCoordinatorPool.class);
    when(coordinatorPool.coordinator(any(ShardId.class))).thenReturn(bulkRetryCoordinator);
    ShardUpsertRequest.Builder builder = new ShardUpsertRequest.Builder(TimeValue.timeValueMillis(10), false, false, null, new Reference[] { fooRef }, UUID.randomUUID());
    final BulkShardProcessor<ShardUpsertRequest> bulkShardProcessor = new BulkShardProcessor<>(clusterService, mock(TransportBulkCreateIndicesAction.class), new IndexNameExpressionResolver(Settings.EMPTY), Settings.EMPTY, coordinatorPool, false, 1, builder, transportShardBulkAction, UUID.randomUUID());
    bulkShardProcessor.add("foo", new ShardUpsertRequest.Item("1", null, new Object[] { "bar1" }, null), null);
    ActionListener<ShardResponse> listener = ref.get();
    listener.onFailure(new RuntimeException("a random exception"));
    assertFalse(bulkShardProcessor.add("foo", new ShardUpsertRequest.Item("2", null, new Object[] { "bar2" }, null), null));
    try {
        bulkShardProcessor.result().get();
    } catch (ExecutionException e) {
        throw e.getCause();
    } finally {
        bulkShardProcessor.close();
    }
}
Also used : TransportBulkCreateIndicesAction(org.elasticsearch.action.admin.indices.create.TransportBulkCreateIndicesAction) ShardId(org.elasticsearch.index.shard.ShardId) CoreMatchers.is(org.hamcrest.CoreMatchers.is) Matchers.isA(org.hamcrest.Matchers.isA) ShardIterator(org.elasticsearch.cluster.routing.ShardIterator) Matchers(org.mockito.Matchers) Mock(org.mockito.Mock) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) AtomicReference(java.util.concurrent.atomic.AtomicReference) Matchers.anyString(org.mockito.Matchers.anyString) ClusterState(org.elasticsearch.cluster.ClusterState) Settings(org.elasticsearch.common.settings.Settings) ClusterService(org.elasticsearch.cluster.ClusterService) TimeValue(org.elasticsearch.common.unit.TimeValue) ShardResponse(io.crate.executor.transport.ShardResponse) ThreadPool(org.elasticsearch.threadpool.ThreadPool) OperationRouting(org.elasticsearch.cluster.routing.OperationRouting) Answers(org.mockito.Answers) java.util.concurrent(java.util.concurrent) Reference(io.crate.metadata.Reference) Test(org.junit.Test) TableIdent(io.crate.metadata.TableIdent) UUID(java.util.UUID) Mockito.when(org.mockito.Mockito.when) CrateUnitTest(io.crate.test.integration.CrateUnitTest) Matchers.any(org.mockito.Matchers.any) ShardUpsertRequest(io.crate.executor.transport.ShardUpsertRequest) RowGranularity(io.crate.metadata.RowGranularity) EsRejectedExecutionException(org.elasticsearch.common.util.concurrent.EsRejectedExecutionException) DataTypes(io.crate.types.DataTypes) ReferenceIdent(io.crate.metadata.ReferenceIdent) IndexNameExpressionResolver(org.elasticsearch.cluster.metadata.IndexNameExpressionResolver) ActionListener(org.elasticsearch.action.ActionListener) Mockito.mock(org.mockito.Mockito.mock) ShardUpsertRequest(io.crate.executor.transport.ShardUpsertRequest) AtomicReference(java.util.concurrent.atomic.AtomicReference) ShardId(org.elasticsearch.index.shard.ShardId) ShardResponse(io.crate.executor.transport.ShardResponse) ActionListener(org.elasticsearch.action.ActionListener) TransportBulkCreateIndicesAction(org.elasticsearch.action.admin.indices.create.TransportBulkCreateIndicesAction) IndexNameExpressionResolver(org.elasticsearch.cluster.metadata.IndexNameExpressionResolver) EsRejectedExecutionException(org.elasticsearch.common.util.concurrent.EsRejectedExecutionException) Test(org.junit.Test) CrateUnitTest(io.crate.test.integration.CrateUnitTest)

Example 3 with Reference

use of io.crate.metadata.Reference in project crate by crate.

the class BulkShardProcessorTest method testThatAddAfterFailureBlocksDueToRetry.

@Test
public void testThatAddAfterFailureBlocksDueToRetry() throws Exception {
    ClusterService clusterService = mock(ClusterService.class);
    OperationRouting operationRouting = mock(OperationRouting.class);
    mockShard(operationRouting, 1);
    mockShard(operationRouting, 2);
    mockShard(operationRouting, 3);
    when(clusterService.operationRouting()).thenReturn(operationRouting);
    // listener will be executed 2 times, once for the successfully added row and once for the failure
    final CountDownLatch listenerLatch = new CountDownLatch(2);
    final AtomicReference<ActionListener<ShardResponse>> ref = new AtomicReference<>();
    BulkRequestExecutor<ShardUpsertRequest> transportShardBulkAction = (request, listener) -> {
        ref.set(listener);
        listenerLatch.countDown();
    };
    BulkRetryCoordinator bulkRetryCoordinator = new BulkRetryCoordinator(threadPool);
    BulkRetryCoordinatorPool coordinatorPool = mock(BulkRetryCoordinatorPool.class);
    when(coordinatorPool.coordinator(any(ShardId.class))).thenReturn(bulkRetryCoordinator);
    ShardUpsertRequest.Builder builder = new ShardUpsertRequest.Builder(TimeValue.timeValueMillis(10), false, false, null, new Reference[] { fooRef }, UUID.randomUUID());
    final BulkShardProcessor<ShardUpsertRequest> bulkShardProcessor = new BulkShardProcessor<>(clusterService, mock(TransportBulkCreateIndicesAction.class), new IndexNameExpressionResolver(Settings.EMPTY), Settings.EMPTY, coordinatorPool, false, 1, builder, transportShardBulkAction, UUID.randomUUID());
    bulkShardProcessor.add("foo", new ShardUpsertRequest.Item("1", null, new Object[] { "bar1" }, null), null);
    final ActionListener<ShardResponse> listener = ref.get();
    listener.onFailure(new EsRejectedExecutionException());
    // wait, failure retry lock is done in decoupled thread
    listenerLatch.await(10, TimeUnit.SECONDS);
    final ScheduledExecutorService scheduledExecutorService = Executors.newScheduledThreadPool(2);
    try {
        final AtomicBoolean hadBlocked = new AtomicBoolean(false);
        final AtomicBoolean hasBlocked = new AtomicBoolean(true);
        final CountDownLatch latch = new CountDownLatch(1);
        scheduledExecutorService.execute(new Runnable() {

            @Override
            public void run() {
                scheduledExecutorService.schedule(new Runnable() {

                    @Override
                    public void run() {
                        hadBlocked.set(hasBlocked.get());
                        latch.countDown();
                    }
                }, 10, TimeUnit.MILLISECONDS);
                bulkShardProcessor.add("foo", new ShardUpsertRequest.Item("2", null, new Object[] { "bar2" }, null), null);
                hasBlocked.set(false);
            }
        });
        latch.await();
        assertTrue(hadBlocked.get());
    } finally {
        scheduledExecutorService.shutdownNow();
    }
}
Also used : TransportBulkCreateIndicesAction(org.elasticsearch.action.admin.indices.create.TransportBulkCreateIndicesAction) ShardId(org.elasticsearch.index.shard.ShardId) CoreMatchers.is(org.hamcrest.CoreMatchers.is) Matchers.isA(org.hamcrest.Matchers.isA) ShardIterator(org.elasticsearch.cluster.routing.ShardIterator) Matchers(org.mockito.Matchers) Mock(org.mockito.Mock) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) AtomicReference(java.util.concurrent.atomic.AtomicReference) Matchers.anyString(org.mockito.Matchers.anyString) ClusterState(org.elasticsearch.cluster.ClusterState) Settings(org.elasticsearch.common.settings.Settings) ClusterService(org.elasticsearch.cluster.ClusterService) TimeValue(org.elasticsearch.common.unit.TimeValue) ShardResponse(io.crate.executor.transport.ShardResponse) ThreadPool(org.elasticsearch.threadpool.ThreadPool) OperationRouting(org.elasticsearch.cluster.routing.OperationRouting) Answers(org.mockito.Answers) java.util.concurrent(java.util.concurrent) Reference(io.crate.metadata.Reference) Test(org.junit.Test) TableIdent(io.crate.metadata.TableIdent) UUID(java.util.UUID) Mockito.when(org.mockito.Mockito.when) CrateUnitTest(io.crate.test.integration.CrateUnitTest) Matchers.any(org.mockito.Matchers.any) ShardUpsertRequest(io.crate.executor.transport.ShardUpsertRequest) RowGranularity(io.crate.metadata.RowGranularity) EsRejectedExecutionException(org.elasticsearch.common.util.concurrent.EsRejectedExecutionException) DataTypes(io.crate.types.DataTypes) ReferenceIdent(io.crate.metadata.ReferenceIdent) IndexNameExpressionResolver(org.elasticsearch.cluster.metadata.IndexNameExpressionResolver) ActionListener(org.elasticsearch.action.ActionListener) Mockito.mock(org.mockito.Mockito.mock) ShardId(org.elasticsearch.index.shard.ShardId) ShardResponse(io.crate.executor.transport.ShardResponse) OperationRouting(org.elasticsearch.cluster.routing.OperationRouting) EsRejectedExecutionException(org.elasticsearch.common.util.concurrent.EsRejectedExecutionException) ShardUpsertRequest(io.crate.executor.transport.ShardUpsertRequest) AtomicReference(java.util.concurrent.atomic.AtomicReference) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) ClusterService(org.elasticsearch.cluster.ClusterService) ActionListener(org.elasticsearch.action.ActionListener) TransportBulkCreateIndicesAction(org.elasticsearch.action.admin.indices.create.TransportBulkCreateIndicesAction) IndexNameExpressionResolver(org.elasticsearch.cluster.metadata.IndexNameExpressionResolver) Test(org.junit.Test) CrateUnitTest(io.crate.test.integration.CrateUnitTest)

Example 4 with Reference

use of io.crate.metadata.Reference in project crate by crate.

the class BulkShardProcessorTest method testKill.

@Test
public void testKill() throws Exception {
    ClusterService clusterService = mock(ClusterService.class);
    OperationRouting operationRouting = mock(OperationRouting.class);
    mockShard(operationRouting, 1);
    mockShard(operationRouting, 2);
    mockShard(operationRouting, 3);
    when(clusterService.operationRouting()).thenReturn(operationRouting);
    final AtomicReference<ActionListener<ShardResponse>> ref = new AtomicReference<>();
    BulkRequestExecutor<ShardUpsertRequest> transportShardBulkAction = (request, listener) -> ref.set(listener);
    BulkRetryCoordinator bulkRetryCoordinator = new BulkRetryCoordinator(threadPool);
    BulkRetryCoordinatorPool coordinatorPool = mock(BulkRetryCoordinatorPool.class);
    when(coordinatorPool.coordinator(any(ShardId.class))).thenReturn(bulkRetryCoordinator);
    ShardUpsertRequest.Builder builder = new ShardUpsertRequest.Builder(TimeValue.timeValueMillis(10), false, false, null, new Reference[] { fooRef }, UUID.randomUUID());
    final BulkShardProcessor<ShardUpsertRequest> bulkShardProcessor = new BulkShardProcessor<>(clusterService, mock(TransportBulkCreateIndicesAction.class), new IndexNameExpressionResolver(Settings.EMPTY), Settings.EMPTY, coordinatorPool, false, 1, builder, transportShardBulkAction, UUID.randomUUID());
    assertThat(bulkShardProcessor.add("foo", new ShardUpsertRequest.Item("1", null, new Object[] { "bar1" }, null), null), is(true));
    bulkShardProcessor.kill(new InterruptedException());
    // A InterruptedException is thrown
    expectedException.expect(ExecutionException.class);
    expectedException.expectCause(isA(InterruptedException.class));
    bulkShardProcessor.result().get();
    // it's not possible to add more
    assertThat(bulkShardProcessor.add("foo", new ShardUpsertRequest.Item("1", null, new Object[] { "bar1" }, null), null), is(false));
}
Also used : TransportBulkCreateIndicesAction(org.elasticsearch.action.admin.indices.create.TransportBulkCreateIndicesAction) ShardId(org.elasticsearch.index.shard.ShardId) CoreMatchers.is(org.hamcrest.CoreMatchers.is) Matchers.isA(org.hamcrest.Matchers.isA) ShardIterator(org.elasticsearch.cluster.routing.ShardIterator) Matchers(org.mockito.Matchers) Mock(org.mockito.Mock) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) AtomicReference(java.util.concurrent.atomic.AtomicReference) Matchers.anyString(org.mockito.Matchers.anyString) ClusterState(org.elasticsearch.cluster.ClusterState) Settings(org.elasticsearch.common.settings.Settings) ClusterService(org.elasticsearch.cluster.ClusterService) TimeValue(org.elasticsearch.common.unit.TimeValue) ShardResponse(io.crate.executor.transport.ShardResponse) ThreadPool(org.elasticsearch.threadpool.ThreadPool) OperationRouting(org.elasticsearch.cluster.routing.OperationRouting) Answers(org.mockito.Answers) java.util.concurrent(java.util.concurrent) Reference(io.crate.metadata.Reference) Test(org.junit.Test) TableIdent(io.crate.metadata.TableIdent) UUID(java.util.UUID) Mockito.when(org.mockito.Mockito.when) CrateUnitTest(io.crate.test.integration.CrateUnitTest) Matchers.any(org.mockito.Matchers.any) ShardUpsertRequest(io.crate.executor.transport.ShardUpsertRequest) RowGranularity(io.crate.metadata.RowGranularity) EsRejectedExecutionException(org.elasticsearch.common.util.concurrent.EsRejectedExecutionException) DataTypes(io.crate.types.DataTypes) ReferenceIdent(io.crate.metadata.ReferenceIdent) IndexNameExpressionResolver(org.elasticsearch.cluster.metadata.IndexNameExpressionResolver) ActionListener(org.elasticsearch.action.ActionListener) Mockito.mock(org.mockito.Mockito.mock) ShardUpsertRequest(io.crate.executor.transport.ShardUpsertRequest) AtomicReference(java.util.concurrent.atomic.AtomicReference) ShardId(org.elasticsearch.index.shard.ShardId) ClusterService(org.elasticsearch.cluster.ClusterService) ActionListener(org.elasticsearch.action.ActionListener) OperationRouting(org.elasticsearch.cluster.routing.OperationRouting) TransportBulkCreateIndicesAction(org.elasticsearch.action.admin.indices.create.TransportBulkCreateIndicesAction) IndexNameExpressionResolver(org.elasticsearch.cluster.metadata.IndexNameExpressionResolver) Test(org.junit.Test) CrateUnitTest(io.crate.test.integration.CrateUnitTest)

Example 5 with Reference

use of io.crate.metadata.Reference in project crate by crate.

the class BlobTableInfoTest method testGetColumnInfo.

@Test
public void testGetColumnInfo() throws Exception {
    Reference foobar = info.getReference(new ColumnIdent("digest"));
    assertNotNull(foobar);
    assertEquals(DataTypes.STRING, foobar.valueType());
}
Also used : ColumnIdent(io.crate.metadata.ColumnIdent) Reference(io.crate.metadata.Reference) Test(org.junit.Test) CrateUnitTest(io.crate.test.integration.CrateUnitTest)

Aggregations

Reference (io.crate.metadata.Reference)84 Test (org.junit.Test)57 CrateUnitTest (io.crate.test.integration.CrateUnitTest)55 TestingHelpers.mapToSortedString (io.crate.testing.TestingHelpers.mapToSortedString)27 HashMap (java.util.HashMap)20 Map (java.util.Map)20 NestedObjectExpression (io.crate.operation.reference.NestedObjectExpression)17 ColumnIdent (io.crate.metadata.ColumnIdent)15 Symbol (io.crate.analyze.symbol.Symbol)14 BytesRef (org.apache.lucene.util.BytesRef)11 TableIdent (io.crate.metadata.TableIdent)9 ReferenceIdent (io.crate.metadata.ReferenceIdent)8 InputColumn (io.crate.analyze.symbol.InputColumn)6 RoutedCollectPhase (io.crate.planner.node.dql.RoutedCollectPhase)6 ShardId (org.elasticsearch.index.shard.ShardId)5 Routing (io.crate.metadata.Routing)4 UUID (java.util.UUID)4 OrderBy (io.crate.analyze.OrderBy)3 ShardResponse (io.crate.executor.transport.ShardResponse)3 ShardUpsertRequest (io.crate.executor.transport.ShardUpsertRequest)3