Search in sources :

Example 6 with OnHeapMemoryManager

use of io.crate.memory.OnHeapMemoryManager in project crate by crate.

the class RootTaskTest method testFailureClosesAllSubContexts.

@Test
public void testFailureClosesAllSubContexts() throws Throwable {
    String localNodeId = "localNodeId";
    RoutedCollectPhase collectPhase = Mockito.mock(RoutedCollectPhase.class);
    Routing routing = Mockito.mock(Routing.class);
    when(routing.containsShards(localNodeId)).thenReturn(false);
    when(collectPhase.phaseId()).thenReturn(1);
    when(collectPhase.routing()).thenReturn(routing);
    when(collectPhase.maxRowGranularity()).thenReturn(RowGranularity.DOC);
    RootTask.Builder builder = new RootTask.Builder(logger, UUID.randomUUID(), "dummy-user", coordinatorNode, Collections.emptySet(), mock(JobsLogs.class));
    CollectTask collectChildTask = new CollectTask(collectPhase, CoordinatorTxnCtx.systemTransactionContext(), mock(MapSideDataCollectOperation.class), RamAccounting.NO_ACCOUNTING, ramAccounting -> new OnHeapMemoryManager(ramAccounting::addBytes), new TestingRowConsumer(), mock(SharedShardContexts.class), Version.CURRENT, 4096);
    TestingRowConsumer batchConsumer = new TestingRowConsumer();
    PageBucketReceiver pageBucketReceiver = new CumulativePageBucketReceiver("n1", 2, Runnable::run, new Streamer[] { IntegerType.INSTANCE.streamer() }, batchConsumer, PassThroughPagingIterator.oneShot(), 1);
    DistResultRXTask distResultRXTask = spy(new DistResultRXTask(2, "dummy", pageBucketReceiver, RamAccounting.NO_ACCOUNTING, 1));
    builder.addTask(collectChildTask);
    builder.addTask(distResultRXTask);
    RootTask rootTask = builder.build();
    Exception failure = new Exception("failure!");
    collectChildTask.kill(failure);
    // other contexts must be killed with same failure
    verify(distResultRXTask, times(1)).kill(failure);
    assertThat(rootTask.getTask(1).completionFuture().isDone(), is(true));
    assertThat(rootTask.getTask(2).completionFuture().isDone(), is(true));
}
Also used : OnHeapMemoryManager(io.crate.memory.OnHeapMemoryManager) MapSideDataCollectOperation(io.crate.execution.engine.collect.MapSideDataCollectOperation) Routing(io.crate.metadata.Routing) CollectTask(io.crate.execution.engine.collect.CollectTask) JobsLogs(io.crate.execution.engine.collect.stats.JobsLogs) RoutedCollectPhase(io.crate.execution.dsl.phases.RoutedCollectPhase) TestingRowConsumer(io.crate.testing.TestingRowConsumer) Test(org.junit.Test)

Example 7 with OnHeapMemoryManager

use of io.crate.memory.OnHeapMemoryManager in project crate by crate.

the class CollectTaskTest method setUp.

@Before
public void setUp() throws Exception {
    super.setUp();
    localNodeId = "dummyLocalNodeId";
    collectPhase = Mockito.mock(RoutedCollectPhase.class);
    Routing routing = Mockito.mock(Routing.class);
    when(routing.containsShards(localNodeId)).thenReturn(true);
    when(collectPhase.routing()).thenReturn(routing);
    when(collectPhase.maxRowGranularity()).thenReturn(RowGranularity.DOC);
    collectOperation = mock(MapSideDataCollectOperation.class);
    Mockito.doAnswer(new Answer<>() {

        @Override
        public Object answer(InvocationOnMock invocation) throws Throwable {
            Runnable runnable = invocation.getArgument(0);
            runnable.run();
            return null;
        }
    }).when(collectOperation).launch(Mockito.any(), Mockito.anyString());
    consumer = new TestingRowConsumer();
    collectTask = new CollectTask(collectPhase, CoordinatorTxnCtx.systemTransactionContext(), collectOperation, RamAccounting.NO_ACCOUNTING, ramAccounting -> new OnHeapMemoryManager(ramAccounting::addBytes), consumer, mock(SharedShardContexts.class), Version.CURRENT, 4096);
}
Also used : CompletableFuture(java.util.concurrent.CompletableFuture) BatchIterator(io.crate.data.BatchIterator) ArgumentMatchers.anyBoolean(org.mockito.ArgumentMatchers.anyBoolean) Answer(org.mockito.stubbing.Answer) InvocationOnMock(org.mockito.invocation.InvocationOnMock) Routing(io.crate.metadata.Routing) SharedShardContexts(io.crate.execution.jobs.SharedShardContexts) ThreadPool(org.elasticsearch.threadpool.ThreadPool) ESTestCase(org.elasticsearch.test.ESTestCase) Before(org.junit.Before) JobKilledException(io.crate.exceptions.JobKilledException) InMemoryBatchIterator(io.crate.data.InMemoryBatchIterator) RoutedCollectPhase(io.crate.execution.dsl.phases.RoutedCollectPhase) Mockito.times(org.mockito.Mockito.times) Test(org.junit.Test) Mockito.when(org.mockito.Mockito.when) RamAccounting(io.crate.breaker.RamAccounting) Mockito.verify(org.mockito.Mockito.verify) TimeUnit(java.util.concurrent.TimeUnit) CountDownLatch(java.util.concurrent.CountDownLatch) Mockito(org.mockito.Mockito) Exceptions(io.crate.exceptions.Exceptions) Version(org.elasticsearch.Version) RowGranularity(io.crate.metadata.RowGranularity) Row(io.crate.data.Row) Matchers.is(org.hamcrest.Matchers.is) OnHeapMemoryManager(io.crate.memory.OnHeapMemoryManager) RefCountedItem(io.crate.common.collections.RefCountedItem) TestingRowConsumer(io.crate.testing.TestingRowConsumer) SentinelRow(io.crate.data.SentinelRow) Mockito.mock(org.mockito.Mockito.mock) IndexSearcher(org.apache.lucene.search.IndexSearcher) CoordinatorTxnCtx(io.crate.metadata.CoordinatorTxnCtx) OnHeapMemoryManager(io.crate.memory.OnHeapMemoryManager) Routing(io.crate.metadata.Routing) InvocationOnMock(org.mockito.invocation.InvocationOnMock) RoutedCollectPhase(io.crate.execution.dsl.phases.RoutedCollectPhase) TestingRowConsumer(io.crate.testing.TestingRowConsumer) Before(org.junit.Before)

Example 8 with OnHeapMemoryManager

use of io.crate.memory.OnHeapMemoryManager in project crate by crate.

the class AbstractWindowFunctionTest method prepareFunctions.

@Before
public void prepareFunctions() {
    DocTableInfo tableInfo = SQLExecutor.tableInfo(new RelationName("doc", "t1"), "create table doc.t1 (x int, y bigint, z string, d double)", clusterService);
    DocTableRelation tableRelation = new DocTableRelation(tableInfo);
    Map<RelationName, AnalyzedRelation> tableSources = Map.of(tableInfo.ident(), tableRelation);
    memoryManager = new OnHeapMemoryManager(bytes -> {
    });
    sqlExpressions = new SqlExpressions(tableSources, tableRelation, User.CRATE_USER, additionalModules);
    inputFactory = new InputFactory(sqlExpressions.nodeCtx);
}
Also used : Input(io.crate.data.Input) TransactionContext(io.crate.metadata.TransactionContext) SqlExpressions(io.crate.testing.SqlExpressions) Arrays(java.util.Arrays) Array(java.lang.reflect.Array) RelationName(io.crate.metadata.RelationName) SENTINEL(io.crate.data.SentinelRow.SENTINEL) BatchIterator(io.crate.data.BatchIterator) ReferenceResolver(io.crate.expression.reference.ReferenceResolver) CrateDummyClusterServiceUnitTest(io.crate.test.integration.CrateDummyClusterServiceUnitTest) RowN(io.crate.data.RowN) Map(java.util.Map) SQLExecutor(io.crate.testing.SQLExecutor) Before(org.junit.Before) DocTableInfo(io.crate.metadata.doc.DocTableInfo) User(io.crate.user.User) AbstractModule(org.elasticsearch.common.inject.AbstractModule) AggregationFunction(io.crate.execution.engine.aggregation.AggregationFunction) InMemoryBatchIterator(io.crate.data.InMemoryBatchIterator) ColumnIdent(io.crate.metadata.ColumnIdent) Comparators.createComparator(io.crate.execution.engine.sort.Comparators.createComparator) RamAccounting(io.crate.breaker.RamAccounting) Collectors(java.util.stream.Collectors) Lists2(io.crate.common.collections.Lists2) InputCollectExpression(io.crate.execution.engine.collect.InputCollectExpression) ExecutionException(java.util.concurrent.ExecutionException) TimeUnit(java.util.concurrent.TimeUnit) Matchers.instanceOf(org.hamcrest.Matchers.instanceOf) ExpressionsInput(io.crate.expression.ExpressionsInput) List(java.util.List) OrderBy(io.crate.analyze.OrderBy) BatchIterators(io.crate.data.BatchIterators) Version(org.elasticsearch.Version) DocTableRelation(io.crate.analyze.relations.DocTableRelation) Row(io.crate.data.Row) Literal(io.crate.expression.symbol.Literal) Symbol(io.crate.expression.symbol.Symbol) FunctionImplementation(io.crate.metadata.FunctionImplementation) Matcher(org.hamcrest.Matcher) AnalyzedRelation(io.crate.analyze.relations.AnalyzedRelation) InputColumns(io.crate.execution.dsl.projection.builder.InputColumns) OnHeapMemoryManager(io.crate.memory.OnHeapMemoryManager) InputFactory(io.crate.expression.InputFactory) Comparator(java.util.Comparator) CoordinatorTxnCtx(io.crate.metadata.CoordinatorTxnCtx) InputFactory(io.crate.expression.InputFactory) DocTableInfo(io.crate.metadata.doc.DocTableInfo) OnHeapMemoryManager(io.crate.memory.OnHeapMemoryManager) RelationName(io.crate.metadata.RelationName) DocTableRelation(io.crate.analyze.relations.DocTableRelation) AnalyzedRelation(io.crate.analyze.relations.AnalyzedRelation) SqlExpressions(io.crate.testing.SqlExpressions) Before(org.junit.Before)

Example 9 with OnHeapMemoryManager

use of io.crate.memory.OnHeapMemoryManager in project crate by crate.

the class ProjectingRowConsumerTest method prepare.

@Before
public void prepare() {
    nodeCtx = createNodeContext();
    memoryManager = new OnHeapMemoryManager(usedBytes -> {
    });
    projectorFactory = new ProjectionToProjectorVisitor(clusterService, new NodeLimits(new ClusterSettings(Settings.EMPTY, ClusterSettings.BUILT_IN_CLUSTER_SETTINGS)), new NoneCircuitBreakerService(), nodeCtx, THREAD_POOL, Settings.EMPTY, mock(TransportActionProvider.class, Answers.RETURNS_DEEP_STUBS), new InputFactory(nodeCtx), new EvaluatingNormalizer(nodeCtx, RowGranularity.SHARD, r -> Literal.ofUnchecked(r.valueType(), r.valueType().implicitCast("1")), null), t -> null, t -> null, Version.CURRENT, new ShardId("dummy", UUID.randomUUID().toString(), 0), Map.of(LocalFsFileOutputFactory.NAME, new LocalFsFileOutputFactory()));
}
Also used : ShardId(org.elasticsearch.index.shard.ShardId) TransactionContext(io.crate.metadata.TransactionContext) java.util(java.util) InputColumn(io.crate.expression.symbol.InputColumn) SENTINEL(io.crate.data.SentinelRow.SENTINEL) CompletableFuture(java.util.concurrent.CompletableFuture) BatchIterator(io.crate.data.BatchIterator) SearchPath(io.crate.metadata.SearchPath) EvaluatingNormalizer(io.crate.expression.eval.EvaluatingNormalizer) CrateDummyClusterServiceUnitTest(io.crate.test.integration.CrateDummyClusterServiceUnitTest) LocalFsFileOutputFactory(io.crate.execution.engine.export.LocalFsFileOutputFactory) NodeLimits(io.crate.execution.jobs.NodeLimits) Settings(org.elasticsearch.common.settings.Settings) NoneCircuitBreakerService(org.elasticsearch.indices.breaker.NoneCircuitBreakerService) UnhandledServerException(io.crate.exceptions.UnhandledServerException) TestingHelpers.createNodeContext(io.crate.testing.TestingHelpers.createNodeContext) Nullable(javax.annotation.Nullable) Before(org.junit.Before) NodeContext(io.crate.metadata.NodeContext) Answers(org.mockito.Answers) GroupProjection(io.crate.execution.dsl.projection.GroupProjection) FilterProjection(io.crate.execution.dsl.projection.FilterProjection) InMemoryBatchIterator(io.crate.data.InMemoryBatchIterator) Test(org.junit.Test) EqOperator(io.crate.expression.operator.EqOperator) Function(io.crate.expression.symbol.Function) RamAccounting(io.crate.breaker.RamAccounting) TransportActionProvider(io.crate.execution.TransportActionProvider) RowConsumer(io.crate.data.RowConsumer) WriterProjection(io.crate.execution.dsl.projection.WriterProjection) ClusterSettings(org.elasticsearch.common.settings.ClusterSettings) Version(org.elasticsearch.Version) RowGranularity(io.crate.metadata.RowGranularity) Row(io.crate.data.Row) Literal(io.crate.expression.symbol.Literal) Symbol(io.crate.expression.symbol.Symbol) AggregateMode(io.crate.expression.symbol.AggregateMode) DataTypes(io.crate.types.DataTypes) Matchers.is(org.hamcrest.Matchers.is) OnHeapMemoryManager(io.crate.memory.OnHeapMemoryManager) InputFactory(io.crate.expression.InputFactory) TestingRowConsumer(io.crate.testing.TestingRowConsumer) Mockito.mock(org.mockito.Mockito.mock) CoordinatorTxnCtx(io.crate.metadata.CoordinatorTxnCtx) ShardId(org.elasticsearch.index.shard.ShardId) InputFactory(io.crate.expression.InputFactory) OnHeapMemoryManager(io.crate.memory.OnHeapMemoryManager) ClusterSettings(org.elasticsearch.common.settings.ClusterSettings) LocalFsFileOutputFactory(io.crate.execution.engine.export.LocalFsFileOutputFactory) EvaluatingNormalizer(io.crate.expression.eval.EvaluatingNormalizer) NodeLimits(io.crate.execution.jobs.NodeLimits) NoneCircuitBreakerService(org.elasticsearch.indices.breaker.NoneCircuitBreakerService) Before(org.junit.Before)

Example 10 with OnHeapMemoryManager

use of io.crate.memory.OnHeapMemoryManager in project crate by crate.

the class GroupingLongCollectorBenchmark method createGroupingCollector.

@Setup
public void createGroupingCollector() throws Exception {
    IndexWriter iw = new IndexWriter(new ByteBuffersDirectory(), new IndexWriterConfig(new StandardAnalyzer()));
    Functions functions = new ModulesBuilder().add(new AggregationImplModule()).createInjector().getInstance(Functions.class);
    SumAggregation<?> sumAgg = (SumAggregation<?>) functions.getQualified(Signature.aggregate(SumAggregation.NAME, DataTypes.INTEGER.getTypeSignature(), DataTypes.LONG.getTypeSignature()), List.of(DataTypes.INTEGER), DataTypes.INTEGER);
    var memoryManager = new OnHeapMemoryManager(bytes -> {
    });
    groupBySumCollector = createGroupBySumCollector(sumAgg, memoryManager);
    int size = 20_000_000;
    rows = new ArrayList<>(size);
    numbers = new long[size];
    for (int i = 0; i < size; i++) {
        long value = (long) i % 200;
        rows.add(new Row1(value));
        numbers[i] = value;
        var doc = new Document();
        doc.add(new NumericDocValuesField("x", value));
        doc.add(new SortedNumericDocValuesField("y", value));
        iw.addDocument(doc);
    }
    iw.commit();
    iw.forceMerge(1, true);
    searcher = new IndexSearcher(DirectoryReader.open(iw));
}
Also used : IndexSearcher(org.apache.lucene.search.IndexSearcher) OnHeapMemoryManager(io.crate.memory.OnHeapMemoryManager) Functions(io.crate.metadata.Functions) Document(org.apache.lucene.document.Document) AggregationImplModule(io.crate.execution.engine.aggregation.impl.AggregationImplModule) Row1(io.crate.data.Row1) SortedNumericDocValuesField(org.apache.lucene.document.SortedNumericDocValuesField) SortedNumericDocValuesField(org.apache.lucene.document.SortedNumericDocValuesField) NumericDocValuesField(org.apache.lucene.document.NumericDocValuesField) IndexWriter(org.apache.lucene.index.IndexWriter) ByteBuffersDirectory(org.apache.lucene.store.ByteBuffersDirectory) StandardAnalyzer(org.apache.lucene.analysis.standard.StandardAnalyzer) SumAggregation(io.crate.execution.engine.aggregation.impl.SumAggregation) ModulesBuilder(org.elasticsearch.common.inject.ModulesBuilder) IndexWriterConfig(org.apache.lucene.index.IndexWriterConfig) Setup(org.openjdk.jmh.annotations.Setup)

Aggregations

OnHeapMemoryManager (io.crate.memory.OnHeapMemoryManager)10 RamAccounting (io.crate.breaker.RamAccounting)7 Row (io.crate.data.Row)6 Literal (io.crate.expression.symbol.Literal)6 BatchIterator (io.crate.data.BatchIterator)5 InMemoryBatchIterator (io.crate.data.InMemoryBatchIterator)5 AggregateMode (io.crate.expression.symbol.AggregateMode)5 CoordinatorTxnCtx (io.crate.metadata.CoordinatorTxnCtx)5 Version (org.elasticsearch.Version)5 Input (io.crate.data.Input)4 SENTINEL (io.crate.data.SentinelRow.SENTINEL)4 InputCollectExpression (io.crate.execution.engine.collect.InputCollectExpression)4 InputFactory (io.crate.expression.InputFactory)4 Functions (io.crate.metadata.Functions)4 RowGranularity (io.crate.metadata.RowGranularity)4 CrateDummyClusterServiceUnitTest (io.crate.test.integration.CrateDummyClusterServiceUnitTest)4 TestingRowConsumer (io.crate.testing.TestingRowConsumer)4 DataTypes (io.crate.types.DataTypes)4 Arrays (java.util.Arrays)4 Collections (java.util.Collections)4