Search in sources :

Example 16 with Functions

use of io.crate.metadata.Functions in project crate by crate.

the class MapSideDataCollectOperationTest method testFileUriCollect.

@Test
public void testFileUriCollect() throws Exception {
    ClusterService clusterService = new NoopClusterService();
    Functions functions = getFunctions();
    CollectSourceResolver collectSourceResolver = mock(CollectSourceResolver.class);
    when(collectSourceResolver.getService(any(RoutedCollectPhase.class))).thenReturn(new FileCollectSource(functions, clusterService, Collections.<String, FileInputFactory>emptyMap()));
    MapSideDataCollectOperation collectOperation = new MapSideDataCollectOperation(collectSourceResolver, threadPool);
    File tmpFile = temporaryFolder.newFile("fileUriCollectOperation.json");
    try (OutputStreamWriter writer = new OutputStreamWriter(new FileOutputStream(tmpFile), StandardCharsets.UTF_8)) {
        writer.write("{\"name\": \"Arthur\", \"id\": 4, \"details\": {\"age\": 38}}\n");
        writer.write("{\"id\": 5, \"name\": \"Trillian\", \"details\": {\"age\": 33}}\n");
    }
    FileUriCollectPhase collectNode = new FileUriCollectPhase(UUID.randomUUID(), 0, "test", Collections.singletonList("noop_id"), Literal.of(Paths.get(tmpFile.toURI()).toUri().toString()), Arrays.<Symbol>asList(createReference("name", DataTypes.STRING), createReference(new ColumnIdent("details", "age"), DataTypes.INTEGER)), Collections.emptyList(), null, false);
    String threadPoolName = JobCollectContext.threadPoolName(collectNode, "noop_id");
    TestingBatchConsumer consumer = new TestingBatchConsumer();
    JobCollectContext jobCollectContext = mock(JobCollectContext.class);
    CrateCollector collectors = collectOperation.createCollector(collectNode, consumer, jobCollectContext);
    collectOperation.launchCollector(collectors, threadPoolName);
    assertThat(new CollectionBucket(consumer.getResult()), contains(isRow("Arthur", 38), isRow("Trillian", 33)));
}
Also used : CollectSourceResolver(io.crate.operation.collect.sources.CollectSourceResolver) Functions(io.crate.metadata.Functions) FileUriCollectPhase(io.crate.planner.node.dql.FileUriCollectPhase) ColumnIdent(io.crate.metadata.ColumnIdent) NoopClusterService(org.elasticsearch.test.cluster.NoopClusterService) ClusterService(org.elasticsearch.cluster.ClusterService) FileCollectSource(io.crate.operation.collect.sources.FileCollectSource) FileOutputStream(java.io.FileOutputStream) FileInputFactory(io.crate.operation.collect.files.FileInputFactory) OutputStreamWriter(java.io.OutputStreamWriter) TestingBatchConsumer(io.crate.testing.TestingBatchConsumer) NoopClusterService(org.elasticsearch.test.cluster.NoopClusterService) File(java.io.File) RoutedCollectPhase(io.crate.planner.node.dql.RoutedCollectPhase) CollectionBucket(io.crate.data.CollectionBucket) Test(org.junit.Test) CrateUnitTest(io.crate.test.integration.CrateUnitTest)

Example 17 with Functions

use of io.crate.metadata.Functions in project crate by crate.

the class CurrentSchemaFunctionTest method testNormalizeCurrentSchemaDefaultSchema.

@Test
public void testNormalizeCurrentSchemaDefaultSchema() throws Exception {
    sqlExpressions = new SqlExpressions(tableSources, new SessionContext(0, Option.NONE, null));
    functions = sqlExpressions.getInstance(Functions.class);
    assertNormalize("current_schema()", isLiteral("doc"), false);
}
Also used : SessionContext(io.crate.action.sql.SessionContext) Functions(io.crate.metadata.Functions) SqlExpressions(io.crate.testing.SqlExpressions) AbstractScalarFunctionsTest(io.crate.operation.scalar.AbstractScalarFunctionsTest) Test(org.junit.Test)

Example 18 with Functions

use of io.crate.metadata.Functions in project crate by crate.

the class GroupingLongCollectorBenchmark method createGroupingCollector.

@Setup
public void createGroupingCollector() throws Exception {
    IndexWriter iw = new IndexWriter(new ByteBuffersDirectory(), new IndexWriterConfig(new StandardAnalyzer()));
    Functions functions = new ModulesBuilder().add(new AggregationImplModule()).createInjector().getInstance(Functions.class);
    SumAggregation<?> sumAgg = (SumAggregation<?>) functions.getQualified(Signature.aggregate(SumAggregation.NAME, DataTypes.INTEGER.getTypeSignature(), DataTypes.LONG.getTypeSignature()), List.of(DataTypes.INTEGER), DataTypes.INTEGER);
    var memoryManager = new OnHeapMemoryManager(bytes -> {
    });
    groupBySumCollector = createGroupBySumCollector(sumAgg, memoryManager);
    int size = 20_000_000;
    rows = new ArrayList<>(size);
    numbers = new long[size];
    for (int i = 0; i < size; i++) {
        long value = (long) i % 200;
        rows.add(new Row1(value));
        numbers[i] = value;
        var doc = new Document();
        doc.add(new NumericDocValuesField("x", value));
        doc.add(new SortedNumericDocValuesField("y", value));
        iw.addDocument(doc);
    }
    iw.commit();
    iw.forceMerge(1, true);
    searcher = new IndexSearcher(DirectoryReader.open(iw));
}
Also used : IndexSearcher(org.apache.lucene.search.IndexSearcher) OnHeapMemoryManager(io.crate.memory.OnHeapMemoryManager) Functions(io.crate.metadata.Functions) Document(org.apache.lucene.document.Document) AggregationImplModule(io.crate.execution.engine.aggregation.impl.AggregationImplModule) Row1(io.crate.data.Row1) SortedNumericDocValuesField(org.apache.lucene.document.SortedNumericDocValuesField) SortedNumericDocValuesField(org.apache.lucene.document.SortedNumericDocValuesField) NumericDocValuesField(org.apache.lucene.document.NumericDocValuesField) IndexWriter(org.apache.lucene.index.IndexWriter) ByteBuffersDirectory(org.apache.lucene.store.ByteBuffersDirectory) StandardAnalyzer(org.apache.lucene.analysis.standard.StandardAnalyzer) SumAggregation(io.crate.execution.engine.aggregation.impl.SumAggregation) ModulesBuilder(org.elasticsearch.common.inject.ModulesBuilder) IndexWriterConfig(org.apache.lucene.index.IndexWriterConfig) Setup(org.openjdk.jmh.annotations.Setup)

Example 19 with Functions

use of io.crate.metadata.Functions in project crate by crate.

the class SQLIntegrationTestCase method assertFunctionIsCreatedOnAll.

public void assertFunctionIsCreatedOnAll(String schema, String name, List<DataType<?>> argTypes) throws Exception {
    SearchPath searchPath = SearchPath.pathWithPGCatalogAndDoc();
    assertBusy(() -> {
        Iterable<Functions> functions = internalCluster().getInstances(Functions.class);
        for (Functions function : functions) {
            FunctionImplementation func = function.get(schema, name, Lists2.map(argTypes, t -> Literal.of(t, null)), searchPath);
            assertThat(func, is(not(nullValue())));
            assertThat(func.info().ident().argumentTypes(), is(equalTo(argTypes)));
        }
    }, 20L, TimeUnit.SECONDS);
}
Also used : ParamTypeHints(io.crate.analyze.ParamTypeHints) SessionContext(io.crate.action.sql.SessionContext) IndexMetadata(org.elasticsearch.cluster.metadata.IndexMetadata) SessionSettings(io.crate.metadata.settings.SessionSettings) Matchers.not(org.hamcrest.Matchers.not) Inherited(java.lang.annotation.Inherited) XContentBuilder(org.elasticsearch.common.xcontent.XContentBuilder) Analyzer(io.crate.analyze.Analyzer) NodeLimits(io.crate.execution.jobs.NodeLimits) DependencyCarrier(io.crate.planner.DependencyCarrier) Functions(io.crate.metadata.Functions) Matchers.nullValue(org.hamcrest.Matchers.nullValue) Map(java.util.Map) SQLTransportExecutor(io.crate.testing.SQLTransportExecutor) RandomStrings(com.carrotsearch.randomizedtesting.generators.RandomStrings) TableInfo(io.crate.metadata.table.TableInfo) RandomizedContext(com.carrotsearch.randomizedtesting.RandomizedContext) TasksService(io.crate.execution.jobs.TasksService) Logger(org.apache.logging.log4j.Logger) Row(io.crate.data.Row) SubQueryResults(io.crate.planner.operators.SubQueryResults) ClusterStateRequest(org.elasticsearch.action.admin.cluster.state.ClusterStateRequest) Matchers.is(org.hamcrest.Matchers.is) XContentFactory(org.elasticsearch.common.xcontent.XContentFactory) ClusterService(org.elasticsearch.cluster.service.ClusterService) BoundTransportAddress(org.elasticsearch.common.transport.BoundTransportAddress) Metadata(org.elasticsearch.cluster.metadata.Metadata) TestName(org.junit.rules.TestName) Timeout(org.junit.rules.Timeout) IndicesService(org.elasticsearch.indices.IndicesService) Nullable(javax.annotation.Nullable) PSQL_PORT_SETTING(io.crate.protocols.postgres.PostgresNetty.PSQL_PORT_SETTING) Before(org.junit.Before) Identifiers(io.crate.sql.Identifiers) Client(org.elasticsearch.client.Client) IndexService(org.elasticsearch.index.IndexService) IndexShard(org.elasticsearch.index.shard.IndexShard) IOException(java.io.IOException) Planner(io.crate.planner.Planner) Field(java.lang.reflect.Field) RoutingProvider(io.crate.metadata.RoutingProvider) Literal(io.crate.expression.symbol.Literal) PlannerContext(io.crate.planner.PlannerContext) Matcher(org.hamcrest.Matcher) Plan(io.crate.planner.Plan) PostgresNetty(io.crate.protocols.postgres.PostgresNetty) KillableCallable(io.crate.execution.jobs.kill.KillableCallable) Schemas(io.crate.metadata.Schemas) UseJdbc(io.crate.testing.UseJdbc) Randomness(org.elasticsearch.common.Randomness) TransportShardAction(io.crate.execution.dml.TransportShardAction) TransportShardUpsertAction(io.crate.execution.dml.upsert.TransportShardUpsertAction) RelationName(io.crate.metadata.RelationName) UseRandomizedSchema(io.crate.testing.UseRandomizedSchema) Random(java.util.Random) RootTask(io.crate.execution.jobs.RootTask) UseHashJoins(io.crate.testing.UseHashJoins) Settings(org.elasticsearch.common.settings.Settings) TestGroup(com.carrotsearch.randomizedtesting.annotations.TestGroup) Locale(java.util.Locale) After(org.junit.After) Documented(java.lang.annotation.Documented) ThreadPool(org.elasticsearch.threadpool.ThreadPool) Method(java.lang.reflect.Method) SQLOperations(io.crate.action.sql.SQLOperations) NodeContext(io.crate.metadata.NodeContext) User(io.crate.user.User) Collection(java.util.Collection) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) UUID(java.util.UUID) Netty4Plugin(org.elasticsearch.transport.Netty4Plugin) InetSocketAddress(java.net.InetSocketAddress) Lists2(io.crate.common.collections.Lists2) List(java.util.List) Session(io.crate.action.sql.Session) ESIntegTestCase(org.elasticsearch.test.ESIntegTestCase) Symbol(io.crate.expression.symbol.Symbol) FunctionImplementation(io.crate.metadata.FunctionImplementation) Matchers.equalTo(org.hamcrest.Matchers.equalTo) Annotation(java.lang.annotation.Annotation) TimeValue(io.crate.common.unit.TimeValue) Paging(io.crate.data.Paging) TestingRowConsumer(io.crate.testing.TestingRowConsumer) CoordinatorTxnCtx(io.crate.metadata.CoordinatorTxnCtx) UserLookup(io.crate.user.UserLookup) SQLResponse(io.crate.testing.SQLResponse) ThreadPoolExecutor(java.util.concurrent.ThreadPoolExecutor) TransportShardDeleteAction(io.crate.execution.dml.delete.TransportShardDeleteAction) SearchPath(io.crate.metadata.SearchPath) Index(org.elasticsearch.index.Index) SETTING_HTTP_COMPRESSION(org.elasticsearch.http.HttpTransportSettings.SETTING_HTTP_COMPRESSION) TaskId(org.elasticsearch.tasks.TaskId) Retention(java.lang.annotation.Retention) Strings(org.elasticsearch.common.Strings) ElasticsearchTimeoutException(org.elasticsearch.ElasticsearchTimeoutException) Constants(io.crate.Constants) Symbols(io.crate.expression.symbol.Symbols) SqlParser(io.crate.sql.parser.SqlParser) Requests(org.elasticsearch.client.Requests) Listeners(com.carrotsearch.randomizedtesting.annotations.Listeners) TestExecutionConfig(io.crate.testing.TestExecutionConfig) ColumnIdent(io.crate.metadata.ColumnIdent) Plugin(org.elasticsearch.plugins.Plugin) Matchers(org.hamcrest.Matchers) DataType(io.crate.types.DataType) TimeUnit(java.util.concurrent.TimeUnit) ClusterStateResponse(org.elasticsearch.action.admin.cluster.state.ClusterStateResponse) Rule(org.junit.Rule) SystemPropsTestLoggingListener(io.crate.test.integration.SystemPropsTestLoggingListener) LogManager(org.apache.logging.log4j.LogManager) ConfigurationException(org.elasticsearch.common.inject.ConfigurationException) RetentionPolicy(java.lang.annotation.RetentionPolicy) SearchPath(io.crate.metadata.SearchPath) Functions(io.crate.metadata.Functions) FunctionImplementation(io.crate.metadata.FunctionImplementation)

Aggregations

Functions (io.crate.metadata.Functions)19 ModulesBuilder (org.elasticsearch.common.inject.ModulesBuilder)7 NodeContext (io.crate.metadata.NodeContext)6 Before (org.junit.Before)6 Setup (org.openjdk.jmh.annotations.Setup)6 InputFactory (io.crate.expression.InputFactory)5 Row (io.crate.data.Row)4 OnHeapMemoryManager (io.crate.memory.OnHeapMemoryManager)4 SessionContext (io.crate.action.sql.SessionContext)3 Input (io.crate.data.Input)3 Row1 (io.crate.data.Row1)3 AggregationImplModule (io.crate.execution.engine.aggregation.impl.AggregationImplModule)3 InputCollectExpression (io.crate.execution.engine.collect.InputCollectExpression)3 Literal (io.crate.expression.symbol.Literal)3 List (java.util.List)3 RamAccounting (io.crate.breaker.RamAccounting)2 LocalFsFileInputFactory (io.crate.execution.engine.collect.files.LocalFsFileInputFactory)2 AggregateMode (io.crate.expression.symbol.AggregateMode)2 ColumnIdent (io.crate.metadata.ColumnIdent)2 Signature (io.crate.metadata.functions.Signature)2