Search in sources :

Example 1 with BoundCreateTable

use of io.crate.analyze.BoundCreateTable in project crate by crate.

the class TableCreator method create.

public CompletableFuture<Long> create(BoundCreateTable createTable) {
    var templateName = createTable.templateName();
    var relationName = createTable.tableIdent();
    var createTableRequest = templateName == null ? new CreateTableRequest(new CreateIndexRequest(relationName.indexNameOrAlias(), createTable.tableParameter().settings()).mapping(Constants.DEFAULT_MAPPING_TYPE, createTable.mapping())) : new CreateTableRequest(new PutIndexTemplateRequest(templateName).mapping(Constants.DEFAULT_MAPPING_TYPE, createTable.mapping()).create(true).settings(createTable.tableParameter().settings()).patterns(Collections.singletonList(createTable.templatePrefix())).order(100).alias(new Alias(relationName.indexNameOrAlias())));
    return transportCreateTableAction.execute(createTableRequest, resp -> {
        if (!resp.isAllShardsAcked() && LOGGER.isWarnEnabled()) {
            LOGGER.warn("CREATE TABLE `{}` was not acknowledged. This could lead to inconsistent state.", relationName.fqn());
        }
        return 1L;
    }).exceptionally(error -> {
        Throwable t = SQLExceptions.unwrap(error);
        String message = t.getMessage();
        Throwable cause = t.getCause();
        if ("mapping [default]".equals(message) && cause != null) {
            // the cause has usually a better more detailed error message
            return Exceptions.rethrowRuntimeException(cause);
        } else if (createTable.ifNotExists() && isTableExistsError(t, templateName)) {
            return 0L;
        } else {
            return Exceptions.rethrowRuntimeException(t);
        }
    });
}
Also used : Alias(org.elasticsearch.action.admin.indices.alias.Alias) CompletableFuture(java.util.concurrent.CompletableFuture) ResourceAlreadyExistsException(org.elasticsearch.ResourceAlreadyExistsException) Inject(org.elasticsearch.common.inject.Inject) Exceptions(io.crate.exceptions.Exceptions) Logger(org.apache.logging.log4j.Logger) Constants(io.crate.Constants) PutIndexTemplateRequest(org.elasticsearch.action.admin.indices.template.put.PutIndexTemplateRequest) Singleton(org.elasticsearch.common.inject.Singleton) CreateIndexRequest(org.elasticsearch.action.admin.indices.create.CreateIndexRequest) SQLExceptions(io.crate.exceptions.SQLExceptions) LogManager(org.apache.logging.log4j.LogManager) Collections(java.util.Collections) BoundCreateTable(io.crate.analyze.BoundCreateTable) Nullable(javax.annotation.Nullable) Alias(org.elasticsearch.action.admin.indices.alias.Alias) PutIndexTemplateRequest(org.elasticsearch.action.admin.indices.template.put.PutIndexTemplateRequest) CreateIndexRequest(org.elasticsearch.action.admin.indices.create.CreateIndexRequest)

Example 2 with BoundCreateTable

use of io.crate.analyze.BoundCreateTable in project crate by crate.

the class CreateTablePlan method executeOrFail.

@Override
public void executeOrFail(DependencyCarrier dependencies, PlannerContext plannerContext, RowConsumer consumer, Row params, SubQueryResults subQueryResults) {
    BoundCreateTable boundCreateTable = bind(createTable, plannerContext.transactionContext(), dependencies.nodeContext(), params, subQueryResults, numberOfShards, schemas, dependencies.fulltextAnalyzerResolver());
    if (boundCreateTable.noOp()) {
        consumer.accept(InMemoryBatchIterator.empty(SENTINEL), null);
        return;
    }
    tableCreator.create(boundCreateTable).whenComplete(new OneRowActionListener<>(consumer, rCount -> new Row1(rCount == null ? -1 : rCount)));
}
Also used : AnalyzedColumnDefinition(io.crate.analyze.AnalyzedColumnDefinition) RelationName(io.crate.metadata.RelationName) IndexMetadata(org.elasticsearch.cluster.metadata.IndexMetadata) SENTINEL(io.crate.data.SentinelRow.SENTINEL) GenericProperties(io.crate.sql.tree.GenericProperties) Function(java.util.function.Function) DependencyCarrier(io.crate.planner.DependencyCarrier) SymbolEvaluator(io.crate.analyze.SymbolEvaluator) Settings(org.elasticsearch.common.settings.Settings) Locale(java.util.Locale) OneRowActionListener(io.crate.execution.support.OneRowActionListener) BoundCreateTable(io.crate.analyze.BoundCreateTable) FulltextAnalyzerResolver(io.crate.metadata.FulltextAnalyzerResolver) Nullable(javax.annotation.Nullable) TableParameter(io.crate.analyze.TableParameter) CreateTable(io.crate.sql.tree.CreateTable) AnalyzedCreateTable(io.crate.analyze.AnalyzedCreateTable) NodeContext(io.crate.metadata.NodeContext) PartitionedBy(io.crate.sql.tree.PartitionedBy) TableCreator(io.crate.execution.ddl.tables.TableCreator) TablePropertiesAnalyzer(io.crate.analyze.TablePropertiesAnalyzer) InMemoryBatchIterator(io.crate.data.InMemoryBatchIterator) ColumnIdent(io.crate.metadata.ColumnIdent) SubQueryAndParamBinder(io.crate.planner.operators.SubQueryAndParamBinder) TableParameters(io.crate.analyze.TableParameters) RowConsumer(io.crate.data.RowConsumer) AnalyzedTableElements(io.crate.analyze.AnalyzedTableElements) NumberOfShards(io.crate.analyze.NumberOfShards) ClusteredBy(io.crate.sql.tree.ClusteredBy) Row(io.crate.data.Row) Symbol(io.crate.expression.symbol.Symbol) PlannerContext(io.crate.planner.PlannerContext) Plan(io.crate.planner.Plan) SubQueryResults(io.crate.planner.operators.SubQueryResults) Optional(java.util.Optional) Schemas(io.crate.metadata.Schemas) VisibleForTesting(io.crate.common.annotations.VisibleForTesting) Row1(io.crate.data.Row1) CoordinatorTxnCtx(io.crate.metadata.CoordinatorTxnCtx) Row1(io.crate.data.Row1) BoundCreateTable(io.crate.analyze.BoundCreateTable)

Example 3 with BoundCreateTable

use of io.crate.analyze.BoundCreateTable in project crate by crate.

the class CreateTablePlan method bind.

@VisibleForTesting
public static BoundCreateTable bind(AnalyzedCreateTable createTable, CoordinatorTxnCtx txnCtx, NodeContext nodeCtx, Row params, SubQueryResults subQueryResults, NumberOfShards numberOfShards, Schemas schemas, FulltextAnalyzerResolver fulltextAnalyzerResolver) {
    Function<? super Symbol, Object> eval = x -> SymbolEvaluator.evaluate(txnCtx, nodeCtx, x, params, subQueryResults);
    CreateTable<Symbol> table = createTable.createTable();
    RelationName relationName = createTable.relationName();
    GenericProperties<Object> properties = table.properties().map(eval);
    AnalyzedTableElements<Object> tableElements = createTable.analyzedTableElements().map(eval);
    TableParameter tableParameter = new TableParameter();
    Optional<ClusteredBy<Object>> mappedClusteredBy = table.clusteredBy().map(x -> x.map(eval));
    Integer numShards = mappedClusteredBy.flatMap(ClusteredBy::numberOfShards).map(numberOfShards::fromNumberOfShards).orElseGet(numberOfShards::defaultNumberOfShards);
    tableParameter.settingsBuilder().put(IndexMetadata.SETTING_NUMBER_OF_SHARDS, numShards);
    // apply default in case it is not specified in the properties,
    // if it is it will get overwritten afterwards.
    TablePropertiesAnalyzer.analyzeWithBoundValues(tableParameter, TableParameters.TABLE_CREATE_PARAMETER_INFO, properties, true);
    AnalyzedTableElements<Symbol> tableElementsWithExpressions = createTable.analyzedTableElementsWithExpressions().map(x -> SubQueryAndParamBinder.convert(x, params, subQueryResults));
    // validate table elements
    AnalyzedTableElements.finalizeAndValidate(relationName, tableElementsWithExpressions, tableElements);
    // update table settings
    Settings tableSettings = AnalyzedTableElements.validateAndBuildSettings(tableElements, fulltextAnalyzerResolver);
    tableParameter.settingsBuilder().put(tableSettings);
    ColumnIdent routingColumn = mappedClusteredBy.map(clusteredBy -> resolveRoutingFromClusteredBy(clusteredBy, tableElements)).orElse(null);
    Optional<PartitionedBy<Object>> partitionedByOptional = table.partitionedBy().map(x -> x.map(eval));
    partitionedByOptional.ifPresent(partitionedBy -> processPartitionedBy(partitionedByOptional.get(), tableElements, relationName, routingColumn));
    return new BoundCreateTable(relationName, tableElements, tableParameter, routingColumn, table.ifNotExists(), schemas);
}
Also used : AnalyzedColumnDefinition(io.crate.analyze.AnalyzedColumnDefinition) RelationName(io.crate.metadata.RelationName) IndexMetadata(org.elasticsearch.cluster.metadata.IndexMetadata) SENTINEL(io.crate.data.SentinelRow.SENTINEL) GenericProperties(io.crate.sql.tree.GenericProperties) Function(java.util.function.Function) DependencyCarrier(io.crate.planner.DependencyCarrier) SymbolEvaluator(io.crate.analyze.SymbolEvaluator) Settings(org.elasticsearch.common.settings.Settings) Locale(java.util.Locale) OneRowActionListener(io.crate.execution.support.OneRowActionListener) BoundCreateTable(io.crate.analyze.BoundCreateTable) FulltextAnalyzerResolver(io.crate.metadata.FulltextAnalyzerResolver) Nullable(javax.annotation.Nullable) TableParameter(io.crate.analyze.TableParameter) CreateTable(io.crate.sql.tree.CreateTable) AnalyzedCreateTable(io.crate.analyze.AnalyzedCreateTable) NodeContext(io.crate.metadata.NodeContext) PartitionedBy(io.crate.sql.tree.PartitionedBy) TableCreator(io.crate.execution.ddl.tables.TableCreator) TablePropertiesAnalyzer(io.crate.analyze.TablePropertiesAnalyzer) InMemoryBatchIterator(io.crate.data.InMemoryBatchIterator) ColumnIdent(io.crate.metadata.ColumnIdent) SubQueryAndParamBinder(io.crate.planner.operators.SubQueryAndParamBinder) TableParameters(io.crate.analyze.TableParameters) RowConsumer(io.crate.data.RowConsumer) AnalyzedTableElements(io.crate.analyze.AnalyzedTableElements) NumberOfShards(io.crate.analyze.NumberOfShards) ClusteredBy(io.crate.sql.tree.ClusteredBy) Row(io.crate.data.Row) Symbol(io.crate.expression.symbol.Symbol) PlannerContext(io.crate.planner.PlannerContext) Plan(io.crate.planner.Plan) SubQueryResults(io.crate.planner.operators.SubQueryResults) Optional(java.util.Optional) Schemas(io.crate.metadata.Schemas) VisibleForTesting(io.crate.common.annotations.VisibleForTesting) Row1(io.crate.data.Row1) CoordinatorTxnCtx(io.crate.metadata.CoordinatorTxnCtx) Symbol(io.crate.expression.symbol.Symbol) TableParameter(io.crate.analyze.TableParameter) PartitionedBy(io.crate.sql.tree.PartitionedBy) ColumnIdent(io.crate.metadata.ColumnIdent) BoundCreateTable(io.crate.analyze.BoundCreateTable) ClusteredBy(io.crate.sql.tree.ClusteredBy) RelationName(io.crate.metadata.RelationName) Settings(org.elasticsearch.common.settings.Settings) VisibleForTesting(io.crate.common.annotations.VisibleForTesting)

Example 4 with BoundCreateTable

use of io.crate.analyze.BoundCreateTable in project crate by crate.

the class DocIndexMetadataTest method getDocIndexMetadataFromStatement.

private DocIndexMetadata getDocIndexMetadataFromStatement(String stmt) throws IOException {
    Statement statement = SqlParser.createStatement(stmt);
    DocTableInfoFactory docTableInfoFactory = new InternalDocTableInfoFactory(nodeCtx, new IndexNameExpressionResolver());
    ViewInfoFactory viewInfoFactory = (ident, state) -> null;
    DocSchemaInfo docSchemaInfo = new DocSchemaInfo(Schemas.DOC_SCHEMA_NAME, clusterService, nodeCtx, udfService, viewInfoFactory, docTableInfoFactory);
    Path homeDir = createTempDir();
    Schemas schemas = new Schemas(Map.of("doc", docSchemaInfo), clusterService, new DocSchemaInfoFactory(docTableInfoFactory, viewInfoFactory, nodeCtx, udfService));
    FulltextAnalyzerResolver fulltextAnalyzerResolver = new FulltextAnalyzerResolver(clusterService, new AnalysisRegistry(new Environment(Settings.builder().put(Environment.PATH_HOME_SETTING.getKey(), homeDir.toString()).build(), homeDir.resolve("config")), emptyMap(), emptyMap(), emptyMap(), emptyMap(), emptyMap(), emptyMap(), emptyMap(), emptyMap(), emptyMap()));
    CreateTableStatementAnalyzer analyzer = new CreateTableStatementAnalyzer(nodeCtx);
    Analysis analysis = new Analysis(new CoordinatorTxnCtx(SessionContext.systemSessionContext()), ParamTypeHints.EMPTY);
    CoordinatorTxnCtx txnCtx = new CoordinatorTxnCtx(SessionContext.systemSessionContext());
    AnalyzedCreateTable analyzedCreateTable = analyzer.analyze((CreateTable<Expression>) statement, analysis.paramTypeHints(), analysis.transactionContext());
    BoundCreateTable analyzedStatement = CreateTablePlan.bind(analyzedCreateTable, txnCtx, nodeCtx, Row.EMPTY, SubQueryResults.EMPTY, new NumberOfShards(clusterService), schemas, fulltextAnalyzerResolver);
    Settings.Builder settingsBuilder = Settings.builder().put("index.number_of_shards", 1).put("index.number_of_replicas", 0).put("index.version.created", org.elasticsearch.Version.CURRENT).put(analyzedStatement.tableParameter().settings());
    IndexMetadata indexMetadata = IndexMetadata.builder(analyzedStatement.tableIdent().name()).settings(settingsBuilder).putMapping(new MappingMetadata(Constants.DEFAULT_MAPPING_TYPE, analyzedStatement.mapping())).build();
    return newMeta(indexMetadata, analyzedStatement.tableIdent().name());
}
Also used : ParamTypeHints(io.crate.analyze.ParamTypeHints) Arrays(java.util.Arrays) SessionContext(io.crate.action.sql.SessionContext) RelationName(io.crate.metadata.RelationName) IndexMetadata(org.elasticsearch.cluster.metadata.IndexMetadata) Environment(org.elasticsearch.env.Environment) Matchers.not(org.hamcrest.Matchers.not) UserDefinedFunctionService(io.crate.expression.udf.UserDefinedFunctionService) CrateDummyClusterServiceUnitTest(io.crate.test.integration.CrateDummyClusterServiceUnitTest) XContentBuilder(org.elasticsearch.common.xcontent.XContentBuilder) Matchers.hasItems(org.hamcrest.Matchers.hasItems) Matchers.hasKey(org.hamcrest.Matchers.hasKey) ArrayType(io.crate.types.ArrayType) Settings(org.elasticsearch.common.settings.Settings) Map(java.util.Map) Matchers.nullValue(org.hamcrest.Matchers.nullValue) CreateTableStatementAnalyzer(io.crate.analyze.CreateTableStatementAnalyzer) Path(java.nio.file.Path) TestingHelpers.createNodeContext(io.crate.testing.TestingHelpers.createNodeContext) BoundCreateTable(io.crate.analyze.BoundCreateTable) FulltextAnalyzerResolver(io.crate.metadata.FulltextAnalyzerResolver) AnalyzedCreateTable(io.crate.analyze.AnalyzedCreateTable) NodeContext(io.crate.metadata.NodeContext) GeneratedReference(io.crate.metadata.GeneratedReference) SymbolMatchers.isFunction(io.crate.testing.SymbolMatchers.isFunction) AnalysisRegistry(org.elasticsearch.index.analysis.AnalysisRegistry) BytesReference(org.elasticsearch.common.bytes.BytesReference) Collectors(java.util.stream.Collectors) Lists2(io.crate.common.collections.Lists2) CheckConstraint(io.crate.sql.tree.CheckConstraint) IsCollectionWithSize.hasSize(org.hamcrest.collection.IsCollectionWithSize.hasSize) SymbolMatchers.isReference(io.crate.testing.SymbolMatchers.isReference) Matchers.instanceOf(org.hamcrest.Matchers.instanceOf) List(java.util.List) Row(io.crate.data.Row) MappingMetadata(org.elasticsearch.cluster.metadata.MappingMetadata) Matchers.contains(org.hamcrest.Matchers.contains) Matchers.containsInAnyOrder(org.hamcrest.Matchers.containsInAnyOrder) DataTypes(io.crate.types.DataTypes) Matchers.equalTo(org.hamcrest.Matchers.equalTo) SubQueryResults(io.crate.planner.operators.SubQueryResults) Statement(io.crate.sql.tree.Statement) Matchers.is(org.hamcrest.Matchers.is) Expression(io.crate.sql.tree.Expression) CoordinatorTxnCtx(io.crate.metadata.CoordinatorTxnCtx) XContentFactory(org.elasticsearch.common.xcontent.XContentFactory) XContentType(org.elasticsearch.common.xcontent.XContentType) Analysis(io.crate.analyze.Analysis) ViewInfoFactory(io.crate.metadata.view.ViewInfoFactory) HashMap(java.util.HashMap) ObjectType(io.crate.types.ObjectType) IndexReference(io.crate.metadata.IndexReference) ArrayList(java.util.ArrayList) XContentHelper(org.elasticsearch.common.xcontent.XContentHelper) Constants(io.crate.Constants) CreateTablePlan(io.crate.planner.node.ddl.CreateTablePlan) ColumnPolicy(io.crate.sql.tree.ColumnPolicy) SqlParser(io.crate.sql.parser.SqlParser) Before(org.junit.Before) CreateTable(io.crate.sql.tree.CreateTable) Collections.emptyMap(java.util.Collections.emptyMap) StringType(io.crate.types.StringType) ColumnIdent(io.crate.metadata.ColumnIdent) Reference(io.crate.metadata.Reference) DataType(io.crate.types.DataType) Matchers(org.hamcrest.Matchers) Test(org.junit.Test) IOException(java.io.IOException) NumberOfShards(io.crate.analyze.NumberOfShards) TreeMap(java.util.TreeMap) Schemas(io.crate.metadata.Schemas) IndexNameExpressionResolver(org.elasticsearch.cluster.metadata.IndexNameExpressionResolver) SymbolMatchers.isLiteral(io.crate.testing.SymbolMatchers.isLiteral) Path(java.nio.file.Path) CoordinatorTxnCtx(io.crate.metadata.CoordinatorTxnCtx) Statement(io.crate.sql.tree.Statement) AnalyzedCreateTable(io.crate.analyze.AnalyzedCreateTable) Schemas(io.crate.metadata.Schemas) NumberOfShards(io.crate.analyze.NumberOfShards) AnalysisRegistry(org.elasticsearch.index.analysis.AnalysisRegistry) CreateTableStatementAnalyzer(io.crate.analyze.CreateTableStatementAnalyzer) BoundCreateTable(io.crate.analyze.BoundCreateTable) ViewInfoFactory(io.crate.metadata.view.ViewInfoFactory) FulltextAnalyzerResolver(io.crate.metadata.FulltextAnalyzerResolver) Expression(io.crate.sql.tree.Expression) Analysis(io.crate.analyze.Analysis) Environment(org.elasticsearch.env.Environment) IndexNameExpressionResolver(org.elasticsearch.cluster.metadata.IndexNameExpressionResolver) IndexMetadata(org.elasticsearch.cluster.metadata.IndexMetadata) MappingMetadata(org.elasticsearch.cluster.metadata.MappingMetadata) Settings(org.elasticsearch.common.settings.Settings)

Aggregations

BoundCreateTable (io.crate.analyze.BoundCreateTable)4 AnalyzedCreateTable (io.crate.analyze.AnalyzedCreateTable)3 NumberOfShards (io.crate.analyze.NumberOfShards)3 Row (io.crate.data.Row)3 ColumnIdent (io.crate.metadata.ColumnIdent)3 CoordinatorTxnCtx (io.crate.metadata.CoordinatorTxnCtx)3 FulltextAnalyzerResolver (io.crate.metadata.FulltextAnalyzerResolver)3 NodeContext (io.crate.metadata.NodeContext)3 RelationName (io.crate.metadata.RelationName)3 Schemas (io.crate.metadata.Schemas)3 SubQueryResults (io.crate.planner.operators.SubQueryResults)3 CreateTable (io.crate.sql.tree.CreateTable)3 Constants (io.crate.Constants)2 AnalyzedColumnDefinition (io.crate.analyze.AnalyzedColumnDefinition)2 AnalyzedTableElements (io.crate.analyze.AnalyzedTableElements)2 SymbolEvaluator (io.crate.analyze.SymbolEvaluator)2 TableParameter (io.crate.analyze.TableParameter)2 TableParameters (io.crate.analyze.TableParameters)2 TablePropertiesAnalyzer (io.crate.analyze.TablePropertiesAnalyzer)2 VisibleForTesting (io.crate.common.annotations.VisibleForTesting)2