Search in sources :

Example 1 with ConceptId

use of com.bakdata.conquery.models.identifiable.ids.specific.ConceptId in project conquery by bakdata.

the class IdTests method testEquals.

@Test
public void testEquals() {
    ConceptTreeChildId idA = new ConceptTreeChildId(new ConceptTreeChildId(new ConceptId(new DatasetId("1"), "2"), "3"), "4");
    ConceptTreeChildId idB = new ConceptTreeChildId(new ConceptTreeChildId(new ConceptId(new DatasetId("1"), "2"), "3"), "4");
    assertThat(idA).isEqualTo(idB);
    assertThat(idA).hasSameHashCodeAs(idB);
    assertThat(idA.toString()).isEqualTo(idB.toString());
}
Also used : ConceptTreeChildId(com.bakdata.conquery.models.identifiable.ids.specific.ConceptTreeChildId) ConceptId(com.bakdata.conquery.models.identifiable.ids.specific.ConceptId) DatasetId(com.bakdata.conquery.models.identifiable.ids.specific.DatasetId) Test(org.junit.jupiter.api.Test) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest)

Example 2 with ConceptId

use of com.bakdata.conquery.models.identifiable.ids.specific.ConceptId in project conquery by bakdata.

the class IdTests method testJacksonBinarySerialization.

@Test
public void testJacksonBinarySerialization() throws JsonParseException, JsonMappingException, JsonProcessingException, IOException {
    ConceptTreeChildId id = new ConceptTreeChildId(new ConceptTreeChildId(new ConceptId(new DatasetId("1"), "2"), "3"), "4");
    ObjectMapper mapper = Jackson.BINARY_MAPPER;
    ConceptTreeChildId copy = mapper.readValue(mapper.writeValueAsBytes(id), ConceptTreeChildId.class);
    assertThat(copy).isEqualTo(id);
    assertThat(copy).hasSameHashCodeAs(id);
    assertThat(copy.toString()).isEqualTo(id.toString());
}
Also used : ConceptTreeChildId(com.bakdata.conquery.models.identifiable.ids.specific.ConceptTreeChildId) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) ConceptId(com.bakdata.conquery.models.identifiable.ids.specific.ConceptId) DatasetId(com.bakdata.conquery.models.identifiable.ids.specific.DatasetId) Test(org.junit.jupiter.api.Test) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest)

Example 3 with ConceptId

use of com.bakdata.conquery.models.identifiable.ids.specific.ConceptId in project conquery by bakdata.

the class DefaultColumnNameTest method checkCombinations.

@ParameterizedTest
@MethodSource("provideCombinations")
void checkCombinations(TestConcept concept, boolean hasCQConceptLabel, String expectedColumnName) {
    doAnswer(invocation -> {
        final ConceptId id = invocation.getArgument(0);
        if (!concept.getId().equals(id)) {
            throw new IllegalStateException("Expected the id " + concept.getId() + " but got " + id);
        }
        return concept;
    }).when(DATASET_REGISTRY).resolve(any());
    final CQConcept cqConcept = concept.createCQConcept(hasCQConceptLabel);
    final UniqueNamer uniqNamer = new UniqueNamer(SETTINGS);
    SelectResultInfo info = new SelectResultInfo(concept.extractSelect(cqConcept), cqConcept);
    assertThat(uniqNamer.getUniqueName(info)).isEqualTo(expectedColumnName);
}
Also used : UniqueNamer(com.bakdata.conquery.models.query.resultinfo.UniqueNamer) CQConcept(com.bakdata.conquery.apiv1.query.concept.specific.CQConcept) SelectResultInfo(com.bakdata.conquery.models.query.resultinfo.SelectResultInfo) ConceptId(com.bakdata.conquery.models.identifiable.ids.specific.ConceptId) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest) MethodSource(org.junit.jupiter.params.provider.MethodSource)

Example 4 with ConceptId

use of com.bakdata.conquery.models.identifiable.ids.specific.ConceptId in project conquery by bakdata.

the class AdminDatasetProcessor method deleteTable.

/**
 * Deletes a table if it has no dependents or not forced to do so.
 */
public synchronized List<ConceptId> deleteTable(Table table, boolean force) {
    final Namespace namespace = datasetRegistry.get(table.getDataset().getId());
    final List<Concept<?>> dependentConcepts = namespace.getStorage().getAllConcepts().stream().flatMap(c -> c.getConnectors().stream()).filter(con -> con.getTable().equals(table)).map(Connector::getConcept).collect(Collectors.toList());
    if (force || dependentConcepts.isEmpty()) {
        for (Concept<?> concept : dependentConcepts) {
            deleteConcept(concept);
        }
        namespace.getStorage().getAllImports().stream().filter(imp -> imp.getTable().equals(table)).forEach(this::deleteImport);
        namespace.getStorage().removeTable(table.getId());
        namespace.sendToAll(new RemoveTable(table));
    }
    return dependentConcepts.stream().map(Concept::getId).collect(Collectors.toList());
}
Also used : Concept(com.bakdata.conquery.models.datasets.concepts.Concept) RemoveConcept(com.bakdata.conquery.models.messages.namespaces.specific.RemoveConcept) UpdateConcept(com.bakdata.conquery.models.messages.namespaces.specific.UpdateConcept) Arrays(java.util.Arrays) ConqueryConfig(com.bakdata.conquery.models.config.ConqueryConfig) SneakyThrows(lombok.SneakyThrows) RequiredArgsConstructor(lombok.RequiredArgsConstructor) EntityIdMap(com.bakdata.conquery.models.identifiable.mapping.EntityIdMap) Import(com.bakdata.conquery.models.datasets.Import) DictionaryId(com.bakdata.conquery.models.identifiable.ids.specific.DictionaryId) ConceptId(com.bakdata.conquery.models.identifiable.ids.specific.ConceptId) DatasetId(com.bakdata.conquery.models.identifiable.ids.specific.DatasetId) InternalOnly(com.bakdata.conquery.io.jackson.InternalOnly) RemoveImportJob(com.bakdata.conquery.models.messages.namespaces.specific.RemoveImportJob) NonNull(lombok.NonNull) Table(com.bakdata.conquery.models.datasets.Table) Concept(com.bakdata.conquery.models.datasets.concepts.Concept) RemoveConcept(com.bakdata.conquery.models.messages.namespaces.specific.RemoveConcept) Set(java.util.Set) Validator(javax.validation.Validator) Collectors(java.util.stream.Collectors) NotFoundException(javax.ws.rs.NotFoundException) RemoveSecondaryId(com.bakdata.conquery.models.messages.namespaces.specific.RemoveSecondaryId) Dataset(com.bakdata.conquery.models.datasets.Dataset) Objects(java.util.Objects) SecondaryIdDescription(com.bakdata.conquery.models.datasets.SecondaryIdDescription) JobManager(com.bakdata.conquery.models.jobs.JobManager) List(java.util.List) Slf4j(lombok.extern.slf4j.Slf4j) Response(javax.ws.rs.core.Response) WebApplicationException(javax.ws.rs.WebApplicationException) Jackson(com.bakdata.conquery.io.jackson.Jackson) MetaStorage(com.bakdata.conquery.io.storage.MetaStorage) Column(com.bakdata.conquery.models.datasets.Column) Namespace(com.bakdata.conquery.models.worker.Namespace) RemoveTable(com.bakdata.conquery.models.messages.namespaces.specific.RemoveTable) Getter(lombok.Getter) AddWorker(com.bakdata.conquery.models.messages.network.specific.AddWorker) UpdateTable(com.bakdata.conquery.models.messages.namespaces.specific.UpdateTable) Connector(com.bakdata.conquery.models.datasets.concepts.Connector) UpdateSecondaryId(com.bakdata.conquery.models.messages.namespaces.specific.UpdateSecondaryId) NamespaceStorage(com.bakdata.conquery.io.storage.NamespaceStorage) IdMutex(com.bakdata.conquery.models.identifiable.IdMutex) TableId(com.bakdata.conquery.models.identifiable.ids.specific.TableId) CsvParser(com.univocity.parsers.csv.CsvParser) UpdateMatchingStatsMessage(com.bakdata.conquery.models.messages.namespaces.specific.UpdateMatchingStatsMessage) Identifiable(com.bakdata.conquery.models.identifiable.Identifiable) SimpleJob(com.bakdata.conquery.models.jobs.SimpleJob) ForbiddenException(javax.ws.rs.ForbiddenException) ImportJob(com.bakdata.conquery.models.jobs.ImportJob) IOException(java.io.IOException) ValidatorHelper(com.bakdata.conquery.models.exceptions.ValidatorHelper) UpdateConcept(com.bakdata.conquery.models.messages.namespaces.specific.UpdateConcept) ShardNodeInformation(com.bakdata.conquery.models.worker.ShardNodeInformation) StructureNode(com.bakdata.conquery.models.datasets.concepts.StructureNode) FilterSearch(com.bakdata.conquery.apiv1.FilterSearch) DatasetRegistry(com.bakdata.conquery.models.worker.DatasetRegistry) Collections(java.util.Collections) InputStream(java.io.InputStream) RemoveTable(com.bakdata.conquery.models.messages.namespaces.specific.RemoveTable) Namespace(com.bakdata.conquery.models.worker.Namespace)

Example 5 with ConceptId

use of com.bakdata.conquery.models.identifiable.ids.specific.ConceptId in project conquery by bakdata.

the class ConceptUpdateAndDeletionTest method execute.

@Override
public void execute(String name, TestConquery testConquery) throws Exception {
    StandaloneSupport conquery = testConquery.getSupport(name);
    // Read two JSONs with different Trees
    final String testJson = In.resource("/tests/query/UPDATE_CONCEPT_TESTS/SIMPLE_TREECONCEPT_Query.json").withUTF8().readAll();
    final String testJson2 = In.resource("/tests/query/UPDATE_CONCEPT_TESTS/SIMPLE_TREECONCEPT_2_Query.json").withUTF8().readAll();
    final Dataset dataset = conquery.getDataset();
    final Namespace namespace = conquery.getNamespace();
    final ConceptId conceptId = ConceptId.Parser.INSTANCE.parse(dataset.getName(), "test_tree");
    final Concept<?> concept;
    final QueryTest test = (QueryTest) JsonIntegrationTest.readJson(dataset, testJson);
    final QueryTest test2 = (QueryTest) JsonIntegrationTest.readJson(dataset, testJson2);
    // Manually import data, so we can do our own work.
    {
        ValidatorHelper.failOnError(log, conquery.getValidator().validate(test));
        importSecondaryIds(conquery, test.getContent().getSecondaryIds());
        conquery.waitUntilWorkDone();
        LoadingUtil.importTables(conquery, test.getContent().getTables());
        conquery.waitUntilWorkDone();
        LoadingUtil.importConcepts(conquery, test.getRawConcepts());
        conquery.waitUntilWorkDone();
        assertThat(namespace.getStorage().getConcept(conceptId)).isNotNull();
        LoadingUtil.importTableContents(conquery, test.getContent().getTables());
        conquery.waitUntilWorkDone();
    }
    final Query query = IntegrationUtils.parseQuery(conquery, test.getRawQuery());
    // State before update.
    {
        log.info("Checking state before update");
        // Must contain the concept.
        assertThat(namespace.getStorage().getAllConcepts()).filteredOn(con -> con.getId().equals(conceptId)).isNotEmpty();
        assertThat(namespace.getStorage().getCentralRegistry().getOptional(conceptId)).isNotEmpty();
        for (ShardNode node : conquery.getShardNodes()) {
            for (Worker value : node.getWorkers().getWorkers().values()) {
                if (!value.getInfo().getDataset().equals(dataset.getId())) {
                    continue;
                }
                final ModificationShieldedWorkerStorage workerStorage = value.getStorage();
                assertThat(workerStorage.getCentralRegistry().getOptional(conceptId)).isNotEmpty();
                assertThat(workerStorage.getAllCBlocks()).describedAs("CBlocks for Worker %s", value.getInfo().getId()).filteredOn(cBlock -> cBlock.getConnector().getConcept().getId().equals(conceptId)).isNotEmpty();
            }
        }
        log.info("Executing query before update");
        IntegrationUtils.assertQueryResult(conquery, query, 1L, ExecutionState.DONE, conquery.getTestUser(), 201);
        conquery.waitUntilWorkDone();
        log.info("Query before update executed");
    }
    // Load a different concept with the same id (it has different children "C1" that are more than "A1")
    // To perform the update, the old concept will be deleted first and the new concept will be added. That means the deletion of concept is also covered here
    {
        log.info("Executing  update");
        LoadingUtil.updateConcepts(conquery, test2.getRawConcepts(), Response.Status.Family.SUCCESSFUL);
        conquery.waitUntilWorkDone();
        log.info("Update executed");
    }
    // Check state after update.
    {
        log.info("Checking state after update");
        // Must contain the concept now.
        assertThat(namespace.getStorage().getAllConcepts()).filteredOn(con -> con.getId().equals(conceptId)).isNotEmpty();
        assertThat(namespace.getStorage().getCentralRegistry().getOptional(conceptId)).isNotEmpty();
        for (ShardNode node : conquery.getShardNodes()) {
            for (Worker value : node.getWorkers().getWorkers().values()) {
                if (!value.getInfo().getDataset().equals(dataset.getId())) {
                    continue;
                }
                final ModificationShieldedWorkerStorage workerStorage = value.getStorage();
                assertThat(workerStorage.getCentralRegistry().getOptional(conceptId)).isNotEmpty();
                assertThat(workerStorage.getAllCBlocks()).describedAs("CBlocks for Worker %s", value.getInfo().getId()).filteredOn(cBlock -> cBlock.getConnector().getConcept().getId().equals(conceptId)).isNotEmpty();
            }
        }
        log.info("Executing query after update");
        // Assert that it now contains 2 instead of 1.
        IntegrationUtils.assertQueryResult(conquery, query, 2L, ExecutionState.DONE, conquery.getTestUser(), 201);
        conquery.waitUntilWorkDone();
        log.info("Query after update executed");
    }
    // new Conquery generated after restarting
    // StandaloneSupport conquery;
    // Restart conquery and assert again, that the data is correct.
    {
        testConquery.shutdown();
        // restart
        testConquery.beforeAll();
        conquery = testConquery.openDataset(dataset.getId());
        log.info("Checking state after re-start");
        {
            // Must contain the concept.
            assertThat(conquery.getNamespace().getStorage().getAllConcepts()).filteredOn(con -> con.getId().equals(conceptId)).isNotEmpty();
            assertThat(conquery.getNamespace().getStorage().getCentralRegistry().getOptional(conceptId)).isNotEmpty();
            for (ShardNode node : conquery.getShardNodes()) {
                for (Worker value : node.getWorkers().getWorkers().values()) {
                    if (!value.getInfo().getDataset().equals(dataset.getId())) {
                        continue;
                    }
                    final ModificationShieldedWorkerStorage workerStorage = value.getStorage();
                    assertThat(workerStorage.getCentralRegistry().getOptional(conceptId)).isNotEmpty();
                    assertThat(workerStorage.getAllCBlocks()).describedAs("CBlocks for Worker %s", value.getInfo().getId()).filteredOn(cBlock -> cBlock.getConnector().getConcept().getId().equals(conceptId)).isNotEmpty();
                }
            }
            log.info("Executing query after restart.");
            // Re-assert state.
            IntegrationUtils.assertQueryResult(conquery, query, 2L, ExecutionState.DONE, conquery.getTestUser(), 201);
            conquery.waitUntilWorkDone();
        }
    }
    // Delete the Concept.
    {
        log.info("Issuing deletion of import {}", conceptId);
        concept = Objects.requireNonNull(conquery.getNamespace().getStorage().getConcept(conceptId));
        conquery.getDatasetsProcessor().deleteConcept(concept);
        conquery.waitUntilWorkDone();
    }
    // Check state after deletion.
    {
        log.info("Checking state after deletion");
        // We've deleted the concept so it and it's associated cblock should be gone.
        assertThat(conquery.getNamespace().getStorage().getAllConcepts()).filteredOn(con -> con.getId().equals(conceptId)).isEmpty();
        assertThat(conquery.getNamespace().getStorage().getCentralRegistry().getOptional(conceptId)).isEmpty();
        assertThat(conquery.getShardNodes().stream().flatMap(node -> node.getWorkers().getWorkers().values().stream()).filter(worker -> worker.getInfo().getDataset().equals(dataset.getId())).map(Worker::getStorage)).noneMatch(workerStorage -> workerStorage.getConcept(conceptId) != null).noneMatch(workerStorage -> workerStorage.getAllCBlocks().stream().anyMatch(cBlock -> cBlock.getConnector().getConcept().getId().equals(conceptId)));
        log.info("Executing query after deletion (EXPECTING AN EXCEPTION IN THE LOGS!)");
        // Issue a query and assert that it is failing.
        IntegrationUtils.assertQueryResult(conquery, query, 0L, ExecutionState.FAILED, conquery.getTestUser(), 400);
    }
    // Restart conquery and assert again, that the state after deletion was maintained.
    {
        {
            testConquery.shutdown();
            // restart
            testConquery.beforeAll();
            conquery = testConquery.openDataset(dataset.getId());
        }
        // Check state after restart.
        {
            log.info("Checking state after restart");
            // We've deleted the concept so it and it's associated cblock should be gone.
            assertThat(conquery.getNamespace().getStorage().getAllConcepts()).filteredOn(con -> con.getId().equals(conceptId)).isEmpty();
            assertThat(conquery.getNamespace().getStorage().getCentralRegistry().getOptional(conceptId)).isEmpty();
            assertThat(conquery.getShardNodes().stream().flatMap(node -> node.getWorkers().getWorkers().values().stream()).filter(worker -> worker.getInfo().getDataset().equals(dataset.getId())).map(Worker::getStorage)).noneMatch(workerStorage -> workerStorage.getConcept(conceptId) != null).noneMatch(workerStorage -> workerStorage.getAllCBlocks().stream().anyMatch(cBlock -> cBlock.getConnector().getConcept().getId().equals(conceptId)));
            log.info("Executing query after restart (EXPECTING AN EXCEPTION IN THE LOGS!)");
            // Issue a query and assert that it is failing.
            IntegrationUtils.assertQueryResult(conquery, query, 0L, ExecutionState.FAILED, conquery.getTestUser(), 400);
        }
    }
}
Also used : IntegrationUtils(com.bakdata.conquery.integration.common.IntegrationUtils) ProgrammaticIntegrationTest(com.bakdata.conquery.integration.tests.ProgrammaticIntegrationTest) ExecutionState(com.bakdata.conquery.models.execution.ExecutionState) TestConquery(com.bakdata.conquery.util.support.TestConquery) Assertions.assertThat(org.assertj.core.api.Assertions.assertThat) ShardNode(com.bakdata.conquery.commands.ShardNode) Concept(com.bakdata.conquery.models.datasets.concepts.Concept) QueryTest(com.bakdata.conquery.integration.json.QueryTest) LoadingUtil(com.bakdata.conquery.integration.common.LoadingUtil) StandaloneSupport(com.bakdata.conquery.util.support.StandaloneSupport) Dataset(com.bakdata.conquery.models.datasets.Dataset) ValidatorHelper(com.bakdata.conquery.models.exceptions.ValidatorHelper) Objects(java.util.Objects) Worker(com.bakdata.conquery.models.worker.Worker) Slf4j(lombok.extern.slf4j.Slf4j) Response(javax.ws.rs.core.Response) ModificationShieldedWorkerStorage(com.bakdata.conquery.io.storage.ModificationShieldedWorkerStorage) In(com.github.powerlibraries.io.In) ConceptId(com.bakdata.conquery.models.identifiable.ids.specific.ConceptId) JsonIntegrationTest(com.bakdata.conquery.integration.json.JsonIntegrationTest) Query(com.bakdata.conquery.apiv1.query.Query) LoadingUtil.importSecondaryIds(com.bakdata.conquery.integration.common.LoadingUtil.importSecondaryIds) Namespace(com.bakdata.conquery.models.worker.Namespace) Query(com.bakdata.conquery.apiv1.query.Query) QueryTest(com.bakdata.conquery.integration.json.QueryTest) ShardNode(com.bakdata.conquery.commands.ShardNode) Dataset(com.bakdata.conquery.models.datasets.Dataset) Worker(com.bakdata.conquery.models.worker.Worker) StandaloneSupport(com.bakdata.conquery.util.support.StandaloneSupport) Namespace(com.bakdata.conquery.models.worker.Namespace) ConceptId(com.bakdata.conquery.models.identifiable.ids.specific.ConceptId) ModificationShieldedWorkerStorage(com.bakdata.conquery.io.storage.ModificationShieldedWorkerStorage)

Aggregations

ConceptId (com.bakdata.conquery.models.identifiable.ids.specific.ConceptId)8 DatasetId (com.bakdata.conquery.models.identifiable.ids.specific.DatasetId)5 ParameterizedTest (org.junit.jupiter.params.ParameterizedTest)5 ConceptTreeChildId (com.bakdata.conquery.models.identifiable.ids.specific.ConceptTreeChildId)4 Test (org.junit.jupiter.api.Test)4 Dataset (com.bakdata.conquery.models.datasets.Dataset)3 Concept (com.bakdata.conquery.models.datasets.concepts.Concept)3 CQConcept (com.bakdata.conquery.apiv1.query.concept.specific.CQConcept)2 QueryTest (com.bakdata.conquery.integration.json.QueryTest)2 MetaStorage (com.bakdata.conquery.io.storage.MetaStorage)2 Connector (com.bakdata.conquery.models.datasets.concepts.Connector)2 ValidatorHelper (com.bakdata.conquery.models.exceptions.ValidatorHelper)2 Namespace (com.bakdata.conquery.models.worker.Namespace)2 StandaloneSupport (com.bakdata.conquery.util.support.StandaloneSupport)2 Objects (java.util.Objects)2 Response (javax.ws.rs.core.Response)2 Slf4j (lombok.extern.slf4j.Slf4j)2 FilterSearch (com.bakdata.conquery.apiv1.FilterSearch)1 FullExecutionStatus (com.bakdata.conquery.apiv1.FullExecutionStatus)1 ConceptQuery (com.bakdata.conquery.apiv1.query.ConceptQuery)1