Search in sources :

Example 56 with Operation

use of uk.gov.gchq.gaffer.operation.Operation in project Gaffer by gchq.

the class OperationChainValidatorTest method shouldNotValidateInvalidOperationChain.

@Test
public void shouldNotValidateInvalidOperationChain() {
    // Given
    Operation operation = Mockito.mock(Operation.class);
    given(operation.validate()).willReturn(new ValidationResult("SparkContext is required"));
    OperationChain opChain = new OperationChain(operation);
    // When
    validateOperationChain(opChain, false);
    // Then
    verify(operation).validate();
}
Also used : OperationChain(uk.gov.gchq.gaffer.operation.OperationChain) Operation(uk.gov.gchq.gaffer.operation.Operation) ValidationResult(uk.gov.gchq.koryphe.ValidationResult) Test(org.junit.jupiter.api.Test)

Example 57 with Operation

use of uk.gov.gchq.gaffer.operation.Operation in project Gaffer by gchq.

the class Graph method updateOperationChainView.

private void updateOperationChainView(final Operations<?> operations) {
    for (final Operation operation : operations.getOperations()) {
        if (operation instanceof Operations) {
            updateOperationChainView((Operations) operation);
        } else if (operation instanceof OperationView) {
            View opView = ((OperationView) operation).getView();
            if (null == opView) {
                opView = config.getView();
            } else if (!(opView instanceof NamedView) && !opView.hasGroups() && !opView.isAllEdges() && !opView.isAllEntities()) {
                // merge with both Entities and Edges
                if (!isEmpty(opView.getGlobalElements()) || (isEmpty(opView.getGlobalEdges()) && isEmpty(opView.getGlobalEntities()))) {
                    opView = new View.Builder().merge(config.getView()).merge(opView).build();
                } else {
                    // We have either global edges or entities in
                    // opView, but not both
                    final View originalView = opView;
                    final View partialConfigView = new View.Builder().merge(config.getView()).removeEdges((x -> isEmpty(originalView.getGlobalEdges()))).removeEntities((x -> isEmpty(originalView.getGlobalEntities()))).build();
                    opView = new View.Builder().merge(partialConfigView).merge(opView).build();
                }
            } else if (opView.isAllEdges() || opView.isAllEntities()) {
                View.Builder opViewBuilder = new View.Builder().merge(opView);
                if (opView.isAllEdges()) {
                    opViewBuilder.edges(getSchema().getEdgeGroups());
                }
                if (opView.isAllEntities()) {
                    opViewBuilder.entities(getSchema().getEntityGroups());
                }
                opView = opViewBuilder.build();
            }
            opView.expandGlobalDefinitions();
            ((OperationView) operation).setView(opView);
        }
    }
}
Also used : Job(uk.gov.gchq.gaffer.jobtracker.Job) LoggerFactory(org.slf4j.LoggerFactory) SchemaException(uk.gov.gchq.gaffer.data.elementdefinition.exception.SchemaException) NamedOperation(uk.gov.gchq.gaffer.named.operation.NamedOperation) OperationChain(uk.gov.gchq.gaffer.operation.OperationChain) URI(java.net.URI) Path(java.nio.file.Path) NamedView(uk.gov.gchq.gaffer.data.elementdefinition.view.NamedView) Pair(uk.gov.gchq.gaffer.commonutil.pair.Pair) Set(java.util.Set) Collectors(java.util.stream.Collectors) IOUtils(org.apache.commons.io.IOUtils) Store(uk.gov.gchq.gaffer.store.Store) ReflectionUtil(uk.gov.gchq.koryphe.util.ReflectionUtil) List(java.util.List) Context(uk.gov.gchq.gaffer.store.Context) View(uk.gov.gchq.gaffer.data.elementdefinition.view.View) CloseableUtil(uk.gov.gchq.gaffer.commonutil.CloseableUtil) NamedOperationResolver(uk.gov.gchq.gaffer.graph.hook.NamedOperationResolver) OperationException(uk.gov.gchq.gaffer.operation.OperationException) StoreTrait(uk.gov.gchq.gaffer.store.StoreTrait) Operations(uk.gov.gchq.gaffer.operation.Operations) GraphHook(uk.gov.gchq.gaffer.graph.hook.GraphHook) JSONSerialiser(uk.gov.gchq.gaffer.jsonserialisation.JSONSerialiser) StoreException(uk.gov.gchq.gaffer.store.StoreException) User(uk.gov.gchq.gaffer.user.User) OperationView(uk.gov.gchq.gaffer.operation.graph.OperationView) UpdateViewHook(uk.gov.gchq.gaffer.graph.hook.UpdateViewHook) ArrayList(java.util.ArrayList) GraphLibrary(uk.gov.gchq.gaffer.store.library.GraphLibrary) Lists(com.google.common.collect.Lists) Output(uk.gov.gchq.gaffer.operation.io.Output) NoGraphLibrary(uk.gov.gchq.gaffer.store.library.NoGraphLibrary) StreamUtil(uk.gov.gchq.gaffer.commonutil.StreamUtil) Logger(org.slf4j.Logger) Properties(java.util.Properties) NamedViewResolver(uk.gov.gchq.gaffer.graph.hook.NamedViewResolver) Files(java.nio.file.Files) FileUtils(org.apache.commons.io.FileUtils) JobDetail(uk.gov.gchq.gaffer.jobtracker.JobDetail) IOException(java.io.IOException) StoreProperties(uk.gov.gchq.gaffer.store.StoreProperties) Operation(uk.gov.gchq.gaffer.operation.Operation) Schema(uk.gov.gchq.gaffer.store.schema.Schema) CollectionUtils.isEmpty(org.apache.commons.collections.CollectionUtils.isEmpty) FunctionAuthoriser(uk.gov.gchq.gaffer.graph.hook.FunctionAuthoriser) FunctionAuthoriserUtil(uk.gov.gchq.gaffer.graph.hook.FunctionAuthoriserUtil) Collections(java.util.Collections) InputStream(java.io.InputStream) OperationView(uk.gov.gchq.gaffer.operation.graph.OperationView) NamedView(uk.gov.gchq.gaffer.data.elementdefinition.view.NamedView) NamedOperation(uk.gov.gchq.gaffer.named.operation.NamedOperation) Operation(uk.gov.gchq.gaffer.operation.Operation) Operations(uk.gov.gchq.gaffer.operation.Operations) NamedView(uk.gov.gchq.gaffer.data.elementdefinition.view.NamedView) View(uk.gov.gchq.gaffer.data.elementdefinition.view.View) OperationView(uk.gov.gchq.gaffer.operation.graph.OperationView)

Example 58 with Operation

use of uk.gov.gchq.gaffer.operation.Operation in project Gaffer by gchq.

the class SchemaMigration method migrateOperation.

private List<Operation> migrateOperation(final Operation op) {
    final OperationView opView = OperationView.class.cast(op);
    final Map<String, ViewMigration> migratedEntities = migrateViewElements(entities, opView.getView()::getEntity);
    final Map<String, ViewMigration> migratedEdges = migrateViewElements(edges, opView.getView()::getEdge);
    final View.Builder viewBuilder = new View.Builder().merge(opView.getView());
    for (final Map.Entry<String, ViewMigration> entry : migratedEntities.entrySet()) {
        viewBuilder.entity(entry.getKey(), entry.getValue().buildViewElementDefinition());
    }
    for (final Map.Entry<String, ViewMigration> entry : migratedEdges.entrySet()) {
        viewBuilder.edge(entry.getKey(), entry.getValue().buildViewElementDefinition());
    }
    viewBuilder.config(ViewValidator.SKIP_VIEW_VALIDATION, TRUE);
    final View updatedView = viewBuilder.build();
    LOGGER.debug("Migrated view: {}", updatedView);
    opView.setView(updatedView);
    final List<Operation> migrationOps = ViewMigration.createMigrationOps(aggregateAfter, migratedEdges.values(), migratedEntities.values());
    if (LOGGER.isDebugEnabled()) {
        try {
            LOGGER.debug("Migrated operations: {}", StringUtil.toString(JSONSerialiser.serialise(new OperationChain<>(migrationOps), true)));
        } catch (final SerialisationException e) {
            LOGGER.debug("Failed to json serialise the migration operations: {}", new OperationChain<>(migrationOps));
        }
    }
    return migrationOps;
}
Also used : SerialisationException(uk.gov.gchq.gaffer.exception.SerialisationException) Operation(uk.gov.gchq.gaffer.operation.Operation) OperationView(uk.gov.gchq.gaffer.operation.graph.OperationView) View(uk.gov.gchq.gaffer.data.elementdefinition.view.View) OperationView(uk.gov.gchq.gaffer.operation.graph.OperationView) OperationChain(uk.gov.gchq.gaffer.operation.OperationChain) HashMap(java.util.HashMap) Map(java.util.Map)

Example 59 with Operation

use of uk.gov.gchq.gaffer.operation.Operation in project Gaffer by gchq.

the class ConditionalTest method shouldNotFailToShallowCloneWhenTransformIsNull.

@Test
public void shouldNotFailToShallowCloneWhenTransformIsNull() {
    // Given
    final Operation transform = null;
    final Conditional conditional = new Conditional(new IsMoreThan(1), transform);
    // When
    final Conditional clone = conditional.shallowClone();
    // Then
    assertNotSame(conditional, clone);
    assertNull(clone.getTransform());
}
Also used : Operation(uk.gov.gchq.gaffer.operation.Operation) IsMoreThan(uk.gov.gchq.koryphe.impl.predicate.IsMoreThan) Test(org.junit.jupiter.api.Test) JSONSerialisationTest(uk.gov.gchq.gaffer.JSONSerialisationTest)

Example 60 with Operation

use of uk.gov.gchq.gaffer.operation.Operation in project Gaffer by gchq.

the class StoreTest method shouldExecuteOperationChainJobAndExportResults.

@Test
public void shouldExecuteOperationChainJobAndExportResults() throws OperationException, InterruptedException, StoreException {
    // Given
    final Operation operation = new GetVariables.Builder().variableNames(Lists.newArrayList()).build();
    final OperationChain<?> opChain = new OperationChain<>(operation);
    final StoreProperties properties = mock(StoreProperties.class);
    given(properties.getJobExecutorThreadCount()).willReturn(1);
    given(properties.getJobTrackerEnabled()).willReturn(true);
    final Store store = new StoreImpl();
    final Schema schema = new Schema();
    store.initialise("graphId", schema, properties);
    // When
    final JobDetail resultJobDetail = store.executeJob(opChain, context);
    // Then
    Thread.sleep(1000);
    final ArgumentCaptor<JobDetail> jobDetail = ArgumentCaptor.forClass(JobDetail.class);
    verify(jobTracker, times(2)).addOrUpdateJob(jobDetail.capture(), eq(user));
    assertEquals(jobDetail.getAllValues().get(0), resultJobDetail);
    assertEquals(JobStatus.FINISHED, jobDetail.getAllValues().get(1).getStatus());
    final ArgumentCaptor<Context> contextCaptor = ArgumentCaptor.forClass(Context.class);
    verify(exportToGafferResultCacheHandler).doOperation(Mockito.any(ExportToGafferResultCache.class), contextCaptor.capture(), eq(store));
    assertSame(user, contextCaptor.getValue().getUser());
}
Also used : Schema(uk.gov.gchq.gaffer.store.schema.Schema) GetSchema(uk.gov.gchq.gaffer.store.operation.GetSchema) NamedOperation(uk.gov.gchq.gaffer.named.operation.NamedOperation) DeleteNamedOperation(uk.gov.gchq.gaffer.named.operation.DeleteNamedOperation) Operation(uk.gov.gchq.gaffer.operation.Operation) AddNamedOperation(uk.gov.gchq.gaffer.named.operation.AddNamedOperation) JobDetail(uk.gov.gchq.gaffer.jobtracker.JobDetail) ValidateOperationChain(uk.gov.gchq.gaffer.operation.impl.ValidateOperationChain) OperationChain(uk.gov.gchq.gaffer.operation.OperationChain) ExportToGafferResultCache(uk.gov.gchq.gaffer.operation.impl.export.resultcache.ExportToGafferResultCache) Test(org.junit.jupiter.api.Test)

Aggregations

Operation (uk.gov.gchq.gaffer.operation.Operation)136 Test (org.junit.jupiter.api.Test)88 OperationChain (uk.gov.gchq.gaffer.operation.OperationChain)49 NamedOperation (uk.gov.gchq.gaffer.named.operation.NamedOperation)44 Schema (uk.gov.gchq.gaffer.store.schema.Schema)41 Context (uk.gov.gchq.gaffer.store.Context)35 GetElements (uk.gov.gchq.gaffer.operation.impl.get.GetElements)34 Store (uk.gov.gchq.gaffer.store.Store)28 StoreProperties (uk.gov.gchq.gaffer.store.StoreProperties)26 LinkedHashMap (java.util.LinkedHashMap)21 GetAdjacentIds (uk.gov.gchq.gaffer.operation.impl.get.GetAdjacentIds)21 User (uk.gov.gchq.gaffer.user.User)21 ArrayList (java.util.ArrayList)18 GetAllElements (uk.gov.gchq.gaffer.operation.impl.get.GetAllElements)18 View (uk.gov.gchq.gaffer.data.elementdefinition.view.View)17 EntitySeed (uk.gov.gchq.gaffer.operation.data.EntitySeed)16 HashSet (java.util.HashSet)15 HashMap (java.util.HashMap)13 TestStore (uk.gov.gchq.gaffer.integration.store.TestStore)13 OperationException (uk.gov.gchq.gaffer.operation.OperationException)13