Search in sources :

Example 1 with GraphFetchCacheByTargetCrossKeys

use of org.finos.legend.engine.plan.execution.cache.graphFetch.GraphFetchCacheByTargetCrossKeys in project legend-engine by finos.

the class TestPlanExecutionWithGraphFetchCrossKeyCache method testCrossCacheWithNoPropertyAccess.

@Test
public void testCrossCacheWithNoPropertyAccess() throws JavaCompileException {
    String fetchFunction = "###Pure\n" + "function test::fetch(): String[1]\n" + "{\n" + "  test::Person.all()\n" + "    ->graphFetch(#{\n" + "      test::Person {\n" + "        fullName\n" + "      }\n" + "    }#, 1)\n" + "    ->serialize(#{\n" + "      test::Person {\n" + "        fullName\n" + "      }\n" + "    }#)\n" + "}";
    SingleExecutionPlan plan = buildPlanForFetchFunction(fetchFunction);
    GraphFetchCacheByTargetCrossKeys firmCache = getFirmEmptyCache(plan);
    PlanExecutionContext context = new PlanExecutionContext(plan, firmCache);
    String expectedRes = "[" + "{\"fullName\":\"P1\"}," + "{\"fullName\":\"P2\"}," + "{\"fullName\":\"P3\"}," + "{\"fullName\":\"P4\"}," + "{\"fullName\":\"P5\"}" + "]";
    Assert.assertEquals(expectedRes, executePlan(plan, context));
    assertCacheStats(firmCache.getExecutionCache(), 0, 0, 0, 0);
}
Also used : GraphFetchCacheByTargetCrossKeys(org.finos.legend.engine.plan.execution.cache.graphFetch.GraphFetchCacheByTargetCrossKeys) SingleExecutionPlan(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.SingleExecutionPlan) PlanExecutionContext(org.finos.legend.engine.plan.execution.PlanExecutionContext) Test(org.junit.Test)

Example 2 with GraphFetchCacheByTargetCrossKeys

use of org.finos.legend.engine.plan.execution.cache.graphFetch.GraphFetchCacheByTargetCrossKeys in project legend-engine by finos.

the class TestPlanExecutionWithGraphFetchCrossKeyCache method testMultiCrossPropertyCaches.

@Test
public void testMultiCrossPropertyCaches() throws JavaCompileException {
    String fetchFunction = "###Pure\n" + "function test::fetch(): String[1]\n" + "{\n" + "  test::Person.all()\n" + "    ->graphFetch(#{\n" + "      test::Person {\n" + "        fullName,\n" + "        firm {\n" + "          name\n" + "        },\n" + "        address {\n" + "          name\n" + "        }\n" + "      }\n" + "    }#, 1)\n" + "    ->serialize(#{\n" + "      test::Person {\n" + "        fullName,\n" + "        firm {\n" + "          name\n" + "        },\n" + "        address {\n" + "          name\n" + "        }\n" + "      }\n" + "    }#)\n" + "}";
    SingleExecutionPlan plan = buildPlanForFetchFunction(fetchFunction);
    GraphFetchCacheByTargetCrossKeys firmCache = getFirmEmptyCache(plan);
    GraphFetchCacheByTargetCrossKeys addressCache = getAddressCache(plan);
    PlanExecutionContext context = new PlanExecutionContext(plan, firmCache, addressCache);
    String expectedRes = "[" + "{\"fullName\":\"P1\",\"firm\":{\"name\":\"F1\"},\"address\":{\"name\":\"A1\"}}," + "{\"fullName\":\"P2\",\"firm\":{\"name\":\"F2\"},\"address\":{\"name\":\"A2\"}}," + "{\"fullName\":\"P3\",\"firm\":null,\"address\":null}," + "{\"fullName\":\"P4\",\"firm\":null,\"address\":{\"name\":\"A3\"}}," + "{\"fullName\":\"P5\",\"firm\":{\"name\":\"F1\"},\"address\":{\"name\":\"A1\"}}" + "]";
    Assert.assertEquals(expectedRes, executePlan(plan, context));
    assertCacheStats(firmCache.getExecutionCache(), 3, 5, 2, 3);
    assertCacheStats(addressCache.getExecutionCache(), 4, 5, 1, 4);
    Assert.assertEquals(expectedRes, executePlan(plan, context));
    assertCacheStats(firmCache.getExecutionCache(), 3, 10, 7, 3);
    assertCacheStats(addressCache.getExecutionCache(), 4, 10, 6, 4);
}
Also used : GraphFetchCacheByTargetCrossKeys(org.finos.legend.engine.plan.execution.cache.graphFetch.GraphFetchCacheByTargetCrossKeys) SingleExecutionPlan(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.SingleExecutionPlan) PlanExecutionContext(org.finos.legend.engine.plan.execution.PlanExecutionContext) Test(org.junit.Test)

Example 3 with GraphFetchCacheByTargetCrossKeys

use of org.finos.legend.engine.plan.execution.cache.graphFetch.GraphFetchCacheByTargetCrossKeys in project legend-engine by finos.

the class TestPlanExecutionWithGraphFetchCrossKeyCache method testToManyCrossPropertyCache.

@Test
public void testToManyCrossPropertyCache() throws JavaCompileException {
    String fetchFunction = "###Pure\n" + "function test::fetch(): String[1]\n" + "{\n" + "  test::Address.all()\n" + "    ->graphFetch(#{\n" + "      test::Address {\n" + "        name,\n" + "        persons {\n" + "          fullName\n" + "        }\n" + "      }\n" + "    }#, 1)\n" + "    ->serialize(#{\n" + "      test::Address {\n" + "        name,\n" + "        persons {\n" + "          fullName\n" + "        }\n" + "      }\n" + "    }#)\n" + "}";
    SingleExecutionPlan plan = buildPlanForFetchFunction(fetchFunction);
    GraphFetchCacheByTargetCrossKeys personCache = getPersonCache(plan);
    PlanExecutionContext context = new PlanExecutionContext(plan, personCache);
    String expectedRes = "[" + "{\"name\":\"A1\",\"persons\":[{\"fullName\":\"P1\"},{\"fullName\":\"P5\"}]}," + "{\"name\":\"A2\",\"persons\":[{\"fullName\":\"P2\"}]}," + "{\"name\":\"A3\",\"persons\":[{\"fullName\":\"P4\"}]}," + "{\"name\":\"A4\",\"persons\":[]}," + "{\"name\":\"A5\",\"persons\":[]}" + "]";
    Assert.assertEquals(expectedRes, executePlan(plan, context));
    assertCacheStats(personCache.getExecutionCache(), 5, 5, 0, 5);
    Assert.assertEquals(expectedRes, executePlan(plan, context));
    assertCacheStats(personCache.getExecutionCache(), 5, 10, 5, 5);
}
Also used : GraphFetchCacheByTargetCrossKeys(org.finos.legend.engine.plan.execution.cache.graphFetch.GraphFetchCacheByTargetCrossKeys) SingleExecutionPlan(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.SingleExecutionPlan) PlanExecutionContext(org.finos.legend.engine.plan.execution.PlanExecutionContext) Test(org.junit.Test)

Example 4 with GraphFetchCacheByTargetCrossKeys

use of org.finos.legend.engine.plan.execution.cache.graphFetch.GraphFetchCacheByTargetCrossKeys in project legend-engine by finos.

the class TestPlanExecutionWithGraphFetchCrossKeyCache method testCrossPropertyCachingWithSerializableCache.

@Test
public void testCrossPropertyCachingWithSerializableCache() throws JavaCompileException, IOException, ClassNotFoundException {
    String fetchFunction = "###Pure\n" + "function test::fetch(): String[1]\n" + "{\n" + "  test::Person.all()\n" + "    ->graphFetch(#{\n" + "      test::Person {\n" + "        fullName,\n" + "        firm {\n" + "          name,\n" + "          address {\n" + "            name\n" + "          }\n" + "        }\n" + "      }\n" + "    }#, 1)\n" + "    ->serialize(#{\n" + "      test::Person {\n" + "        fullName,\n" + "        firm {\n" + "          name,\n" + "          address {\n" + "            name\n" + "          }\n" + "        }\n" + "      }\n" + "    }#)\n" + "}";
    String expectedRes = "[" + "{\"fullName\":\"P1\",\"firm\":{\"name\":\"F1\",\"address\":{\"name\":\"A4\"}}}," + "{\"fullName\":\"P2\",\"firm\":{\"name\":\"F2\",\"address\":{\"name\":\"A3\"}}}," + "{\"fullName\":\"P3\",\"firm\":null}," + "{\"fullName\":\"P4\",\"firm\":null}," + "{\"fullName\":\"P5\",\"firm\":{\"name\":\"F1\",\"address\":{\"name\":\"A4\"}}}" + "]";
    SingleExecutionPlan plan = buildPlanForFetchFunction(fetchFunction);
    EngineJavaCompiler compiler = JavaHelper.compilePlan(plan, null);
    GraphFetchCrossAssociationKeys graphFetchCrossAssociationKeys = GraphFetchCrossAssociationKeys.graphFetchCrossAssociationKeysForPlan(plan).stream().filter(x -> x.getName().equals("<default, root.firm>")).findFirst().orElse(null);
    try (JavaHelper.ThreadContextClassLoaderScope ignored = JavaHelper.withCurrentThreadContextClassLoader(compiler.getClassLoader())) {
        Map<GraphFetchCacheKey, List<Object>> firmMap1 = Maps.mutable.empty();
        GraphFetchCacheByTargetCrossKeys firmCache1 = ExecutionCacheBuilder.buildGraphFetchCacheByTargetCrossKeysFromExecutionCache(buildExecutionCacheFromMap(firmMap1), graphFetchCrossAssociationKeys);
        Assert.assertEquals(expectedRes, executePlan(plan, new PlanExecutionContext(firmCache1)));
        assertCacheStats(firmCache1.getExecutionCache(), 3, 5, 2, 3);
        ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
        ObjectOutputStream objectOutputStream = new ObjectOutputStream(byteArrayOutputStream);
        objectOutputStream.writeObject(firmMap1);
        objectOutputStream.flush();
        ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(byteArrayOutputStream.toByteArray());
        ClassLoaderObjectInputStream objectInputStream = new ClassLoaderObjectInputStream(compiler.getClassLoader(), byteArrayInputStream);
        Object o = objectInputStream.readObject();
        @SuppressWarnings("unchecked") Map<GraphFetchCacheKey, List<Object>> firmMap2 = (Map<GraphFetchCacheKey, List<Object>>) o;
        GraphFetchCacheByTargetCrossKeys firmCache2 = ExecutionCacheBuilder.buildGraphFetchCacheByTargetCrossKeysFromExecutionCache(buildExecutionCacheFromMap(firmMap2), graphFetchCrossAssociationKeys);
        Assert.assertEquals(expectedRes, executePlan(plan, new PlanExecutionContext(firmCache2)));
        assertCacheStats(firmCache2.getExecutionCache(), 3, 5, 5, 0);
    }
}
Also used : EngineJavaCompiler(org.finos.legend.engine.shared.javaCompiler.EngineJavaCompiler) ByteArrayOutputStream(java.io.ByteArrayOutputStream) ClassLoaderObjectInputStream(org.apache.commons.io.input.ClassLoaderObjectInputStream) ObjectOutputStream(java.io.ObjectOutputStream) GraphFetchCrossAssociationKeys(org.finos.legend.engine.plan.execution.cache.graphFetch.GraphFetchCrossAssociationKeys) GraphFetchCacheKey(org.finos.legend.engine.plan.execution.cache.graphFetch.GraphFetchCacheKey) GraphFetchCacheByTargetCrossKeys(org.finos.legend.engine.plan.execution.cache.graphFetch.GraphFetchCacheByTargetCrossKeys) ByteArrayInputStream(java.io.ByteArrayInputStream) List(java.util.List) SingleExecutionPlan(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.SingleExecutionPlan) PlanExecutionContext(org.finos.legend.engine.plan.execution.PlanExecutionContext) Map(java.util.Map) JavaHelper(org.finos.legend.engine.plan.execution.nodes.helpers.platform.JavaHelper) Test(org.junit.Test)

Example 5 with GraphFetchCacheByTargetCrossKeys

use of org.finos.legend.engine.plan.execution.cache.graphFetch.GraphFetchCacheByTargetCrossKeys in project legend-engine by finos.

the class ExecutionNodeExecutor method findGraphFetchCacheByTargetCrossKeys.

private ExecutionCache<GraphFetchCacheKey, List<Object>> findGraphFetchCacheByTargetCrossKeys(GlobalGraphFetchExecutionNode globalGraphFetchExecutionNode) {
    List<GraphFetchCache> graphFetchCaches = this.executionState.graphFetchCaches;
    XStorePropertyFetchDetails fetchDetails = globalGraphFetchExecutionNode.xStorePropertyFetchDetails;
    return graphFetchCaches.stream().filter(GraphFetchCacheByTargetCrossKeys.class::isInstance).map(GraphFetchCacheByTargetCrossKeys.class::cast).filter(cache -> cache.getGraphFetchCrossAssociationKeys() != null).filter(cache -> {
        GraphFetchCrossAssociationKeys c = cache.getGraphFetchCrossAssociationKeys();
        return c.getPropertyPath().equals(fetchDetails.propertyPath) && c.getSourceMappingId().equals(fetchDetails.sourceMappingId) && c.getSourceSetId().equals(fetchDetails.sourceSetId) && c.getTargetMappingId().equals(fetchDetails.targetMappingId) && c.getTargetSetId().equals(fetchDetails.targetSetId) && c.getTargetPropertiesOrdered().equals(fetchDetails.targetPropertiesOrdered) && c.getSubTree().equals(fetchDetails.subTree);
    }).map(GraphFetchCacheByTargetCrossKeys::getExecutionCache).findFirst().orElse(null);
}
Also used : Maps(org.eclipse.collections.impl.factory.Maps) ExecutionNodeVisitor(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.nodes.ExecutionNodeVisitor) Arrays(java.util.Arrays) DefaultExecutionNodeContext(org.finos.legend.engine.plan.execution.nodes.helpers.platform.DefaultExecutionNodeContext) Spliterators(java.util.Spliterators) GraphFetchCacheKey(org.finos.legend.engine.plan.execution.cache.graphFetch.GraphFetchCacheKey) ExecutionNodeSerializerHelper(org.finos.legend.engine.plan.execution.nodes.helpers.ExecutionNodeSerializerHelper) DoubleSummaryStatistics(java.util.DoubleSummaryStatistics) MutableList(org.eclipse.collections.api.list.MutableList) InMemoryCrossStoreGraphFetchExecutionNode(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.nodes.graphFetch.store.inMemory.InMemoryCrossStoreGraphFetchExecutionNode) ErrorExecutionNode(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.nodes.ErrorExecutionNode) IPlatformPureExpressionExecutionNodeSerializeSpecifics(org.finos.legend.engine.plan.dependencies.store.platform.IPlatformPureExpressionExecutionNodeSerializeSpecifics) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) StoreStreamReadingExecutionNode(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.nodes.graphFetch.store.inMemory.StoreStreamReadingExecutionNode) Map(java.util.Map) ConstantResult(org.finos.legend.engine.plan.execution.result.ConstantResult) ExecutionNode(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.nodes.ExecutionNode) FunctionParametersValidationNode(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.nodes.FunctionParametersValidationNode) AppliedFunction(org.finos.legend.engine.protocol.pure.v1.model.valueSpecification.application.AppliedFunction) AllocationExecutionNode(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.nodes.AllocationExecutionNode) ConstantExecutionNode(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.nodes.ConstantExecutionNode) Collection(java.util.Collection) GraphExecutionState(org.finos.legend.engine.plan.execution.nodes.state.GraphExecutionState) FunctionParametersParametersValidation(org.finos.legend.engine.plan.execution.validation.FunctionParametersParametersValidation) SerializationConfig(org.finos.legend.engine.protocol.pure.v1.model.valueSpecification.raw.SerializationConfig) Collectors(java.util.stream.Collectors) Constrained(org.finos.legend.engine.plan.dependencies.domain.dataQuality.Constrained) IExecutionNodeContext(org.finos.legend.engine.plan.dependencies.store.shared.IExecutionNodeContext) Objects(java.util.Objects) InMemoryRootGraphFetchExecutionNode(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.nodes.graphFetch.store.inMemory.InMemoryRootGraphFetchExecutionNode) List(java.util.List) Stream(java.util.stream.Stream) Lists(org.eclipse.collections.impl.factory.Lists) Span(io.opentracing.Span) Scope(io.opentracing.Scope) FreeMarkerConditionalExecutionNode(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.nodes.FreeMarkerConditionalExecutionNode) Spliterator(java.util.Spliterator) FreeMarkerExecutor(org.finos.legend.engine.plan.execution.nodes.helpers.freemarker.FreeMarkerExecutor) IPlatformPureExpressionExecutionNodeGraphFetchUnionSpecifics(org.finos.legend.engine.plan.dependencies.store.platform.IPlatformPureExpressionExecutionNodeGraphFetchUnionSpecifics) StoreType(org.finos.legend.engine.plan.execution.stores.StoreType) MultiResultSequenceExecutionNode(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.nodes.MultiResultSequenceExecutionNode) CommonProfile(org.pac4j.core.profile.CommonProfile) ExecutionState(org.finos.legend.engine.plan.execution.nodes.state.ExecutionState) GraphFetchCache(org.finos.legend.engine.plan.execution.cache.graphFetch.GraphFetchCache) AggregationAwareExecutionNode(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.nodes.AggregationAwareExecutionNode) IPlatformPureExpressionExecutionNodeGraphFetchMergeSpecifics(org.finos.legend.engine.plan.dependencies.store.platform.IPlatformPureExpressionExecutionNodeGraphFetchMergeSpecifics) PartialClassBuilder(org.finos.legend.engine.plan.execution.result.builder._class.PartialClassBuilder) GraphFetchResult(org.finos.legend.engine.plan.execution.result.graphFetch.GraphFetchResult) GraphFetchM2MExecutionNode(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.nodes.GraphFetchM2MExecutionNode) ExecutionNodeResultHelper(org.finos.legend.engine.plan.execution.nodes.helpers.ExecutionNodeResultHelper) GraphFetchCrossAssociationKeys(org.finos.legend.engine.plan.execution.cache.graphFetch.GraphFetchCrossAssociationKeys) ExecutionCache(org.finos.legend.engine.plan.execution.cache.ExecutionCache) PureExpressionPlatformExecutionNode(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.nodes.PureExpressionPlatformExecutionNode) GraphFetchExecutionNode(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.nodes.graphFetch.GraphFetchExecutionNode) StreamingObjectResult(org.finos.legend.engine.plan.execution.result.object.StreamingObjectResult) JavaHelper(org.finos.legend.engine.plan.execution.nodes.helpers.platform.JavaHelper) StreamSupport(java.util.stream.StreamSupport) IChecked(org.finos.legend.engine.plan.dependencies.domain.dataQuality.IChecked) JavaPlatformImplementation(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.nodes.JavaPlatformImplementation) ErrorResult(org.finos.legend.engine.plan.execution.result.ErrorResult) SequenceExecutionNode(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.nodes.SequenceExecutionNode) Result(org.finos.legend.engine.plan.execution.result.Result) ResultVisitor(org.finos.legend.engine.plan.execution.result.ResultVisitor) XStorePropertyFetchDetails(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.nodes.graphFetch.XStorePropertyFetchDetails) GraphFetchCacheByTargetCrossKeys(org.finos.legend.engine.plan.execution.cache.graphFetch.GraphFetchCacheByTargetCrossKeys) ExecutionNodeJavaPlatformHelper(org.finos.legend.engine.plan.execution.nodes.helpers.platform.ExecutionNodeJavaPlatformHelper) GlobalTracer(io.opentracing.util.GlobalTracer) GraphObjectsBatch(org.finos.legend.engine.plan.execution.result.graphFetch.GraphObjectsBatch) InMemoryPropertyGraphFetchExecutionNode(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.nodes.graphFetch.store.inMemory.InMemoryPropertyGraphFetchExecutionNode) ProfileManager(org.pac4j.core.profile.ProfileManager) Consumer(java.util.function.Consumer) AtomicLong(java.util.concurrent.atomic.AtomicLong) MultiResult(org.finos.legend.engine.plan.execution.result.MultiResult) GlobalGraphFetchExecutionNode(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.nodes.graphFetch.GlobalGraphFetchExecutionNode) LocalGraphFetchExecutionNode(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.nodes.graphFetch.LocalGraphFetchExecutionNode) XStorePropertyFetchDetails(org.finos.legend.engine.protocol.pure.v1.model.executionPlan.nodes.graphFetch.XStorePropertyFetchDetails) GraphFetchCacheByTargetCrossKeys(org.finos.legend.engine.plan.execution.cache.graphFetch.GraphFetchCacheByTargetCrossKeys) GraphFetchCache(org.finos.legend.engine.plan.execution.cache.graphFetch.GraphFetchCache) GraphFetchCrossAssociationKeys(org.finos.legend.engine.plan.execution.cache.graphFetch.GraphFetchCrossAssociationKeys)

Aggregations

GraphFetchCacheByTargetCrossKeys (org.finos.legend.engine.plan.execution.cache.graphFetch.GraphFetchCacheByTargetCrossKeys)7 PlanExecutionContext (org.finos.legend.engine.plan.execution.PlanExecutionContext)6 SingleExecutionPlan (org.finos.legend.engine.protocol.pure.v1.model.executionPlan.SingleExecutionPlan)6 Test (org.junit.Test)6 List (java.util.List)2 Map (java.util.Map)2 GraphFetchCacheKey (org.finos.legend.engine.plan.execution.cache.graphFetch.GraphFetchCacheKey)2 GraphFetchCrossAssociationKeys (org.finos.legend.engine.plan.execution.cache.graphFetch.GraphFetchCrossAssociationKeys)2 JavaHelper (org.finos.legend.engine.plan.execution.nodes.helpers.platform.JavaHelper)2 Scope (io.opentracing.Scope)1 Span (io.opentracing.Span)1 GlobalTracer (io.opentracing.util.GlobalTracer)1 ByteArrayInputStream (java.io.ByteArrayInputStream)1 ByteArrayOutputStream (java.io.ByteArrayOutputStream)1 ObjectOutputStream (java.io.ObjectOutputStream)1 Arrays (java.util.Arrays)1 Collection (java.util.Collection)1 DoubleSummaryStatistics (java.util.DoubleSummaryStatistics)1 Objects (java.util.Objects)1 Spliterator (java.util.Spliterator)1