Search in sources :

Example 1 with Output

use of org.kie.kogito.explainability.model.Output in project kogito-apps by kiegroup.

the class TestUtils method getSymbolicArithmeticModel.

public static PredictionProvider getSymbolicArithmeticModel() {
    return inputs -> supplyAsync(() -> {
        List<PredictionOutput> predictionOutputs = new LinkedList<>();
        final String OPERAND_FEATURE_NAME = "operand";
        for (PredictionInput predictionInput : inputs) {
            List<Feature> features = predictionInput.getFeatures();
            // Find a valid operand feature, if any
            Optional<String> operand = features.stream().filter(f -> OPERAND_FEATURE_NAME.equals(f.getName())).map(f -> f.getValue().asString()).findFirst();
            if (!operand.isPresent()) {
                throw new IllegalArgumentException("No valid operand found in features");
            }
            final String operandValue = operand.get();
            double result = 0;
            // Apply the found operand to the rest of the features
            for (Feature feature : features) {
                if (!OPERAND_FEATURE_NAME.equals(feature.getName())) {
                    switch(operandValue) {
                        case "+":
                            result += feature.getValue().asNumber();
                            break;
                        case "-":
                            result -= feature.getValue().asNumber();
                            break;
                        case "*":
                            result *= feature.getValue().asNumber();
                            break;
                        case "/":
                            result /= feature.getValue().asNumber();
                            break;
                    }
                }
            }
            PredictionOutput predictionOutput = new PredictionOutput(List.of(new Output("result", Type.NUMBER, new Value(result), 1d)));
            predictionOutputs.add(predictionOutput);
        }
        return predictionOutputs;
    });
}
Also used : Arrays(java.util.Arrays) LimeExplainer(org.kie.kogito.explainability.local.lime.LimeExplainer) Feature(org.kie.kogito.explainability.model.Feature) Prediction(org.kie.kogito.explainability.model.Prediction) Random(java.util.Random) Mockito.when(org.mockito.Mockito.when) Value(org.kie.kogito.explainability.model.Value) Type(org.kie.kogito.explainability.model.Type) PredictionProvider(org.kie.kogito.explainability.model.PredictionProvider) ArrayList(java.util.ArrayList) PredictionInput(org.kie.kogito.explainability.model.PredictionInput) List(java.util.List) Pair(org.apache.commons.lang3.tuple.Pair) Output(org.kie.kogito.explainability.model.Output) CompletableFuture.supplyAsync(java.util.concurrent.CompletableFuture.supplyAsync) Optional(java.util.Optional) ValidationUtils(org.kie.kogito.explainability.utils.ValidationUtils) LinkedList(java.util.LinkedList) Assertions.assertDoesNotThrow(org.junit.jupiter.api.Assertions.assertDoesNotThrow) PredictionOutput(org.kie.kogito.explainability.model.PredictionOutput) Mockito.mock(org.mockito.Mockito.mock) PredictionInput(org.kie.kogito.explainability.model.PredictionInput) Feature(org.kie.kogito.explainability.model.Feature) LinkedList(java.util.LinkedList) PredictionOutput(org.kie.kogito.explainability.model.PredictionOutput) Output(org.kie.kogito.explainability.model.Output) PredictionOutput(org.kie.kogito.explainability.model.PredictionOutput) Value(org.kie.kogito.explainability.model.Value)

Example 2 with Output

use of org.kie.kogito.explainability.model.Output in project kogito-apps by kiegroup.

the class TestUtils method getNoisySumModel.

public static PredictionProvider getNoisySumModel(Random rn, double noiseMagnitude) {
    return inputs -> supplyAsync(() -> {
        List<PredictionOutput> predictionOutputs = new LinkedList<>();
        for (PredictionInput predictionInput : inputs) {
            List<Feature> features = predictionInput.getFeatures();
            double result = 0;
            for (int i = 0; i < features.size(); i++) {
                result += features.get(i).getValue().asNumber() + ((rn.nextDouble() - .5) * noiseMagnitude);
            }
            PredictionOutput predictionOutput = new PredictionOutput(List.of(new Output("noisy-sum", Type.NUMBER, new Value(result), 1d)));
            predictionOutputs.add(predictionOutput);
        }
        return predictionOutputs;
    });
}
Also used : Arrays(java.util.Arrays) LimeExplainer(org.kie.kogito.explainability.local.lime.LimeExplainer) Feature(org.kie.kogito.explainability.model.Feature) Prediction(org.kie.kogito.explainability.model.Prediction) Random(java.util.Random) Mockito.when(org.mockito.Mockito.when) Value(org.kie.kogito.explainability.model.Value) Type(org.kie.kogito.explainability.model.Type) PredictionProvider(org.kie.kogito.explainability.model.PredictionProvider) ArrayList(java.util.ArrayList) PredictionInput(org.kie.kogito.explainability.model.PredictionInput) List(java.util.List) Pair(org.apache.commons.lang3.tuple.Pair) Output(org.kie.kogito.explainability.model.Output) CompletableFuture.supplyAsync(java.util.concurrent.CompletableFuture.supplyAsync) Optional(java.util.Optional) ValidationUtils(org.kie.kogito.explainability.utils.ValidationUtils) LinkedList(java.util.LinkedList) Assertions.assertDoesNotThrow(org.junit.jupiter.api.Assertions.assertDoesNotThrow) PredictionOutput(org.kie.kogito.explainability.model.PredictionOutput) Mockito.mock(org.mockito.Mockito.mock) PredictionInput(org.kie.kogito.explainability.model.PredictionInput) PredictionOutput(org.kie.kogito.explainability.model.PredictionOutput) Output(org.kie.kogito.explainability.model.Output) PredictionOutput(org.kie.kogito.explainability.model.PredictionOutput) Value(org.kie.kogito.explainability.model.Value) Feature(org.kie.kogito.explainability.model.Feature) LinkedList(java.util.LinkedList)

Example 3 with Output

use of org.kie.kogito.explainability.model.Output in project kogito-apps by kiegroup.

the class TestUtils method getSumThresholdModel.

public static PredictionProvider getSumThresholdModel(double center, double epsilon) {
    return inputs -> supplyAsync(() -> {
        List<PredictionOutput> predictionOutputs = new LinkedList<>();
        for (PredictionInput predictionInput : inputs) {
            List<Feature> features = predictionInput.getFeatures();
            double result = 0;
            for (int i = 0; i < features.size(); i++) {
                result += features.get(i).getValue().asNumber();
            }
            final boolean inside = (result >= center - epsilon && result <= center + epsilon);
            PredictionOutput predictionOutput = new PredictionOutput(List.of(new Output("inside", Type.BOOLEAN, new Value(inside), 1.0 - Math.abs(result - center))));
            predictionOutputs.add(predictionOutput);
        }
        return predictionOutputs;
    });
}
Also used : Arrays(java.util.Arrays) LimeExplainer(org.kie.kogito.explainability.local.lime.LimeExplainer) Feature(org.kie.kogito.explainability.model.Feature) Prediction(org.kie.kogito.explainability.model.Prediction) Random(java.util.Random) Mockito.when(org.mockito.Mockito.when) Value(org.kie.kogito.explainability.model.Value) Type(org.kie.kogito.explainability.model.Type) PredictionProvider(org.kie.kogito.explainability.model.PredictionProvider) ArrayList(java.util.ArrayList) PredictionInput(org.kie.kogito.explainability.model.PredictionInput) List(java.util.List) Pair(org.apache.commons.lang3.tuple.Pair) Output(org.kie.kogito.explainability.model.Output) CompletableFuture.supplyAsync(java.util.concurrent.CompletableFuture.supplyAsync) Optional(java.util.Optional) ValidationUtils(org.kie.kogito.explainability.utils.ValidationUtils) LinkedList(java.util.LinkedList) Assertions.assertDoesNotThrow(org.junit.jupiter.api.Assertions.assertDoesNotThrow) PredictionOutput(org.kie.kogito.explainability.model.PredictionOutput) Mockito.mock(org.mockito.Mockito.mock) PredictionInput(org.kie.kogito.explainability.model.PredictionInput) PredictionOutput(org.kie.kogito.explainability.model.PredictionOutput) Output(org.kie.kogito.explainability.model.Output) PredictionOutput(org.kie.kogito.explainability.model.PredictionOutput) Value(org.kie.kogito.explainability.model.Value) Feature(org.kie.kogito.explainability.model.Feature) LinkedList(java.util.LinkedList)

Example 4 with Output

use of org.kie.kogito.explainability.model.Output in project kogito-apps by kiegroup.

the class AggregatedLimeExplainerTest method testExplainWithMetadata.

@ParameterizedTest
@ValueSource(ints = { 0, 1, 2, 3, 4 })
void testExplainWithMetadata(int seed) throws ExecutionException, InterruptedException {
    Random random = new Random();
    random.setSeed(seed);
    PredictionProvider sumSkipModel = TestUtils.getSumSkipModel(1);
    PredictionProviderMetadata metadata = new PredictionProviderMetadata() {

        @Override
        public DataDistribution getDataDistribution() {
            return DataUtils.generateRandomDataDistribution(3, 100, random);
        }

        @Override
        public PredictionInput getInputShape() {
            List<Feature> features = new LinkedList<>();
            features.add(FeatureFactory.newNumericalFeature("f0", 0));
            features.add(FeatureFactory.newNumericalFeature("f1", 0));
            features.add(FeatureFactory.newNumericalFeature("f2", 0));
            return new PredictionInput(features);
        }

        @Override
        public PredictionOutput getOutputShape() {
            List<Output> outputs = new LinkedList<>();
            outputs.add(new Output("sum-but1", Type.BOOLEAN, new Value(false), 0d));
            return new PredictionOutput(outputs);
        }
    };
    AggregatedLimeExplainer aggregatedLimeExplainer = new AggregatedLimeExplainer();
    Map<String, Saliency> explain = aggregatedLimeExplainer.explainFromMetadata(sumSkipModel, metadata).get();
    assertNotNull(explain);
    assertEquals(1, explain.size());
    assertTrue(explain.containsKey("sum-but1"));
    Saliency saliency = explain.get("sum-but1");
    assertNotNull(saliency);
    List<String> collect = saliency.getPositiveFeatures(2).stream().map(FeatureImportance::getFeature).map(Feature::getName).collect(Collectors.toList());
    // skipped feature should not appear in top two positive features
    assertFalse(collect.contains("f1"));
}
Also used : PredictionInput(org.kie.kogito.explainability.model.PredictionInput) PredictionProviderMetadata(org.kie.kogito.explainability.model.PredictionProviderMetadata) Saliency(org.kie.kogito.explainability.model.Saliency) PredictionProvider(org.kie.kogito.explainability.model.PredictionProvider) Feature(org.kie.kogito.explainability.model.Feature) LinkedList(java.util.LinkedList) Random(java.util.Random) FeatureImportance(org.kie.kogito.explainability.model.FeatureImportance) PredictionOutput(org.kie.kogito.explainability.model.PredictionOutput) PredictionOutput(org.kie.kogito.explainability.model.PredictionOutput) Output(org.kie.kogito.explainability.model.Output) Value(org.kie.kogito.explainability.model.Value) ValueSource(org.junit.jupiter.params.provider.ValueSource) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest)

Example 5 with Output

use of org.kie.kogito.explainability.model.Output in project kogito-apps by kiegroup.

the class CounterfactualExplainerTest method testFinalUniqueIds.

@ParameterizedTest
@ValueSource(ints = { 0, 1, 2 })
void testFinalUniqueIds(int seed) throws ExecutionException, InterruptedException, TimeoutException {
    Random random = new Random();
    random.setSeed(seed);
    final List<Output> goal = new ArrayList<>();
    List<Feature> features = List.of(FeatureFactory.newNumericalFeature("f-num1", 10.0, NumericalFeatureDomain.create(0, 20)));
    PredictionProvider model = TestUtils.getFeaturePassModel(0);
    final TerminationConfig terminationConfig = new TerminationConfig().withScoreCalculationCountLimit(100_000L);
    final SolverConfig solverConfig = SolverConfigBuilder.builder().withTerminationConfig(terminationConfig).build();
    solverConfig.setRandomSeed((long) seed);
    solverConfig.setEnvironmentMode(EnvironmentMode.REPRODUCIBLE);
    final List<UUID> intermediateIds = new ArrayList<>();
    final List<UUID> executionIds = new ArrayList<>();
    final Consumer<CounterfactualResult> captureIntermediateIds = counterfactual -> {
        intermediateIds.add(counterfactual.getSolutionId());
    };
    final Consumer<CounterfactualResult> captureExecutionIds = counterfactual -> {
        executionIds.add(counterfactual.getExecutionId());
    };
    final CounterfactualConfig counterfactualConfig = new CounterfactualConfig().withSolverConfig(solverConfig);
    solverConfig.withEasyScoreCalculatorClass(MockCounterFactualScoreCalculator.class);
    final CounterfactualExplainer counterfactualExplainer = new CounterfactualExplainer(counterfactualConfig);
    PredictionInput input = new PredictionInput(features);
    PredictionOutput output = new PredictionOutput(goal);
    final UUID executionId = UUID.randomUUID();
    Prediction prediction = new CounterfactualPrediction(input, output, null, executionId, null);
    final CounterfactualResult counterfactualResult = counterfactualExplainer.explainAsync(prediction, model, captureIntermediateIds.andThen(captureExecutionIds)).get(Config.INSTANCE.getAsyncTimeout(), Config.INSTANCE.getAsyncTimeUnit());
    for (CounterfactualEntity entity : counterfactualResult.getEntities()) {
        logger.debug("Entity: {}", entity);
    }
    // All intermediate ids should be unique
    assertEquals((int) intermediateIds.stream().distinct().count(), intermediateIds.size());
    // There should be at least one intermediate id
    assertTrue(intermediateIds.size() > 0);
    // There should be at least one execution id
    assertTrue(executionIds.size() > 0);
    // We should have the same number of execution ids as intermediate ids (captured from intermediate results)
    assertEquals(executionIds.size(), intermediateIds.size());
    // All execution ids should be the same
    assertEquals(1, (int) executionIds.stream().distinct().count());
    // The last intermediate id must be different from the final result id
    assertNotEquals(intermediateIds.get(intermediateIds.size() - 1), counterfactualResult.getSolutionId());
    // Captured execution ids should be the same as the one provided
    assertEquals(executionIds.get(0), executionId);
}
Also used : BeforeEach(org.junit.jupiter.api.BeforeEach) FeatureFactory(org.kie.kogito.explainability.model.FeatureFactory) Feature(org.kie.kogito.explainability.model.Feature) LoggerFactory(org.slf4j.LoggerFactory) Assertions.assertNotEquals(org.junit.jupiter.api.Assertions.assertNotEquals) TimeoutException(java.util.concurrent.TimeoutException) Random(java.util.Random) CounterfactualEntity(org.kie.kogito.explainability.local.counterfactual.entities.CounterfactualEntity) Value(org.kie.kogito.explainability.model.Value) TerminationConfig(org.optaplanner.core.config.solver.termination.TerminationConfig) Assertions.assertFalse(org.junit.jupiter.api.Assertions.assertFalse) FeatureDistribution(org.kie.kogito.explainability.model.FeatureDistribution) EmptyFeatureDomain(org.kie.kogito.explainability.model.domain.EmptyFeatureDomain) Mockito.atLeast(org.mockito.Mockito.atLeast) PredictionOutput(org.kie.kogito.explainability.model.PredictionOutput) CategoricalFeatureDomain(org.kie.kogito.explainability.model.domain.CategoricalFeatureDomain) DataUtils(org.kie.kogito.explainability.utils.DataUtils) UUID(java.util.UUID) Collectors(java.util.stream.Collectors) Test(org.junit.jupiter.api.Test) PredictionInput(org.kie.kogito.explainability.model.PredictionInput) List(java.util.List) Stream(java.util.stream.Stream) NormalDistribution(org.apache.commons.math3.distribution.NormalDistribution) Output(org.kie.kogito.explainability.model.Output) Assertions.assertTrue(org.junit.jupiter.api.Assertions.assertTrue) SolverJob(org.optaplanner.core.api.solver.SolverJob) Mockito.mock(org.mockito.Mockito.mock) IntStream(java.util.stream.IntStream) ArgumentMatchers.any(org.mockito.ArgumentMatchers.any) SolverConfig(org.optaplanner.core.config.solver.SolverConfig) Assertions.assertNotNull(org.junit.jupiter.api.Assertions.assertNotNull) PerturbationContext(org.kie.kogito.explainability.model.PerturbationContext) Prediction(org.kie.kogito.explainability.model.Prediction) DataDomain(org.kie.kogito.explainability.model.DataDomain) Assertions.assertNull(org.junit.jupiter.api.Assertions.assertNull) EnvironmentMode(org.optaplanner.core.config.solver.EnvironmentMode) CompletableFuture(java.util.concurrent.CompletableFuture) SolverManager(org.optaplanner.core.api.solver.SolverManager) Function(java.util.function.Function) ArrayList(java.util.ArrayList) MockCounterFactualScoreCalculator(org.kie.kogito.explainability.local.counterfactual.score.MockCounterFactualScoreCalculator) ArgumentCaptor(org.mockito.ArgumentCaptor) NumericFeatureDistribution(org.kie.kogito.explainability.model.NumericFeatureDistribution) CounterfactualPrediction(org.kie.kogito.explainability.model.CounterfactualPrediction) Assertions.assertEquals(org.junit.jupiter.api.Assertions.assertEquals) LinkedList(java.util.LinkedList) BendableBigDecimalScore(org.optaplanner.core.api.score.buildin.bendablebigdecimal.BendableBigDecimalScore) ValueSource(org.junit.jupiter.params.provider.ValueSource) Logger(org.slf4j.Logger) Mockito.when(org.mockito.Mockito.when) Type(org.kie.kogito.explainability.model.Type) PredictionProvider(org.kie.kogito.explainability.model.PredictionProvider) Mockito.verify(org.mockito.Mockito.verify) ExecutionException(java.util.concurrent.ExecutionException) TimeUnit(java.util.concurrent.TimeUnit) Consumer(java.util.function.Consumer) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest) TestUtils(org.kie.kogito.explainability.TestUtils) NumericalFeatureDomain(org.kie.kogito.explainability.model.domain.NumericalFeatureDomain) Config(org.kie.kogito.explainability.Config) Collections(java.util.Collections) FeatureDomain(org.kie.kogito.explainability.model.domain.FeatureDomain) PredictionInput(org.kie.kogito.explainability.model.PredictionInput) Prediction(org.kie.kogito.explainability.model.Prediction) CounterfactualPrediction(org.kie.kogito.explainability.model.CounterfactualPrediction) ArrayList(java.util.ArrayList) PredictionProvider(org.kie.kogito.explainability.model.PredictionProvider) Feature(org.kie.kogito.explainability.model.Feature) CounterfactualPrediction(org.kie.kogito.explainability.model.CounterfactualPrediction) CounterfactualEntity(org.kie.kogito.explainability.local.counterfactual.entities.CounterfactualEntity) TerminationConfig(org.optaplanner.core.config.solver.termination.TerminationConfig) Random(java.util.Random) PredictionOutput(org.kie.kogito.explainability.model.PredictionOutput) PredictionOutput(org.kie.kogito.explainability.model.PredictionOutput) Output(org.kie.kogito.explainability.model.Output) UUID(java.util.UUID) SolverConfig(org.optaplanner.core.config.solver.SolverConfig) ValueSource(org.junit.jupiter.params.provider.ValueSource) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest)

Aggregations

Output (org.kie.kogito.explainability.model.Output)120 PredictionOutput (org.kie.kogito.explainability.model.PredictionOutput)109 Feature (org.kie.kogito.explainability.model.Feature)102 Value (org.kie.kogito.explainability.model.Value)63 Random (java.util.Random)61 ParameterizedTest (org.junit.jupiter.params.ParameterizedTest)59 PredictionInput (org.kie.kogito.explainability.model.PredictionInput)57 PredictionProvider (org.kie.kogito.explainability.model.PredictionProvider)52 ArrayList (java.util.ArrayList)47 ValueSource (org.junit.jupiter.params.provider.ValueSource)47 Prediction (org.kie.kogito.explainability.model.Prediction)46 Test (org.junit.jupiter.api.Test)42 List (java.util.List)39 Type (org.kie.kogito.explainability.model.Type)36 LinkedList (java.util.LinkedList)35 CounterfactualEntity (org.kie.kogito.explainability.local.counterfactual.entities.CounterfactualEntity)23 Mockito.mock (org.mockito.Mockito.mock)20 Optional (java.util.Optional)19 ExecutionException (java.util.concurrent.ExecutionException)19 Collectors (java.util.stream.Collectors)18