Search in sources :

Example 6 with ActionDefinition

use of org.talend.dataprep.api.action.ActionDefinition in project data-prep by Talend.

the class TestI18nKeysForActionsTest method allActionShouldHaveTranslations.

@Test
public void allActionShouldHaveTranslations() {
    for (ActionDefinition actionMetadata : allActions) {
        final String name = actionMetadata.getName();
        assertNotNull(name);
        assertNotEquals("", name);
        actionMetadata.getLabel(Locale.US);
        actionMetadata.getDescription(Locale.US);
        String toString = actionMetadata.getName() + "," + actionMetadata.getCategory(Locale.US) + "," + actionMetadata.getLabel(Locale.US) + "," + actionMetadata.getDescription(Locale.US);
        LOGGER.info(toString);
        for (Parameter param : actionMetadata.getParameters(Locale.US)) {
            assertParameter(param);
        }
    }
}
Also used : Parameter(org.talend.dataprep.parameters.Parameter) SelectParameter(org.talend.dataprep.parameters.SelectParameter) ActionDefinition(org.talend.dataprep.api.action.ActionDefinition) Test(org.junit.Test) TransformationBaseTest(org.talend.dataprep.transformation.TransformationBaseTest)

Example 7 with ActionDefinition

use of org.talend.dataprep.api.action.ActionDefinition in project data-prep by Talend.

the class SimpleSuggestionEngineTest method shouldSuggestionsShouldBeSorted.

@Test
public void shouldSuggestionsShouldBeSorted() throws IOException {
    final String json = IOUtils.toString(this.getClass().getResourceAsStream("sample_column.json"), UTF_8);
    ObjectMapper mapper = new ObjectMapper();
    final ColumnMetadata columnMetadata = mapper.readValue(json, ColumnMetadata.class);
    List<ActionDefinition> actions = new ArrayList<>();
    actions.add(new FillIfEmpty());
    actions.add(new FillInvalid());
    actions.add(new DeleteInvalid());
    actions.add(new DeleteEmpty());
    actions.add(new Absolute());
    actions.add(new UpperCase());
    final Stream<Suggestion> suggestions = engine.score(actions.stream(), columnMetadata);
    int currentScore = Integer.MAX_VALUE;
    for (Suggestion suggestion : suggestions.collect(Collectors.toList())) {
        assertTrue(currentScore >= suggestion.getScore());
        currentScore = suggestion.getScore();
    }
}
Also used : ColumnMetadata(org.talend.dataprep.api.dataset.ColumnMetadata) UpperCase(org.talend.dataprep.transformation.actions.text.UpperCase) ArrayList(java.util.ArrayList) Absolute(org.talend.dataprep.transformation.actions.math.Absolute) ActionDefinition(org.talend.dataprep.api.action.ActionDefinition) DeleteInvalid(org.talend.dataprep.transformation.actions.delete.DeleteInvalid) FillIfEmpty(org.talend.dataprep.transformation.actions.fill.FillIfEmpty) DeleteEmpty(org.talend.dataprep.transformation.actions.delete.DeleteEmpty) FillInvalid(org.talend.dataprep.transformation.actions.fill.FillInvalid) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) Test(org.junit.Test)

Example 8 with ActionDefinition

use of org.talend.dataprep.api.action.ActionDefinition in project data-prep by Talend.

the class ActionParser method parse.

/**
 * Return the parsed actions ready to be run.
 *
 * @param actions the actions to be parsed as string.
 * @return the parsed actions.
 * @throws IllegalArgumentException if <code>actions</code> is null.
 */
public List<RunnableAction> parse(String actions) {
    if (actions == null) {
        // Actions cannot be null (but can be empty string for no op actions).
        throw new IllegalArgumentException("Actions parameter can not be null.");
    }
    if (StringUtils.isEmpty(actions)) {
        return Collections.emptyList();
    }
    try {
        // Parse action JSON
        final Actions parsedActions = mapper.reader(Actions.class).readValue(actions);
        final List<Action> allActions = parsedActions.getActions();
        // Create closures from parsed actions
        final List<RunnableAction> builtActions = new ArrayList<>(allActions.size() + 1);
        // 
        allActions.stream().filter(// 
        parsedAction -> parsedAction != null && parsedAction.getName() != null).forEach(parsedAction -> {
            String actionNameLowerCase = parsedAction.getName().toLowerCase();
            final ActionDefinition metadata = actionRegistry.get(actionNameLowerCase);
            builtActions.add(factory.create(metadata, parsedAction.getParameters()));
        });
        // all set: wraps everything and return to caller
        return builtActions;
    } catch (TalendRuntimeException tpe) {
        // leave TDPException as is
        throw tpe;
    } catch (Exception e) {
        throw new TalendRuntimeException(BaseErrorCodes.UNABLE_TO_PARSE_JSON, e);
    }
}
Also used : StringUtils(org.apache.commons.lang.StringUtils) ActionRegistry(org.talend.dataprep.transformation.pipeline.ActionRegistry) ActionFactory(org.talend.dataprep.transformation.actions.common.ActionFactory) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) Actions(org.talend.dataprep.api.preparation.Actions) ArrayList(java.util.ArrayList) List(java.util.List) ActionDefinition(org.talend.dataprep.api.action.ActionDefinition) BaseErrorCodes(org.talend.dataprep.BaseErrorCodes) Action(org.talend.dataprep.api.preparation.Action) Collections(java.util.Collections) TalendRuntimeException(org.talend.daikon.exception.TalendRuntimeException) RunnableAction(org.talend.dataprep.transformation.actions.common.RunnableAction) TalendRuntimeException(org.talend.daikon.exception.TalendRuntimeException) Action(org.talend.dataprep.api.preparation.Action) RunnableAction(org.talend.dataprep.transformation.actions.common.RunnableAction) Actions(org.talend.dataprep.api.preparation.Actions) RunnableAction(org.talend.dataprep.transformation.actions.common.RunnableAction) ArrayList(java.util.ArrayList) ActionDefinition(org.talend.dataprep.api.action.ActionDefinition) TalendRuntimeException(org.talend.daikon.exception.TalendRuntimeException)

Example 9 with ActionDefinition

use of org.talend.dataprep.api.action.ActionDefinition in project data-prep by Talend.

the class ActionsStaticProfiler method profile.

public ActionsProfile profile(final List<ColumnMetadata> columns, final List<RunnableAction> actions) {
    final Map<Action, ActionDefinition> metadataByAction = getActionMetadataByAction(actions);
    // Compile actions
    final Set<String> originalColumns = columns.stream().map(ColumnMetadata::getId).collect(toSet());
    final Set<String> valueModifiedColumns = new HashSet<>();
    final Set<String> metadataModifiedColumns = new HashSet<>();
    int createColumnActions = 0;
    // Analyze what columns to look at during analysis
    for (Map.Entry<Action, ActionDefinition> entry : metadataByAction.entrySet()) {
        final ActionDefinition actionMetadata = entry.getValue();
        final Action action = entry.getKey();
        Set<ActionDefinition.Behavior> behavior = actionMetadata.getBehavior();
        boolean createColumn = false;
        for (ActionDefinition.Behavior currentBehavior : behavior) {
            switch(currentBehavior) {
                case VALUES_ALL:
                    // All values are going to be changed, and all original columns are going to be modified.
                    valueModifiedColumns.addAll(originalColumns);
                    break;
                case METADATA_CHANGE_TYPE:
                    valueModifiedColumns.add(action.getParameters().get(COLUMN_ID.getKey()));
                    metadataModifiedColumns.add(action.getParameters().get(COLUMN_ID.getKey()));
                    break;
                case VALUES_COLUMN:
                    valueModifiedColumns.add(action.getParameters().get(COLUMN_ID.getKey()));
                    break;
                case VALUES_MULTIPLE_COLUMNS:
                    // Add the action's source column
                    valueModifiedColumns.add(action.getParameters().get(COLUMN_ID.getKey()));
                    // ... then add all column parameter (COLUMN_ID is string, not column)
                    final List<Parameter> parameters = actionMetadata.getParameters(Locale.US);
                    valueModifiedColumns.addAll(// 
                    parameters.stream().filter(// 
                    parameter -> ParameterType.valueOf(parameter.getType().toUpperCase()) == ParameterType.COLUMN).map(// 
                    parameter -> action.getParameters().get(parameter.getName())).collect(Collectors.toList()));
                    break;
                case METADATA_COPY_COLUMNS:
                case METADATA_CREATE_COLUMNS:
                    createColumn = true;
                    break;
                case METADATA_DELETE_COLUMNS:
                case METADATA_CHANGE_NAME:
                    // Do nothing: no need to re-analyze where only name was changed.
                    break;
                default:
                    break;
            }
        }
        if (createColumn || isCreateColumnParameterOn(action)) {
            createColumnActions++;
        }
    }
    // when values are modified, we need to do a full analysis (schema + invalid + stats)
    boolean needFullAnalysis = !valueModifiedColumns.isEmpty() || createColumnActions > 0;
    // when only metadata is modified, we need to re-evaluate the invalids entries
    boolean needOnlyInvalidAnalysis = !needFullAnalysis && !metadataModifiedColumns.isEmpty();
    // only the columns with modified values or new columns need the schema + stats analysis
    SerializablePredicate<ColumnMetadata> filterForFullAnalysis = new FilterForFullAnalysis(originalColumns, valueModifiedColumns);
    // only the columns with metadata change or value changes need to re-evaluate invalids
    Predicate<ColumnMetadata> filterForInvalidAnalysis = new FilterForInvalidAnalysis(filterForFullAnalysis, metadataModifiedColumns);
    return new ActionsProfile(needFullAnalysis, needOnlyInvalidAnalysis, filterForFullAnalysis, filterForInvalidAnalysis, filterForInvalidAnalysis, metadataByAction);
}
Also used : ImplicitParameters(org.talend.dataprep.transformation.actions.common.ImplicitParameters) java.util(java.util) ActionRegistry(org.talend.dataprep.transformation.pipeline.ActionRegistry) Parameter(org.talend.dataprep.parameters.Parameter) Predicate(java.util.function.Predicate) ParameterType(org.talend.dataprep.parameters.ParameterType) Collectors(java.util.stream.Collectors) ScopeCategory(org.talend.dataprep.transformation.actions.category.ScopeCategory) COLUMN_ID(org.talend.dataprep.transformation.actions.common.ImplicitParameters.COLUMN_ID) ActionDefinition(org.talend.dataprep.api.action.ActionDefinition) Action(org.talend.dataprep.api.preparation.Action) Collectors.toSet(java.util.stream.Collectors.toSet) ColumnMetadata(org.talend.dataprep.api.dataset.ColumnMetadata) RunnableAction(org.talend.dataprep.transformation.actions.common.RunnableAction) Action(org.talend.dataprep.api.preparation.Action) RunnableAction(org.talend.dataprep.transformation.actions.common.RunnableAction) ColumnMetadata(org.talend.dataprep.api.dataset.ColumnMetadata) ActionDefinition(org.talend.dataprep.api.action.ActionDefinition) Parameter(org.talend.dataprep.parameters.Parameter)

Example 10 with ActionDefinition

use of org.talend.dataprep.api.action.ActionDefinition in project data-prep by Talend.

the class ActionMetadataValidationTest method checkScopeConsistency_should_throw_exception_on_missing_scope.

@Test
public void checkScopeConsistency_should_throw_exception_on_missing_scope() throws Exception {
    // given
    final Map<String, String> parameters = new HashMap<>();
    parameters.put("column_id", "0001");
    ActionDefinition actionMock = new ActionMetadataExtendingColumn();
    // when
    try {
        validator.checkScopeConsistency(actionMock, parameters);
        fail("should have thrown TDP exception because param scope is missing");
    }// then
     catch (final TalendRuntimeException e) {
        assertThat(e.getCode(), is(MISSING_ACTION_SCOPE));
    }
}
Also used : TalendRuntimeException(org.talend.daikon.exception.TalendRuntimeException) HashMap(java.util.HashMap) ActionDefinition(org.talend.dataprep.api.action.ActionDefinition) Test(org.junit.Test) ServiceBaseTest(org.talend.ServiceBaseTest)

Aggregations

ActionDefinition (org.talend.dataprep.api.action.ActionDefinition)13 Test (org.junit.Test)7 Action (org.talend.dataprep.api.preparation.Action)4 RunnableAction (org.talend.dataprep.transformation.actions.common.RunnableAction)4 ActionRegistry (org.talend.dataprep.transformation.pipeline.ActionRegistry)4 HashMap (java.util.HashMap)3 ObjectMapper (com.fasterxml.jackson.databind.ObjectMapper)2 ArrayList (java.util.ArrayList)2 Collections (java.util.Collections)2 List (java.util.List)2 Collectors (java.util.stream.Collectors)2 StringUtils (org.apache.commons.lang.StringUtils)2 ServiceBaseTest (org.talend.ServiceBaseTest)2 TalendRuntimeException (org.talend.daikon.exception.TalendRuntimeException)2 ColumnMetadata (org.talend.dataprep.api.dataset.ColumnMetadata)2 Owner (org.talend.dataprep.api.share.Owner)2 Parameter (org.talend.dataprep.parameters.Parameter)2 PreparationRepository (org.talend.dataprep.preparation.store.PreparationRepository)2 Security (org.talend.dataprep.security.Security)2 AbstractMetadataBaseTest (org.talend.dataprep.transformation.actions.AbstractMetadataBaseTest)2