Search in sources :

Example 1 with ParsingError

use of org.openforis.collect.io.metadata.parsing.ParsingError in project collect by openforis.

the class SpeciesImportProcessIntegrationTest method testInvalidColumns.

@Test
public void testInvalidColumns() throws Exception {
    SpeciesImportProcess process = importCSVFile(INVALID_MISSING_COLUMNS_TEST_CSV);
    SpeciesImportStatus status = process.getStatus();
    assertTrue(status.isError());
    List<ParsingError> errors = status.getErrors();
    assertEquals(1, errors.size());
    ParsingError error = errors.get(0);
    ErrorType errorType = error.getErrorType();
    assertEquals(ErrorType.MISSING_REQUIRED_COLUMNS, errorType);
}
Also used : ParsingError(org.openforis.collect.io.metadata.parsing.ParsingError) ErrorType(org.openforis.collect.io.metadata.parsing.ParsingError.ErrorType) SpeciesImportProcess(org.openforis.collect.manager.speciesimport.SpeciesImportProcess) SpeciesImportStatus(org.openforis.collect.manager.speciesimport.SpeciesImportStatus) CollectIntegrationTest(org.openforis.collect.CollectIntegrationTest) Test(org.junit.Test)

Example 2 with ParsingError

use of org.openforis.collect.io.metadata.parsing.ParsingError in project collect by openforis.

the class CSVDataImportJob method getParsingErrors.

public List<DataParsingError> getParsingErrors() {
    List<DataParsingError> result = new ArrayList<DataParsingError>();
    for (Worker worker : getTasks()) {
        CSVDataImportTask task = (CSVDataImportTask) worker;
        ReferenceDataImportStatus<ParsingError> dataImportStatus = task.getDataImportStatus();
        if (dataImportStatus != null) {
            List<ParsingError> errors = dataImportStatus.getErrors();
            for (ParsingError parsingError : errors) {
                DataParsingError dataParsingError = new DataParsingError(task.getInput().getFile().getName(), parsingError.getRow(), parsingError.getErrorType(), parsingError.getColumns(), parsingError.getMessage(), parsingError.getMessageArgs());
                result.add(dataParsingError);
            }
        }
    }
    return result;
}
Also used : ParsingError(org.openforis.collect.io.metadata.parsing.ParsingError) ArrayList(java.util.ArrayList) Worker(org.openforis.concurrency.Worker)

Example 3 with ParsingError

use of org.openforis.collect.io.metadata.parsing.ParsingError in project collect by openforis.

the class CSVDataImportProcess method setSRSIdField.

private void setSRSIdField(Attribute<?, ?> attr, String value, long row, String colName) {
    boolean valid = true;
    if (StringUtils.isNotBlank(value)) {
        // check SRS id validity
        Survey survey = attr.getSurvey();
        SpatialReferenceSystem srs = survey.getSpatialReferenceSystem(value);
        if (srs == null) {
            ParsingError parsingError = new ParsingError(ErrorType.INVALID_VALUE, row, colName, SRS_NOT_FOUND_MESSAGE_KEY);
            parsingError.setMessageArgs(new String[] { value });
            status.addParsingError(parsingError);
            valid = false;
        }
    }
    if (valid) {
        Field<String> field = ((CoordinateAttribute) attr).getSrsIdField();
        NodeChangeSet changes = recordUpdater.updateField(field, value);
        if (nodeChangeBatchProcessor != null) {
            nodeChangeBatchProcessor.add(changes, adminUser.getUsername());
        }
    }
}
Also used : NodeChangeSet(org.openforis.collect.model.NodeChangeSet) CollectSurvey(org.openforis.collect.model.CollectSurvey) Survey(org.openforis.idm.metamodel.Survey) ParsingError(org.openforis.collect.io.metadata.parsing.ParsingError) CoordinateAttribute(org.openforis.idm.model.CoordinateAttribute) SpatialReferenceSystem(org.openforis.idm.metamodel.SpatialReferenceSystem)

Example 4 with ParsingError

use of org.openforis.collect.io.metadata.parsing.ParsingError in project collect by openforis.

the class CSVDataImportProcess method createParentEntitySearchError.

private ParsingError createParentEntitySearchError(CollectRecord record, DataLine line, EntityIdentifier<?> identifier, String messageKey) {
    EntityIdentifierDefinition identifierDefn = identifier.getDefinition();
    Survey survey = record.getSurvey();
    Schema schema = survey.getSchema();
    EntityDefinition parentEntityDefn = (EntityDefinition) schema.getDefinitionById(identifierDefn.getEntityDefinitionId());
    String[] colNames = DataCSVReader.getKeyAttributeColumnNames(parentEntityDefn, parentEntityDefn.getKeyAttributeDefinitions());
    ParsingError error = new ParsingError(ErrorType.INVALID_VALUE, line.getLineNumber(), colNames, messageKey);
    List<String> recordKeys = record.getRootEntityKeyValues();
    CollectionUtils.filter(recordKeys, new Predicate() {

        @Override
        public boolean evaluate(Object object) {
            return StringUtils.isNotBlank((String) object);
        }
    });
    String jointRecordKeys = StringUtils.join(recordKeys, ", ");
    String jointParentEntityKeys = identifier instanceof EntityPositionIdentifier ? "[" + ((EntityPositionIdentifier) identifier).getPosition() + "]" : StringUtils.join(((EntityKeysIdentifier) identifier).getKeyValues(), ", ");
    error.setMessageArgs(new String[] { parentEntityDefn.getName(), jointParentEntityKeys, jointRecordKeys });
    return error;
}
Also used : EntityPositionIdentifier(org.openforis.collect.io.data.DataLine.EntityPositionIdentifier) EntityKeysIdentifier(org.openforis.collect.io.data.DataLine.EntityKeysIdentifier) Schema(org.openforis.idm.metamodel.Schema) Predicate(org.apache.commons.collections.Predicate) EntityDefinition(org.openforis.idm.metamodel.EntityDefinition) CollectSurvey(org.openforis.collect.model.CollectSurvey) Survey(org.openforis.idm.metamodel.Survey) ParsingError(org.openforis.collect.io.metadata.parsing.ParsingError) EntityIdentifierDefinition(org.openforis.collect.io.data.DataLine.EntityIdentifierDefinition)

Example 5 with ParsingError

use of org.openforis.collect.io.metadata.parsing.ParsingError in project collect by openforis.

the class CSVDataImportProcessIntegrationTest method invalidHeaderTest.

@Test
public void invalidHeaderTest() throws Exception {
    EntityDefinition clusterDefn = survey.getSchema().getRootEntityDefinition("cluster");
    EntityDefinition plotDefn = (EntityDefinition) clusterDefn.getChildDefinition("plot");
    CSVDataImportProcess process = importCSVFile(INVALID_HEADER_TEST_CSV, plotDefn.getId());
    ReferenceDataImportStatus<ParsingError> status = process.getStatus();
    assertFalse(status.isComplete());
    assertTrue(status.isError());
    List<ParsingError> errors = status.getErrors();
    assertEquals(1, errors.size());
    ParsingError headerError = errors.get(0);
    assertEquals(ErrorType.WRONG_COLUMN_NAME, headerError.getErrorType());
    assertTrue(Arrays.equals(new String[] { "land_usage" }, headerError.getColumns()));
}
Also used : EntityDefinition(org.openforis.idm.metamodel.EntityDefinition) ParsingError(org.openforis.collect.io.metadata.parsing.ParsingError) CollectIntegrationTest(org.openforis.collect.CollectIntegrationTest) Test(org.junit.Test)

Aggregations

ParsingError (org.openforis.collect.io.metadata.parsing.ParsingError)46 EntityDefinition (org.openforis.idm.metamodel.EntityDefinition)19 Test (org.junit.Test)17 CollectIntegrationTest (org.openforis.collect.CollectIntegrationTest)17 CollectRecord (org.openforis.collect.model.CollectRecord)12 ParsingException (org.openforis.collect.io.exception.ParsingException)9 Entity (org.openforis.idm.model.Entity)9 CollectSurvey (org.openforis.collect.model.CollectSurvey)5 RealAttribute (org.openforis.idm.model.RealAttribute)5 DataParsingError (org.openforis.collect.io.data.CSVDataImportJob.DataParsingError)4 RealValue (org.openforis.idm.model.RealValue)4 IOException (java.io.IOException)3 NodeChangeSet (org.openforis.collect.model.NodeChangeSet)3 Survey (org.openforis.idm.metamodel.Survey)3 ArrayList (java.util.ArrayList)2 SpeciesImportProcess (org.openforis.collect.manager.speciesimport.SpeciesImportProcess)2 SpeciesImportStatus (org.openforis.collect.manager.speciesimport.SpeciesImportStatus)2 CollectRecordSummary (org.openforis.collect.model.CollectRecordSummary)2 CodeListItem (org.openforis.idm.metamodel.CodeListItem)2 SpatialReferenceSystem (org.openforis.idm.metamodel.SpatialReferenceSystem)2