Search in sources :

Example 11 with SchemaUtils

use of uk.gov.gchq.gaffer.parquetstore.utils.SchemaUtils in project Gaffer by gchq.

the class ParquetStore method initialise.

@Override
public void initialise(final String graphId, final Schema schema, final StoreProperties properties) throws StoreException {
    if (!(properties instanceof ParquetStoreProperties)) {
        throw new StoreException("ParquetStore must be initialised with properties of class ParquetStoreProperties");
    }
    final ParquetStoreProperties parquetStoreProperties = (ParquetStoreProperties) properties;
    if (null == parquetStoreProperties.getDataDir()) {
        throw new StoreException("The ParquetStoreProperties must contain a non-null data directory (" + ParquetStoreProperties.DATA_DIR + ")");
    }
    if (null == parquetStoreProperties.getTempFilesDir()) {
        throw new StoreException("The ParquetStoreProperties must contain a non-null temporary data directory (" + ParquetStoreProperties.TEMP_FILES_DIR + ")");
    }
    LOGGER.info("Initialising ParquetStore for graph id {}", graphId);
    super.initialise(graphId, schema, parquetStoreProperties);
    try {
        fs = FileSystem.get(new Configuration());
        schemaUtils = new SchemaUtils(getSchema());
        initialise();
        loadGraphPartitioner();
    } catch (final IOException e) {
        throw new StoreException("Could not connect to the file system", e);
    }
}
Also used : SchemaUtils(uk.gov.gchq.gaffer.parquetstore.utils.SchemaUtils) Configuration(org.apache.hadoop.conf.Configuration) IOException(java.io.IOException) StoreException(uk.gov.gchq.gaffer.store.StoreException)

Aggregations

SchemaUtils (uk.gov.gchq.gaffer.parquetstore.utils.SchemaUtils)11 Path (org.apache.hadoop.fs.Path)8 ArrayList (java.util.ArrayList)7 List (java.util.List)6 ParquetStore (uk.gov.gchq.gaffer.parquetstore.ParquetStore)6 IOException (java.io.IOException)5 FileSystem (org.apache.hadoop.fs.FileSystem)4 Test (org.junit.jupiter.api.Test)4 ParquetStoreProperties (uk.gov.gchq.gaffer.parquetstore.ParquetStoreProperties)4 GraphPartitioner (uk.gov.gchq.gaffer.parquetstore.partitioner.GraphPartitioner)4 Schema (uk.gov.gchq.gaffer.store.schema.Schema)4 FileStatus (org.apache.hadoop.fs.FileStatus)3 Logger (org.slf4j.Logger)3 LoggerFactory (org.slf4j.LoggerFactory)3 View (uk.gov.gchq.gaffer.data.elementdefinition.view.View)3 OperationException (uk.gov.gchq.gaffer.operation.OperationException)3 LongVertexOperationsTest (uk.gov.gchq.gaffer.parquetstore.operation.handler.LongVertexOperationsTest)3 CalculatePartitionerTest (uk.gov.gchq.gaffer.parquetstore.operation.handler.utilities.CalculatePartitionerTest)3 StoreException (uk.gov.gchq.gaffer.store.StoreException)3 Arrays (java.util.Arrays)2