Examples with ApplicationException nl.sidnlabs.entrada.exception.ApplicationException used on opensource projects

Example 1 with ApplicationException

use of nl.sidnlabs.entrada.exception.ApplicationException in project entrada by SIDN.

the class AmazonInitializer method initializeStorage.

@Override
public boolean initializeStorage() {
    log.info("Provision AWS storage");
    // create local storage locations
    super.initializeStorage();
    if (!fileManager.supported(output)) {
        throw new ApplicationException("Selected mode is AWS but the ENTRADA output location does not use S3, cannot continue");
    }
    // check if the s3 bucket and required directories exist and if not create these
    if (!BucketNameUtils.isValidV2BucketName(bucket)) {
        throw new ApplicationException("\"" + bucket + "\" is not a valid S3 bucket name, for bucket restrictions and limitations, see: " + bucketRules);
    }
    // only create bucket and lifecycle rules if we are managing the bucket
    if (!manageBucket) {
        return true;
    }
    if (!amazonS3.doesBucketExistV2(bucket)) {
        log.info("Create bucket: " + bucket);
        amazonS3.createBucket(bucket);
        // make sure to block all public access to the bucket
        amazonS3.setPublicAccessBlock(new SetPublicAccessBlockRequest().withBucketName(bucket).withPublicAccessBlockConfiguration(new PublicAccessBlockConfiguration().withBlockPublicAcls(Boolean.TRUE).withIgnorePublicAcls(Boolean.TRUE).withBlockPublicPolicy(Boolean.TRUE).withRestrictPublicBuckets(Boolean.TRUE)));
        enableEncryption();
    }
    // also make sure pcap files archived on S3 have a expiration lifecycle policy
    return amazonS3.doesBucketExistV2(bucket) && enableBucketLifecycle(athenaOutputLocation, "Delete Athena results", outputExpiration, false) && enableBucketLifecycle(archive, "Delete archived pcap-files", archiveExpiration, true);
}

Example 2 with ApplicationException

use of nl.sidnlabs.entrada.exception.ApplicationException in project entrada by SIDN.

the class HDFSFileManagerImpl method createSecureFS.

private FileSystem createSecureFS() {
    Configuration conf = conf();
    conf.set("hadoop.security.authentication", "kerberos");
    UserGroupInformation.setConfiguration(conf);
    try {
        if (StringUtils.isNotBlank(krbKeyTab)) {
            UserGroupInformation.loginUserFromKeytab(hdfsUsername, krbKeyTab);
        }
        // https://stackoverflow.com/questions/20057881/hadoop-filesystem-closed-exception-when-doing-bufferedreader-close/20061797#20061797
        return FileSystem.newInstance(new URI(hdfsNameservice), conf);
    } catch (Exception e) {
        throw new ApplicationException("Cannot create secure HDFS filesystem", e);
    }
}

Example 3 with ApplicationException

use of nl.sidnlabs.entrada.exception.ApplicationException in project entrada by SIDN.

the class HDFSFileManagerImpl method conf.

private Configuration conf() {
    Configuration conf = new Configuration();
    conf.set("fs.defaultFS", hdfsNameservice);
    String coreSiteXml = confDir + "/core-site.xml";
    if (!new File(coreSiteXml).exists()) {
        throw new ApplicationException("Missing core-site.xml, add this to the conf directory");
    }
    String hdfsSiteXml = confDir + "/hdfs-site.xml";
    if (!new File(hdfsSiteXml).exists()) {
        throw new ApplicationException("Missing hdfs-site.xml, add this to the conf directory");
    }
    conf.addResource(new Path("file://" + hdfsSiteXml));
    conf.addResource(new Path("file://" + coreSiteXml));
    return conf;
}

Example 4 with ApplicationException

use of nl.sidnlabs.entrada.exception.ApplicationException in project entrada by SIDN.

the class AbstractParquetRowWriter method schema.

public Schema schema(String schema) {
    if (avroSchema != null) {
        // use cached version of schema
        return avroSchema;
    }
    try {
        Parser parser = new Schema.Parser().setValidate(true);
        avroSchema = parser.parse(new ClassPathResource(schema, getClass()).getInputStream());
    } catch (IOException e) {
        throw new ApplicationException("Cannot load schema from file: " + schema, e);
    }
    return avroSchema;
}

Example 5 with ApplicationException

use of nl.sidnlabs.entrada.exception.ApplicationException in project entrada by SIDN.

the class HDFSFileManagerImpl method createNonSecureFS.

private FileSystem createNonSecureFS() {
    Configuration conf = conf();
    System.setProperty("HADOOP_USER_NAME", hdfsUsername);
    try {
        // https://stackoverflow.com/questions/20057881/hadoop-filesystem-closed-exception-when-doing-bufferedreader-close/20061797#20061797
        return FileSystem.newInstance(conf);
    } catch (IOException e) {
        throw new ApplicationException("Cannot create non-secure HDFS filesystem", e);
    }
}