Search in sources :

Example 1 with TextFile

use of org.hl7.fhir.utilities.TextFile in project pathling by aehrc.

the class ImportExecutor method readStringsFromUrl.

@Nonnull
private Dataset<String> readStringsFromUrl(@Nonnull final ParametersParameterComponent urlParam) {
    final String url = ((UrlType) urlParam.getValue()).getValueAsString();
    final String decodedUrl = URLDecoder.decode(url, StandardCharsets.UTF_8);
    final String convertedUrl = PersistenceScheme.convertS3ToS3aUrl(decodedUrl);
    final Dataset<String> jsonStrings;
    try {
        // Check that the user is authorized to execute the operation.
        accessRules.ifPresent(ar -> ar.checkCanImportFrom(convertedUrl));
        final FilterFunction<String> nonBlanks = s -> !s.isBlank();
        jsonStrings = spark.read().textFile(convertedUrl).filter(nonBlanks);
    } catch (final SecurityError e) {
        throw new InvalidUserInputError("Not allowed to import from URL: " + convertedUrl, e);
    } catch (final Exception e) {
        throw new InvalidUserInputError("Error reading from URL: " + convertedUrl, e);
    }
    return jsonStrings;
}
Also used : UrlType(org.hl7.fhir.r4.model.UrlType) URLDecoder(java.net.URLDecoder) Getter(lombok.Getter) Dataset(org.apache.spark.sql.Dataset) ResourceType(org.hl7.fhir.r4.model.Enumerations.ResourceType) UnsupportedResourceError(au.csiro.pathling.encoders.UnsupportedResourceError) FhirEncoders(au.csiro.pathling.encoders.FhirEncoders) AccessRules(au.csiro.pathling.io.AccessRules) IBaseResource(org.hl7.fhir.instance.model.api.IBaseResource) InvalidUserInputError(au.csiro.pathling.errors.InvalidUserInputError) SecurityError(au.csiro.pathling.errors.SecurityError) Nonnull(javax.annotation.Nonnull) Nullable(javax.annotation.Nullable) MapFunction(org.apache.spark.api.java.function.MapFunction) SparkSession(org.apache.spark.sql.SparkSession) ParametersParameterComponent(org.hl7.fhir.r4.model.Parameters.ParametersParameterComponent) IssueSeverity(org.hl7.fhir.r4.model.OperationOutcome.IssueSeverity) PersistenceScheme(au.csiro.pathling.io.PersistenceScheme) Collectors(java.util.stream.Collectors) Profile(org.springframework.context.annotation.Profile) FhirContextFactory(au.csiro.pathling.fhir.FhirContextFactory) StandardCharsets(java.nio.charset.StandardCharsets) OperationOutcomeIssueComponent(org.hl7.fhir.r4.model.OperationOutcome.OperationOutcomeIssueComponent) ExpressionEncoder(org.apache.spark.sql.catalyst.encoders.ExpressionEncoder) OperationOutcome(org.hl7.fhir.r4.model.OperationOutcome) ResourceParam(ca.uhn.fhir.rest.annotation.ResourceParam) List(java.util.List) Slf4j(lombok.extern.slf4j.Slf4j) Component(org.springframework.stereotype.Component) Database(au.csiro.pathling.io.Database) IssueType(org.hl7.fhir.r4.model.OperationOutcome.IssueType) Optional(java.util.Optional) Parameters(org.hl7.fhir.r4.model.Parameters) CodeType(org.hl7.fhir.r4.model.CodeType) Preconditions.checkUserInput(au.csiro.pathling.utilities.Preconditions.checkUserInput) FilterFunction(org.apache.spark.api.java.function.FilterFunction) InvalidUserInputError(au.csiro.pathling.errors.InvalidUserInputError) UrlType(org.hl7.fhir.r4.model.UrlType) SecurityError(au.csiro.pathling.errors.SecurityError) Nonnull(javax.annotation.Nonnull)

Aggregations

FhirEncoders (au.csiro.pathling.encoders.FhirEncoders)1 UnsupportedResourceError (au.csiro.pathling.encoders.UnsupportedResourceError)1 InvalidUserInputError (au.csiro.pathling.errors.InvalidUserInputError)1 SecurityError (au.csiro.pathling.errors.SecurityError)1 FhirContextFactory (au.csiro.pathling.fhir.FhirContextFactory)1 AccessRules (au.csiro.pathling.io.AccessRules)1 Database (au.csiro.pathling.io.Database)1 PersistenceScheme (au.csiro.pathling.io.PersistenceScheme)1 Preconditions.checkUserInput (au.csiro.pathling.utilities.Preconditions.checkUserInput)1 ResourceParam (ca.uhn.fhir.rest.annotation.ResourceParam)1 URLDecoder (java.net.URLDecoder)1 StandardCharsets (java.nio.charset.StandardCharsets)1 List (java.util.List)1 Optional (java.util.Optional)1 Collectors (java.util.stream.Collectors)1 Nonnull (javax.annotation.Nonnull)1 Nullable (javax.annotation.Nullable)1 Getter (lombok.Getter)1 Slf4j (lombok.extern.slf4j.Slf4j)1 FilterFunction (org.apache.spark.api.java.function.FilterFunction)1