Search in sources :

Example 6 with HashCode

use of org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.HashCode in project beam by apache.

the class PackageUtilTest method makeStagedFile.

private static StagedFile makeStagedFile(String source, String destName) throws IOException {
    File file = new File(source);
    File sourceFile;
    HashCode hashCode;
    if (file.exists()) {
        sourceFile = file.isDirectory() ? zipDirectory(file) : file;
        hashCode = Files.asByteSource(sourceFile).hash(Hashing.sha256());
    } else {
        sourceFile = file;
        hashCode = Hashing.sha256().hashBytes(new byte[] {});
    }
    String destination = destName == null ? Environments.createStagingFileName(file, hashCode) : destName;
    return StagedFile.of(sourceFile.getPath(), hashCode.toString(), destination);
}
Also used : HashCode(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.HashCode) StagedFile(org.apache.beam.runners.dataflow.util.PackageUtil.StagedFile) File(java.io.File)

Example 7 with HashCode

use of org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.HashCode in project beam by apache.

the class GCSUploadMain method main.

public static void main(String[] args) {
    DataflowPipelineOptions options = PipelineOptionsFactory.fromArgs(args).as(DataflowPipelineOptions.class);
    FileSystems.setDefaultPipelineOptions(options);
    GcsStager stager = GcsStager.fromOptions(options);
    stager.stageFiles(options.getFilesToStage().stream().map((String source) -> {
        try {
            File file = new File(source);
            HashCode hashCode = Files.asByteSource(file).hash(Hashing.sha256());
            return PackageUtil.StagedFile.of(source, hashCode.toString(), Environments.createStagingFileName(file, hashCode));
        } catch (IOException e) {
            throw new UncheckedIOException(e);
        }
    }).collect(Collectors.toList()));
}
Also used : DataflowPipelineOptions(org.apache.beam.runners.dataflow.options.DataflowPipelineOptions) HashCode(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.HashCode) UncheckedIOException(java.io.UncheckedIOException) UncheckedIOException(java.io.UncheckedIOException) IOException(java.io.IOException) File(java.io.File)

Aggregations

HashCode (org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.HashCode)7 File (java.io.File)5 IOException (java.io.IOException)4 StagedFile (org.apache.beam.runners.dataflow.util.PackageUtil.StagedFile)3 Map (java.util.Map)2 RunnerApi (org.apache.beam.model.pipeline.v1.RunnerApi)2 StringUtils.byteArrayToJsonString (org.apache.beam.sdk.util.StringUtils.byteArrayToJsonString)2 InvalidProtocolBufferException (org.apache.beam.vendor.grpc.v1p43p2.com.google.protobuf.InvalidProtocolBufferException)2 ImmutableList (org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList)2 ImmutableMap (org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap)2 TableCell (com.google.api.services.bigquery.model.TableCell)1 TableRow (com.google.api.services.bigquery.model.TableRow)1 UncheckedIOException (java.io.UncheckedIOException)1 HashSet (java.util.HashSet)1 LinkedHashSet (java.util.LinkedHashSet)1 ArtifactInformation (org.apache.beam.model.pipeline.v1.RunnerApi.ArtifactInformation)1 SdkComponents (org.apache.beam.runners.core.construction.SdkComponents)1 DataflowPipelineOptions (org.apache.beam.runners.dataflow.options.DataflowPipelineOptions)1 Pipeline (org.apache.beam.sdk.Pipeline)1 ByteString (org.apache.beam.vendor.grpc.v1p43p2.com.google.protobuf.ByteString)1