Search in sources :

Example 1 with LineReader

use of org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.LineReader in project beam by apache.

the class LocalFileSystemTest method testReadWithExistingFile.

@Test
public void testReadWithExistingFile() throws Exception {
    String expected = "my test string";
    File existingFile = temporaryFolder.newFile();
    Files.write(expected, existingFile, StandardCharsets.UTF_8);
    String data;
    try (Reader reader = Channels.newReader(localFileSystem.open(LocalResourceId.fromPath(existingFile.toPath(), false)), StandardCharsets.UTF_8.name())) {
        data = new LineReader(reader).readLine();
    }
    assertEquals(expected, data);
}
Also used : LineReader(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.LineReader) LineReader(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.LineReader) Reader(java.io.Reader) File(java.io.File) Test(org.junit.Test)

Example 2 with LineReader

use of org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.LineReader in project beam by apache.

the class PackageUtilTest method testPackageUploadWithFileSucceeds.

@Test
public void testPackageUploadWithFileSucceeds() throws Exception {
    Pipe pipe = Pipe.open();
    String contents = "This is a test!";
    File tmpFile = makeFileWithContents("file.txt", contents);
    when(mockGcsUtil.getObjects(anyListOf(GcsPath.class))).thenReturn(ImmutableList.of(StorageObjectOrIOException.create(new FileNotFoundException("some/path"))));
    when(mockGcsUtil.create(any(GcsPath.class), any(GcsUtil.CreateOptions.class))).thenReturn(pipe.sink());
    List<DataflowPackage> targets = defaultPackageUtil.stageClasspathElements(ImmutableList.of(makeStagedFile(tmpFile.getAbsolutePath())), STAGING_PATH, createOptions);
    DataflowPackage target = Iterables.getOnlyElement(targets);
    verify(mockGcsUtil).getObjects(anyListOf(GcsPath.class));
    verify(mockGcsUtil).create(any(GcsPath.class), any(GcsUtil.CreateOptions.class));
    verifyNoMoreInteractions(mockGcsUtil);
    assertThat(target.getName(), endsWith(".txt"));
    assertThat(target.getLocation(), equalTo(STAGING_PATH + target.getName()));
    assertThat(new LineReader(Channels.newReader(pipe.source(), StandardCharsets.UTF_8.name())).readLine(), equalTo(contents));
}
Also used : LineReader(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.LineReader) GcsPath(org.apache.beam.sdk.extensions.gcp.util.gcsfs.GcsPath) FileNotFoundException(java.io.FileNotFoundException) Pipe(java.nio.channels.Pipe) StagedFile(org.apache.beam.runners.dataflow.util.PackageUtil.StagedFile) File(java.io.File) StandardCreateOptions(org.apache.beam.sdk.io.fs.CreateOptions.StandardCreateOptions) CreateOptions(org.apache.beam.sdk.io.fs.CreateOptions) DataflowPackage(com.google.api.services.dataflow.model.DataflowPackage) Test(org.junit.Test)

Aggregations

File (java.io.File)2 LineReader (org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.LineReader)2 Test (org.junit.Test)2 DataflowPackage (com.google.api.services.dataflow.model.DataflowPackage)1 FileNotFoundException (java.io.FileNotFoundException)1 Reader (java.io.Reader)1 Pipe (java.nio.channels.Pipe)1 StagedFile (org.apache.beam.runners.dataflow.util.PackageUtil.StagedFile)1 GcsPath (org.apache.beam.sdk.extensions.gcp.util.gcsfs.GcsPath)1 CreateOptions (org.apache.beam.sdk.io.fs.CreateOptions)1 StandardCreateOptions (org.apache.beam.sdk.io.fs.CreateOptions.StandardCreateOptions)1