Examples with Metadata org.apache.beam.sdk.io.fs.MatchResult.Metadata used on opensource projects

Example 36 with Metadata

use of org.apache.beam.sdk.io.fs.MatchResult.Metadata in project beam by apache.

the class FileBasedSourceTest method testSplitAtFraction.

@Test
public void testSplitAtFraction() throws Exception {
    PipelineOptions options = PipelineOptionsFactory.create();
    File file = createFileWithData("file", createStringDataset(3, 100));
    Metadata metadata = FileSystems.matchSingleFileSpec(file.getPath());
    TestFileBasedSource source = new TestFileBasedSource(metadata, 1, 0, file.length(), null);
    // Shouldn't be able to split while unstarted.
    assertSplitAtFractionFails(source, 0, 0.7, options);
    assertSplitAtFractionSucceedsAndConsistent(source, 1, 0.7, options);
    assertSplitAtFractionSucceedsAndConsistent(source, 30, 0.7, options);
    assertSplitAtFractionFails(source, 0, 0.0, options);
    assertSplitAtFractionFails(source, 70, 0.3, options);
    assertSplitAtFractionFails(source, 100, 1.0, options);
    assertSplitAtFractionFails(source, 100, 0.99, options);
    assertSplitAtFractionSucceedsAndConsistent(source, 100, 0.995, options);
}

Example 37 with Metadata

use of org.apache.beam.sdk.io.fs.MatchResult.Metadata in project beam by apache.

the class FileBasedSourceTest method testToStringFile.

@Test
public void testToStringFile() throws Exception {
    File f = createFileWithData("foo", Collections.emptyList());
    Metadata metadata = FileSystems.matchSingleFileSpec(f.getPath());
    TestFileBasedSource source = new TestFileBasedSource(metadata, 1, 0, 10, null);
    assertEquals(String.format("%s range [0, 10)", f.getAbsolutePath()), source.toString());
}

Example 38 with Metadata

use of org.apache.beam.sdk.io.fs.MatchResult.Metadata in project beam by apache.

the class FileBasedSourceTest method testSplitAtFractionExhaustive.

@Test
public void testSplitAtFractionExhaustive() throws Exception {
    PipelineOptions options = PipelineOptionsFactory.create();
    // Smaller file for exhaustive testing.
    File file = createFileWithData("file", createStringDataset(3, 20));
    Metadata metadata = FileSystems.matchSingleFileSpec(file.getPath());
    TestFileBasedSource source = new TestFileBasedSource(metadata, 1, 0, file.length(), null);
    assertSplitAtFractionExhaustive(source, options);
}

Example 39 with Metadata

use of org.apache.beam.sdk.io.fs.MatchResult.Metadata in project beam by apache.

the class MetadataCoderV2Test method testEncodeDecodeWithCustomLastModifiedMills.

@Test
public void testEncodeDecodeWithCustomLastModifiedMills() throws Exception {
    Path filePath = tmpFolder.newFile("somefile").toPath();
    Metadata metadata = Metadata.builder().setResourceId(FileSystems.matchNewResource(filePath.toString(), false)).setIsReadSeekEfficient(true).setSizeBytes(1024).setLastModifiedMillis(1541097000L).build();
    CoderProperties.coderDecodeEncodeEqual(MetadataCoderV2.of(), metadata);
}

Example 40 with Metadata

use of org.apache.beam.sdk.io.fs.MatchResult.Metadata in project beam by apache.

the class MetadataCoderV2Test method testEncodeDecodeWithDefaultLastModifiedMills.

@Test
public void testEncodeDecodeWithDefaultLastModifiedMills() throws Exception {
    Path filePath = tmpFolder.newFile("somefile").toPath();
    Metadata metadata = Metadata.builder().setResourceId(FileSystems.matchNewResource(filePath.toString(), false)).setIsReadSeekEfficient(true).setSizeBytes(1024).build();
    CoderProperties.coderDecodeEncodeEqual(MetadataCoderV2.of(), metadata);
}