Examples with DummyDataset org.apache.gobblin.data.management.dataset.DummyDataset used on opensource projects

Example 1 with DummyDataset

use of org.apache.gobblin.data.management.dataset.DummyDataset in project incubator-gobblin by apache.

the class ConcurrentBoundedWorkUnitListTest method addFiles.

public boolean addFiles(ConcurrentBoundedWorkUnitList list, String fileSetName, int fileNumber) throws IOException {
    FileSet.Builder<CopyEntity> partitionBuilder = new FileSet.Builder<>(fileSetName, new DummyDataset(new Path("/path")));
    List<WorkUnit> workUnits = Lists.newArrayList();
    for (int i = 0; i < fileNumber; i++) {
        CopyEntity cf = createCopyableFile(i);
        partitionBuilder.add(cf);
        WorkUnit workUnit = new WorkUnit();
        CopySource.serializeCopyEntity(workUnit, cf);
        workUnits.add(workUnit);
    }
    return list.addFileSet(partitionBuilder.build(), workUnits);
}

Example 2 with DummyDataset

use of org.apache.gobblin.data.management.dataset.DummyDataset in project incubator-gobblin by apache.

the class FileSetTest method testPartitionBuilder.

@Test
public void testPartitionBuilder() throws Exception {
    String file1 = "file1";
    String file2 = "file2";
    FileSet<TestFile> fileSet = new FileSet.Builder<TestFile>("test", new DummyDataset(new Path("/path"))).add(new TestFile(createFileStatus(file1))).add(Lists.newArrayList(new TestFile(createFileStatus(file2)))).build();
    Assert.assertEquals(fileSet.getFiles().size(), 2);
    Assert.assertEquals(fileSet.getName(), "test");
    Assert.assertEquals(fileSet.getFiles().get(0).getFileStatus().getPath().toString(), file1);
    Assert.assertEquals(fileSet.getFiles().get(1).getFileStatus().getPath().toString(), file2);
    Assert.assertEquals(fileSet.getTotalEntities(), 2);
    Assert.assertEquals(fileSet.getTotalSizeInBytes(), 20);
}