Search in sources :

Example 1 with GoogleDriveSource

use of org.apache.gobblin.source.extractor.extract.google.GoogleDriveSource in project incubator-gobblin by apache.

the class GoogleDriveSourceTest method testGetExtractor.

public void testGetExtractor() throws IOException {
    @SuppressWarnings("rawtypes") GoogleDriveSource source = new GoogleDriveSource<>();
    GoogleDriveFsHelper fsHelper = mock(GoogleDriveFsHelper.class);
    source.fsHelper = fsHelper;
    Extractor extractor = source.getExtractor(new WorkUnitState());
    Assert.assertTrue(extractor instanceof GoogleDriveExtractor);
}
Also used : GoogleDriveFsHelper(org.apache.gobblin.source.extractor.extract.google.GoogleDriveFsHelper) WorkUnitState(org.apache.gobblin.configuration.WorkUnitState) GoogleDriveExtractor(org.apache.gobblin.source.extractor.extract.google.GoogleDriveExtractor) GoogleDriveExtractor(org.apache.gobblin.source.extractor.extract.google.GoogleDriveExtractor) Extractor(org.apache.gobblin.source.extractor.Extractor) GoogleDriveSource(org.apache.gobblin.source.extractor.extract.google.GoogleDriveSource)

Example 2 with GoogleDriveSource

use of org.apache.gobblin.source.extractor.extract.google.GoogleDriveSource in project incubator-gobblin by apache.

the class GoogleDriveSourceTest method testGetcurrentFsSnapshot.

@SuppressWarnings("unchecked")
public void testGetcurrentFsSnapshot() throws FileBasedHelperException {
    @SuppressWarnings("rawtypes") GoogleDriveSource source = new GoogleDriveSource<>();
    GoogleDriveFsHelper fsHelper = mock(GoogleDriveFsHelper.class);
    source.fsHelper = fsHelper;
    List<String> fileIds = ImmutableList.of("test1", "test2", "test3");
    when(fsHelper.ls(anyString())).thenReturn(fileIds);
    long timestamp = System.currentTimeMillis();
    when(fsHelper.getFileMTime(anyString())).thenReturn(timestamp);
    List<String> expected = Lists.newArrayList();
    for (String fileId : fileIds) {
        expected.add(fileId + source.splitPattern + timestamp);
    }
    Assert.assertEquals(expected, source.getcurrentFsSnapshot(new State()));
}
Also used : GoogleDriveFsHelper(org.apache.gobblin.source.extractor.extract.google.GoogleDriveFsHelper) WorkUnitState(org.apache.gobblin.configuration.WorkUnitState) State(org.apache.gobblin.configuration.State) GoogleDriveSource(org.apache.gobblin.source.extractor.extract.google.GoogleDriveSource)

Aggregations

WorkUnitState (org.apache.gobblin.configuration.WorkUnitState)2 GoogleDriveFsHelper (org.apache.gobblin.source.extractor.extract.google.GoogleDriveFsHelper)2 GoogleDriveSource (org.apache.gobblin.source.extractor.extract.google.GoogleDriveSource)2 State (org.apache.gobblin.configuration.State)1 Extractor (org.apache.gobblin.source.extractor.Extractor)1 GoogleDriveExtractor (org.apache.gobblin.source.extractor.extract.google.GoogleDriveExtractor)1