Examples with HiveSource org.apache.gobblin.data.management.conversion.hive.source.HiveSource used on opensource projects

Example 1 with HiveSource

use of org.apache.gobblin.data.management.conversion.hive.source.HiveSource in project incubator-gobblin by apache.

the class HiveSourceTest method setup.

@BeforeClass
public void setup() throws Exception {
    this.hiveMetastoreTestUtils = LocalHiveMetastoreTestUtils.getInstance();
    this.hiveSource = new HiveSource();
}

Example 2 with HiveSource

use of org.apache.gobblin.data.management.conversion.hive.source.HiveSource in project incubator-gobblin by apache.

the class HiveSourceTest method testShouldCreateWorkunitsNewerThanLookback.

@Test
public void testShouldCreateWorkunitsNewerThanLookback() throws Exception {
    long currentTime = System.currentTimeMillis();
    // Default lookback time is 3 days
    long partitionCreateTime = new DateTime(currentTime).minusDays(2).getMillis();
    org.apache.hadoop.hive.ql.metadata.Partition partition = this.hiveMetastoreTestUtils.createDummyPartition(partitionCreateTime);
    SourceState testState = getTestState("testDb7");
    HiveSource source = new HiveSource();
    source.initialize(testState);
    boolean isOlderThanLookback = source.isOlderThanLookback(partition);
    Assert.assertEquals(isOlderThanLookback, false, "Should create workunits newer than lookback");
}

Example 3 with HiveSource

use of org.apache.gobblin.data.management.conversion.hive.source.HiveSource in project incubator-gobblin by apache.

the class HiveSourceTest method testShouldCreateWorkunitsOlderThanLookback.

@Test
public void testShouldCreateWorkunitsOlderThanLookback() throws Exception {
    long currentTime = System.currentTimeMillis();
    long partitionCreateTime = new DateTime(currentTime).minusDays(35).getMillis();
    org.apache.hadoop.hive.ql.metadata.Partition partition = this.hiveMetastoreTestUtils.createDummyPartition(partitionCreateTime);
    SourceState testState = getTestState("testDb6");
    HiveSource source = new HiveSource();
    source.initialize(testState);
    boolean isOlderThanLookback = source.isOlderThanLookback(partition);
    Assert.assertEquals(isOlderThanLookback, true, "Should not create workunits older than lookback");
}

Example 4 with HiveSource

use of org.apache.gobblin.data.management.conversion.hive.source.HiveSource in project incubator-gobblin by apache.

the class HiveSourceTest method testIsOlderThanLookbackForDistcpGenerationTime.

@Test
public void testIsOlderThanLookbackForDistcpGenerationTime() throws Exception {
    long currentTime = System.currentTimeMillis();
    // Default lookback time is 3 days
    long partitionCreateTime = new DateTime(currentTime).minusDays(2).getMillis();
    Map<String, String> parameters = Maps.newHashMap();
    parameters.put(HiveSource.DISTCP_REGISTRATION_GENERATION_TIME_KEY, partitionCreateTime + "");
    org.apache.hadoop.hive.ql.metadata.Partition partition = this.hiveMetastoreTestUtils.createDummyPartition(0);
    partition.getTPartition().setParameters(parameters);
    SourceState testState = getTestState("testDb6");
    HiveSource source = new HiveSource();
    source.initialize(testState);
    boolean isOlderThanLookback = source.isOlderThanLookback(partition);
    Assert.assertEquals(isOlderThanLookback, false, "Should create workunits newer than lookback");
}