Examples with GlobalMetadata org.apache.gobblin.metadata.types.GlobalMetadata used on opensource projects

Example 11 with GlobalMetadata

use of org.apache.gobblin.metadata.types.GlobalMetadata in project incubator-gobblin by apache.

the class RecordWithMetadataToEnvelopedRecordWithMetadata method updateRecordMetadata.

private void updateRecordMetadata(RecordWithMetadata<?> inputRecord) {
    GlobalMetadata md = inputRecord.getMetadata().getGlobalMetadata();
    String origContentType = md.getContentType();
    if (origContentType != null) {
        md.setInnerContentType(origContentType);
    }
    md.setContentType(CONTENT_TYPE);
    md.markImmutable();
}

Example 12 with GlobalMetadata

use of org.apache.gobblin.metadata.types.GlobalMetadata in project incubator-gobblin by apache.

the class BaseDataPublisherTest method checkMetadata.

private void checkMetadata(File file, int expectedNumFiles, int expectedNumRecords, FsWriterMetrics.FileInfo... expectedFileInfo) throws IOException {
    Assert.assertTrue(file.exists(), "Expected file " + file.getAbsolutePath() + " to exist");
    String contents = IOUtils.toString(new FileInputStream(file), StandardCharsets.UTF_8);
    GlobalMetadata metadata = GlobalMetadata.fromJson(contents);
    Assert.assertEquals(metadata.getNumFiles(), expectedNumFiles, "# of files do not match");
    Assert.assertEquals(metadata.getNumRecords(), expectedNumRecords, "# of records do not match");
    for (FsWriterMetrics.FileInfo fileInfo : expectedFileInfo) {
        long recordsInMetadata = ((Number) metadata.getFileMetadata(fileInfo.getFileName(), GlobalMetadata.NUM_RECORDS_KEY)).longValue();
        Assert.assertEquals(recordsInMetadata, fileInfo.getNumRecords(), "# of records in file-level metadata do not match");
    }
}

Example 13 with GlobalMetadata

use of org.apache.gobblin.metadata.types.GlobalMetadata in project incubator-gobblin by apache.

the class GlobalMetadataCollectorTest method handlesNullDefaults.

@Test
public void handlesNullDefaults() {
    GlobalMetadataCollector collectorWithNoDefaults = new GlobalMetadataCollector(-1);
    GlobalMetadata record = buildMetadataWithContentType(CONTENT_TYPE);
    GlobalMetadata newRecord = collectorWithNoDefaults.processMetadata(record);
    Assert.assertEquals(newRecord, record);
    Assert.assertEquals(collectorWithNoDefaults.getMetadataRecords().size(), 1);
}

Example 14 with GlobalMetadata

use of org.apache.gobblin.metadata.types.GlobalMetadata in project incubator-gobblin by apache.

the class GlobalMetadataCollectorTest method testDoesNotStoreRecordTwice.

@Test
public void testDoesNotStoreRecordTwice() {
    GlobalMetadata defaults = buildMetadataWithUrn("DEFAULT");
    GlobalMetadataCollector collectorWithDefaults = new GlobalMetadataCollector(defaults, -1);
    GlobalMetadata r1 = buildMetadataWithContentType(CONTENT_TYPE);
    GlobalMetadata newRecord = collectorWithDefaults.processMetadata(r1);
    Assert.assertNotNull(newRecord);
    GlobalMetadata r2 = buildMetadataWithContentType(CONTENT_TYPE);
    newRecord = collectorWithDefaults.processMetadata(r2);
    Assert.assertNull(newRecord);
    Assert.assertEquals(collectorWithDefaults.getMetadataRecords().size(), 1);
}

Example 15 with GlobalMetadata

use of org.apache.gobblin.metadata.types.GlobalMetadata in project incubator-gobblin by apache.

the class SimpleConfigMetadataProvider method getGlobalMetadataForDataset.

@Override
public GlobalMetadata getGlobalMetadataForDataset(String datasetUrn) {
    GlobalMetadata defaultMetadata = new GlobalMetadata();
    defaultMetadata.setDatasetUrn(datasetUrn);
    PermissionMetadataParser.setPermission(defaultMetadata, permission);
    return defaultMetadata;
}