Search in sources :

Example 1 with SimpleRecordField

use of org.apache.nifi.repository.schema.SimpleRecordField in project nifi by apache.

the class TestSchemaRecordReaderWriter method testFieldRemovedFromSchema.

@Test
public void testFieldRemovedFromSchema() throws IOException {
    final TocWriter tocWriter = new StandardTocWriter(tocFile, false, false);
    try {
        // Create a schema that has the fields modified
        final RecordSchema schemaV1 = ProvenanceEventSchema.PROVENANCE_EVENT_SCHEMA_V1;
        final List<RecordField> fields = new ArrayList<>(schemaV1.getFields());
        fields.remove(new SimpleRecordField(EventFieldNames.UPDATED_ATTRIBUTES, FieldType.STRING, Repetition.EXACTLY_ONE));
        fields.remove(new SimpleRecordField(EventFieldNames.PREVIOUS_ATTRIBUTES, FieldType.STRING, Repetition.EXACTLY_ONE));
        final RecordSchema recordSchema = new RecordSchema(fields);
        // Create a record writer whose schema does not contain updated attributes or previous attributes.
        // This means that we must also override the method that writes out attributes so that we are able
        // to avoid actually writing them out.
        final ByteArraySchemaRecordWriter writer = new ByteArraySchemaRecordWriter(journalFile, idGenerator, tocWriter, false, 0) {

            @Override
            public void writeHeader(long firstEventId, DataOutputStream out) throws IOException {
                final ByteArrayOutputStream baos = new ByteArrayOutputStream();
                recordSchema.writeTo(baos);
                out.writeInt(baos.size());
                baos.writeTo(out);
            }

            @Override
            protected Record createRecord(final ProvenanceEventRecord event, final long eventId) {
                final RecordSchema contentClaimSchema = new RecordSchema(recordSchema.getField(EventFieldNames.CONTENT_CLAIM).getSubFields());
                return new EventRecord(event, eventId, recordSchema, contentClaimSchema);
            }
        };
        try {
            writer.writeHeader(1L);
            writer.writeRecord(createEvent());
            writer.writeRecord(createEvent());
        } finally {
            writer.close();
        }
    } finally {
        tocWriter.close();
    }
    // Read the records in and make sure that they have the info that we expect.
    try (final InputStream in = new FileInputStream(journalFile);
        final TocReader tocReader = new StandardTocReader(tocFile);
        final RecordReader reader = createReader(in, journalFile.getName(), tocReader, 10000)) {
        for (int i = 0; i < 2; i++) {
            final StandardProvenanceEventRecord event = reader.nextRecord();
            assertNotNull(event);
            assertEquals(ProvenanceEventType.RECEIVE, event.getEventType());
            // We will still have a Map<String, String> for updated attributes because the
            // Provenance Event Builder will create an empty map.
            assertNotNull(event.getUpdatedAttributes());
            assertTrue(event.getUpdatedAttributes().isEmpty());
        }
    }
}
Also used : TocReader(org.apache.nifi.provenance.toc.TocReader) StandardTocReader(org.apache.nifi.provenance.toc.StandardTocReader) RecordField(org.apache.nifi.repository.schema.RecordField) SimpleRecordField(org.apache.nifi.repository.schema.SimpleRecordField) StandardTocReader(org.apache.nifi.provenance.toc.StandardTocReader) DataOutputStream(java.io.DataOutputStream) FileInputStream(java.io.FileInputStream) InputStream(java.io.InputStream) RecordReader(org.apache.nifi.provenance.serialization.RecordReader) ArrayList(java.util.ArrayList) ByteArrayOutputStream(java.io.ByteArrayOutputStream) FileInputStream(java.io.FileInputStream) StandardTocWriter(org.apache.nifi.provenance.toc.StandardTocWriter) SimpleRecordField(org.apache.nifi.repository.schema.SimpleRecordField) StandardTocWriter(org.apache.nifi.provenance.toc.StandardTocWriter) NopTocWriter(org.apache.nifi.provenance.toc.NopTocWriter) TocWriter(org.apache.nifi.provenance.toc.TocWriter) EventRecord(org.apache.nifi.provenance.schema.EventRecord) RecordSchema(org.apache.nifi.repository.schema.RecordSchema) Test(org.junit.Test)

Example 2 with SimpleRecordField

use of org.apache.nifi.repository.schema.SimpleRecordField in project nifi by apache.

the class TestSchemaRecordReaderWriter method testAddOneRecordReadTwice.

@Test
public void testAddOneRecordReadTwice() throws IOException {
    final RecordField unitTestField = new SimpleRecordField("Unit Test Field", FieldType.STRING, Repetition.EXACTLY_ONE);
    final Consumer<List<RecordField>> schemaModifier = fields -> fields.add(unitTestField);
    final Map<RecordField, Object> toAdd = new HashMap<>();
    toAdd.put(unitTestField, "hello");
    try (final ByteArraySchemaRecordWriter writer = createSchemaWriter(schemaModifier, toAdd)) {
        writer.writeHeader(1L);
        writer.writeRecord(createEvent());
    }
    try (final InputStream in = new FileInputStream(journalFile);
        final TocReader tocReader = new StandardTocReader(tocFile);
        final RecordReader reader = createReader(in, journalFile.getName(), tocReader, 10000)) {
        final ProvenanceEventRecord firstEvent = reader.nextRecord();
        assertNotNull(firstEvent);
        final ProvenanceEventRecord secondEvent = reader.nextRecord();
        assertNull(secondEvent);
    }
}
Also used : StandardTocWriter(org.apache.nifi.provenance.toc.StandardTocWriter) TocReader(org.apache.nifi.provenance.toc.TocReader) Record(org.apache.nifi.repository.schema.Record) ByteArrayOutputStream(java.io.ByteArrayOutputStream) HashMap(java.util.HashMap) Callable(java.util.concurrent.Callable) RecordWriter(org.apache.nifi.provenance.serialization.RecordWriter) ArrayList(java.util.ArrayList) NopTocWriter(org.apache.nifi.provenance.toc.NopTocWriter) RecordSchema(org.apache.nifi.repository.schema.RecordSchema) StandardTocReader(org.apache.nifi.provenance.toc.StandardTocReader) TocWriter(org.apache.nifi.provenance.toc.TocWriter) FieldType(org.apache.nifi.repository.schema.FieldType) DataOutputStream(java.io.DataOutputStream) Map(java.util.Map) TocUtil(org.apache.nifi.provenance.toc.TocUtil) Repetition(org.apache.nifi.repository.schema.Repetition) Before(org.junit.Before) OutputStream(java.io.OutputStream) NullOutputStream(org.apache.nifi.stream.io.NullOutputStream) Assert.assertNotNull(org.junit.Assert.assertNotNull) EventFieldNames(org.apache.nifi.provenance.schema.EventFieldNames) RecordField(org.apache.nifi.repository.schema.RecordField) Assert.assertTrue(org.junit.Assert.assertTrue) IOException(java.io.IOException) Test(org.junit.Test) FileInputStream(java.io.FileInputStream) UUID(java.util.UUID) File(java.io.File) FieldMapRecord(org.apache.nifi.repository.schema.FieldMapRecord) TimeUnit(java.util.concurrent.TimeUnit) Consumer(java.util.function.Consumer) AtomicLong(java.util.concurrent.atomic.AtomicLong) List(java.util.List) Assert.assertNull(org.junit.Assert.assertNull) EventRecord(org.apache.nifi.provenance.schema.EventRecord) Ignore(org.junit.Ignore) Assert.assertFalse(org.junit.Assert.assertFalse) ProvenanceEventSchema(org.apache.nifi.provenance.schema.ProvenanceEventSchema) RecordReader(org.apache.nifi.provenance.serialization.RecordReader) SimpleRecordField(org.apache.nifi.repository.schema.SimpleRecordField) Assert(org.junit.Assert) Assert.assertEquals(org.junit.Assert.assertEquals) InputStream(java.io.InputStream) TocReader(org.apache.nifi.provenance.toc.TocReader) StandardTocReader(org.apache.nifi.provenance.toc.StandardTocReader) RecordField(org.apache.nifi.repository.schema.RecordField) SimpleRecordField(org.apache.nifi.repository.schema.SimpleRecordField) HashMap(java.util.HashMap) StandardTocReader(org.apache.nifi.provenance.toc.StandardTocReader) FileInputStream(java.io.FileInputStream) InputStream(java.io.InputStream) RecordReader(org.apache.nifi.provenance.serialization.RecordReader) FileInputStream(java.io.FileInputStream) SimpleRecordField(org.apache.nifi.repository.schema.SimpleRecordField) ArrayList(java.util.ArrayList) List(java.util.List) Test(org.junit.Test)

Example 3 with SimpleRecordField

use of org.apache.nifi.repository.schema.SimpleRecordField in project nifi by apache.

the class EventIdFirstHeaderSchema method buildSchema.

private static RecordSchema buildSchema() {
    final List<RecordField> fields = new ArrayList<>();
    fields.add(new SimpleRecordField(FieldNames.FIRST_EVENT_ID, FieldType.LONG, Repetition.EXACTLY_ONE));
    fields.add(new SimpleRecordField(FieldNames.TIMESTAMP_OFFSET, FieldType.LONG, Repetition.EXACTLY_ONE));
    fields.add(new SimpleRecordField(FieldNames.COMPONENT_IDS, FieldType.STRING, Repetition.ZERO_OR_MORE));
    fields.add(new SimpleRecordField(FieldNames.COMPONENT_TYPES, FieldType.STRING, Repetition.ZERO_OR_MORE));
    fields.add(new SimpleRecordField(FieldNames.QUEUE_IDS, FieldType.STRING, Repetition.ZERO_OR_MORE));
    fields.add(new SimpleRecordField(FieldNames.EVENT_TYPES, FieldType.STRING, Repetition.ZERO_OR_MORE));
    return new RecordSchema(fields);
}
Also used : RecordField(org.apache.nifi.repository.schema.RecordField) SimpleRecordField(org.apache.nifi.repository.schema.SimpleRecordField) SimpleRecordField(org.apache.nifi.repository.schema.SimpleRecordField) ArrayList(java.util.ArrayList) RecordSchema(org.apache.nifi.repository.schema.RecordSchema)

Example 4 with SimpleRecordField

use of org.apache.nifi.repository.schema.SimpleRecordField in project nifi by apache.

the class SchemaSwapSerializer method serializeFlowFiles.

@Override
public void serializeFlowFiles(final List<FlowFileRecord> toSwap, final FlowFileQueue queue, final String swapLocation, final OutputStream out) throws IOException {
    schema.writeTo(out);
    long contentSize = 0L;
    long maxFlowFileId = -1L;
    final List<ResourceClaim> resourceClaims = new ArrayList<>();
    for (final FlowFileRecord flowFile : toSwap) {
        contentSize += flowFile.getSize();
        if (flowFile.getId() > maxFlowFileId) {
            maxFlowFileId = flowFile.getId();
        }
        final ContentClaim contentClaim = flowFile.getContentClaim();
        if (contentClaim != null) {
            resourceClaims.add(contentClaim.getResourceClaim());
        }
    }
    final QueueSize queueSize = new QueueSize(toSwap.size(), contentSize);
    final SwapSummary swapSummary = new StandardSwapSummary(queueSize, maxFlowFileId, resourceClaims);
    final Record summaryRecord = new SwapSummaryFieldMap(swapSummary, queue.getIdentifier(), SwapSchema.SWAP_SUMMARY_SCHEMA_V1);
    final List<Record> flowFileRecords = toSwap.stream().map(flowFile -> new FlowFileRecordFieldMap(flowFile, flowFileSchema)).collect(Collectors.toList());
    // Create a simple record to hold the summary and the flowfile contents
    final RecordField summaryField = new SimpleRecordField(SwapSchema.SWAP_SUMMARY, FieldType.COMPLEX, Repetition.EXACTLY_ONE);
    final RecordField contentsField = new ComplexRecordField(SwapSchema.FLOWFILE_CONTENTS, Repetition.ZERO_OR_MORE, FlowFileSchema.FLOWFILE_SCHEMA_V2.getFields());
    final List<RecordField> fields = new ArrayList<>(2);
    fields.add(summaryField);
    fields.add(contentsField);
    final Map<RecordField, Object> swapFileMap = new LinkedHashMap<>();
    swapFileMap.put(summaryField, summaryRecord);
    swapFileMap.put(contentsField, flowFileRecords);
    final Record swapFileRecord = new FieldMapRecord(swapFileMap, new RecordSchema(fields));
    final SchemaRecordWriter writer = new SchemaRecordWriter();
    writer.writeRecord(swapFileRecord, out);
    out.flush();
}
Also used : OutputStream(java.io.OutputStream) ContentClaim(org.apache.nifi.controller.repository.claim.ContentClaim) FlowFileSchema(org.apache.nifi.controller.repository.schema.FlowFileSchema) Record(org.apache.nifi.repository.schema.Record) RecordField(org.apache.nifi.repository.schema.RecordField) ComplexRecordField(org.apache.nifi.repository.schema.ComplexRecordField) FlowFileRecord(org.apache.nifi.controller.repository.FlowFileRecord) IOException(java.io.IOException) Collectors(java.util.stream.Collectors) QueueSize(org.apache.nifi.controller.queue.QueueSize) ArrayList(java.util.ArrayList) FieldMapRecord(org.apache.nifi.repository.schema.FieldMapRecord) RecordSchema(org.apache.nifi.repository.schema.RecordSchema) LinkedHashMap(java.util.LinkedHashMap) SwapSummary(org.apache.nifi.controller.repository.SwapSummary) FieldType(org.apache.nifi.repository.schema.FieldType) List(java.util.List) ResourceClaim(org.apache.nifi.controller.repository.claim.ResourceClaim) Map(java.util.Map) FlowFileRecordFieldMap(org.apache.nifi.controller.repository.schema.FlowFileRecordFieldMap) SimpleRecordField(org.apache.nifi.repository.schema.SimpleRecordField) Repetition(org.apache.nifi.repository.schema.Repetition) SchemaRecordWriter(org.apache.nifi.repository.schema.SchemaRecordWriter) FlowFileQueue(org.apache.nifi.controller.queue.FlowFileQueue) FieldMapRecord(org.apache.nifi.repository.schema.FieldMapRecord) RecordField(org.apache.nifi.repository.schema.RecordField) ComplexRecordField(org.apache.nifi.repository.schema.ComplexRecordField) SimpleRecordField(org.apache.nifi.repository.schema.SimpleRecordField) ArrayList(java.util.ArrayList) SwapSummary(org.apache.nifi.controller.repository.SwapSummary) LinkedHashMap(java.util.LinkedHashMap) QueueSize(org.apache.nifi.controller.queue.QueueSize) ContentClaim(org.apache.nifi.controller.repository.claim.ContentClaim) SimpleRecordField(org.apache.nifi.repository.schema.SimpleRecordField) FlowFileRecordFieldMap(org.apache.nifi.controller.repository.schema.FlowFileRecordFieldMap) ResourceClaim(org.apache.nifi.controller.repository.claim.ResourceClaim) Record(org.apache.nifi.repository.schema.Record) FlowFileRecord(org.apache.nifi.controller.repository.FlowFileRecord) FieldMapRecord(org.apache.nifi.repository.schema.FieldMapRecord) ComplexRecordField(org.apache.nifi.repository.schema.ComplexRecordField) FlowFileRecord(org.apache.nifi.controller.repository.FlowFileRecord) RecordSchema(org.apache.nifi.repository.schema.RecordSchema) SchemaRecordWriter(org.apache.nifi.repository.schema.SchemaRecordWriter)

Example 5 with SimpleRecordField

use of org.apache.nifi.repository.schema.SimpleRecordField in project nifi by apache.

the class SchemaRepositoryRecordSerde method swapOutRecord.

private RepositoryRecord swapOutRecord(final Record record) {
    final Long recordId = (Long) record.getFieldValue(RepositoryRecordSchema.RECORD_ID_FIELD);
    final String queueId = (String) record.getFieldValue(new SimpleRecordField(RepositoryRecordSchema.QUEUE_IDENTIFIER, FieldType.STRING, Repetition.EXACTLY_ONE));
    final String swapLocation = (String) record.getFieldValue(new SimpleRecordField(RepositoryRecordSchema.SWAP_LOCATION, FieldType.STRING, Repetition.EXACTLY_ONE));
    final FlowFileQueue queue = getFlowFileQueue(queueId);
    final FlowFileRecord flowFileRecord = new StandardFlowFileRecord.Builder().id(recordId).build();
    return new StandardRepositoryRecord(queue, flowFileRecord, swapLocation);
}
Also used : SimpleRecordField(org.apache.nifi.repository.schema.SimpleRecordField) FlowFileQueue(org.apache.nifi.controller.queue.FlowFileQueue)

Aggregations

SimpleRecordField (org.apache.nifi.repository.schema.SimpleRecordField)6 ArrayList (java.util.ArrayList)5 RecordField (org.apache.nifi.repository.schema.RecordField)5 RecordSchema (org.apache.nifi.repository.schema.RecordSchema)5 ByteArrayOutputStream (java.io.ByteArrayOutputStream)3 DataOutputStream (java.io.DataOutputStream)3 FileInputStream (java.io.FileInputStream)3 IOException (java.io.IOException)3 InputStream (java.io.InputStream)3 OutputStream (java.io.OutputStream)3 List (java.util.List)3 Map (java.util.Map)3 EventRecord (org.apache.nifi.provenance.schema.EventRecord)3 RecordReader (org.apache.nifi.provenance.serialization.RecordReader)3 NopTocWriter (org.apache.nifi.provenance.toc.NopTocWriter)3 StandardTocReader (org.apache.nifi.provenance.toc.StandardTocReader)3 StandardTocWriter (org.apache.nifi.provenance.toc.StandardTocWriter)3 TocReader (org.apache.nifi.provenance.toc.TocReader)3 TocWriter (org.apache.nifi.provenance.toc.TocWriter)3 FieldMapRecord (org.apache.nifi.repository.schema.FieldMapRecord)3