Search in sources :

Example 6 with ClusterResolvers

use of org.apache.nifi.atlas.resolver.ClusterResolvers in project nifi by apache.

the class TestKafkaTopic method testPublishKafka.

@Test
public void testPublishKafka() {
    final String processorName = "PublishKafka";
    final String transitUri = "PLAINTEXT://0.example.com:6667/topicA";
    final ProvenanceEventRecord record = Mockito.mock(ProvenanceEventRecord.class);
    when(record.getComponentType()).thenReturn(processorName);
    when(record.getTransitUri()).thenReturn(transitUri);
    when(record.getEventType()).thenReturn(ProvenanceEventType.SEND);
    final ClusterResolvers clusterResolvers = Mockito.mock(ClusterResolvers.class);
    when(clusterResolvers.fromHostNames(matches(".+\\.example\\.com"))).thenReturn("cluster1");
    final AnalysisContext context = Mockito.mock(AnalysisContext.class);
    when(context.getClusterResolver()).thenReturn(clusterResolvers);
    final NiFiProvenanceEventAnalyzer analyzer = NiFiProvenanceEventAnalyzerFactory.getAnalyzer(processorName, transitUri, record.getEventType());
    assertNotNull(analyzer);
    final DataSetRefs refs = analyzer.analyze(context, record);
    assertEquals(0, refs.getInputs().size());
    assertEquals(1, refs.getOutputs().size());
    Referenceable ref = refs.getOutputs().iterator().next();
    assertEquals("topicA", ref.get(ATTR_NAME));
    assertEquals("topicA", ref.get("topic"));
    assertEquals("topicA@cluster1", ref.get(ATTR_QUALIFIED_NAME));
}
Also used : Referenceable(org.apache.atlas.typesystem.Referenceable) ClusterResolvers(org.apache.nifi.atlas.resolver.ClusterResolvers) ProvenanceEventRecord(org.apache.nifi.provenance.ProvenanceEventRecord) DataSetRefs(org.apache.nifi.atlas.provenance.DataSetRefs) AnalysisContext(org.apache.nifi.atlas.provenance.AnalysisContext) NiFiProvenanceEventAnalyzer(org.apache.nifi.atlas.provenance.NiFiProvenanceEventAnalyzer) Test(org.junit.Test)

Example 7 with ClusterResolvers

use of org.apache.nifi.atlas.resolver.ClusterResolvers in project nifi by apache.

the class TestKafkaTopic method testConsumeKafka.

@Test
public void testConsumeKafka() {
    final String processorName = "ConsumeKafka";
    final String transitUri = "PLAINTEXT://0.example.com:6667/topicA";
    final ProvenanceEventRecord record = Mockito.mock(ProvenanceEventRecord.class);
    when(record.getComponentType()).thenReturn(processorName);
    when(record.getTransitUri()).thenReturn(transitUri);
    when(record.getEventType()).thenReturn(ProvenanceEventType.RECEIVE);
    final ClusterResolvers clusterResolvers = Mockito.mock(ClusterResolvers.class);
    when(clusterResolvers.fromHostNames(matches(".+\\.example\\.com"))).thenReturn("cluster1");
    final AnalysisContext context = Mockito.mock(AnalysisContext.class);
    when(context.getClusterResolver()).thenReturn(clusterResolvers);
    final NiFiProvenanceEventAnalyzer analyzer = NiFiProvenanceEventAnalyzerFactory.getAnalyzer(processorName, transitUri, record.getEventType());
    assertNotNull(analyzer);
    final DataSetRefs refs = analyzer.analyze(context, record);
    assertEquals(1, refs.getInputs().size());
    assertEquals(0, refs.getOutputs().size());
    Referenceable ref = refs.getInputs().iterator().next();
    assertEquals("kafka_topic", ref.getTypeName());
    assertEquals("topicA", ref.get(ATTR_NAME));
    assertEquals("topicA", ref.get("topic"));
    assertEquals("topicA@cluster1", ref.get(ATTR_QUALIFIED_NAME));
}
Also used : Referenceable(org.apache.atlas.typesystem.Referenceable) ClusterResolvers(org.apache.nifi.atlas.resolver.ClusterResolvers) ProvenanceEventRecord(org.apache.nifi.provenance.ProvenanceEventRecord) DataSetRefs(org.apache.nifi.atlas.provenance.DataSetRefs) AnalysisContext(org.apache.nifi.atlas.provenance.AnalysisContext) NiFiProvenanceEventAnalyzer(org.apache.nifi.atlas.provenance.NiFiProvenanceEventAnalyzer) Test(org.junit.Test)

Example 8 with ClusterResolvers

use of org.apache.nifi.atlas.resolver.ClusterResolvers in project nifi by apache.

the class TestKafkaTopic method testConsumeKafkaRecord_0_10.

@Test
public void testConsumeKafkaRecord_0_10() {
    final String processorName = "ConsumeKafkaRecord_0_10";
    final String transitUri = "PLAINTEXT://0.example.com:6667/topicA";
    final ProvenanceEventRecord record = Mockito.mock(ProvenanceEventRecord.class);
    when(record.getComponentType()).thenReturn(processorName);
    when(record.getTransitUri()).thenReturn(transitUri);
    when(record.getEventType()).thenReturn(ProvenanceEventType.RECEIVE);
    final ClusterResolvers clusterResolvers = Mockito.mock(ClusterResolvers.class);
    when(clusterResolvers.fromHostNames(matches(".+\\.example\\.com"))).thenReturn("cluster1");
    final AnalysisContext context = Mockito.mock(AnalysisContext.class);
    when(context.getClusterResolver()).thenReturn(clusterResolvers);
    final NiFiProvenanceEventAnalyzer analyzer = NiFiProvenanceEventAnalyzerFactory.getAnalyzer(processorName, transitUri, record.getEventType());
    assertNotNull(analyzer);
    final DataSetRefs refs = analyzer.analyze(context, record);
    assertEquals(1, refs.getInputs().size());
    assertEquals(0, refs.getOutputs().size());
    Referenceable ref = refs.getInputs().iterator().next();
    assertEquals("kafka_topic", ref.getTypeName());
    assertEquals("topicA", ref.get(ATTR_NAME));
    assertEquals("topicA", ref.get("topic"));
    assertEquals("topicA@cluster1", ref.get(ATTR_QUALIFIED_NAME));
}
Also used : Referenceable(org.apache.atlas.typesystem.Referenceable) ClusterResolvers(org.apache.nifi.atlas.resolver.ClusterResolvers) ProvenanceEventRecord(org.apache.nifi.provenance.ProvenanceEventRecord) DataSetRefs(org.apache.nifi.atlas.provenance.DataSetRefs) AnalysisContext(org.apache.nifi.atlas.provenance.AnalysisContext) NiFiProvenanceEventAnalyzer(org.apache.nifi.atlas.provenance.NiFiProvenanceEventAnalyzer) Test(org.junit.Test)

Example 9 with ClusterResolvers

use of org.apache.nifi.atlas.resolver.ClusterResolvers in project nifi by apache.

the class TestNiFiRemotePort method testRemoteInputPort.

@Test
public void testRemoteInputPort() {
    final String componentType = "Remote Input Port";
    final String transitUri = "http://0.example.com:8080/nifi-api/data-transfer/input-ports/port-guid/transactions/tx-guid/flow-files";
    final ProvenanceEventRecord sendEvent = Mockito.mock(ProvenanceEventRecord.class);
    when(sendEvent.getEventId()).thenReturn(123L);
    when(sendEvent.getComponentId()).thenReturn("port-guid");
    when(sendEvent.getComponentType()).thenReturn(componentType);
    when(sendEvent.getTransitUri()).thenReturn(transitUri);
    when(sendEvent.getEventType()).thenReturn(ProvenanceEventType.SEND);
    final ClusterResolvers clusterResolvers = Mockito.mock(ClusterResolvers.class);
    when(clusterResolvers.fromHostNames(matches(".+\\.example\\.com"))).thenReturn("cluster1");
    final List<ConnectionStatus> connections = new ArrayList<>();
    final ConnectionStatus connection = new ConnectionStatus();
    connection.setDestinationId("port-guid");
    connection.setDestinationName("inputPortA");
    connections.add(connection);
    final AnalysisContext context = Mockito.mock(AnalysisContext.class);
    when(context.getClusterResolver()).thenReturn(clusterResolvers);
    when(context.findConnectionTo(matches("port-guid"))).thenReturn(connections);
    final NiFiProvenanceEventAnalyzer analyzer = NiFiProvenanceEventAnalyzerFactory.getAnalyzer(componentType, transitUri, sendEvent.getEventType());
    assertNotNull(analyzer);
    final DataSetRefs refs = analyzer.analyze(context, sendEvent);
    assertEquals(0, refs.getInputs().size());
    assertEquals(1, refs.getOutputs().size());
    assertEquals(1, refs.getComponentIds().size());
    // Should report connected componentId.
    assertTrue(refs.getComponentIds().contains("port-guid"));
    Referenceable ref = refs.getOutputs().iterator().next();
    assertEquals(TYPE_NIFI_INPUT_PORT, ref.getTypeName());
    assertEquals("inputPortA", ref.get(ATTR_NAME));
    assertEquals("port-guid@cluster1", ref.get(ATTR_QUALIFIED_NAME));
}
Also used : Referenceable(org.apache.atlas.typesystem.Referenceable) ClusterResolvers(org.apache.nifi.atlas.resolver.ClusterResolvers) ProvenanceEventRecord(org.apache.nifi.provenance.ProvenanceEventRecord) DataSetRefs(org.apache.nifi.atlas.provenance.DataSetRefs) ArrayList(java.util.ArrayList) AnalysisContext(org.apache.nifi.atlas.provenance.AnalysisContext) ConnectionStatus(org.apache.nifi.controller.status.ConnectionStatus) NiFiProvenanceEventAnalyzer(org.apache.nifi.atlas.provenance.NiFiProvenanceEventAnalyzer) Test(org.junit.Test)

Example 10 with ClusterResolvers

use of org.apache.nifi.atlas.resolver.ClusterResolvers in project nifi by apache.

the class ReportLineageToAtlas method initClusterResolvers.

private void initClusterResolvers(ConfigurationContext context) {
    final Set<ClusterResolver> loadedClusterResolvers = new LinkedHashSet<>();
    clusterResolverLoader.forEach(resolver -> {
        resolver.configure(context);
        loadedClusterResolvers.add(resolver);
    });
    clusterResolvers = new ClusterResolvers(Collections.unmodifiableSet(loadedClusterResolvers), defaultClusterName);
}
Also used : LinkedHashSet(java.util.LinkedHashSet) ClusterResolver(org.apache.nifi.atlas.resolver.ClusterResolver) RegexClusterResolver(org.apache.nifi.atlas.resolver.RegexClusterResolver) ClusterResolvers(org.apache.nifi.atlas.resolver.ClusterResolvers)

Aggregations

ClusterResolvers (org.apache.nifi.atlas.resolver.ClusterResolvers)19 AnalysisContext (org.apache.nifi.atlas.provenance.AnalysisContext)18 DataSetRefs (org.apache.nifi.atlas.provenance.DataSetRefs)18 NiFiProvenanceEventAnalyzer (org.apache.nifi.atlas.provenance.NiFiProvenanceEventAnalyzer)18 ProvenanceEventRecord (org.apache.nifi.provenance.ProvenanceEventRecord)18 Test (org.junit.Test)18 Referenceable (org.apache.atlas.typesystem.Referenceable)17 HashMap (java.util.HashMap)5 ArrayList (java.util.ArrayList)4 ConnectionStatus (org.apache.nifi.controller.status.ConnectionStatus)4 LinkedHashSet (java.util.LinkedHashSet)1 ClusterResolver (org.apache.nifi.atlas.resolver.ClusterResolver)1 RegexClusterResolver (org.apache.nifi.atlas.resolver.RegexClusterResolver)1