Search in sources :

Example 1 with RepositoryConfiguration

use of org.apache.nifi.provenance.RepositoryConfiguration in project nifi by apache.

the class TestEventIndexTask method testIndexWriterCommittedWhenAppropriate.

@Test(timeout = 5000)
public void testIndexWriterCommittedWhenAppropriate() throws IOException, InterruptedException {
    final BlockingQueue<StoredDocument> docQueue = new LinkedBlockingQueue<>();
    final RepositoryConfiguration repoConfig = new RepositoryConfiguration();
    final File storageDir = new File("target/storage/TestEventIndexTask/1");
    repoConfig.addStorageDirectory("1", storageDir);
    final AtomicInteger commitCount = new AtomicInteger(0);
    // Mock out an IndexWriter and keep track of the number of events that are indexed.
    final IndexWriter indexWriter = Mockito.mock(IndexWriter.class);
    final EventIndexWriter eventIndexWriter = new LuceneEventIndexWriter(indexWriter, storageDir);
    final IndexManager indexManager = Mockito.mock(IndexManager.class);
    Mockito.when(indexManager.borrowIndexWriter(Mockito.any(File.class))).thenReturn(eventIndexWriter);
    final IndexDirectoryManager directoryManager = new IndexDirectoryManager(repoConfig);
    // Create an EventIndexTask and override the commit(IndexWriter) method so that we can keep track of how
    // many times the index writer gets committed.
    final EventIndexTask task = new EventIndexTask(docQueue, repoConfig, indexManager, directoryManager, 201, EventReporter.NO_OP) {

        @Override
        protected void commit(EventIndexWriter indexWriter) throws IOException {
            commitCount.incrementAndGet();
        }
    };
    // Create 4 threads, each one a daemon thread running the EventIndexTask
    for (int i = 0; i < 4; i++) {
        final Thread t = new Thread(task);
        t.setDaemon(true);
        t.start();
    }
    assertEquals(0, commitCount.get());
    // Index 100 documents with a storage filename of "0.0.prov"
    for (int i = 0; i < 100; i++) {
        final Document document = new Document();
        document.add(new LongField(SearchableFields.EventTime.getSearchableFieldName(), System.currentTimeMillis(), Store.NO));
        final StorageSummary location = new StorageSummary(1L, "0.0.prov", "1", 0, 1000L, 1000L);
        final StoredDocument storedDoc = new StoredDocument(document, location);
        docQueue.add(storedDoc);
    }
    assertEquals(0, commitCount.get());
    // Index 100 documents
    for (int i = 0; i < 100; i++) {
        final Document document = new Document();
        document.add(new LongField(SearchableFields.EventTime.getSearchableFieldName(), System.currentTimeMillis(), Store.NO));
        final StorageSummary location = new StorageSummary(1L, "0.0.prov", "1", 0, 1000L, 1000L);
        final StoredDocument storedDoc = new StoredDocument(document, location);
        docQueue.add(storedDoc);
    }
    // Wait until we've indexed all 200 events
    while (eventIndexWriter.getEventsIndexed() < 200) {
        Thread.sleep(10L);
    }
    // Wait a bit and make sure that we still haven't committed the index writer.
    Thread.sleep(100L);
    assertEquals(0, commitCount.get());
    // Add another document.
    final Document document = new Document();
    document.add(new LongField(SearchableFields.EventTime.getSearchableFieldName(), System.currentTimeMillis(), Store.NO));
    final StorageSummary location = new StorageSummary(1L, "0.0.prov", "1", 0, 1000L, 1000L);
    StoredDocument storedDoc = new StoredDocument(document, location);
    docQueue.add(storedDoc);
    // Wait until index writer is committed.
    while (commitCount.get() == 0) {
        Thread.sleep(10L);
    }
    assertEquals(1, commitCount.get());
    // Add a new IndexableDocument with a count of 1 to ensure that the writer is committed again.
    storedDoc = new StoredDocument(document, location);
    docQueue.add(storedDoc);
    Thread.sleep(100L);
    assertEquals(1, commitCount.get());
    // Add a new IndexableDocument with a count of 3. Index writer should not be committed again.
    storedDoc = new StoredDocument(document, location);
    docQueue.add(storedDoc);
    Thread.sleep(100L);
    assertEquals(1, commitCount.get());
}
Also used : LuceneEventIndexWriter(org.apache.nifi.provenance.lucene.LuceneEventIndexWriter) LinkedBlockingQueue(java.util.concurrent.LinkedBlockingQueue) Document(org.apache.lucene.document.Document) IndexManager(org.apache.nifi.provenance.lucene.IndexManager) LongField(org.apache.lucene.document.LongField) StorageSummary(org.apache.nifi.provenance.serialization.StorageSummary) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) LuceneEventIndexWriter(org.apache.nifi.provenance.lucene.LuceneEventIndexWriter) IndexWriter(org.apache.lucene.index.IndexWriter) EventIndexWriter(org.apache.nifi.provenance.index.EventIndexWriter) LuceneEventIndexWriter(org.apache.nifi.provenance.lucene.LuceneEventIndexWriter) EventIndexWriter(org.apache.nifi.provenance.index.EventIndexWriter) RepositoryConfiguration(org.apache.nifi.provenance.RepositoryConfiguration) File(java.io.File) Test(org.junit.Test)

Example 2 with RepositoryConfiguration

use of org.apache.nifi.provenance.RepositoryConfiguration in project nifi by apache.

the class TestIndexDirectoryManager method testActiveIndexNotLostWhenSizeExceeded.

@Test
public void testActiveIndexNotLostWhenSizeExceeded() throws IOException, InterruptedException {
    final RepositoryConfiguration config = createConfig(2);
    config.setDesiredIndexSize(4096 * 128);
    final File storageDir1 = config.getStorageDirectories().get("1");
    final File storageDir2 = config.getStorageDirectories().get("2");
    final File index1 = new File(storageDir1, "index-1");
    final File index2 = new File(storageDir1, "index-2");
    final File index3 = new File(storageDir2, "index-3");
    final File index4 = new File(storageDir2, "index-4");
    final File[] allIndices = new File[] { index1, index2, index3, index4 };
    for (final File file : allIndices) {
        assertTrue(file.mkdirs() || file.exists());
    }
    try {
        final IndexDirectoryManager mgr = new IndexDirectoryManager(config);
        mgr.initialize();
        File indexDir = mgr.getWritableIndexingDirectory(System.currentTimeMillis(), "1");
        final File newFile = new File(indexDir, "1.bin");
        try (final OutputStream fos = new FileOutputStream(newFile)) {
            final byte[] data = new byte[4096];
            for (int i = 0; i < 1024; i++) {
                fos.write(data);
            }
        }
        try {
            final File newDir = mgr.getWritableIndexingDirectory(System.currentTimeMillis(), "1");
            assertEquals(indexDir, newDir);
        } finally {
            newFile.delete();
        }
    } finally {
        for (final File file : allIndices) {
            file.delete();
        }
    }
}
Also used : OutputStream(java.io.OutputStream) FileOutputStream(java.io.FileOutputStream) FileOutputStream(java.io.FileOutputStream) RepositoryConfiguration(org.apache.nifi.provenance.RepositoryConfiguration) File(java.io.File) Test(org.junit.Test)

Example 3 with RepositoryConfiguration

use of org.apache.nifi.provenance.RepositoryConfiguration in project nifi by apache.

the class TestIndexDirectoryManager method testGetDirectoriesBefore.

@Test
public void testGetDirectoriesBefore() throws InterruptedException {
    final RepositoryConfiguration config = createConfig(2);
    config.setDesiredIndexSize(4096 * 128);
    final File storageDir = config.getStorageDirectories().get("1");
    final File index1 = new File(storageDir, "index-1");
    final File index2 = new File(storageDir, "index-2");
    final File[] allIndices = new File[] { index1, index2 };
    for (final File file : allIndices) {
        if (file.exists()) {
            assertTrue(file.delete());
        }
    }
    assertTrue(index1.mkdirs());
    // Wait 1500 millis because some file systems use only second-precision timestamps instead of millisecond-precision timestamps and
    // we want to ensure that the two directories have different timestamps. Also using a value of 1500 instead of 1000 because sleep()
    // can awake before the given time so we give it a buffer zone.
    Thread.sleep(1500L);
    final long timestamp = System.currentTimeMillis();
    assertTrue(index2.mkdirs());
    try {
        final IndexDirectoryManager mgr = new IndexDirectoryManager(config);
        mgr.initialize();
        final List<File> dirsBefore = mgr.getDirectoriesBefore(timestamp);
        assertEquals(1, dirsBefore.size());
        assertEquals(index1, dirsBefore.get(0));
    } finally {
        for (final File file : allIndices) {
            file.delete();
        }
    }
}
Also used : RepositoryConfiguration(org.apache.nifi.provenance.RepositoryConfiguration) File(java.io.File) Test(org.junit.Test)

Example 4 with RepositoryConfiguration

use of org.apache.nifi.provenance.RepositoryConfiguration in project nifi by apache.

the class TestLuceneEventIndex method testUnauthorizedEventsGetFilteredForQuery.

@Test(timeout = 60000)
public void testUnauthorizedEventsGetFilteredForQuery() throws InterruptedException {
    assumeFalse(isWindowsEnvironment());
    final RepositoryConfiguration repoConfig = createConfig(1);
    repoConfig.setDesiredIndexSize(1L);
    final IndexManager indexManager = new SimpleIndexManager(repoConfig);
    final ArrayListEventStore eventStore = new ArrayListEventStore();
    final LuceneEventIndex index = new LuceneEventIndex(repoConfig, indexManager, 3, EventReporter.NO_OP);
    index.initialize(eventStore);
    for (int i = 0; i < 3; i++) {
        final ProvenanceEventRecord event = createEvent("1234");
        final StorageResult storageResult = eventStore.addEvent(event);
        index.addEvents(storageResult.getStorageLocations());
    }
    final Query query = new Query(UUID.randomUUID().toString());
    final EventAuthorizer authorizer = new EventAuthorizer() {

        @Override
        public boolean isAuthorized(ProvenanceEventRecord event) {
            return event.getEventId() % 2 == 0;
        }

        @Override
        public void authorize(ProvenanceEventRecord event) throws AccessDeniedException {
            throw new AccessDeniedException();
        }
    };
    List<ProvenanceEventRecord> events = Collections.emptyList();
    while (events.size() < 2) {
        final QuerySubmission submission = index.submitQuery(query, authorizer, "unit test");
        assertTrue(submission.getResult().awaitCompletion(5, TimeUnit.SECONDS));
        events = submission.getResult().getMatchingEvents();
        Thread.sleep(25L);
    }
    assertEquals(2, events.size());
}
Also used : AccessDeniedException(org.apache.nifi.authorization.AccessDeniedException) QuerySubmission(org.apache.nifi.provenance.search.QuerySubmission) Query(org.apache.nifi.provenance.search.Query) EventAuthorizer(org.apache.nifi.provenance.authorization.EventAuthorizer) SimpleIndexManager(org.apache.nifi.provenance.lucene.SimpleIndexManager) ArrayListEventStore(org.apache.nifi.provenance.store.ArrayListEventStore) SimpleIndexManager(org.apache.nifi.provenance.lucene.SimpleIndexManager) IndexManager(org.apache.nifi.provenance.lucene.IndexManager) ProvenanceEventRecord(org.apache.nifi.provenance.ProvenanceEventRecord) StandardProvenanceEventRecord(org.apache.nifi.provenance.StandardProvenanceEventRecord) RepositoryConfiguration(org.apache.nifi.provenance.RepositoryConfiguration) StorageResult(org.apache.nifi.provenance.store.StorageResult) Test(org.junit.Test)

Example 5 with RepositoryConfiguration

use of org.apache.nifi.provenance.RepositoryConfiguration in project nifi by apache.

the class TestLuceneEventIndex method testUnauthorizedEventsGetPlaceholdersForFindParents.

@Test(timeout = 60000)
public void testUnauthorizedEventsGetPlaceholdersForFindParents() throws InterruptedException {
    assumeFalse(isWindowsEnvironment());
    final RepositoryConfiguration repoConfig = createConfig(1);
    repoConfig.setDesiredIndexSize(1L);
    final IndexManager indexManager = new SimpleIndexManager(repoConfig);
    final ArrayListEventStore eventStore = new ArrayListEventStore();
    final LuceneEventIndex index = new LuceneEventIndex(repoConfig, indexManager, 3, EventReporter.NO_OP);
    index.initialize(eventStore);
    final ProvenanceEventRecord firstEvent = createEvent("4444");
    final Map<String, String> previousAttributes = new HashMap<>();
    previousAttributes.put("uuid", "4444");
    final Map<String, String> updatedAttributes = new HashMap<>();
    updatedAttributes.put("updated", "true");
    final ProvenanceEventRecord join = new StandardProvenanceEventRecord.Builder().setEventType(ProvenanceEventType.JOIN).setAttributes(previousAttributes, updatedAttributes).addParentUuid("4444").addChildFlowFile("1234").setComponentId("component-1").setComponentType("unit test").setEventId(idGenerator.getAndIncrement()).setEventTime(System.currentTimeMillis()).setFlowFileEntryDate(System.currentTimeMillis()).setFlowFileUUID("1234").setLineageStartDate(System.currentTimeMillis()).setCurrentContentClaim("container", "section", "unit-test-id", 0L, 1024L).build();
    index.addEvents(eventStore.addEvent(firstEvent).getStorageLocations());
    index.addEvents(eventStore.addEvent(join).getStorageLocations());
    for (int i = 0; i < 3; i++) {
        final ProvenanceEventRecord event = createEvent("1234");
        final StorageResult storageResult = eventStore.addEvent(event);
        index.addEvents(storageResult.getStorageLocations());
    }
    final NiFiUser user = createUser();
    final EventAuthorizer allowJoinEvents = new EventAuthorizer() {

        @Override
        public boolean isAuthorized(ProvenanceEventRecord event) {
            return event.getEventType() == ProvenanceEventType.JOIN;
        }

        @Override
        public void authorize(ProvenanceEventRecord event) throws AccessDeniedException {
        }
    };
    List<LineageNode> nodes = Collections.emptyList();
    while (nodes.size() < 2) {
        final ComputeLineageSubmission submission = index.submitExpandParents(1L, user, allowJoinEvents);
        assertTrue(submission.getResult().awaitCompletion(5, TimeUnit.SECONDS));
        nodes = submission.getResult().getNodes();
        Thread.sleep(25L);
    }
    assertEquals(2, nodes.size());
    final Map<ProvenanceEventType, List<LineageNode>> eventMap = nodes.stream().filter(n -> n.getNodeType() == LineageNodeType.PROVENANCE_EVENT_NODE).collect(Collectors.groupingBy(n -> ((ProvenanceEventLineageNode) n).getEventType()));
    assertEquals(2, eventMap.size());
    assertEquals(1, eventMap.get(ProvenanceEventType.JOIN).size());
    assertEquals(1, eventMap.get(ProvenanceEventType.UNKNOWN).size());
    assertEquals("4444", eventMap.get(ProvenanceEventType.UNKNOWN).get(0).getFlowFileUuid());
}
Also used : BeforeClass(org.junit.BeforeClass) Query(org.apache.nifi.provenance.search.Query) Assume.assumeFalse(org.junit.Assume.assumeFalse) AccessDeniedException(org.apache.nifi.authorization.AccessDeniedException) SearchableFields(org.apache.nifi.provenance.SearchableFields) HashMap(java.util.HashMap) ArrayList(java.util.ArrayList) Answer(org.mockito.stubbing.Answer) InvocationOnMock(org.mockito.invocation.InvocationOnMock) QueryResult(org.apache.nifi.provenance.search.QueryResult) StorageResult(org.apache.nifi.provenance.store.StorageResult) NiFiUser(org.apache.nifi.authorization.user.NiFiUser) ProvenanceEventRecord(org.apache.nifi.provenance.ProvenanceEventRecord) TestName(org.junit.rules.TestName) Map(java.util.Map) EventStore(org.apache.nifi.provenance.store.EventStore) LineageNode(org.apache.nifi.provenance.lineage.LineageNode) QuerySubmission(org.apache.nifi.provenance.search.QuerySubmission) StorageSummary(org.apache.nifi.provenance.serialization.StorageSummary) ProvenanceEventType(org.apache.nifi.provenance.ProvenanceEventType) Assert.assertNotNull(org.junit.Assert.assertNotNull) Set(java.util.Set) Assert.assertTrue(org.junit.Assert.assertTrue) Test(org.junit.Test) IOException(java.io.IOException) UUID(java.util.UUID) Collectors(java.util.stream.Collectors) StandardProvenanceEventRecord(org.apache.nifi.provenance.StandardProvenanceEventRecord) SimpleIndexManager(org.apache.nifi.provenance.lucene.SimpleIndexManager) File(java.io.File) TimeUnit(java.util.concurrent.TimeUnit) Mockito(org.mockito.Mockito) AtomicLong(java.util.concurrent.atomic.AtomicLong) List(java.util.List) Rule(org.junit.Rule) Assert.assertNull(org.junit.Assert.assertNull) EventReporter(org.apache.nifi.events.EventReporter) ComputeLineageSubmission(org.apache.nifi.provenance.lineage.ComputeLineageSubmission) RepositoryConfiguration(org.apache.nifi.provenance.RepositoryConfiguration) ProvenanceEventLineageNode(org.apache.nifi.provenance.lineage.ProvenanceEventLineageNode) IndexManager(org.apache.nifi.provenance.lucene.IndexManager) ArrayListEventStore(org.apache.nifi.provenance.store.ArrayListEventStore) Collections(java.util.Collections) EventAuthorizer(org.apache.nifi.provenance.authorization.EventAuthorizer) SearchTerms(org.apache.nifi.provenance.search.SearchTerms) Assert.assertEquals(org.junit.Assert.assertEquals) LineageNodeType(org.apache.nifi.provenance.lineage.LineageNodeType) NiFiUser(org.apache.nifi.authorization.user.NiFiUser) HashMap(java.util.HashMap) EventAuthorizer(org.apache.nifi.provenance.authorization.EventAuthorizer) ComputeLineageSubmission(org.apache.nifi.provenance.lineage.ComputeLineageSubmission) SimpleIndexManager(org.apache.nifi.provenance.lucene.SimpleIndexManager) ArrayListEventStore(org.apache.nifi.provenance.store.ArrayListEventStore) SimpleIndexManager(org.apache.nifi.provenance.lucene.SimpleIndexManager) IndexManager(org.apache.nifi.provenance.lucene.IndexManager) StandardProvenanceEventRecord(org.apache.nifi.provenance.StandardProvenanceEventRecord) ProvenanceEventLineageNode(org.apache.nifi.provenance.lineage.ProvenanceEventLineageNode) ProvenanceEventRecord(org.apache.nifi.provenance.ProvenanceEventRecord) StandardProvenanceEventRecord(org.apache.nifi.provenance.StandardProvenanceEventRecord) ArrayList(java.util.ArrayList) List(java.util.List) LineageNode(org.apache.nifi.provenance.lineage.LineageNode) ProvenanceEventLineageNode(org.apache.nifi.provenance.lineage.ProvenanceEventLineageNode) RepositoryConfiguration(org.apache.nifi.provenance.RepositoryConfiguration) ProvenanceEventType(org.apache.nifi.provenance.ProvenanceEventType) StorageResult(org.apache.nifi.provenance.store.StorageResult) Test(org.junit.Test)

Aggregations

RepositoryConfiguration (org.apache.nifi.provenance.RepositoryConfiguration)27 Test (org.junit.Test)24 ProvenanceEventRecord (org.apache.nifi.provenance.ProvenanceEventRecord)16 File (java.io.File)15 StandardProvenanceEventRecord (org.apache.nifi.provenance.StandardProvenanceEventRecord)15 IndexManager (org.apache.nifi.provenance.lucene.IndexManager)9 ArrayList (java.util.ArrayList)8 SimpleIndexManager (org.apache.nifi.provenance.lucene.SimpleIndexManager)8 ArrayListEventStore (org.apache.nifi.provenance.store.ArrayListEventStore)8 EventAuthorizer (org.apache.nifi.provenance.authorization.EventAuthorizer)6 StorageSummary (org.apache.nifi.provenance.serialization.StorageSummary)6 AtomicLong (java.util.concurrent.atomic.AtomicLong)5 AccessDeniedException (org.apache.nifi.authorization.AccessDeniedException)5 EventIndexWriter (org.apache.nifi.provenance.index.EventIndexWriter)5 Query (org.apache.nifi.provenance.search.Query)5 QuerySubmission (org.apache.nifi.provenance.search.QuerySubmission)5 StorageResult (org.apache.nifi.provenance.store.StorageResult)5 List (java.util.List)4 AtomicInteger (java.util.concurrent.atomic.AtomicInteger)4 InvocationOnMock (org.mockito.invocation.InvocationOnMock)4