Search in sources :

Example 1 with KafkaExporterExecutor

use of org.apache.rya.periodic.notification.exporter.KafkaExporterExecutor in project incubator-rya by apache.

the class PeriodicNotificationApplicationFactory method getPeriodicApplication.

/**
 * Create a PeriodicNotificationApplication.
 * @param conf - Configuration object that specifies the parameters needed to create the application
 * @return PeriodicNotificationApplication to periodically poll Rya Fluo for new results
 * @throws PeriodicApplicationException
 */
public static PeriodicNotificationApplication getPeriodicApplication(final PeriodicNotificationApplicationConfiguration conf) throws PeriodicApplicationException {
    final Properties kafkaConsumerProps = getKafkaConsumerProperties(conf);
    final Properties kafkaProducerProps = getKafkaProducerProperties(conf);
    final BlockingQueue<TimestampedNotification> notifications = new LinkedBlockingQueue<>();
    final BlockingQueue<NodeBin> bins = new LinkedBlockingQueue<>();
    final BlockingQueue<BindingSetRecord> bindingSets = new LinkedBlockingQueue<>();
    FluoClient fluo = null;
    try {
        final PeriodicQueryResultStorage storage = getPeriodicQueryResultStorage(conf);
        fluo = FluoClientFactory.getFluoClient(conf.getFluoAppName(), Optional.of(conf.getFluoTableName()), conf);
        final NotificationCoordinatorExecutor coordinator = getCoordinator(conf.getCoordinatorThreads(), notifications);
        addRegisteredNotices(coordinator, fluo.newSnapshot());
        final KafkaExporterExecutor exporter = getExporter(conf.getExporterThreads(), kafkaProducerProps, bindingSets);
        final PeriodicQueryPrunerExecutor pruner = getPruner(storage, fluo, conf.getPrunerThreads(), bins);
        final NotificationProcessorExecutor processor = getProcessor(storage, notifications, bins, bindingSets, conf.getProcessorThreads());
        final KafkaNotificationProvider provider = getProvider(conf.getProducerThreads(), conf.getNotificationTopic(), coordinator, kafkaConsumerProps);
        return PeriodicNotificationApplication.builder().setCoordinator(coordinator).setProvider(provider).setExporter(exporter).setProcessor(processor).setPruner(pruner).build();
    } catch (AccumuloException | AccumuloSecurityException e) {
        throw new PeriodicApplicationException(e.getMessage());
    }
}
Also used : KafkaNotificationProvider(org.apache.rya.periodic.notification.registration.kafka.KafkaNotificationProvider) AccumuloException(org.apache.accumulo.core.client.AccumuloException) FluoClient(org.apache.fluo.api.client.FluoClient) NodeBin(org.apache.rya.periodic.notification.api.NodeBin) NotificationCoordinatorExecutor(org.apache.rya.periodic.notification.api.NotificationCoordinatorExecutor) PeriodicNotificationCoordinatorExecutor(org.apache.rya.periodic.notification.coordinator.PeriodicNotificationCoordinatorExecutor) BindingSetRecord(org.apache.rya.periodic.notification.api.BindingSetRecord) NotificationProcessorExecutor(org.apache.rya.periodic.notification.processor.NotificationProcessorExecutor) TimestampedNotification(org.apache.rya.periodic.notification.notification.TimestampedNotification) Properties(java.util.Properties) LinkedBlockingQueue(java.util.concurrent.LinkedBlockingQueue) KafkaExporterExecutor(org.apache.rya.periodic.notification.exporter.KafkaExporterExecutor) PeriodicQueryPrunerExecutor(org.apache.rya.periodic.notification.pruner.PeriodicQueryPrunerExecutor) AccumuloSecurityException(org.apache.accumulo.core.client.AccumuloSecurityException) AccumuloPeriodicQueryResultStorage(org.apache.rya.indexing.pcj.storage.accumulo.AccumuloPeriodicQueryResultStorage) PeriodicQueryResultStorage(org.apache.rya.indexing.pcj.storage.PeriodicQueryResultStorage)

Aggregations

Properties (java.util.Properties)1 LinkedBlockingQueue (java.util.concurrent.LinkedBlockingQueue)1 AccumuloException (org.apache.accumulo.core.client.AccumuloException)1 AccumuloSecurityException (org.apache.accumulo.core.client.AccumuloSecurityException)1 FluoClient (org.apache.fluo.api.client.FluoClient)1 PeriodicQueryResultStorage (org.apache.rya.indexing.pcj.storage.PeriodicQueryResultStorage)1 AccumuloPeriodicQueryResultStorage (org.apache.rya.indexing.pcj.storage.accumulo.AccumuloPeriodicQueryResultStorage)1 BindingSetRecord (org.apache.rya.periodic.notification.api.BindingSetRecord)1 NodeBin (org.apache.rya.periodic.notification.api.NodeBin)1 NotificationCoordinatorExecutor (org.apache.rya.periodic.notification.api.NotificationCoordinatorExecutor)1 PeriodicNotificationCoordinatorExecutor (org.apache.rya.periodic.notification.coordinator.PeriodicNotificationCoordinatorExecutor)1 KafkaExporterExecutor (org.apache.rya.periodic.notification.exporter.KafkaExporterExecutor)1 TimestampedNotification (org.apache.rya.periodic.notification.notification.TimestampedNotification)1 NotificationProcessorExecutor (org.apache.rya.periodic.notification.processor.NotificationProcessorExecutor)1 PeriodicQueryPrunerExecutor (org.apache.rya.periodic.notification.pruner.PeriodicQueryPrunerExecutor)1 KafkaNotificationProvider (org.apache.rya.periodic.notification.registration.kafka.KafkaNotificationProvider)1