Search in sources :

Example 1 with ChangelogPartitionManager

use of org.apache.samza.storage.ChangelogPartitionManager in project samza by apache.

the class SamzaTaskProxy method getJobModel.

/**
   * Retrieves the jobModel from the jobCoordinator.
   * @param jobInstance the job instance (jobId, jobName).
   * @return the JobModel fetched from the coordinator stream.
   */
protected JobModel getJobModel(JobInstance jobInstance) {
    CoordinatorStreamSystemConsumer coordinatorSystemConsumer = null;
    CoordinatorStreamSystemProducer coordinatorSystemProducer = null;
    try {
        CoordinatorStreamSystemFactory coordinatorStreamSystemFactory = new CoordinatorStreamSystemFactory();
        Config coordinatorSystemConfig = getCoordinatorSystemConfig(jobInstance);
        LOG.info("Using config: {} to create coordinatorStream producer and consumer.", coordinatorSystemConfig);
        coordinatorSystemConsumer = coordinatorStreamSystemFactory.getCoordinatorStreamSystemConsumer(coordinatorSystemConfig, METRICS_REGISTRY);
        coordinatorSystemProducer = coordinatorStreamSystemFactory.getCoordinatorStreamSystemProducer(coordinatorSystemConfig, METRICS_REGISTRY);
        LOG.info("Registering coordinator system stream consumer.");
        coordinatorSystemConsumer.register();
        LOG.debug("Starting coordinator system stream consumer.");
        coordinatorSystemConsumer.start();
        LOG.debug("Bootstrapping coordinator system stream consumer.");
        coordinatorSystemConsumer.bootstrap();
        LOG.info("Registering coordinator system stream producer.");
        coordinatorSystemProducer.register(SOURCE);
        Config config = coordinatorSystemConsumer.getConfig();
        LOG.info("Got config from coordinatorSystemConsumer: {}.", config);
        ChangelogPartitionManager changelogManager = new ChangelogPartitionManager(coordinatorSystemProducer, coordinatorSystemConsumer, SOURCE);
        changelogManager.start();
        LocalityManager localityManager = new LocalityManager(coordinatorSystemProducer, coordinatorSystemConsumer);
        localityManager.start();
        String jobCoordinatorSystemName = config.get(JobConfig.JOB_COORDINATOR_SYSTEM());
        /**
       * Select job coordinator system properties from config and instantiate SystemAdmin for it alone.
       * Instantiating SystemAdmin's for other input/output systems defined in config is unnecessary.
       */
        Config systemAdminConfig = config.subset(String.format("systems.%s", jobCoordinatorSystemName), false);
        scala.collection.immutable.Map<String, SystemAdmin> systemAdmins = JobModelManager.getSystemAdmins(systemAdminConfig);
        StreamMetadataCache streamMetadataCache = new StreamMetadataCache(systemAdmins, 0, SystemClock.instance());
        Map<TaskName, Integer> changeLogPartitionMapping = changelogManager.readChangeLogPartitionMapping();
        return JobModelManager.readJobModel(config, changeLogPartitionMapping, localityManager, streamMetadataCache, null);
    } finally {
        if (coordinatorSystemConsumer != null) {
            coordinatorSystemConsumer.stop();
        }
        if (coordinatorSystemProducer != null) {
            coordinatorSystemProducer.stop();
        }
    }
}
Also used : StreamMetadataCache(org.apache.samza.system.StreamMetadataCache) JobConfig(org.apache.samza.config.JobConfig) MapConfig(org.apache.samza.config.MapConfig) StorageConfig(org.apache.samza.config.StorageConfig) Config(org.apache.samza.config.Config) CoordinatorStreamSystemFactory(org.apache.samza.coordinator.stream.CoordinatorStreamSystemFactory) CoordinatorStreamSystemProducer(org.apache.samza.coordinator.stream.CoordinatorStreamSystemProducer) CoordinatorStreamSystemConsumer(org.apache.samza.coordinator.stream.CoordinatorStreamSystemConsumer) TaskName(org.apache.samza.container.TaskName) ChangelogPartitionManager(org.apache.samza.storage.ChangelogPartitionManager) SystemAdmin(org.apache.samza.system.SystemAdmin) LocalityManager(org.apache.samza.container.LocalityManager)

Aggregations

Config (org.apache.samza.config.Config)1 JobConfig (org.apache.samza.config.JobConfig)1 MapConfig (org.apache.samza.config.MapConfig)1 StorageConfig (org.apache.samza.config.StorageConfig)1 LocalityManager (org.apache.samza.container.LocalityManager)1 TaskName (org.apache.samza.container.TaskName)1 CoordinatorStreamSystemConsumer (org.apache.samza.coordinator.stream.CoordinatorStreamSystemConsumer)1 CoordinatorStreamSystemFactory (org.apache.samza.coordinator.stream.CoordinatorStreamSystemFactory)1 CoordinatorStreamSystemProducer (org.apache.samza.coordinator.stream.CoordinatorStreamSystemProducer)1 ChangelogPartitionManager (org.apache.samza.storage.ChangelogPartitionManager)1 StreamMetadataCache (org.apache.samza.system.StreamMetadataCache)1 SystemAdmin (org.apache.samza.system.SystemAdmin)1