Search in sources :

Example 1 with MultiThreadMessagingContext

use of io.cdap.cdap.messaging.context.MultiThreadMessagingContext in project cdap by caskdata.

the class RuntimeClientServiceTest method testBasicRelay.

@Test
public void testBasicRelay() throws Exception {
    // Send some messages to multiple topics in the client side TMS, they should get replicated to the server side TMS.
    MessagingContext messagingContext = new MultiThreadMessagingContext(clientMessagingService);
    MessagePublisher messagePublisher = messagingContext.getDirectMessagePublisher();
    ProgramStateWriter programStateWriter = new MessagingProgramStateWriter(clientCConf, clientMessagingService);
    for (Map.Entry<String, String> entry : topicConfigs.entrySet()) {
        // the RuntimeClientService will decode it to watch for program termination
        if (entry.getKey().equals(Constants.AppFabric.PROGRAM_STATUS_EVENT_TOPIC)) {
            // Write a non-terminal state to test basic relaying
            programStateWriter.running(PROGRAM_RUN_ID, null);
        } else {
            messagePublisher.publish(NamespaceId.SYSTEM.getNamespace(), entry.getValue(), entry.getKey(), entry.getKey());
        }
    }
    MessagingContext serverMessagingContext = new MultiThreadMessagingContext(messagingService);
    for (Map.Entry<String, String> entry : topicConfigs.entrySet()) {
        if (entry.getKey().equals(Constants.AppFabric.PROGRAM_STATUS_EVENT_TOPIC)) {
            // Extract the program run status from the Notification
            Tasks.waitFor(Collections.singletonList(ProgramRunStatus.RUNNING), () -> fetchMessages(serverMessagingContext, entry.getValue(), 10, null).stream().map(Message::getPayloadAsString).map(s -> GSON.fromJson(s, Notification.class)).map(n -> n.getProperties().get(ProgramOptionConstants.PROGRAM_STATUS)).map(ProgramRunStatus::valueOf).collect(Collectors.toList()), 5, TimeUnit.SECONDS);
        } else {
            Tasks.waitFor(Arrays.asList(entry.getKey(), entry.getKey()), () -> fetchMessages(serverMessagingContext, entry.getValue(), 10, null).stream().map(Message::getPayloadAsString).collect(Collectors.toList()), 5, TimeUnit.SECONDS);
        }
    }
    // Writes a program terminate message to unblock stopping of the client service
    programStateWriter.completed(PROGRAM_RUN_ID);
}
Also used : MessagingServerRuntimeModule(io.cdap.cdap.messaging.guice.MessagingServerRuntimeModule) Arrays(java.util.Arrays) ConfigModule(io.cdap.cdap.common.guice.ConfigModule) NamespaceId(io.cdap.cdap.proto.id.NamespaceId) Notification(io.cdap.cdap.proto.Notification) Spliterators(java.util.Spliterators) TimeoutException(java.util.concurrent.TimeoutException) MessageFetcher(io.cdap.cdap.api.messaging.MessageFetcher) ProgramStateWriter(io.cdap.cdap.app.runtime.ProgramStateWriter) Gson(com.google.gson.Gson) RuntimeServerModule(io.cdap.cdap.app.guice.RuntimeServerModule) After(org.junit.After) Map(java.util.Map) ClassRule(org.junit.ClassRule) NoOpMetricsCollectionService(io.cdap.cdap.common.metrics.NoOpMetricsCollectionService) Tasks(io.cdap.cdap.common.utils.Tasks) MessagingService(io.cdap.cdap.messaging.MessagingService) ProgramRunStatus(io.cdap.cdap.proto.ProgramRunStatus) Collectors(java.util.stream.Collectors) StandardCharsets(java.nio.charset.StandardCharsets) List(java.util.List) MultiThreadMessagingContext(io.cdap.cdap.messaging.context.MultiThreadMessagingContext) Constants(io.cdap.cdap.common.conf.Constants) MessagingContext(io.cdap.cdap.api.messaging.MessagingContext) ProgramOptionConstants(io.cdap.cdap.internal.app.runtime.ProgramOptionConstants) ListenableFuture(com.google.common.util.concurrent.ListenableFuture) InMemoryDiscoveryService(org.apache.twill.discovery.InMemoryDiscoveryService) LocalLocationModule(io.cdap.cdap.common.guice.LocalLocationModule) MessagingProgramStateWriter(io.cdap.cdap.internal.app.program.MessagingProgramStateWriter) RemoteAuthenticatorModules(io.cdap.cdap.common.guice.RemoteAuthenticatorModules) ProgramRunId(io.cdap.cdap.proto.id.ProgramRunId) DiscoveryServiceClient(org.apache.twill.discovery.DiscoveryServiceClient) StreamSupport(java.util.stream.StreamSupport) Nullable(javax.annotation.Nullable) Before(org.junit.Before) DiscoveryService(org.apache.twill.discovery.DiscoveryService) Message(io.cdap.cdap.api.messaging.Message) RunIds(io.cdap.cdap.common.app.RunIds) Test(org.junit.Test) IOException(java.io.IOException) CloseableIterator(io.cdap.cdap.api.dataset.lib.CloseableIterator) TopicNotFoundException(io.cdap.cdap.api.messaging.TopicNotFoundException) Service(com.google.common.util.concurrent.Service) MetricsCollectionService(io.cdap.cdap.api.metrics.MetricsCollectionService) Injector(com.google.inject.Injector) TimeUnit(java.util.concurrent.TimeUnit) CConfiguration(io.cdap.cdap.common.conf.CConfiguration) MessagePublisher(io.cdap.cdap.api.messaging.MessagePublisher) Guice(com.google.inject.Guice) Assert(org.junit.Assert) Collections(java.util.Collections) AuthenticationContextModules(io.cdap.cdap.security.auth.context.AuthenticationContextModules) TemporaryFolder(org.junit.rules.TemporaryFolder) AbstractModule(com.google.inject.AbstractModule) ProgramRunStatus(io.cdap.cdap.proto.ProgramRunStatus) Message(io.cdap.cdap.api.messaging.Message) ProgramStateWriter(io.cdap.cdap.app.runtime.ProgramStateWriter) MessagingProgramStateWriter(io.cdap.cdap.internal.app.program.MessagingProgramStateWriter) MessagePublisher(io.cdap.cdap.api.messaging.MessagePublisher) MultiThreadMessagingContext(io.cdap.cdap.messaging.context.MultiThreadMessagingContext) MessagingContext(io.cdap.cdap.api.messaging.MessagingContext) MultiThreadMessagingContext(io.cdap.cdap.messaging.context.MultiThreadMessagingContext) MessagingProgramStateWriter(io.cdap.cdap.internal.app.program.MessagingProgramStateWriter) Map(java.util.Map) Notification(io.cdap.cdap.proto.Notification) Test(org.junit.Test)

Example 2 with MultiThreadMessagingContext

use of io.cdap.cdap.messaging.context.MultiThreadMessagingContext in project cdap by caskdata.

the class RuntimeClientServiceTest method testProgramTerminate.

/**
 * Test for {@link RuntimeClientService} that will terminate itself when seeing program completed message.
 */
@Test
public void testProgramTerminate() throws Exception {
    MessagingContext messagingContext = new MultiThreadMessagingContext(clientMessagingService);
    MessagePublisher messagePublisher = messagingContext.getDirectMessagePublisher();
    ProgramStateWriter programStateWriter = new MessagingProgramStateWriter(clientCConf, clientMessagingService);
    // Send a terminate program state first, wait for the service sees the state change,
    // then publish messages to other topics.
    programStateWriter.completed(PROGRAM_RUN_ID);
    Tasks.waitFor(true, () -> runtimeClientService.getProgramFinishTime() >= 0, 2, TimeUnit.SECONDS);
    for (Map.Entry<String, String> entry : topicConfigs.entrySet()) {
        // the RuntimeClientService will decode it to watch for program termination
        if (!entry.getKey().equals(Constants.AppFabric.PROGRAM_STATUS_EVENT_TOPIC)) {
            List<String> payloads = Arrays.asList(entry.getKey(), entry.getKey(), entry.getKey());
            messagePublisher.publish(NamespaceId.SYSTEM.getNamespace(), entry.getValue(), StandardCharsets.UTF_8, payloads.iterator());
        }
    }
    // The client service should get stopped by itself.
    Tasks.waitFor(Service.State.TERMINATED, () -> runtimeClientService.state(), clientCConf.getLong(Constants.RuntimeMonitor.GRACEFUL_SHUTDOWN_MS) + 2000, TimeUnit.MILLISECONDS);
    // All messages should be sent after the runtime client service stopped
    MessagingContext serverMessagingContext = new MultiThreadMessagingContext(messagingService);
    for (Map.Entry<String, String> entry : topicConfigs.entrySet()) {
        if (entry.getKey().equals(Constants.AppFabric.PROGRAM_STATUS_EVENT_TOPIC)) {
            // Extract the program run status from the Notification
            Tasks.waitFor(Collections.singletonList(ProgramRunStatus.COMPLETED), () -> fetchMessages(serverMessagingContext, entry.getValue(), 10, null).stream().map(Message::getPayloadAsString).map(s -> GSON.fromJson(s, Notification.class)).map(n -> n.getProperties().get(ProgramOptionConstants.PROGRAM_STATUS)).map(ProgramRunStatus::valueOf).collect(Collectors.toList()), 5, TimeUnit.SECONDS);
        } else {
            Tasks.waitFor(Arrays.asList(entry.getKey(), entry.getKey(), entry.getKey()), () -> fetchMessages(serverMessagingContext, entry.getValue(), 10, null).stream().map(Message::getPayloadAsString).collect(Collectors.toList()), 5, TimeUnit.SECONDS);
        }
    }
}
Also used : MessagingServerRuntimeModule(io.cdap.cdap.messaging.guice.MessagingServerRuntimeModule) Arrays(java.util.Arrays) ConfigModule(io.cdap.cdap.common.guice.ConfigModule) NamespaceId(io.cdap.cdap.proto.id.NamespaceId) Notification(io.cdap.cdap.proto.Notification) Spliterators(java.util.Spliterators) TimeoutException(java.util.concurrent.TimeoutException) MessageFetcher(io.cdap.cdap.api.messaging.MessageFetcher) ProgramStateWriter(io.cdap.cdap.app.runtime.ProgramStateWriter) Gson(com.google.gson.Gson) RuntimeServerModule(io.cdap.cdap.app.guice.RuntimeServerModule) After(org.junit.After) Map(java.util.Map) ClassRule(org.junit.ClassRule) NoOpMetricsCollectionService(io.cdap.cdap.common.metrics.NoOpMetricsCollectionService) Tasks(io.cdap.cdap.common.utils.Tasks) MessagingService(io.cdap.cdap.messaging.MessagingService) ProgramRunStatus(io.cdap.cdap.proto.ProgramRunStatus) Collectors(java.util.stream.Collectors) StandardCharsets(java.nio.charset.StandardCharsets) List(java.util.List) MultiThreadMessagingContext(io.cdap.cdap.messaging.context.MultiThreadMessagingContext) Constants(io.cdap.cdap.common.conf.Constants) MessagingContext(io.cdap.cdap.api.messaging.MessagingContext) ProgramOptionConstants(io.cdap.cdap.internal.app.runtime.ProgramOptionConstants) ListenableFuture(com.google.common.util.concurrent.ListenableFuture) InMemoryDiscoveryService(org.apache.twill.discovery.InMemoryDiscoveryService) LocalLocationModule(io.cdap.cdap.common.guice.LocalLocationModule) MessagingProgramStateWriter(io.cdap.cdap.internal.app.program.MessagingProgramStateWriter) RemoteAuthenticatorModules(io.cdap.cdap.common.guice.RemoteAuthenticatorModules) ProgramRunId(io.cdap.cdap.proto.id.ProgramRunId) DiscoveryServiceClient(org.apache.twill.discovery.DiscoveryServiceClient) StreamSupport(java.util.stream.StreamSupport) Nullable(javax.annotation.Nullable) Before(org.junit.Before) DiscoveryService(org.apache.twill.discovery.DiscoveryService) Message(io.cdap.cdap.api.messaging.Message) RunIds(io.cdap.cdap.common.app.RunIds) Test(org.junit.Test) IOException(java.io.IOException) CloseableIterator(io.cdap.cdap.api.dataset.lib.CloseableIterator) TopicNotFoundException(io.cdap.cdap.api.messaging.TopicNotFoundException) Service(com.google.common.util.concurrent.Service) MetricsCollectionService(io.cdap.cdap.api.metrics.MetricsCollectionService) Injector(com.google.inject.Injector) TimeUnit(java.util.concurrent.TimeUnit) CConfiguration(io.cdap.cdap.common.conf.CConfiguration) MessagePublisher(io.cdap.cdap.api.messaging.MessagePublisher) Guice(com.google.inject.Guice) Assert(org.junit.Assert) Collections(java.util.Collections) AuthenticationContextModules(io.cdap.cdap.security.auth.context.AuthenticationContextModules) TemporaryFolder(org.junit.rules.TemporaryFolder) AbstractModule(com.google.inject.AbstractModule) Message(io.cdap.cdap.api.messaging.Message) MessagePublisher(io.cdap.cdap.api.messaging.MessagePublisher) MultiThreadMessagingContext(io.cdap.cdap.messaging.context.MultiThreadMessagingContext) MessagingProgramStateWriter(io.cdap.cdap.internal.app.program.MessagingProgramStateWriter) Notification(io.cdap.cdap.proto.Notification) ProgramRunStatus(io.cdap.cdap.proto.ProgramRunStatus) ProgramStateWriter(io.cdap.cdap.app.runtime.ProgramStateWriter) MessagingProgramStateWriter(io.cdap.cdap.internal.app.program.MessagingProgramStateWriter) MultiThreadMessagingContext(io.cdap.cdap.messaging.context.MultiThreadMessagingContext) MessagingContext(io.cdap.cdap.api.messaging.MessagingContext) Map(java.util.Map) Test(org.junit.Test)

Example 3 with MultiThreadMessagingContext

use of io.cdap.cdap.messaging.context.MultiThreadMessagingContext in project cdap by caskdata.

the class MapReduceRunnerTestBase method getDataNotifications.

/**
 * Returns a list of {@link Notification} object fetched from the data event topic in TMS that was published
 * starting from the given time.
 */
protected List<Notification> getDataNotifications(long startTime) throws Exception {
    // Get data notifications from TMS
    List<Notification> notifications = new ArrayList<>();
    MessagingContext messagingContext = new MultiThreadMessagingContext(injector.getInstance(MessagingService.class));
    try (CloseableIterator<Message> messages = messagingContext.getMessageFetcher().fetch(NamespaceId.SYSTEM.getNamespace(), injector.getInstance(CConfiguration.class).get(Constants.Dataset.DATA_EVENT_TOPIC), 10, startTime)) {
        while (messages.hasNext()) {
            notifications.add(GSON.fromJson(new String(messages.next().getPayload(), StandardCharsets.UTF_8), Notification.class));
        }
    }
    return notifications;
}
Also used : Message(io.cdap.cdap.api.messaging.Message) ArrayList(java.util.ArrayList) MultiThreadMessagingContext(io.cdap.cdap.messaging.context.MultiThreadMessagingContext) MessagingContext(io.cdap.cdap.api.messaging.MessagingContext) MultiThreadMessagingContext(io.cdap.cdap.messaging.context.MultiThreadMessagingContext) Notification(io.cdap.cdap.proto.Notification) MessagingService(io.cdap.cdap.messaging.MessagingService)

Example 4 with MultiThreadMessagingContext

use of io.cdap.cdap.messaging.context.MultiThreadMessagingContext in project cdap by caskdata.

the class TetheringRuntimeJobManagerTest method setUp.

@BeforeClass
public static void setUp() throws IOException, TopicAlreadyExistsException {
    CConfiguration cConf = CConfiguration.create();
    cConf.set(Constants.Tethering.TOPIC_PREFIX, "prefix-");
    cConf.set(Constants.CFG_LOCAL_DATA_DIR, TEMP_FOLDER.newFolder().getAbsolutePath());
    Injector injector = Guice.createInjector(new ConfigModule(cConf), new InMemoryDiscoveryModule(), new LocalLocationModule(), new MessagingServerRuntimeModule().getInMemoryModules(), new AbstractModule() {

        @Override
        protected void configure() {
            bind(MetricsCollectionService.class).to(NoOpMetricsCollectionService.class).in(Scopes.SINGLETON);
        }
    });
    messagingService = injector.getInstance(MessagingService.class);
    if (messagingService instanceof Service) {
        ((Service) messagingService).startAndWait();
    }
    TetheringConf conf = TetheringConf.fromProperties(PROPERTIES);
    topicId = new TopicId(NamespaceId.SYSTEM.getNamespace(), cConf.get(Constants.Tethering.TOPIC_PREFIX) + TETHERED_INSTANCE_NAME);
    messagingService.createTopic(new TopicMetadata(topicId, Collections.emptyMap()));
    messageFetcher = new MultiThreadMessagingContext(messagingService).getMessageFetcher();
    runtimeJobManager = new TetheringRuntimeJobManager(conf, cConf, messagingService);
}
Also used : InMemoryDiscoveryModule(io.cdap.cdap.common.guice.InMemoryDiscoveryModule) NoOpMetricsCollectionService(io.cdap.cdap.common.metrics.NoOpMetricsCollectionService) MetricsCollectionService(io.cdap.cdap.api.metrics.MetricsCollectionService) ConfigModule(io.cdap.cdap.common.guice.ConfigModule) NoOpMetricsCollectionService(io.cdap.cdap.common.metrics.NoOpMetricsCollectionService) MessagingService(io.cdap.cdap.messaging.MessagingService) Service(com.google.common.util.concurrent.Service) MetricsCollectionService(io.cdap.cdap.api.metrics.MetricsCollectionService) MessagingServerRuntimeModule(io.cdap.cdap.messaging.guice.MessagingServerRuntimeModule) MultiThreadMessagingContext(io.cdap.cdap.messaging.context.MultiThreadMessagingContext) CConfiguration(io.cdap.cdap.common.conf.CConfiguration) AbstractModule(com.google.inject.AbstractModule) MessagingService(io.cdap.cdap.messaging.MessagingService) TopicMetadata(io.cdap.cdap.messaging.TopicMetadata) LocalLocationModule(io.cdap.cdap.common.guice.LocalLocationModule) TetheringConf(io.cdap.cdap.internal.tethering.runtime.spi.provisioner.TetheringConf) Injector(com.google.inject.Injector) TopicId(io.cdap.cdap.proto.id.TopicId) BeforeClass(org.junit.BeforeClass)

Example 5 with MultiThreadMessagingContext

use of io.cdap.cdap.messaging.context.MultiThreadMessagingContext in project cdap by caskdata.

the class MetadataSubscriberServiceTest method testAppDeletionMessage.

@Test
public void testAppDeletionMessage() throws Exception {
    Injector injector = getInjector();
    // get the alert publisher
    CConfiguration cConf = injector.getInstance(CConfiguration.class);
    MessagingService messagingService = injector.getInstance(MessagingService.class);
    MultiThreadMessagingContext messagingContext = new MultiThreadMessagingContext(messagingService);
    AdminEventPublisher publisher = new AdminEventPublisher(cConf, messagingContext);
    // get the mds and put some workflow metadata in that, the publish of app deletion message should get the metadata
    // deleted
    MetadataStorage mds = injector.getInstance(MetadataStorage.class);
    // use an app with all program types to get all specification tested
    ApplicationId appId = NamespaceId.DEFAULT.app(AllProgramsApp.NAME);
    ProgramId workflowId = appId.workflow(AllProgramsApp.NoOpWorkflow.NAME);
    // generate an app spec from the application
    ApplicationSpecification appSpec = Specifications.from(new AllProgramsApp());
    // need to put metadata on workflow since we currently only set or delete workflow metadata
    mds.apply(new MetadataMutation.Update(workflowId.toMetadataEntity(), new Metadata(MetadataScope.SYSTEM, Collections.singletonMap("profile", ProfileId.NATIVE.getScopedName()))), MutationOptions.DEFAULT);
    Assert.assertEquals(ProfileId.NATIVE.getScopedName(), getProfileProperty(mds, workflowId));
    // publish app deletion message
    publisher.publishAppDeletion(appId, appSpec);
    // Verify the workflow profile metadata is removed because of the publish app deletion message
    Tasks.waitFor(true, () -> mds.read(new Read(workflowId.toMetadataEntity())).isEmpty(), 10, TimeUnit.SECONDS, 100, TimeUnit.MILLISECONDS);
}
Also used : ApplicationSpecification(io.cdap.cdap.api.app.ApplicationSpecification) AdminEventPublisher(io.cdap.cdap.internal.profile.AdminEventPublisher) Metadata(io.cdap.cdap.spi.metadata.Metadata) MultiThreadMessagingContext(io.cdap.cdap.messaging.context.MultiThreadMessagingContext) AllProgramsApp(io.cdap.cdap.AllProgramsApp) ProgramId(io.cdap.cdap.proto.id.ProgramId) CConfiguration(io.cdap.cdap.common.conf.CConfiguration) MessagingService(io.cdap.cdap.messaging.MessagingService) CoreMessagingService(io.cdap.cdap.messaging.service.CoreMessagingService) Read(io.cdap.cdap.spi.metadata.Read) MetadataMutation(io.cdap.cdap.spi.metadata.MetadataMutation) Injector(com.google.inject.Injector) MetadataStorage(io.cdap.cdap.spi.metadata.MetadataStorage) ApplicationId(io.cdap.cdap.proto.id.ApplicationId) Test(org.junit.Test)

Aggregations

MultiThreadMessagingContext (io.cdap.cdap.messaging.context.MultiThreadMessagingContext)8 MessagingService (io.cdap.cdap.messaging.MessagingService)7 CConfiguration (io.cdap.cdap.common.conf.CConfiguration)6 Service (com.google.common.util.concurrent.Service)4 AbstractModule (com.google.inject.AbstractModule)4 Injector (com.google.inject.Injector)4 Message (io.cdap.cdap.api.messaging.Message)4 MessagingContext (io.cdap.cdap.api.messaging.MessagingContext)4 MetricsCollectionService (io.cdap.cdap.api.metrics.MetricsCollectionService)4 ConfigModule (io.cdap.cdap.common.guice.ConfigModule)4 LocalLocationModule (io.cdap.cdap.common.guice.LocalLocationModule)4 NoOpMetricsCollectionService (io.cdap.cdap.common.metrics.NoOpMetricsCollectionService)3 MessagingServerRuntimeModule (io.cdap.cdap.messaging.guice.MessagingServerRuntimeModule)3 Notification (io.cdap.cdap.proto.Notification)3 Test (org.junit.Test)3 ListenableFuture (com.google.common.util.concurrent.ListenableFuture)2 Gson (com.google.gson.Gson)2 Guice (com.google.inject.Guice)2 CloseableIterator (io.cdap.cdap.api.dataset.lib.CloseableIterator)2 MessageFetcher (io.cdap.cdap.api.messaging.MessageFetcher)2