Search in sources :

Example 1 with ResourceUpdatedMessage

use of org.atlasapi.messaging.ResourceUpdatedMessage in project atlas-deer by atlasapi.

the class AbstractContentStore method sendResourceUpdatedMessages.

private void sendResourceUpdatedMessages(Iterable<ResourceUpdatedMessage> messages) {
    Map<Id, Id> resourceGraphIds = getResourceGraphIds(messages);
    for (ResourceUpdatedMessage message : messages) {
        try {
            // Downstream workers are processing the entire equivalence graph for every
            // updated content. Therefore we want to use the graph ID as the partition key
            // to ensure each graph is only processed by a single worker thread.
            // We default to using the updated resource id as the partition key if we can't
            // resolve the resource's graph.
            Id resourceId = message.getUpdatedResource().getId();
            Id partitionId = resourceGraphIds.containsKey(resourceId) ? resourceGraphIds.get(resourceId) : resourceId;
            sender.sendMessage(message, Longs.toByteArray(partitionId.longValue()));
        } catch (Exception e) {
            log.error("Failed to send message " + message.getUpdatedResource().toString(), e);
        }
    }
}
Also used : ResourceUpdatedMessage(org.atlasapi.messaging.ResourceUpdatedMessage) Id(org.atlasapi.entity.Id) MissingResourceException(org.atlasapi.entity.util.MissingResourceException) RuntimeWriteException(org.atlasapi.entity.util.RuntimeWriteException) ExecutionException(java.util.concurrent.ExecutionException) WriteException(org.atlasapi.entity.util.WriteException)

Example 2 with ResourceUpdatedMessage

use of org.atlasapi.messaging.ResourceUpdatedMessage in project atlas-deer by atlasapi.

the class AbstractContentStore method createEntityUpdatedMessages.

private <C extends Content> Iterable<ResourceUpdatedMessage> createEntityUpdatedMessages(WriteResult<C, Content> result) {
    C writtenResource = result.getResource();
    ImmutableList.Builder<ResourceUpdatedMessage> messages = ImmutableList.builder();
    messages.add(new ResourceUpdatedMessage(UUID.randomUUID().toString(), Timestamp.of(result.getWriteTime()), writtenResource.toRef()));
    if (writtenResource instanceof Container && containerSummaryChanged((Container) writtenResource, result.getPrevious())) {
        for (ItemRef itemRef : ((Container) writtenResource).getItemRefs()) {
            messages.add(new ResourceUpdatedMessage(UUID.randomUUID().toString(), Timestamp.of(DateTime.now(DateTimeZone.UTC)), itemRef));
        }
    }
    if (writtenResource instanceof Series) {
        Series series = (Series) writtenResource;
        BrandRef brandRef = series.getBrandRef();
        if (brandRef != null) {
            messages.add(new ResourceUpdatedMessage(UUID.randomUUID().toString(), Timestamp.of(result.getWriteTime()), brandRef));
        }
    }
    if (writtenResource instanceof Item && ((Item) writtenResource).getContainerRef() != null) {
        messages.add(new ResourceUpdatedMessage(UUID.randomUUID().toString(), Timestamp.of(result.getWriteTime()), ((Item) writtenResource).getContainerRef()));
    }
    if (writtenResource instanceof Episode && ((Episode) writtenResource).getSeriesRef() != null) {
        messages.add(new ResourceUpdatedMessage(UUID.randomUUID().toString(), Timestamp.of(result.getWriteTime()), ((Episode) writtenResource).getSeriesRef()));
    }
    return messages.build();
}
Also used : ImmutableList(com.google.common.collect.ImmutableList) ResourceUpdatedMessage(org.atlasapi.messaging.ResourceUpdatedMessage)

Example 3 with ResourceUpdatedMessage

use of org.atlasapi.messaging.ResourceUpdatedMessage in project atlas-deer by atlasapi.

the class CassandraContentStoreIT method testWritingItemWritesRefIntoParent.

@Test
public void testWritingItemWritesRefIntoParent() throws Exception {
    when(clock.now()).thenReturn(new DateTime(DateTimeZones.UTC));
    when(idGenerator.generateRaw()).thenReturn(1234L).thenReturn(1235L);
    Brand brand = create(new Brand());
    store.writeContent(brand);
    Brand resolvedBrand = (Brand) resolve(1234L);
    assertThat(resolvedBrand.getItemRefs(), is(empty()));
    Item item = create(new Item());
    item.setContainer(resolvedBrand);
    store.writeContent(item);
    Item resolvedItem = (Item) resolve(1235L);
    assertThat(resolvedItem.getContainerRef().getId().longValue(), is(1234L));
    ArgumentCaptor<ResourceUpdatedMessage> captor = ArgumentCaptor.forClass(ResourceUpdatedMessage.class);
    verify(sender, times(3)).sendMessage(captor.capture(), any(byte[].class));
    List<ResourceUpdatedMessage> values = captor.getAllValues();
    Long numBrandUpdates = values.stream().filter(msg -> msg.getUpdatedResource().getId().longValue() == 1234L).collect(Collectors.counting());
    Long numItemUpdates = values.stream().filter(msg -> msg.getUpdatedResource().getId().longValue() == 1235L).collect(Collectors.counting());
    assertThat(numBrandUpdates, is(2L));
    assertThat(numItemUpdates, is(1L));
}
Also used : StringSerializer(com.netflix.astyanax.serializers.StringSerializer) DateTimeZone(org.joda.time.DateTimeZone) Matchers.isA(org.hamcrest.Matchers.isA) LongSerializer(com.netflix.astyanax.serializers.LongSerializer) Countries(com.metabroadcast.common.intl.Countries) Resolved(org.atlasapi.entity.util.Resolved) TimeoutException(java.util.concurrent.TimeoutException) Assert.assertThat(org.junit.Assert.assertThat) Logger(org.apache.log4j.Logger) Optional(com.google.common.base.Optional) Session(com.datastax.driver.core.Session) Level(org.apache.log4j.Level) After(org.junit.After) Map(java.util.Map) Matchers.nullValue(org.hamcrest.Matchers.nullValue) IdGenerator(com.metabroadcast.common.ids.IdGenerator) ColumnListMutation(com.netflix.astyanax.ColumnListMutation) Longs(com.google.common.primitives.Longs) ImmutableSet(com.google.common.collect.ImmutableSet) ImmutableMap(com.google.common.collect.ImmutableMap) Matchers.notNullValue(org.hamcrest.Matchers.notNullValue) Set(java.util.Set) UUID(java.util.UUID) Collectors(java.util.stream.Collectors) List(java.util.List) ColumnFamily(com.netflix.astyanax.model.ColumnFamily) MatcherAssert(org.hamcrest.MatcherAssert) Matchers.containsInAnyOrder(org.hamcrest.Matchers.containsInAnyOrder) Cluster(com.datastax.driver.core.Cluster) ConsoleAppender(org.apache.log4j.ConsoleAppender) Assert.assertFalse(org.junit.Assert.assertFalse) Matchers.is(org.hamcrest.Matchers.is) Alias(org.atlasapi.entity.Alias) ContentProtos(org.atlasapi.serialization.protobuf.ContentProtos) Iterables(com.google.common.collect.Iterables) ListenableFuture(com.google.common.util.concurrent.ListenableFuture) BeforeClass(org.junit.BeforeClass) Mock(org.mockito.Mock) Matchers(org.mockito.Matchers) Clock(com.metabroadcast.common.time.Clock) ImmutableOptionalMap(com.metabroadcast.common.collect.ImmutableOptionalMap) ContentHasher(org.atlasapi.hashing.content.ContentHasher) EquivalenceGraph(org.atlasapi.equivalence.EquivalenceGraph) ArgumentCaptor(org.mockito.ArgumentCaptor) ImmutableList(com.google.common.collect.ImmutableList) MutationBatch(com.netflix.astyanax.MutationBatch) Publisher(org.atlasapi.media.entity.Publisher) Keyspace(com.netflix.astyanax.Keyspace) Matchers.hasSize(org.hamcrest.Matchers.hasSize) AstyanaxContext(com.netflix.astyanax.AstyanaxContext) WriteResult(org.atlasapi.entity.util.WriteResult) ConnectionException(com.netflix.astyanax.connectionpool.exceptions.ConnectionException) Before(org.junit.Before) Matchers.empty(org.hamcrest.Matchers.empty) CassandraHelper(org.atlasapi.entity.CassandraHelper) CassandraInit(org.atlasapi.util.CassandraInit) DateTime(org.joda.time.DateTime) Throwables(com.google.common.base.Throwables) Assert.assertTrue(org.junit.Assert.assertTrue) Test(org.junit.Test) IOException(java.io.IOException) Mockito.times(org.mockito.Mockito.times) Id(org.atlasapi.entity.Id) Mockito.when(org.mockito.Mockito.when) ResourceUpdatedMessage(org.atlasapi.messaging.ResourceUpdatedMessage) MessagingException(com.metabroadcast.common.queue.MessagingException) DateTimeZones(com.metabroadcast.common.time.DateTimeZones) Mockito.verify(org.mockito.Mockito.verify) LocalDate(org.joda.time.LocalDate) ExecutionException(java.util.concurrent.ExecutionException) TimeUnit(java.util.concurrent.TimeUnit) Futures(com.google.common.util.concurrent.Futures) Mockito.never(org.mockito.Mockito.never) EquivalenceGraphStore(org.atlasapi.equivalence.EquivalenceGraphStore) MessageSender(com.metabroadcast.common.queue.MessageSender) DatastaxCassandraService(com.metabroadcast.common.persistence.cassandra.DatastaxCassandraService) PatternLayout(org.apache.log4j.PatternLayout) WriteException(org.atlasapi.entity.util.WriteException) Collections(java.util.Collections) Mockito.reset(org.mockito.Mockito.reset) ResourceUpdatedMessage(org.atlasapi.messaging.ResourceUpdatedMessage) DateTime(org.joda.time.DateTime) Test(org.junit.Test)

Example 4 with ResourceUpdatedMessage

use of org.atlasapi.messaging.ResourceUpdatedMessage in project atlas-deer by atlasapi.

the class CassandraContentStoreIT method testUpdatingContainerUpdatesContainerSummaryInChildItemAndSendsResourceUpdatedMessage.

@Test
public void testUpdatingContainerUpdatesContainerSummaryInChildItemAndSendsResourceUpdatedMessage() throws Exception {
    DateTime now = new DateTime(DateTimeZones.UTC);
    Brand brand = create(new Brand());
    when(clock.now()).thenReturn(now);
    when(idGenerator.generateRaw()).thenReturn(1234L);
    WriteResult<Brand, Content> brandWriteResult = store.writeContent(brand);
    Series series1 = create(new Series());
    series1.setBrand(brandWriteResult.getResource());
    when(clock.now()).thenReturn(now.plusHours(1));
    when(idGenerator.generateRaw()).thenReturn(1235L);
    WriteResult<Series, Content> series1WriteResult = store.writeContent(series1);
    Episode episode = create(new Episode());
    episode.setContainer(brandWriteResult.getResource());
    episode.setSeries(series1WriteResult.getResource());
    when(clock.now()).thenReturn(now.plusHours(2));
    when(idGenerator.generateRaw()).thenReturn(1237L);
    store.writeContent(episode);
    when(hasher.hash(argThat(isA(Content.class)))).thenReturn("different").thenReturn("differentAgain");
    Episode resolvedEpisode = (Episode) resolve(1237L);
    assertThat(resolvedEpisode.getFirstSeen(), is(now.plusHours(2)));
    assertThat(resolvedEpisode.getLastUpdated(), is(now.plusHours(2)));
    assertThat(resolvedEpisode.getThisOrChildLastUpdated(), is(now.plusHours(2)));
    assertThat(resolvedEpisode.getContainerRef().getId().longValue(), is(1234L));
    assertThat(resolvedEpisode.getSeriesRef().getId().longValue(), is(1235L));
    assertThat(resolvedEpisode.getContainerSummary().getTitle(), is("Brand"));
    brand.setTitle("NewBrand");
    store.writeContent(brand);
    Brand resolvedBrand = (Brand) resolve(1234L);
    assertThat(resolvedBrand.getTitle(), is("NewBrand"));
    ArgumentCaptor<ResourceUpdatedMessage> captor = ArgumentCaptor.forClass(ResourceUpdatedMessage.class);
    verify(sender, times(8)).sendMessage(captor.capture(), any(byte[].class));
    List<ResourceUpdatedMessage> messagesSent = captor.getAllValues();
    assertThat(messagesSent.size(), is(8));
    assertTrue(Iterables.tryFind(messagesSent, msg -> msg.getUpdatedResource().getId().longValue() == 1234L).isPresent());
    assertTrue(Iterables.tryFind(messagesSent, msg -> msg.getUpdatedResource().getId().longValue() == 1235L).isPresent());
    assertTrue(Iterables.tryFind(messagesSent, msg -> msg.getUpdatedResource().getId().longValue() == 1237L).isPresent());
    Episode newResolvedEpisode = (Episode) resolve(1237L);
    assertThat(newResolvedEpisode.getContainerSummary().getTitle(), is("NewBrand"));
}
Also used : ResourceUpdatedMessage(org.atlasapi.messaging.ResourceUpdatedMessage) DateTime(org.joda.time.DateTime) Test(org.junit.Test)

Example 5 with ResourceUpdatedMessage

use of org.atlasapi.messaging.ResourceUpdatedMessage in project atlas-deer by atlasapi.

the class CassandraContentStoreIT method testWriteSendsResourceUpdatedMessage.

@Test
public void testWriteSendsResourceUpdatedMessage() throws Exception {
    Content content = create(new Item());
    content.setTitle("title");
    DateTime now = new DateTime(DateTimeZones.UTC);
    when(clock.now()).thenReturn(now);
    when(idGenerator.generateRaw()).thenReturn(1234L);
    WriteResult<Content, Content> writeResult = store.writeContent(content);
    assertTrue(writeResult.written());
    assertThat(writeResult.getResource().getId().longValue(), is(1234L));
    assertFalse(writeResult.getPrevious().isPresent());
    ArgumentCaptor<ResourceUpdatedMessage> messageCaptor = ArgumentCaptor.forClass(ResourceUpdatedMessage.class);
    verify(sender).sendMessage(messageCaptor.capture(), eq(Longs.toByteArray(1234L)));
    assertThat(messageCaptor.getValue().getUpdatedResource().getId().longValue(), is(1234L));
}
Also used : ResourceUpdatedMessage(org.atlasapi.messaging.ResourceUpdatedMessage) DateTime(org.joda.time.DateTime) Test(org.junit.Test)

Aggregations

ResourceUpdatedMessage (org.atlasapi.messaging.ResourceUpdatedMessage)26 Test (org.junit.Test)11 Id (org.atlasapi.entity.Id)9 DateTime (org.joda.time.DateTime)8 WriteException (org.atlasapi.entity.util.WriteException)7 BatchStatement (com.datastax.driver.core.BatchStatement)6 ExecutionException (java.util.concurrent.ExecutionException)4 org.atlasapi.content (org.atlasapi.content)4 WriteResult (org.atlasapi.entity.util.WriteResult)4 MetricRegistry (com.codahale.metrics.MetricRegistry)3 Statement (com.datastax.driver.core.Statement)3 Before (org.junit.Before)3 Session (com.datastax.driver.core.Session)2 Optional (com.google.common.base.Optional)2 Throwables (com.google.common.base.Throwables)2 ImmutableList (com.google.common.collect.ImmutableList)2 Longs (com.google.common.primitives.Longs)2 Futures (com.google.common.util.concurrent.Futures)2 ListenableFuture (com.google.common.util.concurrent.ListenableFuture)2 Event (com.metabroadcast.columbus.telescope.api.Event)2