Search in sources :

Example 1 with SynchronizerClientFactory

use of io.pravega.client.SynchronizerClientFactory in project pravega by pravega.

the class RevisionedStreamClientTest method testCreateRevisionedStreamClientError.

@Test
public void testCreateRevisionedStreamClientError() {
    String scope = "scope";
    String stream = "stream";
    JavaSerializer<Serializable> serializer = new JavaSerializer<>();
    @Cleanup MockConnectionFactoryImpl connectionFactory = new MockConnectionFactoryImpl();
    Controller controller = Mockito.mock(Controller.class);
    MockSegmentStreamFactory streamFactory = new MockSegmentStreamFactory();
    @Cleanup SynchronizerClientFactory clientFactory = new ClientFactoryImpl(scope, controller, connectionFactory, streamFactory, streamFactory, streamFactory, streamFactory);
    SynchronizerConfig config = SynchronizerConfig.builder().build();
    // Simulate sealed stream.
    CompletableFuture<StreamSegments> result = new CompletableFuture<>();
    result.complete(new StreamSegments(new TreeMap<>()));
    when(controller.getCurrentSegments(scope, stream)).thenReturn(result);
    assertThrows(InvalidStreamException.class, () -> clientFactory.createRevisionedStreamClient(stream, serializer, config));
    // Simulate invalid stream.
    result = new CompletableFuture<>();
    result.completeExceptionally(new RuntimeException());
    when(controller.getCurrentSegments(scope, stream)).thenReturn(result);
    assertThrows(InvalidStreamException.class, () -> clientFactory.createRevisionedStreamClient(stream, serializer, config));
    // Simulate null result from Controller.
    result = new CompletableFuture<>();
    result.complete(null);
    when(controller.getCurrentSegments(scope, stream)).thenReturn(result);
    assertThrows(InvalidStreamException.class, () -> clientFactory.createRevisionedStreamClient(stream, serializer, config));
}
Also used : Serializable(java.io.Serializable) MockSegmentStreamFactory(io.pravega.client.stream.mock.MockSegmentStreamFactory) MockController(io.pravega.client.stream.mock.MockController) Controller(io.pravega.client.control.impl.Controller) TreeMap(java.util.TreeMap) JavaSerializer(io.pravega.client.stream.impl.JavaSerializer) Cleanup(lombok.Cleanup) SynchronizerClientFactory(io.pravega.client.SynchronizerClientFactory) ClientFactoryImpl(io.pravega.client.stream.impl.ClientFactoryImpl) CompletableFuture(java.util.concurrent.CompletableFuture) MockConnectionFactoryImpl(io.pravega.client.stream.mock.MockConnectionFactoryImpl) SynchronizerConfig(io.pravega.client.state.SynchronizerConfig) StreamSegments(io.pravega.client.stream.impl.StreamSegments) Test(org.junit.Test)

Example 2 with SynchronizerClientFactory

use of io.pravega.client.SynchronizerClientFactory in project pravega by pravega.

the class RevisionedStreamClientTest method testMark.

@Test
public void testMark() {
    String scope = "scope";
    String stream = "stream";
    PravegaNodeUri endpoint = new PravegaNodeUri("localhost", SERVICE_PORT);
    @Cleanup MockConnectionFactoryImpl connectionFactory = new MockConnectionFactoryImpl();
    @Cleanup MockController controller = new MockController(endpoint.getEndpoint(), endpoint.getPort(), connectionFactory, false);
    createScopeAndStream(scope, stream, controller);
    MockSegmentStreamFactory streamFactory = new MockSegmentStreamFactory();
    @Cleanup SynchronizerClientFactory clientFactory = new ClientFactoryImpl(scope, controller, connectionFactory, streamFactory, streamFactory, streamFactory, streamFactory);
    SynchronizerConfig config = SynchronizerConfig.builder().build();
    @Cleanup RevisionedStreamClient<String> client = clientFactory.createRevisionedStreamClient(stream, new JavaSerializer<>(), config);
    client.writeUnconditionally("a");
    Revision ra = client.fetchLatestRevision();
    client.writeUnconditionally("b");
    Revision rb = client.fetchLatestRevision();
    client.writeUnconditionally("c");
    Revision rc = client.fetchLatestRevision();
    assertTrue(client.compareAndSetMark(null, ra));
    assertEquals(ra, client.getMark());
    assertTrue(client.compareAndSetMark(ra, rb));
    assertEquals(rb, client.getMark());
    assertFalse(client.compareAndSetMark(ra, rc));
    assertEquals(rb, client.getMark());
    assertTrue(client.compareAndSetMark(rb, rc));
    assertEquals(rc, client.getMark());
    assertTrue(client.compareAndSetMark(rc, ra));
    assertEquals(ra, client.getMark());
    assertTrue(client.compareAndSetMark(ra, null));
    assertEquals(null, client.getMark());
}
Also used : SynchronizerClientFactory(io.pravega.client.SynchronizerClientFactory) ClientFactoryImpl(io.pravega.client.stream.impl.ClientFactoryImpl) PravegaNodeUri(io.pravega.shared.protocol.netty.PravegaNodeUri) Revision(io.pravega.client.state.Revision) MockSegmentStreamFactory(io.pravega.client.stream.mock.MockSegmentStreamFactory) MockConnectionFactoryImpl(io.pravega.client.stream.mock.MockConnectionFactoryImpl) MockController(io.pravega.client.stream.mock.MockController) Cleanup(lombok.Cleanup) SynchronizerConfig(io.pravega.client.state.SynchronizerConfig) Test(org.junit.Test)

Example 3 with SynchronizerClientFactory

use of io.pravega.client.SynchronizerClientFactory in project pravega by pravega.

the class RevisionedStreamClientTest method testWriteWhileReading.

@Test
public void testWriteWhileReading() {
    String scope = "scope";
    String stream = "stream";
    PravegaNodeUri endpoint = new PravegaNodeUri("localhost", SERVICE_PORT);
    @Cleanup MockConnectionFactoryImpl connectionFactory = new MockConnectionFactoryImpl();
    @Cleanup MockController controller = new MockController(endpoint.getEndpoint(), endpoint.getPort(), connectionFactory, false);
    createScopeAndStream(scope, stream, controller);
    MockSegmentStreamFactory streamFactory = new MockSegmentStreamFactory();
    @Cleanup SynchronizerClientFactory clientFactory = new ClientFactoryImpl(scope, controller, connectionFactory, streamFactory, streamFactory, streamFactory, streamFactory);
    SynchronizerConfig config = SynchronizerConfig.builder().build();
    @Cleanup RevisionedStreamClient<String> client = clientFactory.createRevisionedStreamClient(stream, new JavaSerializer<>(), config);
    Revision initialRevision = client.fetchLatestRevision();
    client.writeUnconditionally("a");
    client.writeUnconditionally("b");
    client.writeUnconditionally("c");
    Iterator<Entry<Revision, String>> iter = client.readFrom(initialRevision);
    assertTrue(iter.hasNext());
    assertEquals("a", iter.next().getValue());
    client.writeUnconditionally("d");
    assertTrue(iter.hasNext());
    assertEquals("b", iter.next().getValue());
    assertTrue(iter.hasNext());
    Entry<Revision, String> entry = iter.next();
    assertEquals("c", entry.getValue());
    assertFalse(iter.hasNext());
    iter = client.readFrom(entry.getKey());
    assertTrue(iter.hasNext());
    assertEquals("d", iter.next().getValue());
    assertFalse(iter.hasNext());
}
Also used : MockSegmentStreamFactory(io.pravega.client.stream.mock.MockSegmentStreamFactory) Cleanup(lombok.Cleanup) SynchronizerClientFactory(io.pravega.client.SynchronizerClientFactory) ClientFactoryImpl(io.pravega.client.stream.impl.ClientFactoryImpl) Entry(java.util.Map.Entry) PravegaNodeUri(io.pravega.shared.protocol.netty.PravegaNodeUri) Revision(io.pravega.client.state.Revision) MockConnectionFactoryImpl(io.pravega.client.stream.mock.MockConnectionFactoryImpl) MockController(io.pravega.client.stream.mock.MockController) SynchronizerConfig(io.pravega.client.state.SynchronizerConfig) Test(org.junit.Test)

Example 4 with SynchronizerClientFactory

use of io.pravega.client.SynchronizerClientFactory in project pravega by pravega.

the class RevisionedStreamClientTest method testTimeoutWithStreamIterator.

@Test
public void testTimeoutWithStreamIterator() throws Exception {
    String scope = "scope";
    String stream = "stream";
    // Setup Environment
    PravegaNodeUri endpoint = new PravegaNodeUri("localhost", SERVICE_PORT);
    JavaSerializer<String> serializer = new JavaSerializer<>();
    @Cleanup MockConnectionFactoryImpl connectionFactory = new MockConnectionFactoryImpl();
    @Cleanup MockController controller = new MockController(endpoint.getEndpoint(), endpoint.getPort(), connectionFactory, false);
    createScopeAndStream(scope, stream, controller);
    MockSegmentStreamFactory streamFactory = new MockSegmentStreamFactory();
    // Setup mock
    SegmentOutputStreamFactory outFactory = mock(SegmentOutputStreamFactory.class);
    SegmentOutputStream out = mock(SegmentOutputStream.class);
    Segment segment = new Segment(scope, stream, 0);
    when(outFactory.createOutputStreamForSegment(eq(segment), any(), any(), any(DelegationTokenProvider.class))).thenReturn(out);
    SegmentInputStreamFactory inFactory = mock(SegmentInputStreamFactory.class);
    EventSegmentReader in = mock(EventSegmentReader.class);
    when(inFactory.createEventReaderForSegment(eq(segment), anyInt())).thenReturn(in);
    when(in.read(anyLong())).thenReturn(null).thenReturn(serializer.serialize("testData"));
    SegmentMetadataClientFactory metaFactory = mock(SegmentMetadataClientFactory.class);
    SegmentMetadataClient metaClient = mock(SegmentMetadataClient.class);
    when(metaFactory.createSegmentMetadataClient(any(Segment.class), any(DelegationTokenProvider.class))).thenReturn(metaClient);
    when(metaClient.getSegmentInfo()).thenReturn(CompletableFuture.completedFuture(new SegmentInfo(segment, 0, 30, false, System.currentTimeMillis())));
    @Cleanup SynchronizerClientFactory clientFactory = new ClientFactoryImpl(scope, controller, connectionFactory, inFactory, streamFactory, streamFactory, metaFactory);
    @Cleanup RevisionedStreamClient<String> client = clientFactory.createRevisionedStreamClient(stream, serializer, SynchronizerConfig.builder().build());
    Iterator<Entry<Revision, String>> iterator = client.readFrom(new RevisionImpl(segment, 15, 1));
    assertTrue("True is expected since offset is less than end offset", iterator.hasNext());
    assertNotNull("Verify the entry is not null", iterator.next());
    verify(in, times(2)).read(anyLong());
}
Also used : MockSegmentStreamFactory(io.pravega.client.stream.mock.MockSegmentStreamFactory) JavaSerializer(io.pravega.client.stream.impl.JavaSerializer) Cleanup(lombok.Cleanup) Segment(io.pravega.client.segment.impl.Segment) SegmentMetadataClient(io.pravega.client.segment.impl.SegmentMetadataClient) SynchronizerClientFactory(io.pravega.client.SynchronizerClientFactory) ClientFactoryImpl(io.pravega.client.stream.impl.ClientFactoryImpl) Entry(java.util.Map.Entry) PravegaNodeUri(io.pravega.shared.protocol.netty.PravegaNodeUri) SegmentOutputStreamFactory(io.pravega.client.segment.impl.SegmentOutputStreamFactory) SegmentOutputStream(io.pravega.client.segment.impl.SegmentOutputStream) EventSegmentReader(io.pravega.client.segment.impl.EventSegmentReader) SegmentMetadataClientFactory(io.pravega.client.segment.impl.SegmentMetadataClientFactory) MockConnectionFactoryImpl(io.pravega.client.stream.mock.MockConnectionFactoryImpl) MockController(io.pravega.client.stream.mock.MockController) SegmentInfo(io.pravega.client.segment.impl.SegmentInfo) SegmentInputStreamFactory(io.pravega.client.segment.impl.SegmentInputStreamFactory) DelegationTokenProvider(io.pravega.client.security.auth.DelegationTokenProvider) Test(org.junit.Test)

Example 5 with SynchronizerClientFactory

use of io.pravega.client.SynchronizerClientFactory in project pravega by pravega.

the class ReaderGroupStateManagerTest method testSegmentsAssigned.

@Test(timeout = 10000)
public void testSegmentsAssigned() throws ReaderNotInReaderGroupException {
    String scope = "scope";
    String stream = "stream";
    PravegaNodeUri endpoint = new PravegaNodeUri("localhost", SERVICE_PORT);
    MockConnectionFactoryImpl connectionFactory = new MockConnectionFactoryImpl();
    MockController controller = new MockController(endpoint.getEndpoint(), endpoint.getPort(), connectionFactory, false);
    createScopeAndStream(scope, stream, controller);
    MockSegmentStreamFactory streamFactory = new MockSegmentStreamFactory();
    @Cleanup SynchronizerClientFactory clientFactory = new ClientFactoryImpl(scope, controller, connectionFactory, streamFactory, streamFactory, streamFactory, streamFactory);
    SynchronizerConfig config = SynchronizerConfig.builder().build();
    @Cleanup StateSynchronizer<ReaderGroupState> stateSynchronizer = createState(stream, clientFactory, config);
    AtomicLong clock = new AtomicLong();
    Map<SegmentWithRange, Long> segments = new HashMap<>();
    segments.put(new SegmentWithRange(new Segment(scope, stream, 0), 0.0, 0.25), 0L);
    segments.put(new SegmentWithRange(new Segment(scope, stream, 1), 0.25, 0.5), 1L);
    segments.put(new SegmentWithRange(new Segment(scope, stream, 2), 0.5, 0.75), 2L);
    segments.put(new SegmentWithRange(new Segment(scope, stream, 3), 0.75, 1.0), 3L);
    stateSynchronizer.initialize(new ReaderGroupState.ReaderGroupStateInit(ReaderGroupConfig.builder().stream(Stream.of(scope, stream)).build(), segments, Collections.emptyMap(), false));
    ReaderGroupStateManager reader1 = new ReaderGroupStateManager(scope, stream, "reader1", stateSynchronizer, controller, clock::get);
    reader1.initializeReader(0);
    ReaderGroupStateManager reader2 = new ReaderGroupStateManager(scope, stream, "reader2", stateSynchronizer, controller, clock::get);
    reader2.initializeReader(0);
    Map<SegmentWithRange, Long> segments1 = reader1.acquireNewSegmentsIfNeeded(0, new PositionImpl(Collections.emptyMap()));
    assertFalse(segments1.isEmpty());
    assertEquals(2, segments1.size());
    assertTrue(reader1.acquireNewSegmentsIfNeeded(0, new PositionImpl(segments1)).isEmpty());
    assertNull(reader1.findSegmentToReleaseIfRequired());
    Map<SegmentWithRange, Long> segments2 = reader2.acquireNewSegmentsIfNeeded(0, new PositionImpl(Collections.emptyMap()));
    assertFalse(segments2.isEmpty());
    assertEquals(2, segments2.size());
    assertTrue(reader2.acquireNewSegmentsIfNeeded(0, new PositionImpl(segments2)).isEmpty());
    assertNull(reader2.findSegmentToReleaseIfRequired());
    assertTrue(Sets.intersection(segments1.keySet(), segments2.keySet()).isEmpty());
    clock.addAndGet(ReaderGroupStateManager.UPDATE_WINDOW.toNanos());
    assertTrue(reader1.acquireNewSegmentsIfNeeded(0, new PositionImpl(segments1)).isEmpty());
    assertNull(reader1.findSegmentToReleaseIfRequired());
    assertTrue(reader2.acquireNewSegmentsIfNeeded(0, new PositionImpl(segments2)).isEmpty());
    assertNull(reader2.findSegmentToReleaseIfRequired());
    reader1.readerShutdown(new PositionImpl(segments1));
    clock.addAndGet(ReaderGroupStateManager.UPDATE_WINDOW.toNanos());
    Map<SegmentWithRange, Long> segmentsRecovered = reader2.acquireNewSegmentsIfNeeded(0, new PositionImpl(segments2));
    assertFalse(segmentsRecovered.isEmpty());
    assertEquals(2, segmentsRecovered.size());
    assertEquals(segments1, segmentsRecovered);
    assertTrue(reader2.acquireNewSegmentsIfNeeded(0, new PositionImpl(segments)).isEmpty());
    assertNull(reader2.findSegmentToReleaseIfRequired());
    segments2.putAll(segmentsRecovered);
    reader2.readerShutdown(new PositionImpl(segments2));
    reader1.initializeReader(0);
    segments1 = reader1.acquireNewSegmentsIfNeeded(0, new PositionImpl(Collections.emptyMap()));
    assertEquals(4, segments1.size());
    assertEquals(segments2, segments1);
}
Also used : HashMap(java.util.HashMap) MockSegmentStreamFactory(io.pravega.client.stream.mock.MockSegmentStreamFactory) Cleanup(lombok.Cleanup) Segment(io.pravega.client.segment.impl.Segment) SynchronizerClientFactory(io.pravega.client.SynchronizerClientFactory) AtomicLong(java.util.concurrent.atomic.AtomicLong) PravegaNodeUri(io.pravega.shared.protocol.netty.PravegaNodeUri) MockConnectionFactoryImpl(io.pravega.client.stream.mock.MockConnectionFactoryImpl) AtomicLong(java.util.concurrent.atomic.AtomicLong) MockController(io.pravega.client.stream.mock.MockController) SynchronizerConfig(io.pravega.client.state.SynchronizerConfig) Test(org.junit.Test)

Aggregations

SynchronizerClientFactory (io.pravega.client.SynchronizerClientFactory)34 Cleanup (lombok.Cleanup)34 Test (org.junit.Test)33 MockConnectionFactoryImpl (io.pravega.client.stream.mock.MockConnectionFactoryImpl)26 MockController (io.pravega.client.stream.mock.MockController)26 MockSegmentStreamFactory (io.pravega.client.stream.mock.MockSegmentStreamFactory)26 PravegaNodeUri (io.pravega.shared.protocol.netty.PravegaNodeUri)25 SynchronizerConfig (io.pravega.client.state.SynchronizerConfig)23 Segment (io.pravega.client.segment.impl.Segment)22 AtomicLong (java.util.concurrent.atomic.AtomicLong)21 HashMap (java.util.HashMap)17 ClientFactoryImpl (io.pravega.client.stream.impl.ClientFactoryImpl)8 Watermark (io.pravega.shared.watermarks.Watermark)7 lombok.val (lombok.val)7 ReaderGroupConfig (io.pravega.client.stream.ReaderGroupConfig)6 Revision (io.pravega.client.state.Revision)5 CreateCheckpoint (io.pravega.client.stream.impl.ReaderGroupState.CreateCheckpoint)5 StreamImpl (io.pravega.client.stream.impl.StreamImpl)5 ArgumentMatchers.anyString (org.mockito.ArgumentMatchers.anyString)5 JavaSerializer (io.pravega.client.stream.impl.JavaSerializer)4