Search in sources :

Example 1 with IngestService

use of com.alibaba.graphscope.groot.ingestor.IngestService in project GraphScope by alibaba.

the class IngestServiceTest method testIngestService.

@Test
void testIngestService() {
    Configs configs = Configs.newBuilder().put(CommonConfig.NODE_IDX.getKey(), "0").put(CommonConfig.STORE_NODE_COUNT.getKey(), "1").put(IngestorConfig.INGESTOR_CHECK_PROCESSOR_INTERVAL_MS.getKey(), "100").build();
    MockDiscovery mockDiscovery = new MockDiscovery();
    MetaService mockMetaService = mock(MetaService.class);
    when(mockMetaService.getQueueIdsForIngestor(0)).thenReturn(Arrays.asList(0));
    LogService mockLogService = mock(LogService.class);
    IngestProgressFetcher mockIngestProgressFetcher = mock(IngestProgressFetcher.class);
    when(mockIngestProgressFetcher.getTailOffsets(Arrays.asList(0))).thenReturn(Arrays.asList(50L));
    StoreWriter mockStoreWriter = mock(StoreWriter.class);
    IngestService spyIngestService = spy(new IngestService(configs, mockDiscovery, mockMetaService, mockLogService, mockIngestProgressFetcher, mockStoreWriter, new MetricsCollector(configs)));
    IngestProcessor mockIngestProcessor = mock(IngestProcessor.class);
    doReturn(mockIngestProcessor).when(spyIngestService).makeIngestProcessor(any(), any(), any(), eq(0), any(), any());
    spyIngestService.start();
    verify(mockIngestProcessor, never()).start();
    mockDiscovery.addNode(RoleType.STORE, Collections.singletonMap(0, null));
    verify(mockIngestProcessor, timeout(5000L)).setTailOffset(50L);
    verify(mockIngestProcessor, timeout(5000L)).start();
    spyIngestService.advanceIngestSnapshotId(5L, null);
    verify(mockIngestProcessor).ingestBatch(eq("marker"), eq(IngestService.MARKER_BATCH), any());
    mockDiscovery.removeNode(RoleType.STORE, Collections.singletonMap(0, null));
    verify(mockIngestProcessor, timeout(5000L).times(2)).stop();
    spyIngestService.stop();
}
Also used : MetricsCollector(com.alibaba.graphscope.groot.metrics.MetricsCollector) IngestService(com.alibaba.graphscope.groot.ingestor.IngestService) MetaService(com.alibaba.graphscope.groot.meta.MetaService) StoreWriter(com.alibaba.graphscope.groot.ingestor.StoreWriter) Configs(com.alibaba.maxgraph.common.config.Configs) IngestProgressFetcher(com.alibaba.graphscope.groot.ingestor.IngestProgressFetcher) LogService(com.alibaba.graphscope.groot.wal.LogService) IngestProcessor(com.alibaba.graphscope.groot.ingestor.IngestProcessor) Test(org.junit.jupiter.api.Test)

Example 2 with IngestService

use of com.alibaba.graphscope.groot.ingestor.IngestService in project GraphScope by alibaba.

the class IngestorRpcTest method testIngestorWriteService.

@Test
void testIngestorWriteService() {
    IngestService ingestService = mock(IngestService.class);
    IngestorWriteService ingestorWriteService = new IngestorWriteService(ingestService);
    WriteIngestorRequest req = WriteIngestorRequest.newBuilder().setQueueId(2).setRequestId("test_req").setOperationBatch(OperationBatch.newBuilder().build().toProto()).build();
    doAnswer(invocation -> {
        IngestCallback callback = invocation.getArgument(3);
        callback.onSuccess(10L);
        return null;
    }).when(ingestService).ingestBatch(eq("test_req"), eq(2), eq(OperationBatch.newBuilder().build()), any());
    StreamObserver<WriteIngestorResponse> observer = mock(StreamObserver.class);
    ingestorWriteService.writeIngestor(req, observer);
    verify(observer).onNext(WriteIngestorResponse.newBuilder().setSnapshotId(10L).build());
    verify(observer).onCompleted();
}
Also used : IngestService(com.alibaba.graphscope.groot.ingestor.IngestService) IngestorWriteService(com.alibaba.graphscope.groot.ingestor.IngestorWriteService) WriteIngestorRequest(com.alibaba.maxgraph.proto.groot.WriteIngestorRequest) WriteIngestorResponse(com.alibaba.maxgraph.proto.groot.WriteIngestorResponse) IngestCallback(com.alibaba.graphscope.groot.ingestor.IngestCallback) Test(org.junit.jupiter.api.Test)

Example 3 with IngestService

use of com.alibaba.graphscope.groot.ingestor.IngestService in project GraphScope by alibaba.

the class IngestorRpcTest method testIngestorSnapshotService.

@Test
void testIngestorSnapshotService() {
    IngestService ingestService = mock(IngestService.class);
    IngestorSnapshotService ingestorSnapshotService = new IngestorSnapshotService(ingestService);
    AdvanceIngestSnapshotIdRequest req = AdvanceIngestSnapshotIdRequest.newBuilder().setSnapshotId(10L).build();
    StreamObserver<AdvanceIngestSnapshotIdResponse> streamObserver = mock(StreamObserver.class);
    doAnswer(invocation -> {
        CompletionCallback<Long> callback = invocation.getArgument(1);
        callback.onCompleted(9L);
        return null;
    }).when(ingestService).advanceIngestSnapshotId(anyLong(), any());
    ingestorSnapshotService.advanceIngestSnapshotId(req, streamObserver);
    verify(streamObserver).onNext(AdvanceIngestSnapshotIdResponse.newBuilder().setPreviousSnapshotId(9L).build());
    verify(streamObserver).onCompleted();
}
Also used : IngestService(com.alibaba.graphscope.groot.ingestor.IngestService) AdvanceIngestSnapshotIdResponse(com.alibaba.maxgraph.proto.groot.AdvanceIngestSnapshotIdResponse) AdvanceIngestSnapshotIdRequest(com.alibaba.maxgraph.proto.groot.AdvanceIngestSnapshotIdRequest) ArgumentMatchers.anyLong(org.mockito.ArgumentMatchers.anyLong) IngestorSnapshotService(com.alibaba.graphscope.groot.ingestor.IngestorSnapshotService) Test(org.junit.jupiter.api.Test)

Aggregations

IngestService (com.alibaba.graphscope.groot.ingestor.IngestService)3 Test (org.junit.jupiter.api.Test)3 IngestCallback (com.alibaba.graphscope.groot.ingestor.IngestCallback)1 IngestProcessor (com.alibaba.graphscope.groot.ingestor.IngestProcessor)1 IngestProgressFetcher (com.alibaba.graphscope.groot.ingestor.IngestProgressFetcher)1 IngestorSnapshotService (com.alibaba.graphscope.groot.ingestor.IngestorSnapshotService)1 IngestorWriteService (com.alibaba.graphscope.groot.ingestor.IngestorWriteService)1 StoreWriter (com.alibaba.graphscope.groot.ingestor.StoreWriter)1 MetaService (com.alibaba.graphscope.groot.meta.MetaService)1 MetricsCollector (com.alibaba.graphscope.groot.metrics.MetricsCollector)1 LogService (com.alibaba.graphscope.groot.wal.LogService)1 Configs (com.alibaba.maxgraph.common.config.Configs)1 AdvanceIngestSnapshotIdRequest (com.alibaba.maxgraph.proto.groot.AdvanceIngestSnapshotIdRequest)1 AdvanceIngestSnapshotIdResponse (com.alibaba.maxgraph.proto.groot.AdvanceIngestSnapshotIdResponse)1 WriteIngestorRequest (com.alibaba.maxgraph.proto.groot.WriteIngestorRequest)1 WriteIngestorResponse (com.alibaba.maxgraph.proto.groot.WriteIngestorResponse)1 ArgumentMatchers.anyLong (org.mockito.ArgumentMatchers.anyLong)1