Search in sources :

Example 1 with BatchId

use of com.alibaba.graphscope.groot.operation.BatchId in project GraphScope by alibaba.

the class FrontendRpcTest method testIngestorWriteClient.

@Test
void testIngestorWriteClient() {
    IngestorWriteGrpc.IngestorWriteBlockingStub stub = mock(IngestorWriteGrpc.IngestorWriteBlockingStub.class);
    IngestorWriteClient client = new IngestorWriteClient(stub);
    when(stub.writeIngestor(any())).thenReturn(WriteIngestorResponse.newBuilder().setSnapshotId(10L).build());
    BatchId batchId = client.writeIngestor("test_req", 2, OperationBatch.newBuilder().build());
    assertEquals(batchId.getSnapshotId(), 10L);
    verify(stub).writeIngestor(WriteIngestorRequest.newBuilder().setRequestId("test_req").setQueueId(2).setOperationBatch(OperationBatch.newBuilder().build().toProto()).build());
}
Also used : BatchId(com.alibaba.graphscope.groot.operation.BatchId) Test(org.junit.jupiter.api.Test)

Example 2 with BatchId

use of com.alibaba.graphscope.groot.operation.BatchId in project GraphScope by alibaba.

the class SchemaManagerTest method testSchemaManager.

@Test
void testSchemaManager() throws IOException, InterruptedException {
    SnapshotManager mockSnapshotManager = mock(SnapshotManager.class);
    doAnswer(invocationOnMock -> {
        SnapshotListener listener = invocationOnMock.getArgument(1);
        listener.onSnapshotAvailable();
        return null;
    }).when(mockSnapshotManager).addSnapshotListener(anyLong(), any());
    when(mockSnapshotManager.increaseWriteSnapshotId()).thenReturn(1L);
    when(mockSnapshotManager.getCurrentWriteSnapshotId()).thenReturn(1L);
    DdlExecutors ddlExecutors = new DdlExecutors();
    DdlWriter mockDdlWriter = mock(DdlWriter.class);
    when(mockDdlWriter.writeOperations(anyString(), any())).thenReturn(new BatchId(1L));
    MetaService mockMetaService = mock(MetaService.class);
    GraphDefFetcher mockGraphDefFetcher = mock(GraphDefFetcher.class);
    GraphDef initialGraphDef = GraphDef.newBuilder().build();
    when(mockGraphDefFetcher.fetchGraphDef()).thenReturn(initialGraphDef);
    SchemaManager schemaManager = new SchemaManager(mockSnapshotManager, ddlExecutors, mockDdlWriter, mockMetaService, mockGraphDefFetcher);
    schemaManager.start();
    assertEquals(initialGraphDef, schemaManager.getGraphDef());
    PropertyValue defaultValue = new PropertyValue(DataType.INT, ByteBuffer.allocate(Integer.BYTES).putInt(1).array());
    PropertyDef propertyDef = new PropertyDef(1, 1, "p1", DataType.INT, defaultValue, true, "property_1");
    TypeDef typeDef = TypeDef.newBuilder().setLabel("vertex1").addPropertyDef(propertyDef).setTypeEnum(TypeEnum.VERTEX).build();
    DdlRequestBatch ddlRequestBatch = DdlRequestBatch.newBuilder().addDdlRequest(new CreateVertexTypeRequest(typeDef)).build();
    CountDownLatch latch = new CountDownLatch(1);
    schemaManager.submitBatchDdl("requestId", "sessionId", ddlRequestBatch, new CompletionCallback<Long>() {

        @Override
        public void onCompleted(Long res) {
            latch.countDown();
        }

        @Override
        public void onError(Throwable t) {
        }
    });
    assertTrue(latch.await(5L, TimeUnit.SECONDS));
    schemaManager.stop();
}
Also used : MetaService(com.alibaba.graphscope.groot.meta.MetaService) PropertyDef(com.alibaba.maxgraph.sdkcommon.schema.PropertyDef) BatchId(com.alibaba.graphscope.groot.operation.BatchId) PropertyValue(com.alibaba.maxgraph.sdkcommon.schema.PropertyValue) SchemaManager(com.alibaba.graphscope.groot.coordinator.SchemaManager) CountDownLatch(java.util.concurrent.CountDownLatch) GraphDefFetcher(com.alibaba.graphscope.groot.coordinator.GraphDefFetcher) GraphDef(com.alibaba.maxgraph.sdkcommon.schema.GraphDef) SnapshotListener(com.alibaba.graphscope.groot.SnapshotListener) DdlWriter(com.alibaba.graphscope.groot.coordinator.DdlWriter) TypeDef(com.alibaba.maxgraph.sdkcommon.schema.TypeDef) DdlExecutors(com.alibaba.graphscope.groot.schema.ddl.DdlExecutors) ArgumentMatchers.anyLong(org.mockito.ArgumentMatchers.anyLong) CreateVertexTypeRequest(com.alibaba.graphscope.groot.schema.request.CreateVertexTypeRequest) DdlRequestBatch(com.alibaba.graphscope.groot.schema.request.DdlRequestBatch) SnapshotManager(com.alibaba.graphscope.groot.coordinator.SnapshotManager) Test(org.junit.jupiter.api.Test)

Example 3 with BatchId

use of com.alibaba.graphscope.groot.operation.BatchId in project GraphScope by alibaba.

the class SchemaManager method submitBatchDdl.

public void submitBatchDdl(String requestId, String sessionId, DdlRequestBatch ddlRequestBatch, CompletionCallback<Long> callback) {
    logger.info("submitBatchDdl requestId [" + requestId + "], sessionId [" + sessionId + "]");
    if (!ready) {
        callback.onError(new IllegalStateException("SchemaManager is recovering"));
        return;
    }
    this.singleThreadExecutor.execute(() -> {
        try {
            if (!ready) {
                callback.onError(new IllegalStateException("SchemaManager is recovering"));
                return;
            }
            GraphDef tmpGraphDef = this.graphDefRef.get();
            DdlResult ddlResult = this.ddlExecutors.executeDdlRequestBatch(ddlRequestBatch, tmpGraphDef, this.partitionCount);
            GraphDef graphDefResult = ddlResult.getGraphDef();
            List<Operation> ddlOperations = ddlResult.getDdlOperations();
            this.snapshotManager.lockWriteSnapshot();
            BatchId batchId;
            try {
                long currentWriteSnapshotId = this.snapshotManager.getCurrentWriteSnapshotId();
                OperationBatch operationBatch = OperationBatch.newBuilder(ddlOperations).setLatestSnapshotId(currentWriteSnapshotId).build();
                batchId = this.ddlWriter.writeOperations(requestId, operationBatch);
            } finally {
                this.snapshotManager.unlockWriteSnapshot();
            }
            long snapshotId = batchId.getSnapshotId();
            CompletableFuture<Void> future = new CompletableFuture<>();
            this.snapshotManager.addSnapshotListener(snapshotId, () -> {
                this.graphDefRef.set(graphDefResult);
                future.complete(null);
            });
            future.get();
            callback.onCompleted(snapshotId);
        } catch (Exception e) {
            logger.error("Error in Ddl requestId [" + requestId + "], sessionId [" + sessionId + "]", e);
            this.ready = false;
            callback.onError(e);
            this.singleThreadExecutor.execute(() -> recover());
        }
    });
}
Also used : CompletableFuture(java.util.concurrent.CompletableFuture) BatchId(com.alibaba.graphscope.groot.operation.BatchId) Operation(com.alibaba.graphscope.groot.operation.Operation) GraphDef(com.alibaba.maxgraph.sdkcommon.schema.GraphDef) ServiceNotReadyException(com.alibaba.maxgraph.compiler.api.exception.ServiceNotReadyException) IOException(java.io.IOException) ExecutionException(java.util.concurrent.ExecutionException) DdlResult(com.alibaba.graphscope.groot.schema.ddl.DdlResult) OperationBatch(com.alibaba.graphscope.groot.operation.OperationBatch)

Example 4 with BatchId

use of com.alibaba.graphscope.groot.operation.BatchId in project GraphScope by alibaba.

the class IngestorWriteClient method writeIngestor.

public BatchId writeIngestor(String requestId, int queueId, OperationBatch operationBatch) {
    WriteIngestorRequest request = WriteIngestorRequest.newBuilder().setRequestId(requestId).setQueueId(queueId).setOperationBatch(operationBatch.toProto()).build();
    WriteIngestorResponse response = this.stub.writeIngestor(request);
    return new BatchId(response.getSnapshotId());
}
Also used : BatchId(com.alibaba.graphscope.groot.operation.BatchId) WriteIngestorRequest(com.alibaba.maxgraph.proto.groot.WriteIngestorRequest) WriteIngestorResponse(com.alibaba.maxgraph.proto.groot.WriteIngestorResponse)

Aggregations

BatchId (com.alibaba.graphscope.groot.operation.BatchId)4 GraphDef (com.alibaba.maxgraph.sdkcommon.schema.GraphDef)2 Test (org.junit.jupiter.api.Test)2 SnapshotListener (com.alibaba.graphscope.groot.SnapshotListener)1 DdlWriter (com.alibaba.graphscope.groot.coordinator.DdlWriter)1 GraphDefFetcher (com.alibaba.graphscope.groot.coordinator.GraphDefFetcher)1 SchemaManager (com.alibaba.graphscope.groot.coordinator.SchemaManager)1 SnapshotManager (com.alibaba.graphscope.groot.coordinator.SnapshotManager)1 MetaService (com.alibaba.graphscope.groot.meta.MetaService)1 Operation (com.alibaba.graphscope.groot.operation.Operation)1 OperationBatch (com.alibaba.graphscope.groot.operation.OperationBatch)1 DdlExecutors (com.alibaba.graphscope.groot.schema.ddl.DdlExecutors)1 DdlResult (com.alibaba.graphscope.groot.schema.ddl.DdlResult)1 CreateVertexTypeRequest (com.alibaba.graphscope.groot.schema.request.CreateVertexTypeRequest)1 DdlRequestBatch (com.alibaba.graphscope.groot.schema.request.DdlRequestBatch)1 ServiceNotReadyException (com.alibaba.maxgraph.compiler.api.exception.ServiceNotReadyException)1 WriteIngestorRequest (com.alibaba.maxgraph.proto.groot.WriteIngestorRequest)1 WriteIngestorResponse (com.alibaba.maxgraph.proto.groot.WriteIngestorResponse)1 PropertyDef (com.alibaba.maxgraph.sdkcommon.schema.PropertyDef)1 PropertyValue (com.alibaba.maxgraph.sdkcommon.schema.PropertyValue)1