Examples with TransientBlobKey org.apache.flink.runtime.blob.TransientBlobKey used on opensource projects

Example 1 with TransientBlobKey

use of org.apache.flink.runtime.blob.TransientBlobKey in project flink by apache.

the class TaskExecutorTest method testLogNotFoundHandling.

@Test(timeout = 10000L)
public void testLogNotFoundHandling() throws Throwable {
    try (NetUtils.Port port = NetUtils.getAvailablePort()) {
        int dataPort = port.getPort();
        configuration.setInteger(NettyShuffleEnvironmentOptions.DATA_PORT, dataPort);
        configuration.setInteger(NettyShuffleEnvironmentOptions.NETWORK_REQUEST_BACKOFF_INITIAL, 100);
        configuration.setInteger(NettyShuffleEnvironmentOptions.NETWORK_REQUEST_BACKOFF_MAX, 200);
        configuration.setString(ConfigConstants.TASK_MANAGER_LOG_PATH_KEY, "/i/dont/exist");
        try (TaskSubmissionTestEnvironment env = new Builder(jobId).setConfiguration(configuration).setLocalCommunication(false).build()) {
            TaskExecutorGateway tmGateway = env.getTaskExecutorGateway();
            try {
                CompletableFuture<TransientBlobKey> logFuture = tmGateway.requestFileUploadByType(FileType.LOG, timeout);
                logFuture.get();
            } catch (Exception e) {
                assertThat(e.getMessage(), containsString("The file LOG does not exist on the TaskExecutor."));
            }
        }
    }
}

Example 2 with TransientBlobKey

use of org.apache.flink.runtime.blob.TransientBlobKey in project flink by apache.

the class TaskExecutor method putTransientBlobStream.

private CompletableFuture<TransientBlobKey> putTransientBlobStream(InputStream inputStream, String fileTag) {
    final TransientBlobService transientBlobService = taskExecutorBlobService.getTransientBlobService();
    final TransientBlobKey transientBlobKey;
    try {
        transientBlobKey = transientBlobService.putTransient(inputStream);
    } catch (IOException e) {
        log.debug("Could not upload file {}.", fileTag, e);
        return FutureUtils.completedExceptionally(new FlinkException("Could not upload file " + fileTag + '.', e));
    }
    return CompletableFuture.completedFuture(transientBlobKey);
}

Example 3 with TransientBlobKey

use of org.apache.flink.runtime.blob.TransientBlobKey in project flink by apache.

the class AbstractTaskManagerFileHandler method respondToRequest.

@Override
protected CompletableFuture<Void> respondToRequest(ChannelHandlerContext ctx, HttpRequest httpRequest, HandlerRequest<EmptyRequestBody> handlerRequest, RestfulGateway gateway) throws RestHandlerException {
    final ResourceID taskManagerId = handlerRequest.getPathParameter(TaskManagerIdPathParameter.class);
    String filename = getFileName(handlerRequest);
    final Tuple2<ResourceID, String> taskManagerIdAndFileName = new Tuple2<>(taskManagerId, filename);
    final CompletableFuture<TransientBlobKey> blobKeyFuture;
    try {
        blobKeyFuture = fileBlobKeys.get(taskManagerIdAndFileName);
    } catch (ExecutionException e) {
        final Throwable cause = ExceptionUtils.stripExecutionException(e);
        throw new RestHandlerException("Could not retrieve file blob key future.", HttpResponseStatus.INTERNAL_SERVER_ERROR, cause);
    }
    final CompletableFuture<Void> resultFuture = blobKeyFuture.thenAcceptAsync((TransientBlobKey blobKey) -> {
        final File file;
        try {
            file = transientBlobService.getFile(blobKey);
        } catch (IOException e) {
            throw new CompletionException(new FlinkException("Could not retrieve file from transient blob store.", e));
        }
        try {
            HandlerUtils.transferFile(ctx, file, httpRequest);
        } catch (FlinkException e) {
            throw new CompletionException(new FlinkException("Could not transfer file to client.", e));
        }
    }, ctx.executor());
    return resultFuture.whenComplete((Void ignored, Throwable throwable) -> {
        if (throwable != null) {
            log.error("Failed to transfer file from TaskExecutor {}.", taskManagerId, throwable);
            fileBlobKeys.invalidate(taskManagerId);
            final Throwable strippedThrowable = ExceptionUtils.stripCompletionException(throwable);
            if (strippedThrowable instanceof UnknownTaskExecutorException) {
                throw new CompletionException(new NotFoundException(String.format("Failed to transfer file from TaskExecutor %s because it was unknown.", taskManagerId), strippedThrowable));
            } else {
                throw new CompletionException(new FlinkException(String.format("Failed to transfer file from TaskExecutor %s.", taskManagerId), strippedThrowable));
            }
        }
    });
}