Search in sources :

Example 6 with HttpFlowFileServerProtocol

use of org.apache.nifi.remote.protocol.http.HttpFlowFileServerProtocol in project nifi by apache.

the class DataTransferResource method receiveFlowFiles.

@POST
@Consumes(MediaType.APPLICATION_OCTET_STREAM)
@Produces(MediaType.TEXT_PLAIN)
@Path("input-ports/{portId}/transactions/{transactionId}/flow-files")
@ApiOperation(value = "Transfer flow files to the input port", response = String.class, authorizations = { @Authorization(value = "Write - /data-transfer/input-ports/{uuid}") })
@ApiResponses(value = { @ApiResponse(code = 400, message = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), @ApiResponse(code = 401, message = "Client could not be authenticated."), @ApiResponse(code = 403, message = "Client is not authorized to make this request."), @ApiResponse(code = 404, message = "The specified resource could not be found."), @ApiResponse(code = 409, message = "The request was valid but NiFi was not in the appropriate state to process it. Retrying the same request later may be successful."), @ApiResponse(code = 503, message = "NiFi instance is not ready for serving request, or temporarily overloaded. Retrying the same request later may be successful") })
public Response receiveFlowFiles(@ApiParam(value = "The input port id.", required = true) @PathParam("portId") String portId, @PathParam("transactionId") String transactionId, @Context HttpServletRequest req, @Context ServletContext context, InputStream inputStream) {
    // authorize access
    serviceFacade.authorizeAccess(lookup -> {
        authorizeDataTransfer(lookup, ResourceType.InputPort, portId);
    });
    final ValidateRequestResult validationResult = validateResult(req, portId, transactionId);
    if (validationResult.errResponse != null) {
        return validationResult.errResponse;
    }
    logger.debug("receiveFlowFiles request: portId={}", portId);
    final ByteArrayOutputStream out = new ByteArrayOutputStream();
    final Peer peer = constructPeer(req, inputStream, out, portId, transactionId);
    final int transportProtocolVersion = validationResult.transportProtocolVersion;
    try {
        HttpFlowFileServerProtocol serverProtocol = initiateServerProtocol(req, peer, transportProtocolVersion);
        int numOfFlowFiles = serverProtocol.getPort().receiveFlowFiles(peer, serverProtocol);
        logger.debug("finished receiving flow files, numOfFlowFiles={}", numOfFlowFiles);
        if (numOfFlowFiles < 1) {
            return Response.status(Response.Status.BAD_REQUEST).entity("Client should send request when there is data to send. There was no flow file sent.").build();
        }
    } catch (HandshakeException e) {
        return responseCreator.handshakeExceptionResponse(e);
    } catch (NotAuthorizedException e) {
        return responseCreator.unauthorizedResponse(e);
    } catch (BadRequestException | RequestExpiredException e) {
        return responseCreator.badRequestResponse(e);
    } catch (Exception e) {
        return responseCreator.unexpectedErrorResponse(portId, e);
    }
    String serverChecksum = ((HttpServerCommunicationsSession) peer.getCommunicationsSession()).getChecksum();
    return responseCreator.acceptedResponse(transactionManager, serverChecksum, transportProtocolVersion);
}
Also used : HttpServerCommunicationsSession(org.apache.nifi.remote.io.http.HttpServerCommunicationsSession) Peer(org.apache.nifi.remote.Peer) StandardHttpFlowFileServerProtocol(org.apache.nifi.remote.protocol.http.StandardHttpFlowFileServerProtocol) HttpFlowFileServerProtocol(org.apache.nifi.remote.protocol.http.HttpFlowFileServerProtocol) RequestExpiredException(org.apache.nifi.remote.exception.RequestExpiredException) ByteArrayOutputStream(org.apache.nifi.stream.io.ByteArrayOutputStream) NotAuthorizedException(org.apache.nifi.remote.exception.NotAuthorizedException) BadRequestException(org.apache.nifi.remote.exception.BadRequestException) HandshakeException(org.apache.nifi.remote.exception.HandshakeException) WebApplicationException(javax.ws.rs.WebApplicationException) AccessDeniedException(org.apache.nifi.authorization.AccessDeniedException) NotAuthorizedException(org.apache.nifi.remote.exception.NotAuthorizedException) IOException(java.io.IOException) UnknownHostException(java.net.UnknownHostException) RequestExpiredException(org.apache.nifi.remote.exception.RequestExpiredException) BadRequestException(org.apache.nifi.remote.exception.BadRequestException) HandshakeException(org.apache.nifi.remote.exception.HandshakeException) Path(javax.ws.rs.Path) POST(javax.ws.rs.POST) Consumes(javax.ws.rs.Consumes) Produces(javax.ws.rs.Produces) ApiOperation(io.swagger.annotations.ApiOperation) ApiResponses(io.swagger.annotations.ApiResponses)

Example 7 with HttpFlowFileServerProtocol

use of org.apache.nifi.remote.protocol.http.HttpFlowFileServerProtocol in project nifi by apache.

the class DataTransferResource method initiateServerProtocol.

private HttpFlowFileServerProtocol initiateServerProtocol(final HttpServletRequest req, final Peer peer, final Integer transportProtocolVersion) throws IOException {
    // Switch transaction protocol version based on transport protocol version.
    TransportProtocolVersionNegotiator negotiatedTransportProtocolVersion = new TransportProtocolVersionNegotiator(transportProtocolVersion);
    VersionNegotiator versionNegotiator = new StandardVersionNegotiator(negotiatedTransportProtocolVersion.getTransactionProtocolVersion());
    final String dataTransferUrl = req.getRequestURL().toString();
    ((HttpCommunicationsSession) peer.getCommunicationsSession()).setDataTransferUrl(dataTransferUrl);
    HttpFlowFileServerProtocol serverProtocol = getHttpFlowFileServerProtocol(versionNegotiator);
    HttpRemoteSiteListener.getInstance(nifiProperties).setupServerProtocol(serverProtocol);
    serverProtocol.handshake(peer);
    return serverProtocol;
}
Also used : TransportProtocolVersionNegotiator(org.apache.nifi.remote.client.http.TransportProtocolVersionNegotiator) VersionNegotiator(org.apache.nifi.remote.VersionNegotiator) TransportProtocolVersionNegotiator(org.apache.nifi.remote.client.http.TransportProtocolVersionNegotiator) StandardVersionNegotiator(org.apache.nifi.remote.StandardVersionNegotiator) HttpCommunicationsSession(org.apache.nifi.remote.io.http.HttpCommunicationsSession) StandardHttpFlowFileServerProtocol(org.apache.nifi.remote.protocol.http.StandardHttpFlowFileServerProtocol) HttpFlowFileServerProtocol(org.apache.nifi.remote.protocol.http.HttpFlowFileServerProtocol) StandardVersionNegotiator(org.apache.nifi.remote.StandardVersionNegotiator)

Example 8 with HttpFlowFileServerProtocol

use of org.apache.nifi.remote.protocol.http.HttpFlowFileServerProtocol in project nifi by apache.

the class DataTransferResource method commitInputPortTransaction.

@DELETE
@Consumes(MediaType.APPLICATION_OCTET_STREAM)
@Produces(MediaType.APPLICATION_JSON)
@Path("input-ports/{portId}/transactions/{transactionId}")
@ApiOperation(value = "Commit or cancel the specified transaction", response = TransactionResultEntity.class, authorizations = { @Authorization(value = "Write - /data-transfer/input-ports/{uuid}") })
@ApiResponses(value = { @ApiResponse(code = 400, message = "NiFi was unable to complete the request because it was invalid. The request should not be retried without modification."), @ApiResponse(code = 401, message = "Client could not be authenticated."), @ApiResponse(code = 403, message = "Client is not authorized to make this request."), @ApiResponse(code = 404, message = "The specified resource could not be found."), @ApiResponse(code = 409, message = "The request was valid but NiFi was not in the appropriate state to process it. Retrying the same request later may be successful."), @ApiResponse(code = 503, message = "NiFi instance is not ready for serving request, or temporarily overloaded. Retrying the same request later may be successful") })
public Response commitInputPortTransaction(@ApiParam(value = "The response code. Available values are BAD_CHECKSUM(19), CONFIRM_TRANSACTION(12) or CANCEL_TRANSACTION(15).", required = true) @QueryParam(RESPONSE_CODE) Integer responseCode, @ApiParam(value = "The input port id.", required = true) @PathParam("portId") String portId, @ApiParam(value = "The transaction id.", required = true) @PathParam("transactionId") String transactionId, @Context HttpServletRequest req, @Context ServletContext context, InputStream inputStream) {
    // authorize access
    serviceFacade.authorizeAccess(lookup -> {
        authorizeDataTransfer(lookup, ResourceType.InputPort, portId);
    });
    final ValidateRequestResult validationResult = validateResult(req, portId, transactionId);
    if (validationResult.errResponse != null) {
        return validationResult.errResponse;
    }
    logger.debug("commitInputPortTransaction request: portId={}, transactionId={}, responseCode={}", portId, transactionId, responseCode);
    final int transportProtocolVersion = validationResult.transportProtocolVersion;
    final ByteArrayOutputStream out = new ByteArrayOutputStream();
    final Peer peer = constructPeer(req, inputStream, out, portId, transactionId);
    final TransactionResultEntity entity = new TransactionResultEntity();
    try {
        HttpFlowFileServerProtocol serverProtocol = initiateServerProtocol(req, peer, transportProtocolVersion);
        HttpServerCommunicationsSession commsSession = (HttpServerCommunicationsSession) peer.getCommunicationsSession();
        // Pass the response code sent from the client.
        String inputErrMessage = null;
        if (responseCode == null) {
            inputErrMessage = "responseCode is required.";
        } else if (ResponseCode.BAD_CHECKSUM.getCode() != responseCode && ResponseCode.CONFIRM_TRANSACTION.getCode() != responseCode && ResponseCode.CANCEL_TRANSACTION.getCode() != responseCode) {
            inputErrMessage = "responseCode " + responseCode + " is invalid. ";
        }
        if (inputErrMessage != null) {
            entity.setMessage(inputErrMessage);
            entity.setResponseCode(ResponseCode.ABORT.getCode());
            return Response.status(Response.Status.BAD_REQUEST).entity(entity).build();
        }
        if (ResponseCode.CANCEL_TRANSACTION.getCode() == responseCode) {
            return cancelTransaction(transactionId, entity);
        }
        commsSession.setResponseCode(ResponseCode.fromCode(responseCode));
        try {
            int flowFileSent = serverProtocol.commitReceiveTransaction(peer);
            entity.setResponseCode(commsSession.getResponseCode().getCode());
            entity.setFlowFileSent(flowFileSent);
        } catch (IOException e) {
            if (ResponseCode.BAD_CHECKSUM.getCode() == responseCode && e.getMessage().contains("Received a BadChecksum response")) {
                // AbstractFlowFileServerProtocol throws IOException after it canceled transaction.
                // This is a known behavior and if we return 500 with this exception,
                // it's not clear if there is an issue at server side, or cancel operation has been accomplished.
                // Above conditions can guarantee this is the latter case, we return 200 OK here.
                entity.setResponseCode(ResponseCode.CANCEL_TRANSACTION.getCode());
                return noCache(Response.ok(entity)).build();
            } else {
                return responseCreator.unexpectedErrorResponse(portId, transactionId, e);
            }
        }
    } catch (HandshakeException e) {
        return responseCreator.handshakeExceptionResponse(e);
    } catch (Exception e) {
        return responseCreator.unexpectedErrorResponse(portId, transactionId, e);
    }
    return noCache(setCommonHeaders(Response.ok(entity), transportProtocolVersion, transactionManager)).build();
}
Also used : TransactionResultEntity(org.apache.nifi.web.api.entity.TransactionResultEntity) HttpServerCommunicationsSession(org.apache.nifi.remote.io.http.HttpServerCommunicationsSession) Peer(org.apache.nifi.remote.Peer) StandardHttpFlowFileServerProtocol(org.apache.nifi.remote.protocol.http.StandardHttpFlowFileServerProtocol) HttpFlowFileServerProtocol(org.apache.nifi.remote.protocol.http.HttpFlowFileServerProtocol) ByteArrayOutputStream(org.apache.nifi.stream.io.ByteArrayOutputStream) IOException(java.io.IOException) HandshakeException(org.apache.nifi.remote.exception.HandshakeException) BadRequestException(org.apache.nifi.remote.exception.BadRequestException) HandshakeException(org.apache.nifi.remote.exception.HandshakeException) WebApplicationException(javax.ws.rs.WebApplicationException) AccessDeniedException(org.apache.nifi.authorization.AccessDeniedException) NotAuthorizedException(org.apache.nifi.remote.exception.NotAuthorizedException) IOException(java.io.IOException) UnknownHostException(java.net.UnknownHostException) RequestExpiredException(org.apache.nifi.remote.exception.RequestExpiredException) Path(javax.ws.rs.Path) DELETE(javax.ws.rs.DELETE) Consumes(javax.ws.rs.Consumes) Produces(javax.ws.rs.Produces) ApiOperation(io.swagger.annotations.ApiOperation) ApiResponses(io.swagger.annotations.ApiResponses)

Example 9 with HttpFlowFileServerProtocol

use of org.apache.nifi.remote.protocol.http.HttpFlowFileServerProtocol in project nifi by apache.

the class TestDataTransferResource method testCommitOutputPortTransactionBadChecksum.

@Test
public void testCommitOutputPortTransactionBadChecksum() throws Exception {
    final HttpServletRequest req = createCommonHttpServletRequest();
    final DataTransferResource resource = getDataTransferResource();
    final HttpFlowFileServerProtocol serverProtocol = resource.getHttpFlowFileServerProtocol(null);
    doThrow(new HandshakeException(ResponseCode.BAD_CHECKSUM, "Bad checksum.")).when(serverProtocol).commitTransferTransaction(any(), any());
    final ServletContext context = null;
    final InputStream inputStream = null;
    final HttpRemoteSiteListener transactionManager = HttpRemoteSiteListener.getInstance(NiFiProperties.createBasicNiFiProperties(null, null));
    final String transactionId = transactionManager.createTransaction();
    final Response response = resource.commitOutputPortTransaction(ResponseCode.CONFIRM_TRANSACTION.getCode(), "client-checksum", "port-id", transactionId, req, context, inputStream);
    transactionManager.cancelTransaction(transactionId);
    TransactionResultEntity resultEntity = (TransactionResultEntity) response.getEntity();
    assertEquals(400, response.getStatus());
    assertEquals(ResponseCode.BAD_CHECKSUM.getCode(), resultEntity.getResponseCode());
}
Also used : HttpServletRequest(javax.servlet.http.HttpServletRequest) HttpServletResponse(javax.servlet.http.HttpServletResponse) Response(javax.ws.rs.core.Response) TransactionResultEntity(org.apache.nifi.web.api.entity.TransactionResultEntity) InputStream(java.io.InputStream) HttpRemoteSiteListener(org.apache.nifi.remote.HttpRemoteSiteListener) HttpFlowFileServerProtocol(org.apache.nifi.remote.protocol.http.HttpFlowFileServerProtocol) ServletContext(javax.servlet.ServletContext) HandshakeException(org.apache.nifi.remote.exception.HandshakeException) Test(org.junit.Test)

Example 10 with HttpFlowFileServerProtocol

use of org.apache.nifi.remote.protocol.http.HttpFlowFileServerProtocol in project nifi by apache.

the class TestDataTransferResource method getDataTransferResource.

private DataTransferResource getDataTransferResource() {
    final NiFiServiceFacade serviceFacade = mock(NiFiServiceFacade.class);
    final HttpFlowFileServerProtocol serverProtocol = mock(HttpFlowFileServerProtocol.class);
    final DataTransferResource resource = new DataTransferResource(NiFiProperties.createBasicNiFiProperties(null, null)) {

        @Override
        protected void authorizeDataTransfer(AuthorizableLookup lookup, ResourceType resourceType, String identifier) {
        }

        @Override
        HttpFlowFileServerProtocol getHttpFlowFileServerProtocol(VersionNegotiator versionNegotiator) {
            return serverProtocol;
        }
    };
    resource.setProperties(NiFiProperties.createBasicNiFiProperties(null, null));
    resource.setServiceFacade(serviceFacade);
    return resource;
}
Also used : AuthorizableLookup(org.apache.nifi.authorization.AuthorizableLookup) NiFiServiceFacade(org.apache.nifi.web.NiFiServiceFacade) VersionNegotiator(org.apache.nifi.remote.VersionNegotiator) HttpFlowFileServerProtocol(org.apache.nifi.remote.protocol.http.HttpFlowFileServerProtocol) ResourceType(org.apache.nifi.authorization.resource.ResourceType)

Aggregations

HttpFlowFileServerProtocol (org.apache.nifi.remote.protocol.http.HttpFlowFileServerProtocol)12 HandshakeException (org.apache.nifi.remote.exception.HandshakeException)8 InputStream (java.io.InputStream)6 ServletContext (javax.servlet.ServletContext)6 HttpServletRequest (javax.servlet.http.HttpServletRequest)6 HttpServletResponse (javax.servlet.http.HttpServletResponse)6 Response (javax.ws.rs.core.Response)6 Peer (org.apache.nifi.remote.Peer)6 TransactionResultEntity (org.apache.nifi.web.api.entity.TransactionResultEntity)6 Test (org.junit.Test)6 StandardHttpFlowFileServerProtocol (org.apache.nifi.remote.protocol.http.StandardHttpFlowFileServerProtocol)5 ApiOperation (io.swagger.annotations.ApiOperation)4 ApiResponses (io.swagger.annotations.ApiResponses)4 IOException (java.io.IOException)4 UnknownHostException (java.net.UnknownHostException)4 Consumes (javax.ws.rs.Consumes)4 Path (javax.ws.rs.Path)4 Produces (javax.ws.rs.Produces)4 WebApplicationException (javax.ws.rs.WebApplicationException)4 AccessDeniedException (org.apache.nifi.authorization.AccessDeniedException)4