Search in sources :

Example 1 with SinkSpec

use of org.apache.pulsar.functions.proto.Function.SinkSpec in project pulsar by apache.

the class FunctionActioner method getBuiltinArchive.

private File getBuiltinArchive(FunctionDetails.Builder functionDetails) throws IOException, ClassNotFoundException {
    if (functionDetails.hasSource()) {
        SourceSpec sourceSpec = functionDetails.getSource();
        if (!StringUtils.isEmpty(sourceSpec.getBuiltin())) {
            Connector connector = connectorsManager.getConnector(sourceSpec.getBuiltin());
            File archive = connector.getArchivePath().toFile();
            String sourceClass = connector.getConnectorDefinition().getSourceClass();
            SourceSpec.Builder builder = SourceSpec.newBuilder(functionDetails.getSource());
            builder.setClassName(sourceClass);
            functionDetails.setSource(builder);
            fillSourceTypeClass(functionDetails, connector.getClassLoader(), sourceClass);
            return archive;
        }
    }
    if (functionDetails.hasSink()) {
        SinkSpec sinkSpec = functionDetails.getSink();
        if (!StringUtils.isEmpty(sinkSpec.getBuiltin())) {
            Connector connector = connectorsManager.getConnector(sinkSpec.getBuiltin());
            File archive = connector.getArchivePath().toFile();
            String sinkClass = connector.getConnectorDefinition().getSinkClass();
            SinkSpec.Builder builder = SinkSpec.newBuilder(functionDetails.getSink());
            builder.setClassName(sinkClass);
            functionDetails.setSink(builder);
            fillSinkTypeClass(functionDetails, connector.getClassLoader(), sinkClass);
            return archive;
        }
    }
    if (!StringUtils.isEmpty(functionDetails.getBuiltin())) {
        return functionsManager.getFunctionArchive(functionDetails.getBuiltin()).toFile();
    }
    throw new IOException("Could not find built in archive definition");
}
Also used : Connector(org.apache.pulsar.functions.utils.io.Connector) SourceSpec(org.apache.pulsar.functions.proto.Function.SourceSpec) IOException(java.io.IOException) File(java.io.File) SinkSpec(org.apache.pulsar.functions.proto.Function.SinkSpec)

Example 2 with SinkSpec

use of org.apache.pulsar.functions.proto.Function.SinkSpec in project pulsar by apache.

the class FunctionApiV2ResourceTest method testGetFunctionSuccess.

@Test
public void testGetFunctionSuccess() throws IOException {
    mockInstanceUtils();
    when(mockedManager.containsFunction(eq(tenant), eq(namespace), eq(function))).thenReturn(true);
    SinkSpec sinkSpec = SinkSpec.newBuilder().setTopic(outputTopic).setSerDeClassName(outputSerdeClassName).build();
    FunctionDetails functionDetails = FunctionDetails.newBuilder().setClassName(className).setSink(sinkSpec).setName(function).setNamespace(namespace).setProcessingGuarantees(ProcessingGuarantees.ATMOST_ONCE).setTenant(tenant).setParallelism(parallelism).setSource(SourceSpec.newBuilder().setSubscriptionType(subscriptionType).putAllTopicsToSerDeClassName(topicsToSerDeClassName)).build();
    FunctionMetaData metaData = FunctionMetaData.newBuilder().setCreateTime(System.currentTimeMillis()).setFunctionDetails(functionDetails).setPackageLocation(PackageLocationMetaData.newBuilder().setPackagePath("/path/to/package")).setVersion(1234).build();
    when(mockedManager.getFunctionMetaData(eq(tenant), eq(namespace), eq(function))).thenReturn(metaData);
    FunctionDetails actual = getDefaultFunctionInfo();
    assertEquals(functionDetails, actual);
}
Also used : FunctionMetaData(org.apache.pulsar.functions.proto.Function.FunctionMetaData) FunctionDetails(org.apache.pulsar.functions.proto.Function.FunctionDetails) SinkSpec(org.apache.pulsar.functions.proto.Function.SinkSpec) Test(org.testng.annotations.Test)

Example 3 with SinkSpec

use of org.apache.pulsar.functions.proto.Function.SinkSpec in project pulsar by apache.

the class SourceApiV3ResourceTest method testGetSourceSuccess.

@Test
public void testGetSourceSuccess() {
    when(mockedManager.containsFunction(eq(tenant), eq(namespace), eq(source))).thenReturn(true);
    SourceSpec sourceSpec = SourceSpec.newBuilder().setBuiltin("jdbc").build();
    SinkSpec sinkSpec = SinkSpec.newBuilder().setTopic(outputTopic).setSerDeClassName(outputSerdeClassName).build();
    FunctionDetails functionDetails = FunctionDetails.newBuilder().setClassName(IdentityFunction.class.getName()).setSink(sinkSpec).setName(source).setNamespace(namespace).setProcessingGuarantees(ProcessingGuarantees.ATLEAST_ONCE).setRuntime(FunctionDetails.Runtime.JAVA).setAutoAck(true).setTenant(tenant).setParallelism(parallelism).setSource(sourceSpec).build();
    FunctionMetaData metaData = FunctionMetaData.newBuilder().setCreateTime(System.currentTimeMillis()).setFunctionDetails(functionDetails).setPackageLocation(PackageLocationMetaData.newBuilder().setPackagePath("/path/to/package")).setVersion(1234).build();
    when(mockedManager.getFunctionMetaData(eq(tenant), eq(namespace), eq(source))).thenReturn(metaData);
    SourceConfig config = getDefaultSourceInfo();
    assertEquals(SourceConfigUtils.convertFromDetails(functionDetails), config);
}
Also used : FunctionMetaData(org.apache.pulsar.functions.proto.Function.FunctionMetaData) FunctionDetails(org.apache.pulsar.functions.proto.Function.FunctionDetails) SourceSpec(org.apache.pulsar.functions.proto.Function.SourceSpec) IdentityFunction(org.apache.pulsar.functions.api.utils.IdentityFunction) SourceConfig(org.apache.pulsar.common.io.SourceConfig) SinkSpec(org.apache.pulsar.functions.proto.Function.SinkSpec) Test(org.testng.annotations.Test)

Example 4 with SinkSpec

use of org.apache.pulsar.functions.proto.Function.SinkSpec in project pulsar by yahoo.

the class FunctionActioner method getBuiltinArchive.

private File getBuiltinArchive(FunctionDetails.Builder functionDetails) throws IOException, ClassNotFoundException {
    if (functionDetails.hasSource()) {
        SourceSpec sourceSpec = functionDetails.getSource();
        if (!StringUtils.isEmpty(sourceSpec.getBuiltin())) {
            Connector connector = connectorsManager.getConnector(sourceSpec.getBuiltin());
            File archive = connector.getArchivePath().toFile();
            String sourceClass = connector.getConnectorDefinition().getSourceClass();
            SourceSpec.Builder builder = SourceSpec.newBuilder(functionDetails.getSource());
            builder.setClassName(sourceClass);
            functionDetails.setSource(builder);
            fillSourceTypeClass(functionDetails, connector.getClassLoader(), sourceClass);
            return archive;
        }
    }
    if (functionDetails.hasSink()) {
        SinkSpec sinkSpec = functionDetails.getSink();
        if (!StringUtils.isEmpty(sinkSpec.getBuiltin())) {
            Connector connector = connectorsManager.getConnector(sinkSpec.getBuiltin());
            File archive = connector.getArchivePath().toFile();
            String sinkClass = connector.getConnectorDefinition().getSinkClass();
            SinkSpec.Builder builder = SinkSpec.newBuilder(functionDetails.getSink());
            builder.setClassName(sinkClass);
            functionDetails.setSink(builder);
            fillSinkTypeClass(functionDetails, connector.getClassLoader(), sinkClass);
            return archive;
        }
    }
    if (!StringUtils.isEmpty(functionDetails.getBuiltin())) {
        return functionsManager.getFunctionArchive(functionDetails.getBuiltin()).toFile();
    }
    throw new IOException("Could not find built in archive definition");
}
Also used : Connector(org.apache.pulsar.functions.utils.io.Connector) SourceSpec(org.apache.pulsar.functions.proto.Function.SourceSpec) IOException(java.io.IOException) File(java.io.File) SinkSpec(org.apache.pulsar.functions.proto.Function.SinkSpec)

Example 5 with SinkSpec

use of org.apache.pulsar.functions.proto.Function.SinkSpec in project pulsar by yahoo.

the class FunctionActioner method fillSourceTypeClass.

private void fillSourceTypeClass(FunctionDetails.Builder functionDetails, ClassLoader narClassLoader, String className) throws ClassNotFoundException {
    String typeArg = getSourceType(className, narClassLoader).getName();
    SourceSpec.Builder sourceBuilder = SourceSpec.newBuilder(functionDetails.getSource());
    sourceBuilder.setTypeClassName(typeArg);
    functionDetails.setSource(sourceBuilder);
    SinkSpec sinkSpec = functionDetails.getSink();
    if (null == sinkSpec || StringUtils.isEmpty(sinkSpec.getTypeClassName())) {
        SinkSpec.Builder sinkBuilder = SinkSpec.newBuilder(sinkSpec);
        sinkBuilder.setTypeClassName(typeArg);
        functionDetails.setSink(sinkBuilder);
    }
}
Also used : SourceSpec(org.apache.pulsar.functions.proto.Function.SourceSpec) SinkSpec(org.apache.pulsar.functions.proto.Function.SinkSpec)

Aggregations

SinkSpec (org.apache.pulsar.functions.proto.Function.SinkSpec)18 FunctionDetails (org.apache.pulsar.functions.proto.Function.FunctionDetails)9 FunctionMetaData (org.apache.pulsar.functions.proto.Function.FunctionMetaData)9 SourceSpec (org.apache.pulsar.functions.proto.Function.SourceSpec)9 Test (org.testng.annotations.Test)9 IOException (java.io.IOException)6 TypeReference (com.fasterxml.jackson.core.type.TypeReference)3 File (java.io.File)3 PulsarClientException (org.apache.pulsar.client.api.PulsarClientException)3 FunctionConfig (org.apache.pulsar.common.functions.FunctionConfig)3 ProducerConfig (org.apache.pulsar.common.functions.ProducerConfig)3 SourceConfig (org.apache.pulsar.common.io.SourceConfig)3 Function (org.apache.pulsar.functions.api.Function)3 IdentityFunction (org.apache.pulsar.functions.api.utils.IdentityFunction)3 PulsarSink (org.apache.pulsar.functions.sink.PulsarSink)3 PulsarSinkConfig (org.apache.pulsar.functions.sink.PulsarSinkConfig)3 Connector (org.apache.pulsar.functions.utils.io.Connector)3 Sink (org.apache.pulsar.io.core.Sink)3