Search in sources :

Example 1 with ProcessorMetaSupplier

use of com.hazelcast.jet.core.ProcessorMetaSupplier in project hazelcast by hazelcast.

the class StreamSqlConnector method fullScanReader.

@Nonnull
@Override
public Vertex fullScanReader(@Nonnull DAG dag, @Nonnull Table table0, @Nullable Expression<Boolean> predicate, @Nonnull List<Expression<?>> projections, @Nullable FunctionEx<ExpressionEvalContext, EventTimePolicy<JetSqlRow>> eventTimePolicyProvider) {
    if (eventTimePolicyProvider != null) {
        throw QueryException.error("Ordering functions are not supported on top of " + TYPE_NAME + " mappings");
    }
    StreamTable table = (StreamTable) table0;
    StreamSourceTransform<JetSqlRow> source = (StreamSourceTransform<JetSqlRow>) table.items(predicate, projections);
    ProcessorMetaSupplier pms = source.metaSupplierFn.apply(EventTimePolicy.noEventTime());
    return dag.newUniqueVertex(table.toString(), pms);
}
Also used : StreamSourceTransform(com.hazelcast.jet.impl.pipeline.transform.StreamSourceTransform) JetSqlRow(com.hazelcast.sql.impl.row.JetSqlRow) ProcessorMetaSupplier(com.hazelcast.jet.core.ProcessorMetaSupplier) Nonnull(javax.annotation.Nonnull)

Example 2 with ProcessorMetaSupplier

use of com.hazelcast.jet.core.ProcessorMetaSupplier in project hazelcast by hazelcast.

the class SeriesSqlConnector method fullScanReader.

@Nonnull
@Override
public Vertex fullScanReader(@Nonnull DAG dag, @Nonnull Table table0, @Nullable Expression<Boolean> predicate, @Nonnull List<Expression<?>> projections, @Nullable FunctionEx<ExpressionEvalContext, EventTimePolicy<JetSqlRow>> eventTimePolicyProvider) {
    if (eventTimePolicyProvider != null) {
        throw QueryException.error("Ordering functions are not supported on top of " + TYPE_NAME + " mappings");
    }
    SeriesTable table = (SeriesTable) table0;
    BatchSource<JetSqlRow> source = table.items(predicate, projections);
    ProcessorMetaSupplier pms = ((BatchSourceTransform<JetSqlRow>) source).metaSupplier;
    return dag.newUniqueVertex(table.toString(), pms);
}
Also used : JetSqlRow(com.hazelcast.sql.impl.row.JetSqlRow) BatchSourceTransform(com.hazelcast.jet.impl.pipeline.transform.BatchSourceTransform) ProcessorMetaSupplier(com.hazelcast.jet.core.ProcessorMetaSupplier) Nonnull(javax.annotation.Nonnull)

Example 3 with ProcessorMetaSupplier

use of com.hazelcast.jet.core.ProcessorMetaSupplier in project hazelcast by hazelcast.

the class JoinByEquiJoinProcessorTest method runTest.

private void runTest(JoinRelType joinType, Expression<Boolean> rowProjectorCondition, Expression<?> rowProjectorProjection, Expression<Boolean> nonEquiCondition, List<JetSqlRow> input, List<JetSqlRow> output) {
    KvRowProjector.Supplier projectorSupplier = KvRowProjector.supplier(new QueryPath[] { QueryPath.KEY_PATH, QueryPath.VALUE_PATH }, new QueryDataType[] { QueryDataType.INT, VARCHAR }, GenericQueryTargetDescriptor.DEFAULT, GenericQueryTargetDescriptor.DEFAULT, rowProjectorCondition, singletonList(rowProjectorProjection));
    ProcessorMetaSupplier processor = JoinByEquiJoinProcessorSupplier.supplier(new JetJoinInfo(joinType, new int[] { 0 }, new int[] { 0 }, nonEquiCondition, null), MAP_NAME, projectorSupplier);
    TestSupport.verifyProcessor(adaptSupplier(processor)).jobConfig(new JobConfig().setArgument(SQL_ARGUMENTS_KEY_NAME, emptyList())).input(input).hazelcastInstance(instance()).outputChecker(SqlTestSupport::compareRowLists).disableProgressAssertion().expectOutput(output);
}
Also used : JetJoinInfo(com.hazelcast.jet.sql.impl.JetJoinInfo) KvRowProjector(com.hazelcast.jet.sql.impl.connector.keyvalue.KvRowProjector) SqlTestSupport(com.hazelcast.jet.sql.SqlTestSupport) ProcessorMetaSupplier(com.hazelcast.jet.core.ProcessorMetaSupplier) JobConfig(com.hazelcast.jet.config.JobConfig)

Example 4 with ProcessorMetaSupplier

use of com.hazelcast.jet.core.ProcessorMetaSupplier in project hazelcast by hazelcast.

the class JmsSourceBuilder method build.

/**
 * Creates and returns the JMS {@link StreamSource} with the supplied
 * components and the projection function {@code projectionFn}.
 * <p>
 * The given function must be stateless.
 *
 * @param projectionFn the function which creates output object from each
 *                    message
 * @param <T> the type of the items the source emits
 */
@Nonnull
public <T> StreamSource<T> build(@Nonnull FunctionEx<? super Message, ? extends T> projectionFn) {
    String usernameLocal = username;
    String passwordLocal = password;
    String destinationLocal = destinationName;
    ProcessingGuarantee maxGuaranteeLocal = maxGuarantee;
    @SuppressWarnings("UnnecessaryLocalVariable") boolean isTopicLocal = isTopic;
    if (connectionFn == null) {
        connectionFn = factory -> requireNonNull(usernameLocal != null || passwordLocal != null ? factory.createConnection(usernameLocal, passwordLocal) : factory.createConnection());
    }
    if (consumerFn == null) {
        checkNotNull(destinationLocal, "neither consumerFn nor destinationName set");
        consumerFn = session -> session.createConsumer(isTopicLocal ? session.createTopic(destinationLocal) : session.createQueue(destinationLocal));
        if (isTopic) {
            // the user didn't specify a custom consumerFn and we know we're using a non-durable consumer
            // for a topic - there's no point in using any guarantee, see `maxGuarantee`
            maxGuaranteeLocal = ProcessingGuarantee.NONE;
        }
    }
    ProcessingGuarantee maxGuaranteeFinal = maxGuaranteeLocal;
    FunctionEx<? super ConnectionFactory, ? extends Connection> connectionFnLocal = connectionFn;
    @SuppressWarnings("UnnecessaryLocalVariable") SupplierEx<? extends ConnectionFactory> factorySupplierLocal = factorySupplier;
    SupplierEx<? extends Connection> newConnectionFn = () -> connectionFnLocal.apply(factorySupplierLocal.get());
    FunctionEx<? super Session, ? extends MessageConsumer> consumerFnLocal = consumerFn;
    boolean isSharedConsumerLocal = isSharedConsumer;
    FunctionEx<? super Message, ?> messageIdFnLocal = messageIdFn;
    FunctionEx<EventTimePolicy<? super T>, ProcessorMetaSupplier> metaSupplierFactory = policy -> isTopicLocal ? streamJmsTopicP(destinationLocal, isSharedConsumerLocal, maxGuaranteeFinal, policy, newConnectionFn, consumerFnLocal, messageIdFnLocal, projectionFn) : streamJmsQueueP(destinationLocal, maxGuaranteeFinal, policy, newConnectionFn, consumerFnLocal, messageIdFnLocal, projectionFn);
    return Sources.streamFromProcessorWithWatermarks(sourceName(), true, metaSupplierFactory);
}
Also used : ProcessingGuarantee(com.hazelcast.jet.config.ProcessingGuarantee) FunctionEx(com.hazelcast.function.FunctionEx) Connection(javax.jms.Connection) Util.checkSerializable(com.hazelcast.jet.impl.util.Util.checkSerializable) ProcessorMetaSupplier(com.hazelcast.jet.core.ProcessorMetaSupplier) EventTimePolicy(com.hazelcast.jet.core.EventTimePolicy) Preconditions.checkNotNull(com.hazelcast.internal.util.Preconditions.checkNotNull) SupplierEx(com.hazelcast.function.SupplierEx) Session(javax.jms.Session) MessageConsumer(javax.jms.MessageConsumer) SourceProcessors.streamJmsTopicP(com.hazelcast.jet.core.processor.SourceProcessors.streamJmsTopicP) Objects.requireNonNull(java.util.Objects.requireNonNull) SourceProcessors.streamJmsQueueP(com.hazelcast.jet.core.processor.SourceProcessors.streamJmsQueueP) ProcessingGuarantee(com.hazelcast.jet.config.ProcessingGuarantee) Nonnull(javax.annotation.Nonnull) Message(javax.jms.Message) Nullable(javax.annotation.Nullable) ConnectionFactory(javax.jms.ConnectionFactory) EventTimePolicy(com.hazelcast.jet.core.EventTimePolicy) ProcessorMetaSupplier(com.hazelcast.jet.core.ProcessorMetaSupplier) Nonnull(javax.annotation.Nonnull)

Example 5 with ProcessorMetaSupplier

use of com.hazelcast.jet.core.ProcessorMetaSupplier in project hazelcast by hazelcast.

the class SourceProcessors method streamJmsTopicP.

/**
 * Returns a supplier of processors for {@link Sources#jmsTopicBuilder}.
 *
 * @param isSharedConsumer true, if {@code createSharedConsumer()} or
 *      {@code createSharedDurableConsumer()} was used to create the
 *      consumer in the {@code consumerFn}
 * @param maxGuarantee maximum processing guarantee for the source. You can
 *      use it to disable acknowledging in transactions to save transaction
 *      overhead
 */
@Nonnull
public static <T> ProcessorMetaSupplier streamJmsTopicP(@Nullable String destination, boolean isSharedConsumer, @Nonnull ProcessingGuarantee maxGuarantee, @Nonnull EventTimePolicy<? super T> eventTimePolicy, @Nonnull SupplierEx<? extends Connection> newConnectionFn, @Nonnull FunctionEx<? super Session, ? extends MessageConsumer> consumerFn, @Nonnull FunctionEx<? super Message, ?> messageIdFn, @Nonnull FunctionEx<? super Message, ? extends T> projectionFn) {
    ProcessorSupplier pSupplier = new StreamJmsP.Supplier<>(destination, maxGuarantee, eventTimePolicy, newConnectionFn, consumerFn, messageIdFn, projectionFn);
    ConnectorPermission permission = ConnectorPermission.jms(destination, ACTION_READ);
    return isSharedConsumer ? ProcessorMetaSupplier.preferLocalParallelismOne(permission, pSupplier) : ProcessorMetaSupplier.forceTotalParallelismOne(pSupplier, newUnsecureUuidString(), permission);
}
Also used : ConnectorPermission(com.hazelcast.security.permission.ConnectorPermission) ProcessorSupplier(com.hazelcast.jet.core.ProcessorSupplier) ProcessorMetaSupplier(com.hazelcast.jet.core.ProcessorMetaSupplier) StreamEventJournalP.streamRemoteCacheSupplier(com.hazelcast.jet.impl.connector.StreamEventJournalP.streamRemoteCacheSupplier) ProcessorSupplier(com.hazelcast.jet.core.ProcessorSupplier) Nonnull(javax.annotation.Nonnull)

Aggregations

ProcessorMetaSupplier (com.hazelcast.jet.core.ProcessorMetaSupplier)21 ProcessorSupplier (com.hazelcast.jet.core.ProcessorSupplier)11 Nonnull (javax.annotation.Nonnull)8 Test (org.junit.Test)6 Vertex (com.hazelcast.jet.core.Vertex)5 QuickTest (com.hazelcast.test.annotation.QuickTest)5 List (java.util.List)5 Address (com.hazelcast.cluster.Address)4 FunctionEx (com.hazelcast.function.FunctionEx)4 DAG (com.hazelcast.jet.core.DAG)4 TestProcessorContext (com.hazelcast.jet.core.test.TestProcessorContext)4 JetSqlRow (com.hazelcast.sql.impl.row.JetSqlRow)4 ParallelJVMTest (com.hazelcast.test.annotation.ParallelJVMTest)4 Map (java.util.Map)4 EventTimePolicy (com.hazelcast.jet.core.EventTimePolicy)3 Function (java.util.function.Function)3 Nullable (javax.annotation.Nullable)3 HazelcastInstance (com.hazelcast.core.HazelcastInstance)2 MemberInfo (com.hazelcast.internal.cluster.MemberInfo)2 EdgeConfig (com.hazelcast.jet.config.EdgeConfig)2