Search in sources :

Example 36 with JetInstance

use of com.hazelcast.jet.JetInstance in project beam by apache.

the class TestJetRunner method run.

@Override
public PipelineResult run(Pipeline pipeline) {
    Collection<JetInstance> instances = initMemberInstances(factory);
    try {
        PipelineResult result = delegate.run(pipeline);
        if (result instanceof FailedRunningPipelineResults) {
            throw ((FailedRunningPipelineResults) result).getCause();
        }
        result.waitUntilFinish();
        return result;
    } finally {
        killMemberInstances(instances, factory);
    }
}
Also used : JetInstance(com.hazelcast.jet.JetInstance) PipelineResult(org.apache.beam.sdk.PipelineResult)

Example 37 with JetInstance

use of com.hazelcast.jet.JetInstance in project hazelcast by hazelcast.

the class TestJob method main.

public static void main(String[] args) {
    DAG dag = new DAG();
    dag.newVertex("v", StreamNoopProcessor::new);
    JetInstance instance = Jet.bootstrappedInstance();
    instance.newJob(dag);
}
Also used : JetInstance(com.hazelcast.jet.JetInstance) DAG(com.hazelcast.jet.core.DAG)

Example 38 with JetInstance

use of com.hazelcast.jet.JetInstance in project gora by apache.

the class JetTest method testNewJetSource.

@Test
public void testNewJetSource() throws Exception {
    DataStore<Long, Pageview> dataStoreIn;
    dataStoreIn = DataStoreFactory.getDataStore(Long.class, Pageview.class, utility.getConfiguration());
    dataStoreOut = DataStoreFactory.getDataStore(Long.class, ResultPageView.class, utility.getConfiguration());
    query = dataStoreIn.newQuery();
    query.setStartKey(0L);
    query.setEndKey(55L);
    JetEngine<Long, Pageview, Long, ResultPageView> jetEngine = new JetEngine<>();
    BatchSource<JetInputOutputFormat<Long, Pageview>> fileSource = jetEngine.createDataSource(dataStoreIn, query);
    Pipeline p = Pipeline.create();
    p.drawFrom(fileSource).filter(item -> item.getValue().getIp().toString().equals("88.240.129.183")).map(e -> {
        ResultPageView resultPageView = new ResultPageView();
        resultPageView.setIp(e.getValue().getIp());
        resultPageView.setTimestamp(e.getValue().getTimestamp());
        resultPageView.setUrl(e.getValue().getUrl());
        return new JetInputOutputFormat<Long, ResultPageView>(e.getValue().getTimestamp(), resultPageView);
    }).drainTo(jetEngine.createDataSink(dataStoreOut));
    JetInstance jet = Jet.newJetInstance();
    Jet.newJetInstance();
    try {
        jet.newJob(p).join();
    } finally {
        Jet.shutdownAll();
    }
    Query<Long, ResultPageView> query = dataStoreOut.newQuery();
    Result<Long, ResultPageView> result = query.execute();
    int noOfOutputRecords = 0;
    String ip = "";
    while (result.next()) {
        noOfOutputRecords++;
        ip = result.get().getIp().toString();
        assertEquals("88.240.129.183", ip);
    }
    assertEquals(2, noOfOutputRecords);
}
Also used : AggregateOperations.counting(com.hazelcast.jet.aggregate.AggregateOperations.counting) JetInstance(com.hazelcast.jet.JetInstance) BatchSource(com.hazelcast.jet.pipeline.BatchSource) BeforeClass(org.junit.BeforeClass) Pipeline(com.hazelcast.jet.pipeline.Pipeline) DataStore(org.apache.gora.store.DataStore) Functions.wholeItem(com.hazelcast.jet.function.Functions.wholeItem) Sinks(com.hazelcast.jet.pipeline.Sinks) Test(org.junit.Test) Pageview(org.apache.gora.jet.generated.Pageview) ResultPageView(org.apache.gora.jet.generated.ResultPageView) Result(org.apache.gora.query.Result) DataStoreFactory(org.apache.gora.store.DataStoreFactory) IMap(com.hazelcast.core.IMap) HBaseTestingUtility(org.apache.hadoop.hbase.HBaseTestingUtility) Query(org.apache.gora.query.Query) Traversers.traverseArray(com.hazelcast.jet.Traversers.traverseArray) Jet(com.hazelcast.jet.Jet) Pattern(java.util.regex.Pattern) GoraException(org.apache.gora.util.GoraException) Assert.assertEquals(org.junit.Assert.assertEquals) JetInstance(com.hazelcast.jet.JetInstance) Pipeline(com.hazelcast.jet.pipeline.Pipeline) Pageview(org.apache.gora.jet.generated.Pageview) ResultPageView(org.apache.gora.jet.generated.ResultPageView) Test(org.junit.Test)

Example 39 with JetInstance

use of com.hazelcast.jet.JetInstance in project hazelcast by hazelcast.

the class HazelcastBootstrapTest method testJet_bootstrappedInstance.

@Test
public void testJet_bootstrappedInstance() {
    JetInstance jet = Jet.bootstrappedInstance();
    executeWithBootstrappedInstance(jet);
}
Also used : JetInstance(com.hazelcast.jet.JetInstance) ParallelJVMTest(com.hazelcast.test.annotation.ParallelJVMTest) QuickTest(com.hazelcast.test.annotation.QuickTest) Test(org.junit.Test)

Example 40 with JetInstance

use of com.hazelcast.jet.JetInstance in project hazelcast-jet by hazelcast.

the class StartServer method main.

/**
 * Creates a server instance of Hazelcast Jet. If the system property
 * {@code print.port} is set, the server writes the port number of the
 * Hazelcast instance to a file named by the property.
 */
public static void main(String[] args) throws Exception {
    JetInstance jet = Jet.newJetInstance();
    printMemberPort(jet.getHazelcastInstance());
}
Also used : JetInstance(com.hazelcast.jet.JetInstance)

Aggregations

JetInstance (com.hazelcast.jet.JetInstance)84 Test (org.junit.Test)45 Job (com.hazelcast.jet.Job)32 JetConfig (com.hazelcast.jet.config.JetConfig)22 MockPS (com.hazelcast.jet.core.TestProcessors.MockPS)14 Pipeline (com.hazelcast.jet.pipeline.Pipeline)14 JobConfig (com.hazelcast.jet.config.JobConfig)13 StuckProcessor (com.hazelcast.jet.core.TestProcessors.StuckProcessor)12 DAG (com.hazelcast.jet.core.DAG)10 ExpectedException (org.junit.rules.ExpectedException)10 Jet (com.hazelcast.jet.Jet)9 Assert.assertEquals (org.junit.Assert.assertEquals)9 JetService (com.hazelcast.jet.impl.JetService)8 Map (java.util.Map)8 CountDownLatch (java.util.concurrent.CountDownLatch)8 JobRepository (com.hazelcast.jet.impl.JobRepository)7 Sources (com.hazelcast.jet.pipeline.Sources)7 CancellationException (java.util.concurrent.CancellationException)7 Assert.assertNotNull (org.junit.Assert.assertNotNull)7 Before (org.junit.Before)7