Search in sources :

Example 11 with DynamicConfigurationAwareConfig

use of com.hazelcast.internal.dynamicconfig.DynamicConfigurationAwareConfig in project hazelcast by hazelcast.

the class DynamicConfigurationAwareConfigConstructorTest method testConstructor.

@Test
public void testConstructor() {
    Config config = new Config().setInstanceName("myInstanceName").addMapConfig(new MapConfig("myMap")).addListConfig(new ListConfig("myList")).addListenerConfig(new ListenerConfig("com.hazelcast.test.MyListenerConfig")).setProperties(buildPropertiesWithDefaults());
    HazelcastProperties properties = new HazelcastProperties(config);
    DynamicConfigurationAwareConfig dynamicConfig = new DynamicConfigurationAwareConfig(config, properties);
    DynamicConfigurationAwareConfigConstructor constructor = new DynamicConfigurationAwareConfigConstructor(DynamicConfigurationAwareConfig.class);
    DynamicConfigurationAwareConfig clonedDynamicConfig = (DynamicConfigurationAwareConfig) constructor.createNew(dynamicConfig);
    assertEquals(dynamicConfig.getInstanceName(), clonedDynamicConfig.getInstanceName());
    assertEquals(dynamicConfig.getMapConfigs().size(), clonedDynamicConfig.getMapConfigs().size());
    assertEquals(dynamicConfig.getListConfigs().size(), clonedDynamicConfig.getListConfigs().size());
    assertEquals(dynamicConfig.getListenerConfigs().size(), clonedDynamicConfig.getListenerConfigs().size());
    assertPropertiesEquals(dynamicConfig.getProperties(), clonedDynamicConfig.getProperties());
}
Also used : ListenerConfig(com.hazelcast.config.ListenerConfig) HazelcastProperties(com.hazelcast.spi.properties.HazelcastProperties) Config(com.hazelcast.config.Config) ListConfig(com.hazelcast.config.ListConfig) ListenerConfig(com.hazelcast.config.ListenerConfig) DynamicConfigurationAwareConfig(com.hazelcast.internal.dynamicconfig.DynamicConfigurationAwareConfig) MapConfig(com.hazelcast.config.MapConfig) DynamicConfigurationAwareConfig(com.hazelcast.internal.dynamicconfig.DynamicConfigurationAwareConfig) DynamicConfigurationAwareConfigConstructor(com.hazelcast.test.starter.constructor.DynamicConfigurationAwareConfigConstructor) MapConfig(com.hazelcast.config.MapConfig) ListConfig(com.hazelcast.config.ListConfig) ParallelJVMTest(com.hazelcast.test.annotation.ParallelJVMTest) QuickTest(com.hazelcast.test.annotation.QuickTest) Test(org.junit.Test)

Example 12 with DynamicConfigurationAwareConfig

use of com.hazelcast.internal.dynamicconfig.DynamicConfigurationAwareConfig in project hazelcast by hazelcast.

the class GracefulShutdownTest method when_shutdownGracefulWhileRestartGraceful_then_restartsFromTerminalSnapshot.

@Test
public void when_shutdownGracefulWhileRestartGraceful_then_restartsFromTerminalSnapshot() throws Exception {
    MapConfig mapConfig = new MapConfig(JobRepository.SNAPSHOT_DATA_MAP_PREFIX + "*");
    mapConfig.getMapStoreConfig().setClassName(BlockingMapStore.class.getName()).setEnabled(true);
    Config config = instances[0].getConfig();
    ((DynamicConfigurationAwareConfig) config).getStaticConfig().addMapConfig(mapConfig);
    BlockingMapStore.shouldBlock = false;
    BlockingMapStore.wasBlocked = false;
    DAG dag = new DAG();
    int numItems = 5000;
    Vertex source = dag.newVertex("source", throttle(() -> new EmitIntegersP(numItems), 500));
    Vertex sink = dag.newVertex("sink", SinkProcessors.writeListP("sink"));
    dag.edge(between(source, sink));
    source.localParallelism(1);
    Job job = instances[0].getJet().newJob(dag, new JobConfig().setProcessingGuarantee(EXACTLY_ONCE).setSnapshotIntervalMillis(2000));
    // wait for the first snapshot
    JetServiceBackend jetServiceBackend = getNode(instances[0]).nodeEngine.getService(JetServiceBackend.SERVICE_NAME);
    JobRepository jobRepository = jetServiceBackend.getJobCoordinationService().jobRepository();
    assertJobStatusEventually(job, RUNNING);
    assertTrueEventually(() -> assertTrue(jobRepository.getJobExecutionRecord(job.getId()).dataMapIndex() >= 0));
    // When
    BlockingMapStore.shouldBlock = true;
    job.restart();
    assertTrueEventually(() -> assertTrue("blocking did not happen", BlockingMapStore.wasBlocked), 5);
    Future shutdownFuture = spawn(() -> instances[1].shutdown());
    logger.info("savedCounters=" + EmitIntegersP.savedCounters);
    int minCounter = EmitIntegersP.savedCounters.values().stream().mapToInt(Integer::intValue).min().getAsInt();
    BlockingMapStore.shouldBlock = false;
    shutdownFuture.get();
    // Then
    job.join();
    Map<Integer, Integer> actual = new ArrayList<>(instances[0].<Integer>getList("sink")).stream().collect(Collectors.toMap(Function.identity(), item -> 1, Integer::sum));
    Map<Integer, Integer> expected = IntStream.range(0, numItems).boxed().collect(Collectors.toMap(Function.identity(), item -> item < minCounter ? 2 : 1));
    assertEquals(expected, actual);
}
Also used : IntStream(java.util.stream.IntStream) ParallelJVMTest(com.hazelcast.test.annotation.ParallelJVMTest) RunWith(org.junit.runner.RunWith) Function(java.util.function.Function) HazelcastSerialClassRunner(com.hazelcast.test.HazelcastSerialClassRunner) MapStore(com.hazelcast.map.MapStore) ArrayList(java.util.ArrayList) ConcurrentMap(java.util.concurrent.ConcurrentMap) MapConfig(com.hazelcast.config.MapConfig) Future(java.util.concurrent.Future) TestUtil.throttle(com.hazelcast.jet.core.TestUtil.throttle) Map(java.util.Map) Nonnull(javax.annotation.Nonnull) Job(com.hazelcast.jet.Job) Before(org.junit.Before) JobRepository(com.hazelcast.jet.impl.JobRepository) Config(com.hazelcast.config.Config) HazelcastInstance(com.hazelcast.core.HazelcastInstance) Collection(java.util.Collection) EXACTLY_ONCE(com.hazelcast.jet.config.ProcessingGuarantee.EXACTLY_ONCE) JobConfig(com.hazelcast.jet.config.JobConfig) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) Assert.assertTrue(org.junit.Assert.assertTrue) Test(org.junit.Test) SlowTest(com.hazelcast.test.annotation.SlowTest) Category(org.junit.experimental.categories.Category) Collectors(java.util.stream.Collectors) SupplierEx(com.hazelcast.function.SupplierEx) NoOutputSourceP(com.hazelcast.jet.core.TestProcessors.NoOutputSourceP) NONE(com.hazelcast.jet.config.ProcessingGuarantee.NONE) DynamicConfigurationAwareConfig(com.hazelcast.internal.dynamicconfig.DynamicConfigurationAwareConfig) BroadcastKey.broadcastKey(com.hazelcast.jet.core.BroadcastKey.broadcastKey) SinkProcessors(com.hazelcast.jet.core.processor.SinkProcessors) HOURS(java.util.concurrent.TimeUnit.HOURS) RUNNING(com.hazelcast.jet.core.JobStatus.RUNNING) Assert.assertEquals(org.junit.Assert.assertEquals) JetServiceBackend(com.hazelcast.jet.impl.JetServiceBackend) Edge.between(com.hazelcast.jet.core.Edge.between) MapConfig(com.hazelcast.config.MapConfig) Config(com.hazelcast.config.Config) JobConfig(com.hazelcast.jet.config.JobConfig) DynamicConfigurationAwareConfig(com.hazelcast.internal.dynamicconfig.DynamicConfigurationAwareConfig) ArrayList(java.util.ArrayList) JobRepository(com.hazelcast.jet.impl.JobRepository) JobConfig(com.hazelcast.jet.config.JobConfig) Future(java.util.concurrent.Future) MapConfig(com.hazelcast.config.MapConfig) Job(com.hazelcast.jet.Job) JetServiceBackend(com.hazelcast.jet.impl.JetServiceBackend) ParallelJVMTest(com.hazelcast.test.annotation.ParallelJVMTest) Test(org.junit.Test) SlowTest(com.hazelcast.test.annotation.SlowTest)

Example 13 with DynamicConfigurationAwareConfig

use of com.hazelcast.internal.dynamicconfig.DynamicConfigurationAwareConfig in project hazelcast by hazelcast.

the class ManualRestartTest method when_terminalSnapshotFails_then_previousSnapshotUsed.

@Test
public void when_terminalSnapshotFails_then_previousSnapshotUsed() {
    MapConfig mapConfig = new MapConfig(JobRepository.SNAPSHOT_DATA_MAP_PREFIX + "*");
    mapConfig.getMapStoreConfig().setClassName(FailingMapStore.class.getName()).setEnabled(true);
    Config config = instances[0].getConfig();
    ((DynamicConfigurationAwareConfig) config).getStaticConfig().addMapConfig(mapConfig);
    FailingMapStore.fail = false;
    FailingMapStore.failed = false;
    DAG dag = new DAG();
    Vertex source = dag.newVertex("source", throttle(() -> new SequencesInPartitionsGeneratorP(2, 10000, true), 1000));
    Vertex sink = dag.newVertex("sink", writeListP("sink"));
    dag.edge(between(source, sink));
    source.localParallelism(1);
    Job job = instances[0].getJet().newJob(dag, new JobConfig().setProcessingGuarantee(EXACTLY_ONCE).setSnapshotIntervalMillis(2000));
    // wait for the first snapshot
    JetServiceBackend jetServiceBackend = getNode(instances[0]).nodeEngine.getService(JetServiceBackend.SERVICE_NAME);
    JobRepository jobRepository = jetServiceBackend.getJobCoordinationService().jobRepository();
    assertJobStatusEventually(job, RUNNING);
    assertTrueEventually(() -> assertTrue(jobRepository.getJobExecutionRecord(job.getId()).dataMapIndex() >= 0));
    // When
    sleepMillis(100);
    FailingMapStore.fail = true;
    job.restart();
    assertTrueEventually(() -> assertTrue(FailingMapStore.failed));
    FailingMapStore.fail = false;
    job.join();
    Map<Integer, Integer> actual = new ArrayList<>(instances[0].<Entry<Integer, Integer>>getList("sink")).stream().filter(// we'll only check partition 0
    e -> e.getKey() == 0).map(Entry::getValue).collect(Collectors.toMap(e -> e, e -> 1, (o, n) -> o + n, TreeMap::new));
    assertEquals("first item != 1, " + actual.toString(), (Integer) 1, actual.get(0));
    assertEquals("last item != 1, " + actual.toString(), (Integer) 1, actual.get(9999));
    // the result should be some ones, then some twos and then some ones. The twos should be during the time
    // since the last successful snapshot until the actual termination, when there was reprocessing.
    boolean sawTwo = false;
    boolean sawOneAgain = false;
    for (Integer v : actual.values()) {
        if (v == 1) {
            if (sawTwo) {
                sawOneAgain = true;
            }
        } else if (v == 2) {
            assertFalse("got a 2 in another group", sawOneAgain);
            sawTwo = true;
        } else {
            fail("v=" + v);
        }
    }
    assertTrue("didn't see any 2s", sawTwo);
}
Also used : ParallelJVMTest(com.hazelcast.test.annotation.ParallelJVMTest) QuickTest(com.hazelcast.test.annotation.QuickTest) RunWith(org.junit.runner.RunWith) JetInitDataSerializerHook(com.hazelcast.jet.impl.execution.init.JetInitDataSerializerHook) HazelcastSerialClassRunner(com.hazelcast.test.HazelcastSerialClassRunner) ArrayList(java.util.ArrayList) Collections.singletonList(java.util.Collections.singletonList) MapConfig(com.hazelcast.config.MapConfig) SequencesInPartitionsGeneratorP(com.hazelcast.jet.core.JobRestartWithSnapshotTest.SequencesInPartitionsGeneratorP) PacketFiltersUtil.rejectOperationsBetween(com.hazelcast.test.PacketFiltersUtil.rejectOperationsBetween) Assertions.assertThatThrownBy(org.assertj.core.api.Assertions.assertThatThrownBy) TestUtil.throttle(com.hazelcast.jet.core.TestUtil.throttle) Map(java.util.Map) STARTING(com.hazelcast.jet.core.JobStatus.STARTING) Assert.fail(org.junit.Assert.fail) ExpectedException(org.junit.rules.ExpectedException) Job(com.hazelcast.jet.Job) Before(org.junit.Before) JobRepository(com.hazelcast.jet.impl.JobRepository) Config(com.hazelcast.config.Config) HazelcastInstance(com.hazelcast.core.HazelcastInstance) MockPS(com.hazelcast.jet.core.TestProcessors.MockPS) EXACTLY_ONCE(com.hazelcast.jet.config.ProcessingGuarantee.EXACTLY_ONCE) JobConfig(com.hazelcast.jet.config.JobConfig) Assert.assertTrue(org.junit.Assert.assertTrue) Test(org.junit.Test) Category(org.junit.experimental.categories.Category) AMapStore(com.hazelcast.client.map.helpers.AMapStore) Collectors(java.util.stream.Collectors) NoOutputSourceP(com.hazelcast.jet.core.TestProcessors.NoOutputSourceP) DynamicConfigurationAwareConfig(com.hazelcast.internal.dynamicconfig.DynamicConfigurationAwareConfig) Serializable(java.io.Serializable) Rule(org.junit.Rule) TreeMap(java.util.TreeMap) Assert.assertFalse(org.junit.Assert.assertFalse) Entry(java.util.Map.Entry) RUNNING(com.hazelcast.jet.core.JobStatus.RUNNING) PacketFiltersUtil.resetPacketFiltersFrom(com.hazelcast.test.PacketFiltersUtil.resetPacketFiltersFrom) Assert.assertEquals(org.junit.Assert.assertEquals) JetServiceBackend(com.hazelcast.jet.impl.JetServiceBackend) Edge.between(com.hazelcast.jet.core.Edge.between) SinkProcessors.writeListP(com.hazelcast.jet.core.processor.SinkProcessors.writeListP) MapConfig(com.hazelcast.config.MapConfig) Config(com.hazelcast.config.Config) JobConfig(com.hazelcast.jet.config.JobConfig) DynamicConfigurationAwareConfig(com.hazelcast.internal.dynamicconfig.DynamicConfigurationAwareConfig) ArrayList(java.util.ArrayList) JobRepository(com.hazelcast.jet.impl.JobRepository) JobConfig(com.hazelcast.jet.config.JobConfig) SequencesInPartitionsGeneratorP(com.hazelcast.jet.core.JobRestartWithSnapshotTest.SequencesInPartitionsGeneratorP) MapConfig(com.hazelcast.config.MapConfig) Job(com.hazelcast.jet.Job) JetServiceBackend(com.hazelcast.jet.impl.JetServiceBackend) ParallelJVMTest(com.hazelcast.test.annotation.ParallelJVMTest) QuickTest(com.hazelcast.test.annotation.QuickTest) Test(org.junit.Test)

Example 14 with DynamicConfigurationAwareConfig

use of com.hazelcast.internal.dynamicconfig.DynamicConfigurationAwareConfig in project hazelcast by hazelcast.

the class AddMultiMapConfigMessageTask method checkStaticConfigDoesNotExist.

@Override
protected boolean checkStaticConfigDoesNotExist(IdentifiedDataSerializable config) {
    DynamicConfigurationAwareConfig nodeConfig = (DynamicConfigurationAwareConfig) nodeEngine.getConfig();
    MultiMapConfig multiMapConfig = (MultiMapConfig) config;
    return nodeConfig.checkStaticConfigDoesNotExist(nodeConfig.getStaticConfig().getMultiMapConfigs(), multiMapConfig.getName(), multiMapConfig);
}
Also used : DynamicConfigurationAwareConfig(com.hazelcast.internal.dynamicconfig.DynamicConfigurationAwareConfig) MultiMapConfig(com.hazelcast.config.MultiMapConfig)

Example 15 with DynamicConfigurationAwareConfig

use of com.hazelcast.internal.dynamicconfig.DynamicConfigurationAwareConfig in project hazelcast by hazelcast.

the class AddReplicatedMapConfigMessageTask method checkStaticConfigDoesNotExist.

@Override
protected boolean checkStaticConfigDoesNotExist(IdentifiedDataSerializable config) {
    DynamicConfigurationAwareConfig nodeConfig = (DynamicConfigurationAwareConfig) nodeEngine.getConfig();
    ReplicatedMapConfig replicatedMapConfig = (ReplicatedMapConfig) config;
    return nodeConfig.checkStaticConfigDoesNotExist(nodeConfig.getStaticConfig().getReplicatedMapConfigs(), replicatedMapConfig.getName(), replicatedMapConfig);
}
Also used : ReplicatedMapConfig(com.hazelcast.config.ReplicatedMapConfig) DynamicConfigurationAwareConfig(com.hazelcast.internal.dynamicconfig.DynamicConfigurationAwareConfig)

Aggregations

DynamicConfigurationAwareConfig (com.hazelcast.internal.dynamicconfig.DynamicConfigurationAwareConfig)19 MapConfig (com.hazelcast.config.MapConfig)4 Config (com.hazelcast.config.Config)3 ParallelJVMTest (com.hazelcast.test.annotation.ParallelJVMTest)3 Test (org.junit.Test)3 ListConfig (com.hazelcast.config.ListConfig)2 HazelcastInstance (com.hazelcast.core.HazelcastInstance)2 Job (com.hazelcast.jet.Job)2 JobConfig (com.hazelcast.jet.config.JobConfig)2 EXACTLY_ONCE (com.hazelcast.jet.config.ProcessingGuarantee.EXACTLY_ONCE)2 Edge.between (com.hazelcast.jet.core.Edge.between)2 RUNNING (com.hazelcast.jet.core.JobStatus.RUNNING)2 NoOutputSourceP (com.hazelcast.jet.core.TestProcessors.NoOutputSourceP)2 TestUtil.throttle (com.hazelcast.jet.core.TestUtil.throttle)2 JetServiceBackend (com.hazelcast.jet.impl.JetServiceBackend)2 JobRepository (com.hazelcast.jet.impl.JobRepository)2 HazelcastSerialClassRunner (com.hazelcast.test.HazelcastSerialClassRunner)2 QuickTest (com.hazelcast.test.annotation.QuickTest)2 ArrayList (java.util.ArrayList)2 Map (java.util.Map)2