Search in sources :

Example 36 with FutureCallback

use of org.apache.kafka.connect.util.FutureCallback in project kafka by apache.

the class ConnectClusterStateImpl method connectorConfig.

@Override
public Map<String, String> connectorConfig(String connName) {
    FutureCallback<Map<String, String>> connectorConfigCallback = new FutureCallback<>();
    herder.connectorConfig(connName, connectorConfigCallback);
    try {
        return new HashMap<>(connectorConfigCallback.get(herderRequestTimeoutMs, TimeUnit.MILLISECONDS));
    } catch (InterruptedException | ExecutionException | TimeoutException e) {
        throw new ConnectException(String.format("Failed to retrieve configuration for connector '%s'", connName), e);
    }
}
Also used : HashMap(java.util.HashMap) ExecutionException(java.util.concurrent.ExecutionException) HashMap(java.util.HashMap) Map(java.util.Map) FutureCallback(org.apache.kafka.connect.util.FutureCallback) TimeoutException(java.util.concurrent.TimeoutException) ConnectException(org.apache.kafka.connect.errors.ConnectException)

Example 37 with FutureCallback

use of org.apache.kafka.connect.util.FutureCallback in project kafka by apache.

the class DistributedHerderTest method testRestartConnectorAndTasksUnknownConnector.

@Test
public void testRestartConnectorAndTasksUnknownConnector() throws Exception {
    String connectorName = "UnknownConnector";
    RestartRequest restartRequest = new RestartRequest(connectorName, false, true);
    // get the initial assignment
    EasyMock.expect(member.memberId()).andStubReturn("leader");
    EasyMock.expect(member.currentProtocolVersion()).andStubReturn(CONNECT_PROTOCOL_V0);
    expectRebalance(1, Collections.emptyList(), Collections.emptyList());
    expectPostRebalanceCatchup(SNAPSHOT);
    member.poll(EasyMock.anyInt());
    PowerMock.expectLastCall();
    // now handle the connector restart
    member.wakeup();
    PowerMock.expectLastCall();
    member.ensureActive();
    PowerMock.expectLastCall();
    member.poll(EasyMock.anyInt());
    PowerMock.expectLastCall();
    PowerMock.replayAll();
    herder.tick();
    FutureCallback<ConnectorStateInfo> callback = new FutureCallback<>();
    herder.restartConnectorAndTasks(restartRequest, callback);
    herder.tick();
    ExecutionException ee = assertThrows(ExecutionException.class, () -> callback.get(1000L, TimeUnit.MILLISECONDS));
    assertTrue(ee.getCause() instanceof NotFoundException);
    assertTrue(ee.getMessage().contains("Unknown connector:"));
    PowerMock.verifyAll();
}
Also used : RestartRequest(org.apache.kafka.connect.runtime.RestartRequest) NotFoundException(org.apache.kafka.connect.errors.NotFoundException) ExecutionException(java.util.concurrent.ExecutionException) ConnectorStateInfo(org.apache.kafka.connect.runtime.rest.entities.ConnectorStateInfo) FutureCallback(org.apache.kafka.connect.util.FutureCallback) PrepareForTest(org.powermock.core.classloader.annotations.PrepareForTest) Test(org.junit.Test)

Example 38 with FutureCallback

use of org.apache.kafka.connect.util.FutureCallback in project kafka by apache.

the class DistributedHerderTest method testRestartConnectorAndTasksSuccess.

@Test
public void testRestartConnectorAndTasksSuccess() throws Exception {
    RestartPlan restartPlan = PowerMock.createMock(RestartPlan.class);
    ConnectorStateInfo connectorStateInfo = PowerMock.createMock(ConnectorStateInfo.class);
    EasyMock.expect(restartPlan.restartConnectorStateInfo()).andReturn(connectorStateInfo).anyTimes();
    RestartRequest restartRequest = new RestartRequest(CONN1, false, true);
    EasyMock.expect(herder.buildRestartPlan(restartRequest)).andReturn(Optional.of(restartPlan)).anyTimes();
    configBackingStore.putRestartRequest(restartRequest);
    PowerMock.expectLastCall();
    // get the initial assignment
    EasyMock.expect(member.memberId()).andStubReturn("leader");
    EasyMock.expect(member.currentProtocolVersion()).andStubReturn(CONNECT_PROTOCOL_V0);
    expectRebalance(1, Collections.emptyList(), Collections.emptyList());
    expectPostRebalanceCatchup(SNAPSHOT);
    member.poll(EasyMock.anyInt());
    PowerMock.expectLastCall();
    // now handle the connector restart
    member.wakeup();
    PowerMock.expectLastCall();
    member.ensureActive();
    PowerMock.expectLastCall();
    member.poll(EasyMock.anyInt());
    PowerMock.expectLastCall();
    PowerMock.replayAll();
    herder.tick();
    FutureCallback<ConnectorStateInfo> callback = new FutureCallback<>();
    herder.restartConnectorAndTasks(restartRequest, callback);
    herder.tick();
    assertEquals(connectorStateInfo, callback.get(1000L, TimeUnit.MILLISECONDS));
    PowerMock.verifyAll();
}
Also used : RestartPlan(org.apache.kafka.connect.runtime.RestartPlan) RestartRequest(org.apache.kafka.connect.runtime.RestartRequest) ConnectorStateInfo(org.apache.kafka.connect.runtime.rest.entities.ConnectorStateInfo) FutureCallback(org.apache.kafka.connect.util.FutureCallback) PrepareForTest(org.powermock.core.classloader.annotations.PrepareForTest) Test(org.junit.Test)

Example 39 with FutureCallback

use of org.apache.kafka.connect.util.FutureCallback in project kafka by apache.

the class WorkerTest method testStartConnectorFailure.

@Test
public void testStartConnectorFailure() throws Exception {
    expectConverters();
    expectStartStorage();
    expectFileConfigProvider();
    final String nonConnectorClass = "java.util.HashMap";
    // Bad connector class name
    connectorProps.put(ConnectorConfig.CONNECTOR_CLASS_CONFIG, nonConnectorClass);
    Exception exception = new ConnectException("Failed to find Connector");
    EasyMock.expect(plugins.currentThreadLoader()).andReturn(delegatingLoader);
    EasyMock.expect(plugins.delegatingLoader()).andReturn(delegatingLoader);
    EasyMock.expect(delegatingLoader.connectorLoader(nonConnectorClass)).andReturn(delegatingLoader);
    EasyMock.expect(Plugins.compareAndSwapLoaders(delegatingLoader)).andReturn(delegatingLoader).times(2);
    EasyMock.expect(plugins.newConnector(EasyMock.anyString())).andThrow(exception);
    connectorStatusListener.onFailure(EasyMock.eq(CONNECTOR_ID), EasyMock.<ConnectException>anyObject());
    EasyMock.expectLastCall();
    expectClusterId();
    PowerMock.replayAll();
    worker = new Worker(WORKER_ID, new MockTime(), plugins, config, offsetBackingStore, noneConnectorClientConfigOverridePolicy);
    worker.herder = herder;
    worker.start();
    assertStatistics(worker, 0, 0);
    FutureCallback<TargetState> onStart = new FutureCallback<>();
    worker.startConnector(CONNECTOR_ID, connectorProps, ctx, connectorStatusListener, TargetState.STARTED, onStart);
    try {
        onStart.get(0, TimeUnit.MILLISECONDS);
        fail("Should have failed to start connector");
    } catch (ExecutionException e) {
        assertEquals(exception, e.getCause());
    }
    assertStartupStatistics(worker, 1, 1, 0, 0);
    assertEquals(Collections.emptySet(), worker.connectorNames());
    assertStatistics(worker, 0, 0);
    assertStartupStatistics(worker, 1, 1, 0, 0);
    worker.stopAndAwaitConnector(CONNECTOR_ID);
    assertStatistics(worker, 0, 0);
    assertStartupStatistics(worker, 1, 1, 0, 0);
    PowerMock.verifyAll();
}
Also used : ExecutionException(java.util.concurrent.ExecutionException) ExecutionException(java.util.concurrent.ExecutionException) ConnectException(org.apache.kafka.connect.errors.ConnectException) ConfigException(org.apache.kafka.common.config.ConfigException) MockTime(org.apache.kafka.common.utils.MockTime) FutureCallback(org.apache.kafka.connect.util.FutureCallback) ConnectException(org.apache.kafka.connect.errors.ConnectException) ThreadedTest(org.apache.kafka.connect.util.ThreadedTest) Test(org.junit.Test) ParameterizedTest(org.apache.kafka.connect.util.ParameterizedTest) PrepareForTest(org.powermock.core.classloader.annotations.PrepareForTest)

Example 40 with FutureCallback

use of org.apache.kafka.connect.util.FutureCallback in project kafka by apache.

the class StandaloneHerderTest method testRestartConnectorAndTasksNoRestarts.

@Test
public void testRestartConnectorAndTasksNoRestarts() throws Exception {
    RestartRequest restartRequest = new RestartRequest(CONNECTOR_NAME, false, true);
    RestartPlan restartPlan = PowerMock.createMock(RestartPlan.class);
    ConnectorStateInfo connectorStateInfo = PowerMock.createMock(ConnectorStateInfo.class);
    EasyMock.expect(restartPlan.shouldRestartConnector()).andReturn(false).anyTimes();
    EasyMock.expect(restartPlan.shouldRestartTasks()).andReturn(false).anyTimes();
    EasyMock.expect(restartPlan.restartConnectorStateInfo()).andReturn(connectorStateInfo).anyTimes();
    EasyMock.expect(herder.buildRestartPlan(restartRequest)).andReturn(Optional.of(restartPlan)).anyTimes();
    connector = PowerMock.createMock(BogusSinkConnector.class);
    expectAdd(SourceSink.SINK);
    Map<String, String> connectorConfig = connectorConfig(SourceSink.SINK);
    Connector connectorMock = PowerMock.createMock(SinkConnector.class);
    expectConfigValidation(connectorMock, true, connectorConfig);
    PowerMock.replayAll();
    herder.putConnectorConfig(CONNECTOR_NAME, connectorConfig, false, createCallback);
    Herder.Created<ConnectorInfo> connectorInfo = createCallback.get(1000L, TimeUnit.SECONDS);
    assertEquals(createdInfo(SourceSink.SINK), connectorInfo.result());
    FutureCallback<ConnectorStateInfo> restartCallback = new FutureCallback<>();
    herder.restartConnectorAndTasks(restartRequest, restartCallback);
    assertEquals(connectorStateInfo, restartCallback.get(1000L, TimeUnit.MILLISECONDS));
    PowerMock.verifyAll();
}
Also used : SourceConnector(org.apache.kafka.connect.source.SourceConnector) WorkerConnector(org.apache.kafka.connect.runtime.WorkerConnector) Connector(org.apache.kafka.connect.connector.Connector) SinkConnector(org.apache.kafka.connect.sink.SinkConnector) ConnectorInfo(org.apache.kafka.connect.runtime.rest.entities.ConnectorInfo) RestartPlan(org.apache.kafka.connect.runtime.RestartPlan) RestartRequest(org.apache.kafka.connect.runtime.RestartRequest) Herder(org.apache.kafka.connect.runtime.Herder) ConnectorStateInfo(org.apache.kafka.connect.runtime.rest.entities.ConnectorStateInfo) FutureCallback(org.apache.kafka.connect.util.FutureCallback) PrepareForTest(org.powermock.core.classloader.annotations.PrepareForTest) Test(org.junit.Test)

Aggregations

FutureCallback (org.apache.kafka.connect.util.FutureCallback)46 Test (org.junit.Test)29 PrepareForTest (org.powermock.core.classloader.annotations.PrepareForTest)29 ConnectorInfo (org.apache.kafka.connect.runtime.rest.entities.ConnectorInfo)24 Herder (org.apache.kafka.connect.runtime.Herder)23 Connector (org.apache.kafka.connect.connector.Connector)18 SinkConnector (org.apache.kafka.connect.sink.SinkConnector)18 SourceConnector (org.apache.kafka.connect.source.SourceConnector)18 ExecutionException (java.util.concurrent.ExecutionException)17 WorkerConnector (org.apache.kafka.connect.runtime.WorkerConnector)17 ConnectorTaskId (org.apache.kafka.connect.util.ConnectorTaskId)13 HashMap (java.util.HashMap)11 Path (javax.ws.rs.Path)11 RestartRequest (org.apache.kafka.connect.runtime.RestartRequest)11 ConnectorStateInfo (org.apache.kafka.connect.runtime.rest.entities.ConnectorStateInfo)11 Map (java.util.Map)9 NotFoundException (org.apache.kafka.connect.errors.NotFoundException)9 POST (javax.ws.rs.POST)7 ConnectException (org.apache.kafka.connect.errors.ConnectException)7 URI (java.net.URI)6