Search in sources :

Example 11 with VoidBlobStore

use of org.apache.flink.runtime.blob.VoidBlobStore in project flink-mirror by flink-ci.

the class DefaultExecutionGraphDeploymentWithSmallBlobCacheSizeLimitTest method setupBlobServer.

@Before
@Override
public void setupBlobServer() throws IOException {
    Configuration config = new Configuration();
    // Always offload the serialized JobInformation, TaskInformation and cached
    // ShuffleDescriptors
    config.setInteger(BlobServerOptions.OFFLOAD_MINSIZE, 0);
    blobServer = new BlobServer(config, TEMPORARY_FOLDER.newFolder(), new VoidBlobStore());
    blobServer.start();
    blobWriter = blobServer;
    InetSocketAddress serverAddress = new InetSocketAddress("localhost", blobServer.getPort());
    // Set the size limit of the blob cache to 1
    BlobCacheSizeTracker blobCacheSizeTracker = new BlobCacheSizeTracker(1L);
    blobCache = new PermanentBlobCache(config, TEMPORARY_FOLDER.newFolder(), new VoidBlobStore(), serverAddress, blobCacheSizeTracker);
}
Also used : VoidBlobStore(org.apache.flink.runtime.blob.VoidBlobStore) PermanentBlobCache(org.apache.flink.runtime.blob.PermanentBlobCache) Configuration(org.apache.flink.configuration.Configuration) InetSocketAddress(java.net.InetSocketAddress) BlobCacheSizeTracker(org.apache.flink.runtime.blob.BlobCacheSizeTracker) BlobServer(org.apache.flink.runtime.blob.BlobServer) Before(org.junit.Before)

Example 12 with VoidBlobStore

use of org.apache.flink.runtime.blob.VoidBlobStore in project flink-mirror by flink-ci.

the class DefaultExecutionGraphDeploymentWithBlobServerTest method setupBlobServer.

@Before
public void setupBlobServer() throws IOException {
    Configuration config = new Configuration();
    // always offload the serialized job and task information
    config.setInteger(BlobServerOptions.OFFLOAD_MINSIZE, 0);
    blobServer = Mockito.spy(new BlobServer(config, TEMPORARY_FOLDER.newFolder(), new VoidBlobStore()));
    blobWriter = blobServer;
    blobCache = blobServer;
    seenHashes.clear();
    // verify that we do not upload the same content more than once
    doAnswer(invocation -> {
        PermanentBlobKey key = (PermanentBlobKey) invocation.callRealMethod();
        assertTrue(seenHashes.add(key.getHash()));
        return key;
    }).when(blobServer).putPermanent(any(JobID.class), Matchers.<byte[]>any());
    blobServer.start();
}
Also used : VoidBlobStore(org.apache.flink.runtime.blob.VoidBlobStore) Configuration(org.apache.flink.configuration.Configuration) PermanentBlobKey(org.apache.flink.runtime.blob.PermanentBlobKey) BlobServer(org.apache.flink.runtime.blob.BlobServer) JobID(org.apache.flink.api.common.JobID) Before(org.junit.Before)

Example 13 with VoidBlobStore

use of org.apache.flink.runtime.blob.VoidBlobStore in project flink-mirror by flink-ci.

the class KubernetesHaServicesTest method testInternalCloseShouldCloseFlinkKubeClient.

@Test
public void testInternalCloseShouldCloseFlinkKubeClient() throws Exception {
    new Context() {

        {
            runTest(() -> {
                final KubernetesHaServices kubernetesHaServices = new KubernetesHaServices(flinkKubeClient, executorService, configuration, new VoidBlobStore());
                kubernetesHaServices.internalClose();
                assertThat(closeKubeClientFuture.isDone(), is(true));
            });
        }
    };
}
Also used : VoidBlobStore(org.apache.flink.runtime.blob.VoidBlobStore) Test(org.junit.Test)

Example 14 with VoidBlobStore

use of org.apache.flink.runtime.blob.VoidBlobStore in project flink-mirror by flink-ci.

the class MiniDispatcherTest method setupClass.

@BeforeClass
public static void setupClass() throws IOException {
    jobGraph = JobGraphTestUtils.singleNoOpJobGraph();
    executionGraphInfo = new ExecutionGraphInfo(new ArchivedExecutionGraphBuilder().setJobID(jobGraph.getJobID()).setState(JobStatus.FINISHED).build());
    rpcService = new TestingRpcService();
    configuration = new Configuration();
    blobServer = new BlobServer(configuration, temporaryFolder.newFolder(), new VoidBlobStore());
}
Also used : VoidBlobStore(org.apache.flink.runtime.blob.VoidBlobStore) Configuration(org.apache.flink.configuration.Configuration) ExecutionGraphInfo(org.apache.flink.runtime.scheduler.ExecutionGraphInfo) TestingRpcService(org.apache.flink.runtime.rpc.TestingRpcService) ArchivedExecutionGraphBuilder(org.apache.flink.runtime.rest.handler.legacy.utils.ArchivedExecutionGraphBuilder) BlobServer(org.apache.flink.runtime.blob.BlobServer) BeforeClass(org.junit.BeforeClass)

Example 15 with VoidBlobStore

use of org.apache.flink.runtime.blob.VoidBlobStore in project flink-mirror by flink-ci.

the class ClientUtilsTest method setup.

@BeforeClass
public static void setup() throws IOException {
    Configuration config = new Configuration();
    blobServer = new BlobServer(config, temporaryFolder.newFolder(), new VoidBlobStore());
    blobServer.start();
}
Also used : VoidBlobStore(org.apache.flink.runtime.blob.VoidBlobStore) Configuration(org.apache.flink.configuration.Configuration) BlobServer(org.apache.flink.runtime.blob.BlobServer) BeforeClass(org.junit.BeforeClass)

Aggregations

VoidBlobStore (org.apache.flink.runtime.blob.VoidBlobStore)51 Configuration (org.apache.flink.configuration.Configuration)42 BlobServer (org.apache.flink.runtime.blob.BlobServer)39 Before (org.junit.Before)18 Test (org.junit.Test)18 InetSocketAddress (java.net.InetSocketAddress)15 JobID (org.apache.flink.api.common.JobID)15 PermanentBlobCache (org.apache.flink.runtime.blob.PermanentBlobCache)15 PermanentBlobKey (org.apache.flink.runtime.blob.PermanentBlobKey)12 UserCodeClassLoader (org.apache.flink.util.UserCodeClassLoader)9 BeforeClass (org.junit.BeforeClass)9 URL (java.net.URL)6 ArrayList (java.util.ArrayList)6 File (java.io.File)3 IOException (java.io.IOException)3 Collection (java.util.Collection)3 TestingServer (org.apache.curator.test.TestingServer)3 KubernetesConfigMap (org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMap)3 BlobCacheSizeTracker (org.apache.flink.runtime.blob.BlobCacheSizeTracker)3 StandaloneCheckpointRecoveryFactory (org.apache.flink.runtime.checkpoint.StandaloneCheckpointRecoveryFactory)3