Search in sources :

Example 16 with PermanentBlobCache

use of org.apache.flink.runtime.blob.PermanentBlobCache in project flink by apache.

the class DefaultExecutionGraphDeploymentWithSmallBlobCacheSizeLimitTest method testDeployMultipleTasksWithSmallBlobCacheSizeLimit.

/**
 * Test the deployment works well even the size limit of {@link BlobCacheSizeTracker} in {@link
 * PermanentBlobCache} is set to the minimum value.
 *
 * <p>In this extreme case, since the size limit is 1, every time a task is deployed, all the
 * existing **tracked** BLOBs on the cache must be untracked and deleted before the new BLOB is
 * stored onto the cache.
 *
 * <p>This extreme case covers the situation of the normal case, where the size limit is much
 * larger than 1 and the deletion won't happen so frequently.
 */
@Test
public void testDeployMultipleTasksWithSmallBlobCacheSizeLimit() throws Exception {
    final int numberOfVertices = 4;
    final int parallelism = 10;
    final ExecutionGraph eg = createAndSetupExecutionGraph(numberOfVertices, parallelism);
    final SimpleAckingTaskManagerGateway taskManagerGateway = new SimpleAckingTaskManagerGateway();
    final BlockingQueue<TaskDeploymentDescriptor> tdds = new ArrayBlockingQueue<>(numberOfVertices * parallelism);
    taskManagerGateway.setSubmitConsumer(FunctionUtils.uncheckedConsumer(taskDeploymentDescriptor -> {
        taskDeploymentDescriptor.loadBigData(blobCache);
        tdds.offer(taskDeploymentDescriptor);
    }));
    for (ExecutionJobVertex ejv : eg.getVerticesTopologically()) {
        for (ExecutionVertex ev : ejv.getTaskVertices()) {
            assertEquals(ExecutionState.CREATED, ev.getExecutionState());
            LogicalSlot slot = new TestingLogicalSlotBuilder().setTaskManagerGateway(taskManagerGateway).createTestingLogicalSlot();
            final Execution execution = ev.getCurrentExecutionAttempt();
            execution.transitionState(ExecutionState.SCHEDULED);
            execution.registerProducedPartitions(slot.getTaskManagerLocation(), true).get();
            ev.deployToSlot(slot);
            assertEquals(ExecutionState.DEPLOYING, ev.getExecutionState());
            TaskDeploymentDescriptor tdd = tdds.take();
            assertNotNull(tdd);
            List<InputGateDeploymentDescriptor> igdds = tdd.getInputGates();
            assertEquals(ev.getAllConsumedPartitionGroups().size(), igdds.size());
            if (igdds.size() > 0) {
                checkShuffleDescriptors(igdds.get(0), ev.getConsumedPartitionGroup(0));
            }
        }
    }
}
Also used : ComponentMainThreadExecutorServiceAdapter(org.apache.flink.runtime.concurrent.ComponentMainThreadExecutorServiceAdapter) IntermediateResultPartitionID(org.apache.flink.runtime.jobgraph.IntermediateResultPartitionID) BlobServer(org.apache.flink.runtime.blob.BlobServer) JobVertex(org.apache.flink.runtime.jobgraph.JobVertex) BlobCacheSizeTracker(org.apache.flink.runtime.blob.BlobCacheSizeTracker) JobGraph(org.apache.flink.runtime.jobgraph.JobGraph) ResultPartitionType(org.apache.flink.runtime.io.network.partition.ResultPartitionType) TaskDeploymentDescriptor(org.apache.flink.runtime.deployment.TaskDeploymentDescriptor) ArrayList(java.util.ArrayList) JobVertexID(org.apache.flink.runtime.jobgraph.JobVertexID) TestingLogicalSlotBuilder(org.apache.flink.runtime.jobmaster.TestingLogicalSlotBuilder) DirectScheduledExecutorService(org.apache.flink.runtime.testutils.DirectScheduledExecutorService) JobException(org.apache.flink.runtime.JobException) FunctionUtils(org.apache.flink.util.function.FunctionUtils) JobGraphTestUtils(org.apache.flink.runtime.jobgraph.JobGraphTestUtils) JobExecutionException(org.apache.flink.runtime.client.JobExecutionException) DistributionPattern(org.apache.flink.runtime.jobgraph.DistributionPattern) Before(org.junit.Before) BlobServerOptions(org.apache.flink.configuration.BlobServerOptions) VoidBlobStore(org.apache.flink.runtime.blob.VoidBlobStore) Assert.assertNotNull(org.junit.Assert.assertNotNull) Configuration(org.apache.flink.configuration.Configuration) ExecutionState(org.apache.flink.runtime.execution.ExecutionState) LogicalSlot(org.apache.flink.runtime.jobmaster.LogicalSlot) Test(org.junit.Test) IOException(java.io.IOException) BlockingQueue(java.util.concurrent.BlockingQueue) InetSocketAddress(java.net.InetSocketAddress) SimpleAckingTaskManagerGateway(org.apache.flink.runtime.executiongraph.utils.SimpleAckingTaskManagerGateway) ArrayBlockingQueue(java.util.concurrent.ArrayBlockingQueue) List(java.util.List) ConsumedPartitionGroup(org.apache.flink.runtime.scheduler.strategy.ConsumedPartitionGroup) PermanentBlobCache(org.apache.flink.runtime.blob.PermanentBlobCache) BatchTask(org.apache.flink.runtime.operators.BatchTask) InputGateDeploymentDescriptor(org.apache.flink.runtime.deployment.InputGateDeploymentDescriptor) Assert.assertEquals(org.junit.Assert.assertEquals) InputGateDeploymentDescriptor(org.apache.flink.runtime.deployment.InputGateDeploymentDescriptor) LogicalSlot(org.apache.flink.runtime.jobmaster.LogicalSlot) SimpleAckingTaskManagerGateway(org.apache.flink.runtime.executiongraph.utils.SimpleAckingTaskManagerGateway) ArrayBlockingQueue(java.util.concurrent.ArrayBlockingQueue) TaskDeploymentDescriptor(org.apache.flink.runtime.deployment.TaskDeploymentDescriptor) TestingLogicalSlotBuilder(org.apache.flink.runtime.jobmaster.TestingLogicalSlotBuilder) Test(org.junit.Test)

Example 17 with PermanentBlobCache

use of org.apache.flink.runtime.blob.PermanentBlobCache in project flink by apache.

the class DefaultExecutionGraphDeploymentWithSmallBlobCacheSizeLimitTest method setupBlobServer.

@Before
@Override
public void setupBlobServer() throws IOException {
    Configuration config = new Configuration();
    // Always offload the serialized JobInformation, TaskInformation and cached
    // ShuffleDescriptors
    config.setInteger(BlobServerOptions.OFFLOAD_MINSIZE, 0);
    blobServer = new BlobServer(config, TEMPORARY_FOLDER.newFolder(), new VoidBlobStore());
    blobServer.start();
    blobWriter = blobServer;
    InetSocketAddress serverAddress = new InetSocketAddress("localhost", blobServer.getPort());
    // Set the size limit of the blob cache to 1
    BlobCacheSizeTracker blobCacheSizeTracker = new BlobCacheSizeTracker(1L);
    blobCache = new PermanentBlobCache(config, TEMPORARY_FOLDER.newFolder(), new VoidBlobStore(), serverAddress, blobCacheSizeTracker);
}
Also used : VoidBlobStore(org.apache.flink.runtime.blob.VoidBlobStore) PermanentBlobCache(org.apache.flink.runtime.blob.PermanentBlobCache) Configuration(org.apache.flink.configuration.Configuration) InetSocketAddress(java.net.InetSocketAddress) BlobCacheSizeTracker(org.apache.flink.runtime.blob.BlobCacheSizeTracker) BlobServer(org.apache.flink.runtime.blob.BlobServer) Before(org.junit.Before)

Example 18 with PermanentBlobCache

use of org.apache.flink.runtime.blob.PermanentBlobCache in project flink by apache.

the class BlobLibraryCacheManagerTest method testLibraryCacheManagerDifferentJobsCleanup.

/**
 * Tests that the {@link BlobLibraryCacheManager} cleans up after the class loader leases for
 * different jobs are closed.
 */
@Test
public void testLibraryCacheManagerDifferentJobsCleanup() throws Exception {
    JobID jobId1 = new JobID();
    JobID jobId2 = new JobID();
    List<PermanentBlobKey> keys1 = new ArrayList<>();
    List<PermanentBlobKey> keys2 = new ArrayList<>();
    BlobServer server = null;
    PermanentBlobCache cache = null;
    BlobLibraryCacheManager libCache = null;
    final byte[] buf = new byte[128];
    try {
        Configuration config = new Configuration();
        config.setLong(BlobServerOptions.CLEANUP_INTERVAL, 1L);
        server = new BlobServer(config, temporaryFolder.newFolder(), new VoidBlobStore());
        server.start();
        InetSocketAddress serverAddress = new InetSocketAddress("localhost", server.getPort());
        cache = new PermanentBlobCache(config, temporaryFolder.newFolder(), new VoidBlobStore(), serverAddress);
        keys1.add(server.putPermanent(jobId1, buf));
        buf[0] += 1;
        keys1.add(server.putPermanent(jobId1, buf));
        keys2.add(server.putPermanent(jobId2, buf));
        libCache = createBlobLibraryCacheManager(cache);
        cache.registerJob(jobId1);
        cache.registerJob(jobId2);
        assertEquals(0, libCache.getNumberOfManagedJobs());
        assertEquals(0, libCache.getNumberOfReferenceHolders(jobId1));
        checkFileCountForJob(2, jobId1, server);
        checkFileCountForJob(0, jobId1, cache);
        checkFileCountForJob(1, jobId2, server);
        checkFileCountForJob(0, jobId2, cache);
        final LibraryCacheManager.ClassLoaderLease classLoaderLeaseJob1 = libCache.registerClassLoaderLease(jobId1);
        final UserCodeClassLoader classLoader1 = classLoaderLeaseJob1.getOrResolveClassLoader(keys1, Collections.emptyList());
        assertEquals(1, libCache.getNumberOfManagedJobs());
        assertEquals(1, libCache.getNumberOfReferenceHolders(jobId1));
        assertEquals(0, libCache.getNumberOfReferenceHolders(jobId2));
        assertEquals(2, checkFilesExist(jobId1, keys1, cache, true));
        checkFileCountForJob(2, jobId1, server);
        checkFileCountForJob(2, jobId1, cache);
        assertEquals(0, checkFilesExist(jobId2, keys2, cache, false));
        checkFileCountForJob(1, jobId2, server);
        checkFileCountForJob(0, jobId2, cache);
        final LibraryCacheManager.ClassLoaderLease classLoaderLeaseJob2 = libCache.registerClassLoaderLease(jobId2);
        final UserCodeClassLoader classLoader2 = classLoaderLeaseJob2.getOrResolveClassLoader(keys2, Collections.emptyList());
        assertThat(classLoader1, not(sameInstance(classLoader2)));
        try {
            classLoaderLeaseJob2.getOrResolveClassLoader(keys1, Collections.<URL>emptyList());
            fail("Should fail with an IllegalStateException");
        } catch (IllegalStateException e) {
        // that's what we want
        }
        try {
            classLoaderLeaseJob2.getOrResolveClassLoader(keys2, Collections.singletonList(new URL("file:///tmp/does-not-exist")));
            fail("Should fail with an IllegalStateException");
        } catch (IllegalStateException e) {
        // that's what we want
        }
        assertEquals(2, libCache.getNumberOfManagedJobs());
        assertEquals(1, libCache.getNumberOfReferenceHolders(jobId1));
        assertEquals(1, libCache.getNumberOfReferenceHolders(jobId2));
        assertEquals(2, checkFilesExist(jobId1, keys1, cache, true));
        checkFileCountForJob(2, jobId1, server);
        checkFileCountForJob(2, jobId1, cache);
        assertEquals(1, checkFilesExist(jobId2, keys2, cache, true));
        checkFileCountForJob(1, jobId2, server);
        checkFileCountForJob(1, jobId2, cache);
        classLoaderLeaseJob1.release();
        assertEquals(1, libCache.getNumberOfManagedJobs());
        assertEquals(0, libCache.getNumberOfReferenceHolders(jobId1));
        assertEquals(1, libCache.getNumberOfReferenceHolders(jobId2));
        assertEquals(2, checkFilesExist(jobId1, keys1, cache, true));
        checkFileCountForJob(2, jobId1, server);
        checkFileCountForJob(2, jobId1, cache);
        assertEquals(1, checkFilesExist(jobId2, keys2, cache, true));
        checkFileCountForJob(1, jobId2, server);
        checkFileCountForJob(1, jobId2, cache);
        classLoaderLeaseJob2.release();
        assertEquals(0, libCache.getNumberOfManagedJobs());
        assertEquals(0, libCache.getNumberOfReferenceHolders(jobId1));
        assertEquals(0, libCache.getNumberOfReferenceHolders(jobId2));
        assertEquals(2, checkFilesExist(jobId1, keys1, cache, true));
        checkFileCountForJob(2, jobId1, server);
        checkFileCountForJob(2, jobId1, cache);
        assertEquals(1, checkFilesExist(jobId2, keys2, cache, true));
        checkFileCountForJob(1, jobId2, server);
        checkFileCountForJob(1, jobId2, cache);
    // only PermanentBlobCache#releaseJob() calls clean up files (tested in
    // BlobCacheCleanupTest etc.
    } finally {
        if (libCache != null) {
            libCache.shutdown();
        }
        // should have been closed by the libraryCacheManager, but just in case
        if (cache != null) {
            cache.close();
        }
        if (server != null) {
            server.close();
        }
    }
}
Also used : Configuration(org.apache.flink.configuration.Configuration) InetSocketAddress(java.net.InetSocketAddress) ArrayList(java.util.ArrayList) URL(java.net.URL) UserCodeClassLoader(org.apache.flink.util.UserCodeClassLoader) VoidBlobStore(org.apache.flink.runtime.blob.VoidBlobStore) PermanentBlobCache(org.apache.flink.runtime.blob.PermanentBlobCache) PermanentBlobKey(org.apache.flink.runtime.blob.PermanentBlobKey) BlobServer(org.apache.flink.runtime.blob.BlobServer) JobID(org.apache.flink.api.common.JobID) Test(org.junit.Test)

Example 19 with PermanentBlobCache

use of org.apache.flink.runtime.blob.PermanentBlobCache in project flink by apache.

the class BlobLibraryCacheRecoveryITCase method testRecoveryRegisterAndDownload.

/**
 * Tests that with {@link HighAvailabilityMode#ZOOKEEPER} distributed JARs are recoverable from
 * any participating BlobLibraryCacheManager.
 */
@Test
public void testRecoveryRegisterAndDownload() throws Exception {
    Random rand = new Random();
    BlobServer[] server = new BlobServer[2];
    InetSocketAddress[] serverAddress = new InetSocketAddress[2];
    BlobLibraryCacheManager[] libServer = new BlobLibraryCacheManager[2];
    PermanentBlobCache cache = null;
    BlobStoreService blobStoreService = null;
    Configuration config = new Configuration();
    config.setString(HighAvailabilityOptions.HA_MODE, "ZOOKEEPER");
    config.setString(HighAvailabilityOptions.HA_STORAGE_PATH, temporaryFolder.newFolder().getAbsolutePath());
    config.setLong(BlobServerOptions.CLEANUP_INTERVAL, 3_600L);
    final ExecutorService executorService = Executors.newSingleThreadExecutor();
    try {
        blobStoreService = BlobUtils.createBlobStoreFromConfig(config);
        final BlobLibraryCacheManager.ClassLoaderFactory classLoaderFactory = BlobLibraryCacheManager.defaultClassLoaderFactory(FlinkUserCodeClassLoaders.ResolveOrder.CHILD_FIRST, new String[0], null, true);
        for (int i = 0; i < server.length; i++) {
            server[i] = new BlobServer(config, temporaryFolder.newFolder(), blobStoreService);
            server[i].start();
            serverAddress[i] = new InetSocketAddress("localhost", server[i].getPort());
            libServer[i] = new BlobLibraryCacheManager(server[i], classLoaderFactory);
        }
        // Random data
        byte[] expected = new byte[1024];
        rand.nextBytes(expected);
        ArrayList<PermanentBlobKey> keys = new ArrayList<>(2);
        JobID jobId = new JobID();
        // Upload some data (libraries)
        // Request 1
        keys.add(server[0].putPermanent(jobId, expected));
        byte[] expected2 = Arrays.copyOfRange(expected, 32, 288);
        // Request 2
        keys.add(server[0].putPermanent(jobId, expected2));
        // The cache
        cache = new PermanentBlobCache(config, temporaryFolder.newFolder(), blobStoreService, serverAddress[0]);
        // Register uploaded libraries
        final LibraryCacheManager.ClassLoaderLease classLoaderLease = libServer[0].registerClassLoaderLease(jobId);
        classLoaderLease.getOrResolveClassLoader(keys, Collections.emptyList());
        // Verify key 1
        File f = cache.getFile(jobId, keys.get(0));
        assertEquals(expected.length, f.length());
        try (FileInputStream fis = new FileInputStream(f)) {
            for (int i = 0; i < expected.length && fis.available() > 0; i++) {
                assertEquals(expected[i], (byte) fis.read());
            }
            assertEquals(0, fis.available());
        }
        // Shutdown cache and start with other server
        cache.close();
        cache = new PermanentBlobCache(config, temporaryFolder.newFolder(), blobStoreService, serverAddress[1]);
        // Verify key 1
        f = cache.getFile(jobId, keys.get(0));
        assertEquals(expected.length, f.length());
        try (FileInputStream fis = new FileInputStream(f)) {
            for (int i = 0; i < expected.length && fis.available() > 0; i++) {
                assertEquals(expected[i], (byte) fis.read());
            }
            assertEquals(0, fis.available());
        }
        // Verify key 2
        f = cache.getFile(jobId, keys.get(1));
        assertEquals(expected2.length, f.length());
        try (FileInputStream fis = new FileInputStream(f)) {
            for (int i = 0; i < 256 && fis.available() > 0; i++) {
                assertEquals(expected2[i], (byte) fis.read());
            }
            assertEquals(0, fis.available());
        }
        // Remove blobs again
        server[1].globalCleanupAsync(jobId, executorService).join();
        // Verify everything is clean below recoveryDir/<cluster_id>
        final String clusterId = config.getString(HighAvailabilityOptions.HA_CLUSTER_ID);
        String haBlobStorePath = config.getString(HighAvailabilityOptions.HA_STORAGE_PATH);
        File haBlobStoreDir = new File(haBlobStorePath, clusterId);
        File[] recoveryFiles = haBlobStoreDir.listFiles();
        assertNotNull("HA storage directory does not exist", recoveryFiles);
        assertEquals("Unclean state backend: " + Arrays.toString(recoveryFiles), 0, recoveryFiles.length);
    } finally {
        assertThat(executorService.shutdownNow(), IsEmptyCollection.empty());
        for (BlobLibraryCacheManager s : libServer) {
            if (s != null) {
                s.shutdown();
            }
        }
        for (BlobServer s : server) {
            if (s != null) {
                s.close();
            }
        }
        if (cache != null) {
            cache.close();
        }
        if (blobStoreService != null) {
            blobStoreService.closeAndCleanupAllData();
        }
    }
}
Also used : Configuration(org.apache.flink.configuration.Configuration) InetSocketAddress(java.net.InetSocketAddress) ArrayList(java.util.ArrayList) BlobStoreService(org.apache.flink.runtime.blob.BlobStoreService) FileInputStream(java.io.FileInputStream) PermanentBlobCache(org.apache.flink.runtime.blob.PermanentBlobCache) Random(java.util.Random) PermanentBlobKey(org.apache.flink.runtime.blob.PermanentBlobKey) ExecutorService(java.util.concurrent.ExecutorService) BlobServer(org.apache.flink.runtime.blob.BlobServer) File(java.io.File) JobID(org.apache.flink.api.common.JobID) Test(org.junit.Test)

Example 20 with PermanentBlobCache

use of org.apache.flink.runtime.blob.PermanentBlobCache in project flink-mirror by flink-ci.

the class DefaultExecutionGraphDeploymentWithBlobCacheTest method setupBlobServer.

@Before
@Override
public void setupBlobServer() throws IOException {
    Configuration config = new Configuration();
    // always offload the serialized job and task information
    config.setInteger(BlobServerOptions.OFFLOAD_MINSIZE, 0);
    blobServer = new BlobServer(config, TEMPORARY_FOLDER.newFolder(), new VoidBlobStore());
    blobServer.start();
    blobWriter = blobServer;
    InetSocketAddress serverAddress = new InetSocketAddress("localhost", blobServer.getPort());
    blobCache = new PermanentBlobCache(config, TEMPORARY_FOLDER.newFolder(), new VoidBlobStore(), serverAddress);
}
Also used : VoidBlobStore(org.apache.flink.runtime.blob.VoidBlobStore) PermanentBlobCache(org.apache.flink.runtime.blob.PermanentBlobCache) Configuration(org.apache.flink.configuration.Configuration) InetSocketAddress(java.net.InetSocketAddress) BlobServer(org.apache.flink.runtime.blob.BlobServer) Before(org.junit.Before)

Aggregations

InetSocketAddress (java.net.InetSocketAddress)21 Configuration (org.apache.flink.configuration.Configuration)21 BlobServer (org.apache.flink.runtime.blob.BlobServer)21 PermanentBlobCache (org.apache.flink.runtime.blob.PermanentBlobCache)21 VoidBlobStore (org.apache.flink.runtime.blob.VoidBlobStore)18 Test (org.junit.Test)15 ArrayList (java.util.ArrayList)12 JobID (org.apache.flink.api.common.JobID)12 PermanentBlobKey (org.apache.flink.runtime.blob.PermanentBlobKey)12 UserCodeClassLoader (org.apache.flink.util.UserCodeClassLoader)9 Before (org.junit.Before)9 File (java.io.File)6 IOException (java.io.IOException)6 URL (java.net.URL)6 BlobCacheSizeTracker (org.apache.flink.runtime.blob.BlobCacheSizeTracker)6 FileInputStream (java.io.FileInputStream)3 Collection (java.util.Collection)3 List (java.util.List)3 Random (java.util.Random)3 ArrayBlockingQueue (java.util.concurrent.ArrayBlockingQueue)3