Search in sources :

Example 6 with CheckpointStorageLocation

use of org.apache.flink.runtime.state.CheckpointStorageLocation in project flink by apache.

the class AbstractFileCheckpointStorageAccessTestBase method testPersistMultipleMetadataOnlyCheckpoints.

// ------------------------------------------------------------------------
// checkpoints
// ------------------------------------------------------------------------
/**
 * Validates that multiple checkpoints from different jobs with the same checkpoint ID do not
 * interfere with each other.
 */
@Test
public void testPersistMultipleMetadataOnlyCheckpoints() throws Exception {
    final FileSystem fs = FileSystem.getLocalFileSystem();
    final Path checkpointDir = new Path(tmp.newFolder().toURI());
    final long checkpointId = 177;
    final CheckpointStorageAccess storage1 = createCheckpointStorage(checkpointDir);
    storage1.initializeBaseLocationsForCheckpoint();
    final CheckpointStorageAccess storage2 = createCheckpointStorage(checkpointDir);
    storage2.initializeBaseLocationsForCheckpoint();
    final CheckpointStorageLocation loc1 = storage1.initializeLocationForCheckpoint(checkpointId);
    final CheckpointStorageLocation loc2 = storage2.initializeLocationForCheckpoint(checkpointId);
    final byte[] data1 = { 77, 66, 55, 99, 88 };
    final byte[] data2 = { 1, 3, 2, 5, 4 };
    final CompletedCheckpointStorageLocation completedLocation1;
    try (CheckpointMetadataOutputStream out = loc1.createMetadataOutputStream()) {
        out.write(data1);
        completedLocation1 = out.closeAndFinalizeCheckpoint();
    }
    final String result1 = completedLocation1.getExternalPointer();
    final CompletedCheckpointStorageLocation completedLocation2;
    try (CheckpointMetadataOutputStream out = loc2.createMetadataOutputStream()) {
        out.write(data2);
        completedLocation2 = out.closeAndFinalizeCheckpoint();
    }
    final String result2 = completedLocation2.getExternalPointer();
    // check that this went to a file, but in a nested directory structure
    // one directory per storage
    FileStatus[] files = fs.listStatus(checkpointDir);
    assertEquals(2, files.length);
    // in each per-storage directory, one for the checkpoint
    FileStatus[] job1Files = fs.listStatus(files[0].getPath());
    FileStatus[] job2Files = fs.listStatus(files[1].getPath());
    assertTrue(job1Files.length >= 1);
    assertTrue(job2Files.length >= 1);
    assertTrue(fs.exists(new Path(result1, AbstractFsCheckpointStorageAccess.METADATA_FILE_NAME)));
    assertTrue(fs.exists(new Path(result2, AbstractFsCheckpointStorageAccess.METADATA_FILE_NAME)));
    // check that both storages can resolve each others contents
    validateContents(storage1.resolveCheckpoint(result1).getMetadataHandle(), data1);
    validateContents(storage1.resolveCheckpoint(result2).getMetadataHandle(), data2);
    validateContents(storage2.resolveCheckpoint(result1).getMetadataHandle(), data1);
    validateContents(storage2.resolveCheckpoint(result2).getMetadataHandle(), data2);
}
Also used : Path(org.apache.flink.core.fs.Path) FileStatus(org.apache.flink.core.fs.FileStatus) FileSystem(org.apache.flink.core.fs.FileSystem) CheckpointMetadataOutputStream(org.apache.flink.runtime.state.CheckpointMetadataOutputStream) CheckpointStorageLocation(org.apache.flink.runtime.state.CheckpointStorageLocation) CompletedCheckpointStorageLocation(org.apache.flink.runtime.state.CompletedCheckpointStorageLocation) CompletedCheckpointStorageLocation(org.apache.flink.runtime.state.CompletedCheckpointStorageLocation) CheckpointStorageAccess(org.apache.flink.runtime.state.CheckpointStorageAccess) MemoryBackendCheckpointStorageAccess(org.apache.flink.runtime.state.memory.MemoryBackendCheckpointStorageAccess) Test(org.junit.Test)

Example 7 with CheckpointStorageLocation

use of org.apache.flink.runtime.state.CheckpointStorageLocation in project flink by apache.

the class AbstractFileCheckpointStorageAccessTestBase method writeToAlreadyExistingCheckpointFails.

@Test
public void writeToAlreadyExistingCheckpointFails() throws Exception {
    final byte[] data = { 8, 8, 4, 5, 2, 6, 3 };
    final long checkpointId = 177;
    final CheckpointStorageAccess storage = createCheckpointStorage(randomTempPath());
    storage.initializeBaseLocationsForCheckpoint();
    final CheckpointStorageLocation loc = storage.initializeLocationForCheckpoint(checkpointId);
    try (CheckpointMetadataOutputStream out = loc.createMetadataOutputStream()) {
        out.write(data);
        out.closeAndFinalizeCheckpoint();
    }
    // create another writer to the metadata file for the checkpoint
    try {
        loc.createMetadataOutputStream();
        fail("this should fail with an exception");
    } catch (IOException ignored) {
    }
}
Also used : CheckpointMetadataOutputStream(org.apache.flink.runtime.state.CheckpointMetadataOutputStream) CheckpointStorageLocation(org.apache.flink.runtime.state.CheckpointStorageLocation) CompletedCheckpointStorageLocation(org.apache.flink.runtime.state.CompletedCheckpointStorageLocation) IOException(java.io.IOException) CheckpointStorageAccess(org.apache.flink.runtime.state.CheckpointStorageAccess) MemoryBackendCheckpointStorageAccess(org.apache.flink.runtime.state.memory.MemoryBackendCheckpointStorageAccess) Test(org.junit.Test)

Aggregations

CheckpointStorageLocation (org.apache.flink.runtime.state.CheckpointStorageLocation)7 CompletedCheckpointStorageLocation (org.apache.flink.runtime.state.CompletedCheckpointStorageLocation)7 CheckpointMetadataOutputStream (org.apache.flink.runtime.state.CheckpointMetadataOutputStream)5 CheckpointStorageAccess (org.apache.flink.runtime.state.CheckpointStorageAccess)3 MemoryBackendCheckpointStorageAccess (org.apache.flink.runtime.state.memory.MemoryBackendCheckpointStorageAccess)3 Test (org.junit.Test)3 File (java.io.File)2 IOException (java.io.IOException)2 JobID (org.apache.flink.api.common.JobID)2 URI (java.net.URI)1 ArrayDeque (java.util.ArrayDeque)1 ArrayList (java.util.ArrayList)1 Collection (java.util.Collection)1 Collections (java.util.Collections)1 HashMap (java.util.HashMap)1 HashSet (java.util.HashSet)1 LinkedHashMap (java.util.LinkedHashMap)1 List (java.util.List)1 Map (java.util.Map)1 Optional (java.util.Optional)1