Examples with SequenceNumber org.apache.flink.runtime.state.changelog.SequenceNumber used on opensource projects

Example 16 with SequenceNumber

use of org.apache.flink.runtime.state.changelog.SequenceNumber in project flink by apache.

the class FsStateChangelogWriterTest method testPersistNewlyAppended.

@Test
public void testPersistNewlyAppended() throws Exception {
    withWriter((writer, uploader) -> {
        SequenceNumber sqn = append(writer, getBytes());
        writer.persist(sqn);
        uploader.reset();
        byte[] bytes = getBytes();
        sqn = append(writer, bytes);
        writer.persist(sqn);
        assertSubmittedOnly(uploader, bytes);
    });
}

Example 17 with SequenceNumber

use of org.apache.flink.runtime.state.changelog.SequenceNumber in project flink by apache.

the class FsStateChangelogWriterTest method testPersistFailedChanges.

@Test(expected = IOException.class)
public void testPersistFailedChanges() throws Exception {
    withWriter((writer, uploader) -> {
        byte[] bytes = getBytes();
        SequenceNumber sqn = append(writer, bytes);
        // future result ignored
        writer.persist(sqn);
        uploader.failUpload(new RuntimeException("test"));
        // should fail right away
        writer.persist(sqn);
    });
}

Example 18 with SequenceNumber

use of org.apache.flink.runtime.state.changelog.SequenceNumber in project flink by apache.

the class FsStateChangelogWriterTest method append.

private SequenceNumber append(FsStateChangelogWriter writer, byte[] bytes) throws IOException {
    SequenceNumber sequenceNumber = writer.nextSequenceNumber();
    writer.append(KEY_GROUP, bytes);
    return sequenceNumber;
}

Example 19 with SequenceNumber

use of org.apache.flink.runtime.state.changelog.SequenceNumber in project flink by apache.

the class FsStateChangelogWriter method persistInternal.

private CompletableFuture<ChangelogStateHandleStreamImpl> persistInternal(SequenceNumber from) throws IOException {
    synchronized (lock) {
        ensureCanPersist(from);
        rollover();
        Map<SequenceNumber, StateChangeSet> toUpload = drainTailMap(notUploaded, from);
        NavigableMap<SequenceNumber, UploadResult> readyToReturn = uploaded.tailMap(from, true);
        LOG.debug("collected readyToReturn: {}, toUpload: {}", readyToReturn, toUpload);
        SequenceNumberRange range = SequenceNumberRange.generic(from, activeSequenceNumber);
        if (range.size() == readyToReturn.size()) {
            checkState(toUpload.isEmpty());
            return completedFuture(buildHandle(keyGroupRange, readyToReturn, 0L));
        } else {
            CompletableFuture<ChangelogStateHandleStreamImpl> future = new CompletableFuture<>();
            uploadCompletionListeners.add(new UploadCompletionListener(keyGroupRange, range, readyToReturn, future));
            if (!toUpload.isEmpty()) {
                uploader.upload(new UploadTask(toUpload.values(), this::handleUploadSuccess, this::handleUploadFailure));
            }
            return future;
        }
    }
}

Example 20 with SequenceNumber

use of org.apache.flink.runtime.state.changelog.SequenceNumber in project flink by apache.

the class ChangelogStateBackendTestUtils method materialize.

/**
 * Explicitly trigger materialization. Materialization is expected to complete before returning
 * from this method by the use of direct executor when constructing materializer.
 * Automatic/periodic triggering is disabled by NOT starting the periodicMaterializationManager.
 *
 * <p>Additionally, verify changelog truncation happened upon completion.
 */
private static void materialize(ChangelogKeyedStateBackend<Integer> keyedBackend, PeriodicMaterializationManager periodicMaterializationManager) {
    StateChangelogWriter<? extends ChangelogStateHandle> writer = keyedBackend.getChangelogWriter();
    SequenceNumber sqn = writer.nextSequenceNumber();
    periodicMaterializationManager.triggerMaterialization();
    assertTrue("Materialization didn't truncate the changelog", sqn.compareTo(writer.getLowestSequenceNumber()) <= 0);
}