Search in sources :

Example 1 with KeyedStateCheckpointOutputStream

use of org.apache.flink.runtime.state.KeyedStateCheckpointOutputStream in project beam by apache.

the class DedupingOperator method snapshotState.

@Override
public void snapshotState(StateSnapshotContext context) throws Exception {
    // copy from AbstractStreamOperator
    if (getKeyedStateBackend() != null) {
        KeyedStateCheckpointOutputStream out;
        try {
            out = context.getRawKeyedOperatorStateOutput();
        } catch (Exception exception) {
            throw new Exception("Could not open raw keyed operator state stream for " + getOperatorName() + '.', exception);
        }
        try {
            KeyGroupsList allKeyGroups = out.getKeyGroupList();
            for (int keyGroupIdx : allKeyGroups) {
                out.startNewKeyGroup(keyGroupIdx);
                DataOutputViewStreamWrapper dov = new DataOutputViewStreamWrapper(out);
                // if (this instanceof KeyGroupCheckpointedOperator)
                snapshotKeyGroupState(keyGroupIdx, dov);
            }
        } catch (Exception exception) {
            throw new Exception("Could not write timer service of " + getOperatorName() + " to checkpoint state stream.", exception);
        } finally {
            try {
                out.close();
            } catch (Exception closeException) {
                LOG.warn("Could not close raw keyed operator state stream for {}. This " + "might have prevented deleting some state data.", getOperatorName(), closeException);
            }
        }
    }
}
Also used : DataOutputViewStreamWrapper(org.apache.flink.core.memory.DataOutputViewStreamWrapper) KeyGroupsList(org.apache.flink.runtime.state.KeyGroupsList) ExecutionException(java.util.concurrent.ExecutionException) KeyedStateCheckpointOutputStream(org.apache.flink.runtime.state.KeyedStateCheckpointOutputStream)

Example 2 with KeyedStateCheckpointOutputStream

use of org.apache.flink.runtime.state.KeyedStateCheckpointOutputStream in project beam by apache.

the class DoFnOperator method snapshotState.

@Override
public void snapshotState(StateSnapshotContext context) throws Exception {
    // copy from AbstractStreamOperator
    if (getKeyedStateBackend() != null) {
        KeyedStateCheckpointOutputStream out;
        try {
            out = context.getRawKeyedOperatorStateOutput();
        } catch (Exception exception) {
            throw new Exception("Could not open raw keyed operator state stream for " + getOperatorName() + '.', exception);
        }
        try {
            KeyGroupsList allKeyGroups = out.getKeyGroupList();
            for (int keyGroupIdx : allKeyGroups) {
                out.startNewKeyGroup(keyGroupIdx);
                DataOutputViewStreamWrapper dov = new DataOutputViewStreamWrapper(out);
                // if (this instanceof KeyGroupCheckpointedOperator)
                snapshotKeyGroupState(keyGroupIdx, dov);
                // Maybe this is a normal DoFn that has no timerService
                if (keyCoder != null) {
                    timerService.snapshotTimersForKeyGroup(dov, keyGroupIdx);
                }
            }
        } catch (Exception exception) {
            throw new Exception("Could not write timer service of " + getOperatorName() + " to checkpoint state stream.", exception);
        } finally {
            try {
                out.close();
            } catch (Exception closeException) {
                LOG.warn("Could not close raw keyed operator state stream for {}. This " + "might have prevented deleting some state data.", getOperatorName(), closeException);
            }
        }
    }
}
Also used : DataOutputViewStreamWrapper(org.apache.flink.core.memory.DataOutputViewStreamWrapper) KeyGroupsList(org.apache.flink.runtime.state.KeyGroupsList) KeyedStateCheckpointOutputStream(org.apache.flink.runtime.state.KeyedStateCheckpointOutputStream)

Example 3 with KeyedStateCheckpointOutputStream

use of org.apache.flink.runtime.state.KeyedStateCheckpointOutputStream in project flink by apache.

the class AbstractStreamOperator method snapshotState.

/**
	 * Stream operators with state, which want to participate in a snapshot need to override this hook method.
	 *
	 * @param context context that provides information and means required for taking a snapshot
	 */
public void snapshotState(StateSnapshotContext context) throws Exception {
    if (getKeyedStateBackend() != null) {
        KeyedStateCheckpointOutputStream out;
        try {
            out = context.getRawKeyedOperatorStateOutput();
        } catch (Exception exception) {
            throw new Exception("Could not open raw keyed operator state stream for " + getOperatorName() + '.', exception);
        }
        try {
            KeyGroupsList allKeyGroups = out.getKeyGroupList();
            for (int keyGroupIdx : allKeyGroups) {
                out.startNewKeyGroup(keyGroupIdx);
                timeServiceManager.snapshotStateForKeyGroup(new DataOutputViewStreamWrapper(out), keyGroupIdx);
            }
        } catch (Exception exception) {
            throw new Exception("Could not write timer service of " + getOperatorName() + " to checkpoint state stream.", exception);
        } finally {
            try {
                out.close();
            } catch (Exception closeException) {
                LOG.warn("Could not close raw keyed operator state stream for {}. This " + "might have prevented deleting some state data.", getOperatorName(), closeException);
            }
        }
    }
}
Also used : DataOutputViewStreamWrapper(org.apache.flink.core.memory.DataOutputViewStreamWrapper) KeyGroupsList(org.apache.flink.runtime.state.KeyGroupsList) ConcurrentModificationException(java.util.ConcurrentModificationException) IOException(java.io.IOException) KeyedStateCheckpointOutputStream(org.apache.flink.runtime.state.KeyedStateCheckpointOutputStream)

Example 4 with KeyedStateCheckpointOutputStream

use of org.apache.flink.runtime.state.KeyedStateCheckpointOutputStream in project flink by apache.

the class StateSnapshotContextSynchronousImplTest method testCreateRawKeyedStateOutput.

@Test
public void testCreateRawKeyedStateOutput() throws Exception {
    KeyedStateCheckpointOutputStream stream = snapshotContext.getRawKeyedOperatorStateOutput();
    Assert.assertNotNull(stream);
}
Also used : KeyedStateCheckpointOutputStream(org.apache.flink.runtime.state.KeyedStateCheckpointOutputStream) Test(org.junit.Test)

Aggregations

KeyedStateCheckpointOutputStream (org.apache.flink.runtime.state.KeyedStateCheckpointOutputStream)4 DataOutputViewStreamWrapper (org.apache.flink.core.memory.DataOutputViewStreamWrapper)3 KeyGroupsList (org.apache.flink.runtime.state.KeyGroupsList)3 IOException (java.io.IOException)1 ConcurrentModificationException (java.util.ConcurrentModificationException)1 ExecutionException (java.util.concurrent.ExecutionException)1 Test (org.junit.Test)1