Search in sources :

Example 41 with OperatorState

use of org.apache.flink.runtime.checkpoint.OperatorState in project flink by apache.

the class ExistingSavepoint method readKeyedState.

/**
 * Read keyed state from an operator in a {@code Savepoint}.
 *
 * @param uid The uid of the operator.
 * @param function The {@link KeyedStateReaderFunction} that is called for each key in state.
 * @param keyTypeInfo The type information of the key in state.
 * @param outTypeInfo The type information of the output of the transform reader function.
 * @param <K> The type of the key in state.
 * @param <OUT> The output type of the transform function.
 * @return A {@code DataSet} of objects read from keyed state.
 * @throws IOException If the savepoint does not contain operator state with the given uid.
 */
public <K, OUT> DataSource<OUT> readKeyedState(String uid, KeyedStateReaderFunction<K, OUT> function, TypeInformation<K> keyTypeInfo, TypeInformation<OUT> outTypeInfo) throws IOException {
    OperatorState operatorState = metadata.getOperatorState(uid);
    KeyedStateInputFormat<K, VoidNamespace, OUT> inputFormat = new KeyedStateInputFormat<>(operatorState, stateBackend, env.getConfiguration(), new KeyedStateReaderOperator<>(function, keyTypeInfo));
    return env.createInput(inputFormat, outTypeInfo);
}
Also used : KeyedStateInputFormat(org.apache.flink.state.api.input.KeyedStateInputFormat) VoidNamespace(org.apache.flink.runtime.state.VoidNamespace) OperatorState(org.apache.flink.runtime.checkpoint.OperatorState)

Example 42 with OperatorState

use of org.apache.flink.runtime.checkpoint.OperatorState in project flink by apache.

the class ExistingSavepoint method readBroadcastState.

/**
 * Read operator {@code BroadcastState} from a {@code Savepoint} when a custom serializer was
 * used; e.g., a different serializer than the one returned by {@code
 * TypeInformation#createSerializer}.
 *
 * @param uid The uid of the operator.
 * @param name The (unique) name for the state.
 * @param keyTypeInfo The type information for the keys in the state.
 * @param valueTypeInfo The type information for the values in the state.
 * @param keySerializer The type serializer used to write keys into the state.
 * @param valueSerializer The type serializer used to write values into the state.
 * @param <K> The type of keys in state.
 * @param <V> The type of values in state.
 * @return A {@code DataSet} of key-value pairs from state.
 * @throws IOException If the savepoint path is invalid or the uid does not exist.
 */
public <K, V> DataSource<Tuple2<K, V>> readBroadcastState(String uid, String name, TypeInformation<K> keyTypeInfo, TypeInformation<V> valueTypeInfo, TypeSerializer<K> keySerializer, TypeSerializer<V> valueSerializer) throws IOException {
    OperatorState operatorState = metadata.getOperatorState(uid);
    MapStateDescriptor<K, V> descriptor = new MapStateDescriptor<>(name, keySerializer, valueSerializer);
    BroadcastStateInputFormat<K, V> inputFormat = new BroadcastStateInputFormat<>(operatorState, env.getConfiguration(), stateBackend, descriptor);
    return env.createInput(inputFormat, new TupleTypeInfo<>(keyTypeInfo, valueTypeInfo));
}
Also used : MapStateDescriptor(org.apache.flink.api.common.state.MapStateDescriptor) BroadcastStateInputFormat(org.apache.flink.state.api.input.BroadcastStateInputFormat) OperatorState(org.apache.flink.runtime.checkpoint.OperatorState)

Example 43 with OperatorState

use of org.apache.flink.runtime.checkpoint.OperatorState in project flink by apache.

the class ExistingSavepoint method readUnionState.

/**
 * Read operator {@code UnionState} from a {@code Savepoint}.
 *
 * @param uid The uid of the operator.
 * @param name The (unique) name for the state.
 * @param typeInfo The type of the elements in the state.
 * @param <T> The type of the values that are in the union state.
 * @return A {@code DataSet} representing the elements in state.
 * @throws IOException If the savepoint path is invalid or the uid does not exist.
 */
public <T> DataSource<T> readUnionState(String uid, String name, TypeInformation<T> typeInfo) throws IOException {
    OperatorState operatorState = metadata.getOperatorState(uid);
    ListStateDescriptor<T> descriptor = new ListStateDescriptor<>(name, typeInfo);
    UnionStateInputFormat<T> inputFormat = new UnionStateInputFormat<>(operatorState, env.getConfiguration(), stateBackend, descriptor);
    return env.createInput(inputFormat, typeInfo);
}
Also used : ListStateDescriptor(org.apache.flink.api.common.state.ListStateDescriptor) OperatorState(org.apache.flink.runtime.checkpoint.OperatorState) UnionStateInputFormat(org.apache.flink.state.api.input.UnionStateInputFormat)

Example 44 with OperatorState

use of org.apache.flink.runtime.checkpoint.OperatorState in project flink by apache.

the class SavepointReader method read.

/**
 * Loads an existing savepoint. Useful if you want to query the state of an existing
 * application. The savepoint will be read using the state backend defined via the clusters
 * configuration.
 *
 * @param env The execution environment used to transform the savepoint.
 * @param path The path to an existing savepoint on disk.
 * @return A {@link SavepointReader}.
 */
public static SavepointReader read(StreamExecutionEnvironment env, String path) throws IOException {
    CheckpointMetadata metadata = SavepointLoader.loadSavepointMetadata(path);
    int maxParallelism = metadata.getOperatorStates().stream().map(OperatorState::getMaxParallelism).max(Comparator.naturalOrder()).orElseThrow(() -> new RuntimeException("Savepoint must contain at least one operator state."));
    SavepointMetadataV2 savepointMetadata = new SavepointMetadataV2(maxParallelism, metadata.getMasterStates(), metadata.getOperatorStates());
    return new SavepointReader(env, savepointMetadata, null);
}
Also used : SavepointMetadataV2(org.apache.flink.state.api.runtime.metadata.SavepointMetadataV2) OperatorState(org.apache.flink.runtime.checkpoint.OperatorState) CheckpointMetadata(org.apache.flink.runtime.checkpoint.metadata.CheckpointMetadata)

Example 45 with OperatorState

use of org.apache.flink.runtime.checkpoint.OperatorState in project flink by apache.

the class SavepointReader method readListState.

/**
 * Read operator {@code ListState} from a {@code Savepoint} when a custom serializer was used;
 * e.g., a different serializer than the one returned by {@code
 * TypeInformation#createSerializer}.
 *
 * @param uid The uid of the operator.
 * @param name The (unique) name for the state.
 * @param typeInfo The type of the elements in the state.
 * @param serializer The serializer used to write the elements into state.
 * @param <T> The type of the values that are in the list state.
 * @return A {@code DataStream} representing the elements in state.
 * @throws IOException If the savepoint path is invalid or the uid does not exist.
 */
public <T> DataStream<T> readListState(String uid, String name, TypeInformation<T> typeInfo, TypeSerializer<T> serializer) throws IOException {
    OperatorState operatorState = metadata.getOperatorState(uid);
    ListStateDescriptor<T> descriptor = new ListStateDescriptor<>(name, serializer);
    ListStateInputFormat<T> inputFormat = new ListStateInputFormat<>(operatorState, MutableConfig.of(env.getConfiguration()), stateBackend, descriptor);
    return SourceBuilder.fromFormat(env, inputFormat, typeInfo);
}
Also used : ListStateInputFormat(org.apache.flink.state.api.input.ListStateInputFormat) ListStateDescriptor(org.apache.flink.api.common.state.ListStateDescriptor) OperatorState(org.apache.flink.runtime.checkpoint.OperatorState)

Aggregations

OperatorState (org.apache.flink.runtime.checkpoint.OperatorState)63 Test (org.junit.Test)22 Configuration (org.apache.flink.configuration.Configuration)17 OperatorSubtaskState (org.apache.flink.runtime.checkpoint.OperatorSubtaskState)14 OperatorID (org.apache.flink.runtime.jobgraph.OperatorID)14 MemoryStateBackend (org.apache.flink.runtime.state.memory.MemoryStateBackend)14 CheckpointMetadata (org.apache.flink.runtime.checkpoint.metadata.CheckpointMetadata)11 MasterState (org.apache.flink.runtime.checkpoint.MasterState)9 Random (java.util.Random)8 ListStateDescriptor (org.apache.flink.api.common.state.ListStateDescriptor)8 KeyGroupRangeInputSplit (org.apache.flink.state.api.input.splits.KeyGroupRangeInputSplit)7 ArrayList (java.util.ArrayList)6 KeyedStateReaderFunction (org.apache.flink.state.api.functions.KeyedStateReaderFunction)6 ThreadLocalRandom (java.util.concurrent.ThreadLocalRandom)5 TimeWindow (org.apache.flink.streaming.api.windowing.windows.TimeWindow)5 MapStateDescriptor (org.apache.flink.api.common.state.MapStateDescriptor)4 BroadcastStateInputFormat (org.apache.flink.state.api.input.BroadcastStateInputFormat)4 ListStateInputFormat (org.apache.flink.state.api.input.ListStateInputFormat)4 UnionStateInputFormat (org.apache.flink.state.api.input.UnionStateInputFormat)4 PassThroughReader (org.apache.flink.state.api.input.operator.window.PassThroughReader)4