Search in sources :

Example 1 with RescaleMappings

use of org.apache.flink.runtime.checkpoint.RescaleMappings in project flink by apache.

the class ResultSubpartitionRecoveredStateHandler method calculateMapping.

private List<RecoveredInputChannel> calculateMapping(InputChannelInfo info) {
    final RescaleMappings oldToNewMapping = oldToNewMappings.computeIfAbsent(info.getGateIdx(), idx -> channelMapping.getChannelMapping(idx).invert());
    final List<RecoveredInputChannel> channels = Arrays.stream(oldToNewMapping.getMappedIndexes(info.getInputChannelIdx())).mapToObj(newChannelIndex -> getChannel(info.getGateIdx(), newChannelIndex)).collect(Collectors.toList());
    if (channels.isEmpty()) {
        throw new IllegalStateException("Recovered a buffer from old " + info + " that has no mapping in " + channelMapping.getChannelMapping(info.getGateIdx()));
    }
    return channels;
}
Also used : RescaleMappings(org.apache.flink.runtime.checkpoint.RescaleMappings) InputGate(org.apache.flink.runtime.io.network.partition.consumer.InputGate) Arrays(java.util.Arrays) EventSerializer(org.apache.flink.runtime.io.network.api.serialization.EventSerializer) CheckpointedResultPartition(org.apache.flink.runtime.io.network.partition.CheckpointedResultPartition) SubtaskConnectionDescriptor(org.apache.flink.runtime.io.network.api.SubtaskConnectionDescriptor) BufferConsumer(org.apache.flink.runtime.io.network.buffer.BufferConsumer) ResultPartitionWriter(org.apache.flink.runtime.io.network.api.writer.ResultPartitionWriter) CheckpointedResultSubpartition(org.apache.flink.runtime.io.network.partition.CheckpointedResultSubpartition) IOException(java.io.IOException) HashMap(java.util.HashMap) Collectors(java.util.stream.Collectors) Buffer(org.apache.flink.runtime.io.network.buffer.Buffer) RecoveredInputChannel(org.apache.flink.runtime.io.network.partition.consumer.RecoveredInputChannel) ChannelStateByteBuffer.wrap(org.apache.flink.runtime.checkpoint.channel.ChannelStateByteBuffer.wrap) InflightDataRescalingDescriptor(org.apache.flink.runtime.checkpoint.InflightDataRescalingDescriptor) RescaleMappings(org.apache.flink.runtime.checkpoint.RescaleMappings) List(java.util.List) BufferBuilder(org.apache.flink.runtime.io.network.buffer.BufferBuilder) InputChannel(org.apache.flink.runtime.io.network.partition.consumer.InputChannel) Map(java.util.Map) RecoveredInputChannel(org.apache.flink.runtime.io.network.partition.consumer.RecoveredInputChannel)

Example 2 with RescaleMappings

use of org.apache.flink.runtime.checkpoint.RescaleMappings in project flink by apache.

the class ResultSubpartitionRecoveredStateHandler method calculateMapping.

private List<CheckpointedResultSubpartition> calculateMapping(ResultSubpartitionInfo info) {
    final RescaleMappings oldToNewMapping = oldToNewMappings.computeIfAbsent(info.getPartitionIdx(), idx -> channelMapping.getChannelMapping(idx).invert());
    final List<CheckpointedResultSubpartition> subpartitions = Arrays.stream(oldToNewMapping.getMappedIndexes(info.getSubPartitionIdx())).mapToObj(newIndexes -> getSubpartition(info.getPartitionIdx(), newIndexes)).collect(Collectors.toList());
    if (subpartitions.isEmpty()) {
        throw new IllegalStateException("Recovered a buffer from old " + info + " that has no mapping in " + channelMapping.getChannelMapping(info.getPartitionIdx()));
    }
    return subpartitions;
}
Also used : RescaleMappings(org.apache.flink.runtime.checkpoint.RescaleMappings) InputGate(org.apache.flink.runtime.io.network.partition.consumer.InputGate) Arrays(java.util.Arrays) EventSerializer(org.apache.flink.runtime.io.network.api.serialization.EventSerializer) CheckpointedResultPartition(org.apache.flink.runtime.io.network.partition.CheckpointedResultPartition) SubtaskConnectionDescriptor(org.apache.flink.runtime.io.network.api.SubtaskConnectionDescriptor) BufferConsumer(org.apache.flink.runtime.io.network.buffer.BufferConsumer) ResultPartitionWriter(org.apache.flink.runtime.io.network.api.writer.ResultPartitionWriter) CheckpointedResultSubpartition(org.apache.flink.runtime.io.network.partition.CheckpointedResultSubpartition) IOException(java.io.IOException) HashMap(java.util.HashMap) Collectors(java.util.stream.Collectors) Buffer(org.apache.flink.runtime.io.network.buffer.Buffer) RecoveredInputChannel(org.apache.flink.runtime.io.network.partition.consumer.RecoveredInputChannel) ChannelStateByteBuffer.wrap(org.apache.flink.runtime.checkpoint.channel.ChannelStateByteBuffer.wrap) InflightDataRescalingDescriptor(org.apache.flink.runtime.checkpoint.InflightDataRescalingDescriptor) RescaleMappings(org.apache.flink.runtime.checkpoint.RescaleMappings) List(java.util.List) BufferBuilder(org.apache.flink.runtime.io.network.buffer.BufferBuilder) InputChannel(org.apache.flink.runtime.io.network.partition.consumer.InputChannel) Map(java.util.Map) CheckpointedResultSubpartition(org.apache.flink.runtime.io.network.partition.CheckpointedResultSubpartition)

Aggregations

IOException (java.io.IOException)2 Arrays (java.util.Arrays)2 HashMap (java.util.HashMap)2 List (java.util.List)2 Map (java.util.Map)2 Collectors (java.util.stream.Collectors)2 InflightDataRescalingDescriptor (org.apache.flink.runtime.checkpoint.InflightDataRescalingDescriptor)2 RescaleMappings (org.apache.flink.runtime.checkpoint.RescaleMappings)2 ChannelStateByteBuffer.wrap (org.apache.flink.runtime.checkpoint.channel.ChannelStateByteBuffer.wrap)2 SubtaskConnectionDescriptor (org.apache.flink.runtime.io.network.api.SubtaskConnectionDescriptor)2 EventSerializer (org.apache.flink.runtime.io.network.api.serialization.EventSerializer)2 ResultPartitionWriter (org.apache.flink.runtime.io.network.api.writer.ResultPartitionWriter)2 Buffer (org.apache.flink.runtime.io.network.buffer.Buffer)2 BufferBuilder (org.apache.flink.runtime.io.network.buffer.BufferBuilder)2 BufferConsumer (org.apache.flink.runtime.io.network.buffer.BufferConsumer)2 CheckpointedResultPartition (org.apache.flink.runtime.io.network.partition.CheckpointedResultPartition)2 CheckpointedResultSubpartition (org.apache.flink.runtime.io.network.partition.CheckpointedResultSubpartition)2 InputChannel (org.apache.flink.runtime.io.network.partition.consumer.InputChannel)2 InputGate (org.apache.flink.runtime.io.network.partition.consumer.InputGate)2 RecoveredInputChannel (org.apache.flink.runtime.io.network.partition.consumer.RecoveredInputChannel)2