Examples with Offloaded org.apache.flink.runtime.deployment.TaskDeploymentDescriptor.Offloaded used on opensource projects

Example 1 with Offloaded

use of org.apache.flink.runtime.deployment.TaskDeploymentDescriptor.Offloaded in project flink by apache.

the class InputGateDeploymentDescriptor method loadBigData.

public void loadBigData(@Nullable PermanentBlobService blobService, JobID jobId) throws IOException, ClassNotFoundException {
    if (serializedInputChannels instanceof Offloaded) {
        PermanentBlobKey blobKey = ((Offloaded<ShuffleDescriptor[]>) serializedInputChannels).serializedValueKey;
        Preconditions.checkNotNull(blobService);
        // NOTE: Do not delete the ShuffleDescriptor BLOBs since it may be needed again during
        // recovery. (it is deleted automatically on the BLOB server and cache when its
        // partition is no longer available or the job enters a terminal state)
        CompressedSerializedValue<ShuffleDescriptor[]> serializedValue = CompressedSerializedValue.fromBytes(blobService.readFile(jobId, blobKey));
        serializedInputChannels = new NonOffloaded<>(serializedValue);
        Preconditions.checkNotNull(serializedInputChannels);
    }
}

Example 2 with Offloaded

use of org.apache.flink.runtime.deployment.TaskDeploymentDescriptor.Offloaded in project flink by apache.

the class IntermediateResult method clearCachedInformationForPartitionGroup.

public void clearCachedInformationForPartitionGroup(ConsumedPartitionGroup consumedPartitionGroup) {
    // When a ConsumedPartitionGroup changes, the cache of ShuffleDescriptors for this
    // partition group is no longer valid and needs to be removed.
    // 
    // Currently, there are two scenarios:
    // 1. The ConsumedPartitionGroup is released
    // 2. Its producer encounters a failover
    // Remove the cache for the ConsumedPartitionGroup and notify the BLOB writer to delete the
    // cache if it is offloaded
    final MaybeOffloaded<ShuffleDescriptor[]> cache = this.shuffleDescriptorCache.remove(consumedPartitionGroup);
    if (cache instanceof Offloaded) {
        PermanentBlobKey blobKey = ((Offloaded<ShuffleDescriptor[]>) cache).serializedValueKey;
        this.producer.getGraph().deleteBlobs(Collections.singletonList(blobKey));
    }
}