Examples with UserCodeClassLoader org.apache.flink.util.UserCodeClassLoader used on opensource projects

Example 6 with UserCodeClassLoader

use of org.apache.flink.util.UserCodeClassLoader in project flink by apache.

the class BatchTask method initBroadcastInputsSerializers.

/**
 * Creates all the serializers and iterators for the broadcast inputs.
 */
protected void initBroadcastInputsSerializers(int numBroadcastInputs) {
    this.broadcastInputSerializers = new TypeSerializerFactory<?>[numBroadcastInputs];
    ClassLoader userCodeClassLoader = getUserCodeClassLoader();
    for (int i = 0; i < numBroadcastInputs; i++) {
        // ---------------- create the serializer first ---------------------
        final TypeSerializerFactory<?> serializerFactory = this.config.getBroadcastInputSerializer(i, userCodeClassLoader);
        this.broadcastInputSerializers[i] = serializerFactory;
    }
}

Example 7 with UserCodeClassLoader

use of org.apache.flink.util.UserCodeClassLoader in project flink by apache.

the class KafkaSource method createReader.

@VisibleForTesting
SourceReader<OUT, KafkaPartitionSplit> createReader(SourceReaderContext readerContext, Consumer<Collection<String>> splitFinishedHook) throws Exception {
    FutureCompletingBlockingQueue<RecordsWithSplitIds<ConsumerRecord<byte[], byte[]>>> elementsQueue = new FutureCompletingBlockingQueue<>();
    deserializationSchema.open(new DeserializationSchema.InitializationContext() {

        @Override
        public MetricGroup getMetricGroup() {
            return readerContext.metricGroup().addGroup("deserializer");
        }

        @Override
        public UserCodeClassLoader getUserCodeClassLoader() {
            return readerContext.getUserCodeClassLoader();
        }
    });
    final KafkaSourceReaderMetrics kafkaSourceReaderMetrics = new KafkaSourceReaderMetrics(readerContext.metricGroup());
    Supplier<KafkaPartitionSplitReader> splitReaderSupplier = () -> new KafkaPartitionSplitReader(props, readerContext, kafkaSourceReaderMetrics);
    KafkaRecordEmitter<OUT> recordEmitter = new KafkaRecordEmitter<>(deserializationSchema);
    return new KafkaSourceReader<>(elementsQueue, new KafkaSourceFetcherManager(elementsQueue, splitReaderSupplier::get, splitFinishedHook), recordEmitter, toConfiguration(props), readerContext, kafkaSourceReaderMetrics);
}

Example 8 with UserCodeClassLoader

use of org.apache.flink.util.UserCodeClassLoader in project flink by apache.

the class Task method createUserCodeClassloader.

private UserCodeClassLoader createUserCodeClassloader() throws Exception {
    long startDownloadTime = System.currentTimeMillis();
    // triggers the download of all missing jar files from the job manager
    final UserCodeClassLoader userCodeClassLoader = classLoaderHandle.getOrResolveClassLoader(requiredJarFiles, requiredClasspaths);
    LOG.debug("Getting user code class loader for task {} at library cache manager took {} milliseconds", executionId, System.currentTimeMillis() - startDownloadTime);
    return userCodeClassLoader;
}

Example 9 with UserCodeClassLoader

use of org.apache.flink.util.UserCodeClassLoader in project flink by apache.

the class SourceOperator method initReader.

/**
 * Initializes the reader. The code from this method should ideally happen in the constructor or
 * in the operator factory even. It has to happen here at a slightly later stage, because of the
 * lazy metric initialization.
 *
 * <p>Calling this method explicitly is an optional way to have the reader initialization a bit
 * earlier than in open(), as needed by the {@link
 * org.apache.flink.streaming.runtime.tasks.SourceOperatorStreamTask}
 *
 * <p>This code should move to the constructor once the metric groups are available at task
 * setup time.
 */
public void initReader() throws Exception {
    if (sourceReader != null) {
        return;
    }
    final int subtaskIndex = getRuntimeContext().getIndexOfThisSubtask();
    final SourceReaderContext context = new SourceReaderContext() {

        @Override
        public SourceReaderMetricGroup metricGroup() {
            return sourceMetricGroup;
        }

        @Override
        public Configuration getConfiguration() {
            return configuration;
        }

        @Override
        public String getLocalHostName() {
            return localHostname;
        }

        @Override
        public int getIndexOfSubtask() {
            return subtaskIndex;
        }

        @Override
        public void sendSplitRequest() {
            operatorEventGateway.sendEventToCoordinator(new RequestSplitEvent(getLocalHostName()));
        }

        @Override
        public void sendSourceEventToCoordinator(SourceEvent event) {
            operatorEventGateway.sendEventToCoordinator(new SourceEventWrapper(event));
        }

        @Override
        public UserCodeClassLoader getUserCodeClassLoader() {
            return new UserCodeClassLoader() {

                @Override
                public ClassLoader asClassLoader() {
                    return getRuntimeContext().getUserCodeClassLoader();
                }

                @Override
                public void registerReleaseHookIfAbsent(String releaseHookName, Runnable releaseHook) {
                    getRuntimeContext().registerUserCodeClassLoaderReleaseHookIfAbsent(releaseHookName, releaseHook);
                }
            };
        }
    };
    sourceReader = readerFactory.apply(context);
}

Example 10 with UserCodeClassLoader

use of org.apache.flink.util.UserCodeClassLoader in project flink by apache.

the class BlobLibraryCacheManagerTest method differentLeasesForSameJob_returnSameClassLoader.

@Test
public void differentLeasesForSameJob_returnSameClassLoader() throws IOException {
    final BlobLibraryCacheManager libraryCacheManager = createSimpleBlobLibraryCacheManager();
    final JobID jobId = new JobID();
    final LibraryCacheManager.ClassLoaderLease classLoaderLease1 = libraryCacheManager.registerClassLoaderLease(jobId);
    final LibraryCacheManager.ClassLoaderLease classLoaderLease2 = libraryCacheManager.registerClassLoaderLease(jobId);
    final UserCodeClassLoader classLoader1 = classLoaderLease1.getOrResolveClassLoader(Collections.emptyList(), Collections.emptyList());
    final UserCodeClassLoader classLoader2 = classLoaderLease2.getOrResolveClassLoader(Collections.emptyList(), Collections.emptyList());
    assertThat(classLoader1, sameInstance(classLoader2));
}