Examples with SparkEnv$ org.apache.spark.SparkEnv$ used on opensource projects

Example 1 with SparkEnv$

use of org.apache.spark.SparkEnv$ in project beam by apache.

the class GlobalWatermarkHolder method clear.

@VisibleForTesting
public static synchronized void clear() {
    sourceTimes.clear();
    lastWatermarkedBatchTime = 0;
    writeLocalWatermarkCopy(null);
    final SparkEnv sparkEnv = SparkEnv.get();
    if (sparkEnv != null) {
        final BlockManager blockManager = sparkEnv.blockManager();
        blockManager.removeBlock(WATERMARKS_BLOCK_ID, true);
    }
}

Example 2 with SparkEnv$

use of org.apache.spark.SparkEnv$ in project TileDB-Spark by TileDB-Inc.

the class MetricsUpdater method getSource.

/**
 * Fetch a TileDBMetricSource from the task or evn context. If we don't find a TileDBMetricSource
 * this means that tiledb metrics are not enabled
 *
 * @param task task or null if not executor
 * @return TileDBMetricSource if found
 */
Optional<TileDBMetricsSource> getSource(TaskContext task) {
    if (task != null) {
        Option<Source> source = task.getMetricsSources(sourceName).headOption();
        if (source.isDefined()) {
            return Optional.of((TileDBMetricsSource) source.get());
        }
    } else {
        SparkEnv env = SparkEnv.get();
        Seq<Source> sources = env.metricsSystem().getSourcesByName(sourceName);
        if (sources.length() > 0)
            return Optional.of((TileDBMetricsSource) sources.head());
    }
    return Optional.empty();
}

Example 3 with SparkEnv$

use of org.apache.spark.SparkEnv$ in project OpenLineage by OpenLineage.

the class OpenLineageSparkListener method onApplicationStart.

/**
 * Check the {@link SparkConf} for open lineage configuration.
 *
 * @param applicationStart
 */
@Override
public void onApplicationStart(SparkListenerApplicationStart applicationStart) {
    if (contextFactory != null) {
        return;
    }
    SparkEnv sparkEnv = SparkEnv$.MODULE$.get();
    if (sparkEnv != null) {
        try {
            ArgumentParser args = parseConf(sparkEnv.conf());
            contextFactory = new ContextFactory(new EventEmitter(args));
        } catch (URISyntaxException e) {
            log.error("Unable to parse open lineage endpoint. Lineage events will not be collected", e);
        }
    } else {
        log.warn("Open lineage listener instantiated, but no configuration could be found. " + "Lineage events will not be collected");
    }
}