Examples with FirehoseFactoryToInputSourceAdaptor org.apache.druid.data.input.FirehoseFactoryToInputSourceAdaptor used on opensource projects

Example 1 with FirehoseFactoryToInputSourceAdaptor

use of org.apache.druid.data.input.FirehoseFactoryToInputSourceAdaptor in project druid by druid-io.

the class InputSourceSplitParallelIndexTaskRunner method newTaskSpec.

final SubTaskSpec<T> newTaskSpec(InputSplit split) {
    final FirehoseFactory firehoseFactory;
    final InputSource inputSource;
    if (baseInputSource instanceof FirehoseFactoryToInputSourceAdaptor) {
        firehoseFactory = ((FirehoseFactoryToInputSourceAdaptor) baseInputSource).getFirehoseFactory().withSplit(split);
        inputSource = null;
    } else {
        firehoseFactory = null;
        inputSource = baseInputSource.withSplit(split);
    }
    final ParallelIndexIngestionSpec subTaskIngestionSpec = new ParallelIndexIngestionSpec(ingestionSchema.getDataSchema(), new ParallelIndexIOConfig(firehoseFactory, inputSource, ingestionSchema.getIOConfig().getInputFormat(), ingestionSchema.getIOConfig().isAppendToExisting(), ingestionSchema.getIOConfig().isDropExisting()), ingestionSchema.getTuningConfig());
    return createSubTaskSpec(getBaseSubtaskSpecName() + "_" + getAndIncrementNextSpecId(), getGroupId(), getTaskId(), getContext(), split, subTaskIngestionSpec);
}

Example 2 with FirehoseFactoryToInputSourceAdaptor

use of org.apache.druid.data.input.FirehoseFactoryToInputSourceAdaptor in project druid by druid-io.

the class SinglePhaseParallelIndexTaskRunner method newTaskSpec.

@VisibleForTesting
SubTaskSpec<SinglePhaseSubTask> newTaskSpec(InputSplit split) {
    final FirehoseFactory firehoseFactory;
    final InputSource inputSource;
    if (baseInputSource instanceof FirehoseFactoryToInputSourceAdaptor) {
        firehoseFactory = ((FirehoseFactoryToInputSourceAdaptor) baseInputSource).getFirehoseFactory().withSplit(split);
        inputSource = null;
    } else {
        firehoseFactory = null;
        inputSource = baseInputSource.withSplit(split);
    }
    final Map<String, Object> subtaskContext = new HashMap<>(getContext());
    return new SinglePhaseSubTaskSpec(getBaseSubtaskSpecName() + "_" + getAndIncrementNextSpecId(), getGroupId(), getTaskId(), new ParallelIndexIngestionSpec(ingestionSchema.getDataSchema(), new ParallelIndexIOConfig(firehoseFactory, inputSource, ingestionSchema.getIOConfig().getInputFormat(), ingestionSchema.getIOConfig().isAppendToExisting(), ingestionSchema.getIOConfig().isDropExisting()), ingestionSchema.getTuningConfig()), subtaskContext, split);
}

Example 3 with FirehoseFactoryToInputSourceAdaptor

use of org.apache.druid.data.input.FirehoseFactoryToInputSourceAdaptor in project druid by druid-io.

the class SeekableStreamSamplerSpec method sample.

@Override
public SamplerResponse sample() {
    final InputSource inputSource;
    final InputFormat inputFormat;
    if (dataSchema.getParser() != null) {
        inputSource = new FirehoseFactoryToInputSourceAdaptor(new SeekableStreamSamplerFirehoseFactory(), dataSchema.getParser());
        inputFormat = null;
    } else {
        RecordSupplier<PartitionIdType, SequenceOffsetType, RecordType> recordSupplier;
        try {
            recordSupplier = createRecordSupplier();
        } catch (Exception e) {
            throw new SamplerException(e, "Unable to create RecordSupplier: %s", Throwables.getRootCause(e).getMessage());
        }
        inputSource = new RecordSupplierInputSource<>(ioConfig.getStream(), recordSupplier, ioConfig.isUseEarliestSequenceNumber());
        inputFormat = Preconditions.checkNotNull(ioConfig.getInputFormat(), "[spec.ioConfig.inputFormat] is required");
    }
    return inputSourceSampler.sample(inputSource, inputFormat, dataSchema, samplerConfig);
}