Examples with PipeConfiguration co.cask.cdap.filetailer.config.PipeConfiguration used on opensource projects

Example 6 with PipeConfiguration

use of co.cask.cdap.filetailer.config.PipeConfiguration in project cdap-ingest by caskdata.

the class PipeManager method setupPipes.

/**
   * Pipes setup
   *
   * @throws IOException in case a client stream cannot be created
   */
private ServiceManager setupPipes() throws IOException {
    StreamClient client = null;
    StreamWriter writer = null;
    try {
        List<PipeConfiguration> pipeConfList = getPipeConfigs();
        for (PipeConfiguration pipeConf : pipeConfList) {
            FileTailerQueue queue = new FileTailerQueue(pipeConf.getQueueSize());
            client = pipeConf.getSinkConfiguration().getStreamClient();
            String streamName = pipeConf.getSinkConfiguration().getStreamName();
            writer = getStreamWriterForPipe(client, streamName);
            FileTailerStateProcessor stateProcessor = new FileTailerStateProcessorImpl(pipeConf.getDaemonDir(), pipeConf.getStateFile());
            FileTailerMetricsProcessor metricsProcessor = new FileTailerMetricsProcessor(pipeConf.getDaemonDir(), pipeConf.getStatisticsFile(), pipeConf.getStatisticsSleepInterval(), pipeConf.getPipeName(), pipeConf.getSourceConfiguration().getFileName());
            pipeList.add(new Pipe(new LogTailer(pipeConf, queue, stateProcessor, metricsProcessor, null), new FileTailerSink(queue, writer, SinkStrategy.LOADBALANCE, stateProcessor, metricsProcessor, null, pipeConf.getSinkConfiguration().getPackSize()), metricsProcessor));
            client = null;
            writer = null;
        }
        return new ServiceManager(pipeList);
    } catch (ConfigurationLoadingException e) {
        throw new ConfigurationLoadingException("Error during loading configuration from file: " + confFile.getAbsolutePath() + e.getMessage());
    } finally {
        if (client != null) {
            client.close();
        }
        if (writer != null) {
            writer.close();
        }
    }
}

Example 7 with PipeConfiguration

use of co.cask.cdap.filetailer.config.PipeConfiguration in project cdap-ingest by caskdata.

the class PollingListenerImpl method setupPipe.

/**
   * Pipe setup
   *
   * @throws IOException if can not setup pipe
   */
private Pipe setupPipe(File file) throws IOException {
    PipeConfiguration pipeConfiguration = observerConf.getPipeConfiguration(file.getName());
    FileTailerQueue queue = new FileTailerQueue(pipeConfiguration.getQueueSize());
    StreamWriter writer = getStreamWriterForPipe(pipeConfiguration);
    FileTailerStateProcessor stateProcessor = new FileTailerStateProcessorImpl(observerConf.getDaemonDir(), pipeConfiguration.getStateFile());
    PipeListener pipeListener = new PipeListenerImpl(pipeConfiguration.getSourceConfiguration().getWorkDir(), file.getAbsolutePath(), observerConf.getDaemonDir() + "/" + pipeConfiguration.getStateFile());
    Pipe pipe = new Pipe(new LogTailer(pipeConfiguration, queue, stateProcessor, metricsProcessor, pipeListener), new FileTailerSink(queue, writer, SinkStrategy.LOADBALANCE, stateProcessor, metricsProcessor, pipeListener, pipeConfiguration.getSinkConfiguration().getPackSize()));
    pipeListener.setPipe(pipe);
    return pipe;
}

Example 8 with PipeConfiguration

use of co.cask.cdap.filetailer.config.PipeConfiguration in project cdap-ingest by caskdata.

the class TailerLogUtils method clearTestDir.

public static void clearTestDir() throws IOException {
    PipeConfiguration flowConf = loadConfig();
    File dir = flowConf.getSourceConfiguration().getWorkDir();
    FileUtils.cleanDirectory(dir);
}

Example 9 with PipeConfiguration

use of co.cask.cdap.filetailer.config.PipeConfiguration in project cdap-ingest by caskdata.

the class TailerLogUtils method deleteTestDir.

public static void deleteTestDir() throws IOException {
    PipeConfiguration flowConf = loadConfig();
    File dir = flowConf.getSourceConfiguration().getWorkDir();
    FileUtils.deleteDirectory(dir);
}

Example 10 with PipeConfiguration

use of co.cask.cdap.filetailer.config.PipeConfiguration in project cdap-ingest by caskdata.

the class TailerLogUtils method createTestDirIfNeed.

public static void createTestDirIfNeed() throws ConfigurationLoadingException {
    PipeConfiguration flowConf = loadConfig();
    File dir = flowConf.getSourceConfiguration().getWorkDir();
    if (!dir.exists()) {
        dir.mkdir();
    }
}