Search in sources :

Example 1 with IntStream.range

use of java.util.stream.IntStream.range in project hazelcast-jet by hazelcast.

the class ReadHdfsPTest method writeToFile.

private void writeToFile() throws IOException {
    Configuration conf = new Configuration();
    LocalFileSystem local = FileSystem.getLocal(conf);
    IntStream.range(0, 4).mapToObj(i -> createPath()).forEach(path -> uncheckRun(() -> {
        paths.add(path);
        if (SequenceFileInputFormat.class.equals(inputFormatClass)) {
            writeToSequenceFile(conf, path);
        } else {
            writeToTextFile(local, path);
        }
    }));
}
Also used : Arrays(java.util.Arrays) HdfsProcessors.readHdfsP(com.hazelcast.jet.hadoop.HdfsProcessors.readHdfsP) Util.uncheckRun(com.hazelcast.jet.impl.util.Util.uncheckRun) IntStream.range(java.util.stream.IntStream.range) FileSystem(org.apache.hadoop.fs.FileSystem) Text(org.apache.hadoop.io.Text) SequenceFile(org.apache.hadoop.io.SequenceFile) DistributedBiFunction(com.hazelcast.jet.function.DistributedBiFunction) Future(java.util.concurrent.Future) SequenceFileInputFormat(org.apache.hadoop.mapred.SequenceFileInputFormat) Configuration(org.apache.hadoop.conf.Configuration) InputFormat(org.apache.hadoop.mapred.InputFormat) Path(org.apache.hadoop.fs.Path) DAG(com.hazelcast.jet.core.DAG) IntWritable(org.apache.hadoop.io.IntWritable) Parameterized(org.junit.runners.Parameterized) ExceptionUtil(com.hazelcast.jet.impl.util.ExceptionUtil) FileInputFormat(org.apache.hadoop.mapred.FileInputFormat) Collection(java.util.Collection) Set(java.util.Set) Category(org.junit.experimental.categories.Category) DistributedStream(com.hazelcast.jet.stream.DistributedStream) IntStream(java.util.stream.IntStream) TextInputFormat(org.apache.hadoop.mapred.TextInputFormat) JetInstance(com.hazelcast.jet.JetInstance) RunWith(org.junit.runner.RunWith) HashSet(java.util.HashSet) OutputStreamWriter(java.io.OutputStreamWriter) IList(com.hazelcast.core.IList) Before(org.junit.Before) Option(org.apache.hadoop.io.SequenceFile.Writer.Option) Files(java.nio.file.Files) BufferedWriter(java.io.BufferedWriter) Assert.assertTrue(org.junit.Assert.assertTrue) IOException(java.io.IOException) Test(org.junit.Test) IListJet(com.hazelcast.jet.IListJet) ParallelTest(com.hazelcast.test.annotation.ParallelTest) Integer.parseInt(java.lang.Integer.parseInt) JobConf(org.apache.hadoop.mapred.JobConf) Vertex(com.hazelcast.jet.core.Vertex) HdfsSources(com.hazelcast.jet.hadoop.HdfsSources) DistributedCollectors(com.hazelcast.jet.stream.DistributedCollectors) Writer(org.apache.hadoop.io.SequenceFile.Writer) HazelcastParametersRunnerFactory(com.hazelcast.test.HazelcastParametersRunnerFactory) LocalFileSystem(org.apache.hadoop.fs.LocalFileSystem) Util(com.hazelcast.jet.Util) Assert.assertEquals(org.junit.Assert.assertEquals) Edge.between(com.hazelcast.jet.core.Edge.between) SinkProcessors.writeListP(com.hazelcast.jet.core.processor.SinkProcessors.writeListP) Configuration(org.apache.hadoop.conf.Configuration) LocalFileSystem(org.apache.hadoop.fs.LocalFileSystem) SequenceFileInputFormat(org.apache.hadoop.mapred.SequenceFileInputFormat)

Aggregations

IList (com.hazelcast.core.IList)1 IListJet (com.hazelcast.jet.IListJet)1 JetInstance (com.hazelcast.jet.JetInstance)1 Util (com.hazelcast.jet.Util)1 DAG (com.hazelcast.jet.core.DAG)1 Edge.between (com.hazelcast.jet.core.Edge.between)1 Vertex (com.hazelcast.jet.core.Vertex)1 SinkProcessors.writeListP (com.hazelcast.jet.core.processor.SinkProcessors.writeListP)1 DistributedBiFunction (com.hazelcast.jet.function.DistributedBiFunction)1 HdfsProcessors.readHdfsP (com.hazelcast.jet.hadoop.HdfsProcessors.readHdfsP)1 HdfsSources (com.hazelcast.jet.hadoop.HdfsSources)1 ExceptionUtil (com.hazelcast.jet.impl.util.ExceptionUtil)1 Util.uncheckRun (com.hazelcast.jet.impl.util.Util.uncheckRun)1 DistributedCollectors (com.hazelcast.jet.stream.DistributedCollectors)1 DistributedStream (com.hazelcast.jet.stream.DistributedStream)1 HazelcastParametersRunnerFactory (com.hazelcast.test.HazelcastParametersRunnerFactory)1 ParallelTest (com.hazelcast.test.annotation.ParallelTest)1 BufferedWriter (java.io.BufferedWriter)1 IOException (java.io.IOException)1 OutputStreamWriter (java.io.OutputStreamWriter)1