Search in sources :

Example 6 with Path

use of org.apache.hadoop.fs.Path in project camel by apache.

the class HdfsConsumerTest method testReadBytes.

@Test
public void testReadBytes() throws Exception {
    if (!canTest()) {
        return;
    }
    final Path file = new Path(new File("target/test/test-camel-bytes").getAbsolutePath());
    Configuration conf = new Configuration();
    FileSystem fs1 = FileSystem.get(file.toUri(), conf);
    SequenceFile.Writer writer = createWriter(fs1, conf, file, NullWritable.class, BytesWritable.class);
    NullWritable keyWritable = NullWritable.get();
    BytesWritable valueWritable = new BytesWritable();
    String value = "CIAO!";
    valueWritable.set(value.getBytes(), 0, value.getBytes().length);
    writer.append(keyWritable, valueWritable);
    writer.sync();
    writer.close();
    MockEndpoint resultEndpoint = context.getEndpoint("mock:result", MockEndpoint.class);
    resultEndpoint.expectedMessageCount(1);
    context.addRoutes(new RouteBuilder() {

        public void configure() {
            from("hdfs:localhost/" + file.toUri() + "?fileSystemType=LOCAL&fileType=SEQUENCE_FILE&initialDelay=0").to("mock:result");
        }
    });
    context.start();
    resultEndpoint.assertIsSatisfied();
}
Also used : Path(org.apache.hadoop.fs.Path) Configuration(org.apache.hadoop.conf.Configuration) SequenceFile(org.apache.hadoop.io.SequenceFile) RouteBuilder(org.apache.camel.builder.RouteBuilder) MockEndpoint(org.apache.camel.component.mock.MockEndpoint) FileSystem(org.apache.hadoop.fs.FileSystem) BytesWritable(org.apache.hadoop.io.BytesWritable) ArrayFile(org.apache.hadoop.io.ArrayFile) SequenceFile(org.apache.hadoop.io.SequenceFile) File(java.io.File) NullWritable(org.apache.hadoop.io.NullWritable) Test(org.junit.Test)

Example 7 with Path

use of org.apache.hadoop.fs.Path in project camel by apache.

the class HdfsConsumerTest method tearDown.

@Override
public void tearDown() throws Exception {
    if (!canTest()) {
        return;
    }
    super.tearDown();
    Thread.sleep(100);
    Configuration conf = new Configuration();
    Path dir = new Path("target/test");
    FileSystem fs = FileSystem.get(dir.toUri(), conf);
    fs.delete(dir, true);
}
Also used : Path(org.apache.hadoop.fs.Path) Configuration(org.apache.hadoop.conf.Configuration) FileSystem(org.apache.hadoop.fs.FileSystem)

Example 8 with Path

use of org.apache.hadoop.fs.Path in project camel by apache.

the class HdfsProducerSplitTest method tearDown.

@Override
public void tearDown() throws Exception {
    if (!canTest()) {
        return;
    }
    super.tearDown();
    Thread.sleep(100);
    Configuration conf = new Configuration();
    Path dir = new Path("target/test");
    FileSystem fs = FileSystem.get(dir.toUri(), conf);
    fs.delete(dir, true);
}
Also used : Path(org.apache.hadoop.fs.Path) Configuration(org.apache.hadoop.conf.Configuration) FileSystem(org.apache.hadoop.fs.FileSystem)

Example 9 with Path

use of org.apache.hadoop.fs.Path in project camel by apache.

the class HdfsProducerTest method testBloomMapWriteText.

@Test
public void testBloomMapWriteText() throws Exception {
    if (!canTest()) {
        return;
    }
    String txtKey = "THEKEY";
    String txtValue = "CIAO MONDO !";
    template.sendBodyAndHeader("direct:write_text5", txtValue, "KEY", txtKey);
    Configuration conf = new Configuration();
    Path file1 = new Path("file:///" + TEMP_DIR.toUri() + "/test-camel-text5");
    FileSystem fs1 = FileSystem.get(file1.toUri(), conf);
    BloomMapFile.Reader reader = new BloomMapFile.Reader(fs1, "file:///" + TEMP_DIR.toUri() + "/test-camel-text5", conf);
    Text key = (Text) ReflectionUtils.newInstance(reader.getKeyClass(), conf);
    Text value = (Text) ReflectionUtils.newInstance(reader.getValueClass(), conf);
    reader.next(key, value);
    assertEquals(key.toString(), txtKey);
    assertEquals(value.toString(), txtValue);
    IOHelper.close(reader);
}
Also used : Path(org.apache.hadoop.fs.Path) Configuration(org.apache.hadoop.conf.Configuration) FileSystem(org.apache.hadoop.fs.FileSystem) BloomMapFile(org.apache.hadoop.io.BloomMapFile) Text(org.apache.hadoop.io.Text) Test(org.junit.Test)

Example 10 with Path

use of org.apache.hadoop.fs.Path in project camel by apache.

the class HdfsProducerTest method testMapWriteTextWithKey.

@Test
public void testMapWriteTextWithKey() throws Exception {
    if (!canTest()) {
        return;
    }
    String txtKey = "THEKEY";
    String txtValue = "CIAO MONDO !";
    template.sendBodyAndHeader("direct:write_text3", txtValue, "KEY", txtKey);
    Configuration conf = new Configuration();
    Path file1 = new Path("file:///" + TEMP_DIR.toUri() + "/test-camel-text3");
    FileSystem fs1 = FileSystem.get(file1.toUri(), conf);
    MapFile.Reader reader = new MapFile.Reader(fs1, "file:///" + TEMP_DIR.toUri() + "/test-camel-text3", conf);
    Text key = (Text) ReflectionUtils.newInstance(reader.getKeyClass(), conf);
    Text value = (Text) ReflectionUtils.newInstance(reader.getValueClass(), conf);
    reader.next(key, value);
    assertEquals(key.toString(), txtKey);
    assertEquals(value.toString(), txtValue);
    IOHelper.close(reader);
}
Also used : Path(org.apache.hadoop.fs.Path) Configuration(org.apache.hadoop.conf.Configuration) FileSystem(org.apache.hadoop.fs.FileSystem) BloomMapFile(org.apache.hadoop.io.BloomMapFile) MapFile(org.apache.hadoop.io.MapFile) Text(org.apache.hadoop.io.Text) Test(org.junit.Test)

Aggregations

Path (org.apache.hadoop.fs.Path)7063 Test (org.junit.Test)2926 FileSystem (org.apache.hadoop.fs.FileSystem)2223 Configuration (org.apache.hadoop.conf.Configuration)1608 IOException (java.io.IOException)1574 FileStatus (org.apache.hadoop.fs.FileStatus)912 FSDataOutputStream (org.apache.hadoop.fs.FSDataOutputStream)662 ArrayList (java.util.ArrayList)644 File (java.io.File)518 MiniDFSCluster (org.apache.hadoop.hdfs.MiniDFSCluster)342 JobConf (org.apache.hadoop.mapred.JobConf)332 Job (org.apache.hadoop.mapreduce.Job)322 FileNotFoundException (java.io.FileNotFoundException)319 FSDataInputStream (org.apache.hadoop.fs.FSDataInputStream)293 HashMap (java.util.HashMap)279 DistributedFileSystem (org.apache.hadoop.hdfs.DistributedFileSystem)276 FsPermission (org.apache.hadoop.fs.permission.FsPermission)270 URI (java.net.URI)267 HdfsConfiguration (org.apache.hadoop.hdfs.HdfsConfiguration)220 Text (org.apache.hadoop.io.Text)185