Search in sources :

Example 1 with PcapReader

use of net.ripe.hadoop.pcap.PcapReader in project hadoop-pcap by RIPE-NCC.

the class PcapInputFormat method initPcapRecordReader.

public static PcapRecordReader initPcapRecordReader(Path path, long start, long length, TaskAttemptContext context) throws IOException {
    Configuration conf = context.getConfiguration();
    FileSystem fs = path.getFileSystem(conf);
    FSDataInputStream baseStream = fs.open(path);
    DataInputStream stream = baseStream;
    CompressionCodecFactory compressionCodecs = new CompressionCodecFactory(conf);
    final CompressionCodec codec = compressionCodecs.getCodec(path);
    if (codec != null)
        stream = new DataInputStream(codec.createInputStream(stream));
    PcapReader reader = initPcapReader(stream, conf);
    return new PcapRecordReader(reader, start, length, baseStream, stream, context);
}
Also used : Configuration(org.apache.hadoop.conf.Configuration) CompressionCodecFactory(org.apache.hadoop.io.compress.CompressionCodecFactory) PcapReader(net.ripe.hadoop.pcap.PcapReader) FileSystem(org.apache.hadoop.fs.FileSystem) FSDataInputStream(org.apache.hadoop.fs.FSDataInputStream) PcapRecordReader(net.ripe.hadoop.pcap.io.reader.PcapRecordReader) CompressionCodec(org.apache.hadoop.io.compress.CompressionCodec) DataInputStream(java.io.DataInputStream) FSDataInputStream(org.apache.hadoop.fs.FSDataInputStream)

Example 2 with PcapReader

use of net.ripe.hadoop.pcap.PcapReader in project hadoop-pcap by RIPE-NCC.

the class PcapInputFormat method initPcapRecordReader.

public static PcapRecordReader initPcapRecordReader(Path path, long start, long length, Reporter reporter, Configuration conf) throws IOException {
    FileSystem fs = path.getFileSystem(conf);
    FSDataInputStream baseStream = fs.open(path);
    DataInputStream stream = baseStream;
    CompressionCodecFactory compressionCodecs = new CompressionCodecFactory(conf);
    final CompressionCodec codec = compressionCodecs.getCodec(path);
    if (codec != null)
        stream = new DataInputStream(codec.createInputStream(stream));
    PcapReader reader = initPcapReader(stream, conf);
    return new PcapRecordReader(reader, start, length, baseStream, stream, reporter);
}
Also used : CompressionCodecFactory(org.apache.hadoop.io.compress.CompressionCodecFactory) PcapReader(net.ripe.hadoop.pcap.PcapReader) FileSystem(org.apache.hadoop.fs.FileSystem) FSDataInputStream(org.apache.hadoop.fs.FSDataInputStream) PcapRecordReader(net.ripe.hadoop.pcap.mr1.io.reader.PcapRecordReader) CompressionCodec(org.apache.hadoop.io.compress.CompressionCodec) DataInputStream(java.io.DataInputStream) FSDataInputStream(org.apache.hadoop.fs.FSDataInputStream)

Example 3 with PcapReader

use of net.ripe.hadoop.pcap.PcapReader in project hadoop-pcap by RIPE-NCC.

the class PcapReaderTest method findIPStartNULL.

@Test
public void findIPStartNULL() {
    PcapReader xreader = new PcapReader(PcapReader.LinkType.NULL);
    assertEquals(4, xreader.findIPStart(null));
}
Also used : PcapReader(net.ripe.hadoop.pcap.PcapReader) Test(org.junit.Test)

Example 4 with PcapReader

use of net.ripe.hadoop.pcap.PcapReader in project hadoop-pcap by RIPE-NCC.

the class PcapReaderTest method findIPStartEN10MB_8021Q.

@Test
public void findIPStartEN10MB_8021Q() {
    byte[] packet = new byte[20];
    PcapReader xreader = new PcapReader(PcapReader.LinkType.EN10MB);
    byte[] ethernetType8021Q = PcapReaderUtil.convertShort(PcapReader.ETHERNET_TYPE_8021Q);
    packet[12] = ethernetType8021Q[0];
    packet[13] = ethernetType8021Q[1];
    byte[] ethernetTypeIp = PcapReaderUtil.convertShort(PcapReader.ETHERNET_TYPE_IP);
    packet[16] = ethernetTypeIp[0];
    packet[17] = ethernetTypeIp[1];
    assertEquals(18, xreader.findIPStart(packet));
}
Also used : PcapReader(net.ripe.hadoop.pcap.PcapReader) Test(org.junit.Test)

Example 5 with PcapReader

use of net.ripe.hadoop.pcap.PcapReader in project hadoop-pcap by RIPE-NCC.

the class PcapReaderTest method findIPStartEN10MB.

@Test
public void findIPStartEN10MB() {
    byte[] packet = new byte[20];
    PcapReader xreader = new PcapReader(PcapReader.LinkType.EN10MB);
    byte[] ethernetType = PcapReaderUtil.convertShort(PcapReader.ETHERNET_TYPE_IP);
    packet[12] = ethernetType[0];
    packet[13] = ethernetType[1];
    assertEquals(14, xreader.findIPStart(packet));
}
Also used : PcapReader(net.ripe.hadoop.pcap.PcapReader) Test(org.junit.Test)

Aggregations

PcapReader (net.ripe.hadoop.pcap.PcapReader)15 Test (org.junit.Test)9 DataInputStream (java.io.DataInputStream)5 FSDataInputStream (org.apache.hadoop.fs.FSDataInputStream)5 FileSystem (org.apache.hadoop.fs.FileSystem)5 PcapRecordReader (net.ripe.hadoop.pcap.io.reader.PcapRecordReader)4 FileInputStream (java.io.FileInputStream)3 Packet (net.ripe.hadoop.pcap.packet.Packet)3 Path (org.apache.hadoop.fs.Path)3 JobConf (org.apache.hadoop.mapred.JobConf)3 Before (org.junit.Before)3 CompressionCodec (org.apache.hadoop.io.compress.CompressionCodec)2 CompressionCodecFactory (org.apache.hadoop.io.compress.CompressionCodecFactory)2 Joiner (com.google.common.base.Joiner)1 InputStream (java.io.InputStream)1 GZIPInputStream (java.util.zip.GZIPInputStream)1 PcapRecordReader (net.ripe.hadoop.pcap.mr1.io.reader.PcapRecordReader)1 Configuration (org.apache.hadoop.conf.Configuration)1