Search in sources :

Example 6 with JavaHBaseContext

use of org.apache.hadoop.hbase.spark.JavaHBaseContext in project hbase by apache.

the class JavaHBaseStreamingBulkPutExample method main.

public static void main(String[] args) {
    if (args.length < 4) {
        System.out.println("JavaHBaseBulkPutExample  " + "{host} {port} {tableName}");
        return;
    }
    String host = args[0];
    String port = args[1];
    String tableName = args[2];
    SparkConf sparkConf = new SparkConf().setAppName("JavaHBaseStreamingBulkPutExample " + tableName + ":" + port + ":" + tableName);
    JavaSparkContext jsc = new JavaSparkContext(sparkConf);
    try {
        JavaStreamingContext jssc = new JavaStreamingContext(jsc, new Duration(1000));
        JavaReceiverInputDStream<String> javaDstream = jssc.socketTextStream(host, Integer.parseInt(port));
        Configuration conf = HBaseConfiguration.create();
        JavaHBaseContext hbaseContext = new JavaHBaseContext(jsc, conf);
        hbaseContext.streamBulkPut(javaDstream, TableName.valueOf(tableName), new PutFunction());
    } finally {
        jsc.stop();
    }
}
Also used : JavaStreamingContext(org.apache.spark.streaming.api.java.JavaStreamingContext) HBaseConfiguration(org.apache.hadoop.hbase.HBaseConfiguration) Configuration(org.apache.hadoop.conf.Configuration) Duration(org.apache.spark.streaming.Duration) JavaHBaseContext(org.apache.hadoop.hbase.spark.JavaHBaseContext) JavaSparkContext(org.apache.spark.api.java.JavaSparkContext) SparkConf(org.apache.spark.SparkConf)

Aggregations

Configuration (org.apache.hadoop.conf.Configuration)6 HBaseConfiguration (org.apache.hadoop.hbase.HBaseConfiguration)6 JavaHBaseContext (org.apache.hadoop.hbase.spark.JavaHBaseContext)6 SparkConf (org.apache.spark.SparkConf)6 JavaSparkContext (org.apache.spark.api.java.JavaSparkContext)6 ArrayList (java.util.ArrayList)4 Tuple2 (scala.Tuple2)2 Iterator (java.util.Iterator)1 BufferedMutator (org.apache.hadoop.hbase.client.BufferedMutator)1 Connection (org.apache.hadoop.hbase.client.Connection)1 Get (org.apache.hadoop.hbase.client.Get)1 Put (org.apache.hadoop.hbase.client.Put)1 Result (org.apache.hadoop.hbase.client.Result)1 Scan (org.apache.hadoop.hbase.client.Scan)1 Table (org.apache.hadoop.hbase.client.Table)1 Duration (org.apache.spark.streaming.Duration)1 JavaStreamingContext (org.apache.spark.streaming.api.java.JavaStreamingContext)1