Examples with ResultItemsCollectorPipeline us.codecraft.webmagic.pipeline.ResultItemsCollectorPipeline used on opensource projects

Example 1 with ResultItemsCollectorPipeline

use of us.codecraft.webmagic.pipeline.ResultItemsCollectorPipeline in project webmagic by code4craft.

the class PhantomJSPageProcessor method main.

public static void main(String[] args) throws Exception {
    PhantomJSDownloader phantomDownloader = new PhantomJSDownloader().setRetryNum(3);
    CollectorPipeline<ResultItems> collectorPipeline = new ResultItemsCollectorPipeline();
    Spider.create(new PhantomJSPageProcessor()).addUrl(//%B6%AC%D7%B0为冬装的GBK编码
    "http://s.taobao.com/search?q=%B6%AC%D7%B0&sort=sale-desc").setDownloader(phantomDownloader).addPipeline(collectorPipeline).thread((Runtime.getRuntime().availableProcessors() - 1) << 1).run();
    List<ResultItems> resultItemsList = collectorPipeline.getCollected();
    System.out.println(resultItemsList.get(0).get("html").toString());
}