Search in sources :

Example 1 with PhantomJSDownloader

use of us.codecraft.webmagic.downloader.PhantomJSDownloader in project webmagic by code4craft.

the class PhantomJSPageProcessor method main.

public static void main(String[] args) throws Exception {
    PhantomJSDownloader phantomDownloader = new PhantomJSDownloader().setRetryNum(3);
    CollectorPipeline<ResultItems> collectorPipeline = new ResultItemsCollectorPipeline();
    Spider.create(new PhantomJSPageProcessor()).addUrl(//%B6%AC%D7%B0为冬装的GBK编码
    "http://s.taobao.com/search?q=%B6%AC%D7%B0&sort=sale-desc").setDownloader(phantomDownloader).addPipeline(collectorPipeline).thread((Runtime.getRuntime().availableProcessors() - 1) << 1).run();
    List<ResultItems> resultItemsList = collectorPipeline.getCollected();
    System.out.println(resultItemsList.get(0).get("html").toString());
}
Also used : PhantomJSDownloader(us.codecraft.webmagic.downloader.PhantomJSDownloader) ResultItemsCollectorPipeline(us.codecraft.webmagic.pipeline.ResultItemsCollectorPipeline) ResultItems(us.codecraft.webmagic.ResultItems)

Aggregations

ResultItems (us.codecraft.webmagic.ResultItems)1 PhantomJSDownloader (us.codecraft.webmagic.downloader.PhantomJSDownloader)1 ResultItemsCollectorPipeline (us.codecraft.webmagic.pipeline.ResultItemsCollectorPipeline)1