Search in sources :

Example 1 with SpiderMonitor

use of us.codecraft.webmagic.monitor.SpiderMonitor in project webmagic by code4craft.

the class Kr36NewsModel method main.

public static void main(String[] args) throws IOException, JMException {
    //Just for benchmark
    Spider thread = OOSpider.create(Site.me().addStartUrl("http://www.36kr.com/").setSleepTime(0), new PageModelPipeline() {

        @Override
        public void process(Object o, Task task) {
        }
    }, Kr36NewsModel.class).thread(20);
    thread.start();
    SpiderMonitor spiderMonitor = SpiderMonitor.instance();
    spiderMonitor.register(thread);
}
Also used : PageModelPipeline(us.codecraft.webmagic.pipeline.PageModelPipeline) Task(us.codecraft.webmagic.Task) SpiderMonitor(us.codecraft.webmagic.monitor.SpiderMonitor) Spider(us.codecraft.webmagic.Spider) OOSpider(us.codecraft.webmagic.model.OOSpider)

Aggregations

Spider (us.codecraft.webmagic.Spider)1 Task (us.codecraft.webmagic.Task)1 OOSpider (us.codecraft.webmagic.model.OOSpider)1 SpiderMonitor (us.codecraft.webmagic.monitor.SpiderMonitor)1 PageModelPipeline (us.codecraft.webmagic.pipeline.PageModelPipeline)1