Search in sources :

Example 1 with GithubRepoPageProcessor

use of us.codecraft.webmagic.processor.example.GithubRepoPageProcessor in project webmagic by code4craft.

the class SpiderMonitorTest method testInherit.

@Test
public void testInherit() throws Exception {
    SpiderMonitor spiderMonitor = new SpiderMonitor() {

        @Override
        protected SpiderStatusMXBean getSpiderStatusMBean(Spider spider, MonitorSpiderListener monitorSpiderListener) {
            return new CustomSpiderStatus(spider, monitorSpiderListener);
        }
    };
    Spider zhihuSpider = Spider.create(new ZhihuPageProcessor()).addUrl("http://my.oschina.net/flashsword/blog").thread(2);
    Spider githubSpider = Spider.create(new GithubRepoPageProcessor()).addUrl("https://github.com/code4craft");
    spiderMonitor.register(zhihuSpider, githubSpider);
}
Also used : GithubRepoPageProcessor(us.codecraft.webmagic.processor.example.GithubRepoPageProcessor) Spider(us.codecraft.webmagic.Spider) ZhihuPageProcessor(us.codecraft.webmagic.processor.example.ZhihuPageProcessor) Test(org.junit.Test)

Example 2 with GithubRepoPageProcessor

use of us.codecraft.webmagic.processor.example.GithubRepoPageProcessor in project webmagic by code4craft.

the class MonitorExample method main.

public static void main(String[] args) throws Exception {
    Spider zhihuSpider = Spider.create(new ZhihuPageProcessor()).addUrl("http://my.oschina.net/flashsword/blog");
    Spider githubSpider = Spider.create(new GithubRepoPageProcessor()).addUrl("https://github.com/code4craft");
    SpiderMonitor.instance().register(zhihuSpider);
    SpiderMonitor.instance().register(githubSpider);
    zhihuSpider.start();
    githubSpider.start();
}
Also used : GithubRepoPageProcessor(us.codecraft.webmagic.processor.example.GithubRepoPageProcessor) Spider(us.codecraft.webmagic.Spider) ZhihuPageProcessor(us.codecraft.webmagic.processor.example.ZhihuPageProcessor)

Aggregations

Spider (us.codecraft.webmagic.Spider)2 GithubRepoPageProcessor (us.codecraft.webmagic.processor.example.GithubRepoPageProcessor)2 ZhihuPageProcessor (us.codecraft.webmagic.processor.example.ZhihuPageProcessor)2 Test (org.junit.Test)1