Examples with GithubRepoPageProcessor us.codecraft.webmagic.processor.example.GithubRepoPageProcessor used on opensource projects

Example 1 with GithubRepoPageProcessor

use of us.codecraft.webmagic.processor.example.GithubRepoPageProcessor in project webmagic by code4craft.

the class SpiderMonitorTest method testInherit.

@Test
public void testInherit() throws Exception {
    SpiderMonitor spiderMonitor = new SpiderMonitor() {

        @Override
        protected SpiderStatusMXBean getSpiderStatusMBean(Spider spider, MonitorSpiderListener monitorSpiderListener) {
            return new CustomSpiderStatus(spider, monitorSpiderListener);
        }
    };
    Spider zhihuSpider = Spider.create(new ZhihuPageProcessor()).addUrl("http://my.oschina.net/flashsword/blog").thread(2);
    Spider githubSpider = Spider.create(new GithubRepoPageProcessor()).addUrl("https://github.com/code4craft");
    spiderMonitor.register(zhihuSpider, githubSpider);
}

Example 2 with GithubRepoPageProcessor

use of us.codecraft.webmagic.processor.example.GithubRepoPageProcessor in project webmagic by code4craft.

the class MonitorExample method main.

public static void main(String[] args) throws Exception {
    Spider zhihuSpider = Spider.create(new ZhihuPageProcessor()).addUrl("http://my.oschina.net/flashsword/blog");
    Spider githubSpider = Spider.create(new GithubRepoPageProcessor()).addUrl("https://github.com/code4craft");
    SpiderMonitor.instance().register(zhihuSpider);
    SpiderMonitor.instance().register(githubSpider);
    zhihuSpider.start();
    githubSpider.start();
}