Examples with MultiPagePipeline us.codecraft.webmagic.pipeline.MultiPagePipeline used on opensource projects

Example 1 with MultiPagePipeline

use of us.codecraft.webmagic.pipeline.MultiPagePipeline in project webmagic by code4craft.

the class QuickStarter method main.

public static void main(String[] args) {
    init();
    String key = null;
    key = readKey(key);
    System.out.println("The demo started and will last 20 seconds...");
    //Start spider
    OOSpider.create(Site.me(), clazzMap.get(key)).addUrl(urlMap.get(key)).addPipeline(new MultiPagePipeline()).addPipeline(new ConsolePipeline()).runAsync();
    try {
        Thread.sleep(20000);
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
    System.out.println("The demo stopped!");
    System.out.println("To more usage, try to customize your own Spider!");
    System.exit(0);
}