Examples with Request us.codecraft.webmagic.Request used on opensource projects

Example 41 with Request

use of us.codecraft.webmagic.Request in project webmagic by code4craft.

the class ProcessorBenchmark method test.

@Ignore
@Test
public void test() {
    ModelPageProcessor modelPageProcessor = ModelPageProcessor.create(Site.me(), OschinaBlog.class);
    Page page = new Page();
    page.setRequest(new Request("http://my.oschina.net/flashsword/blog"));
    page.setUrl(new PlainText("http://my.oschina.net/flashsword/blog"));
    page.setHtml(new Html(html));
    long time = System.currentTimeMillis();
    for (int i = 0; i < 1000; i++) {
        modelPageProcessor.process(page);
    }
    System.out.println(System.currentTimeMillis() - time);
    time = System.currentTimeMillis();
    for (int i = 0; i < 1000; i++) {
        modelPageProcessor.process(page);
    }
    System.out.println(System.currentTimeMillis() - time);
}

Example 42 with Request

use of us.codecraft.webmagic.Request in project webmagic by code4craft.

the class RedisSchedulerTest method test.

@Ignore("environment depended")
@Test
public void test() {
    Task task = new Task() {

        @Override
        public String getUUID() {
            return "1";
        }

        @Override
        public Site getSite() {
            return null;
        }
    };
    Request request = new Request("http://www.ibm.com/developerworks/cn/java/j-javadev2-22/");
    request.putExtra("1", "2");
    redisScheduler.push(request, task);
    Request poll = redisScheduler.poll(task);
    assertThat(poll).isEqualTo(request);
}

Example 43 with Request

use of us.codecraft.webmagic.Request in project webmagic by code4craft.

the class RequestUtilsTest method test_generate_range.

@Test
public void test_generate_range() throws Exception {
    List<Request> requests = RequestUtils.from("http://angularjs.cn/api/article/latest?p=[1-3]&s=20");
    assertThat(requests).containsExactly(new Request("http://angularjs.cn/api/article/latest?p=1&s=20"), new Request("http://angularjs.cn/api/article/latest?p=2&s=20"), new Request("http://angularjs.cn/api/article/latest?p=3&s=20"));
}

Example 44 with Request

use of us.codecraft.webmagic.Request in project webmagic by code4craft.

the class RequestUtils method from.

public static List<Request> from(String exp) {
    Matcher matcher = p4Range.matcher(exp);
    if (!matcher.find()) {
        return Collections.singletonList(new Request(exp));
    }
    int rangeFrom = Integer.parseInt(matcher.group(1));
    int rangeTo = Integer.parseInt(matcher.group(2));
    if (rangeFrom > rangeTo) {
        return Collections.emptyList();
    }
    List<Request> requests = new ArrayList<Request>(rangeTo - rangeFrom + 1);
    for (int i = rangeFrom; i <= rangeTo; i++) {
        requests.add(new Request(matcher.replaceAll(String.valueOf(i))));
    }
    return requests;
}

Example 45 with Request

use of us.codecraft.webmagic.Request in project webmagic by code4craft.

the class PageMocker method getMockPage.

public Page getMockPage() throws IOException {
    Page page = new Page();
    page.setRawText(IOUtils.toString(PageMocker.class.getClassLoader().getResourceAsStream("html/mock-webmagic.html")));
    page.setRequest(new Request("http://webmagic.io/list/0"));
    page.setUrl(new PlainText("http://webmagic.io/list/0"));
    return page;
}