Examples with HtmlParseData edu.uci.ics.crawler4j.parser.HtmlParseData used on opensource projects

Example 6 with HtmlParseData

use of edu.uci.ics.crawler4j.parser.HtmlParseData in project mastering-java by Kingminghuang.

the class Crawler method visit.

@Override
public void visit(Page page) {
    String url = page.getWebURL().getURL();
    System.out.println("URL: " + url);
    if (page.getParseData() instanceof HtmlParseData) {
        HtmlParseData htmlParseData = (HtmlParseData) page.getParseData();
        String text = htmlParseData.getText();
        String html = htmlParseData.getHtml();
        Set<WebURL> links = htmlParseData.getOutgoingUrls();
        System.out.println("Text length: " + text.length());
        System.out.println("HTML length: " + html.length());
        System.out.println("Number of outgoing links: " + links.size());
    }
}

Example 7 with HtmlParseData

use of edu.uci.ics.crawler4j.parser.HtmlParseData in project yyl_example by Relucent.

the class MyCrawler method visit.

/**
	 * 当URL下载完成会调用这个方法
	 */
@Override
public void visit(Page page) {
    String url = page.getWebURL().getURL();
    System.out.println("URL: " + url);
    if (page.getParseData() instanceof HtmlParseData) {
        HtmlParseData htmlParseData = (HtmlParseData) page.getParseData();
        String text = htmlParseData.getText();
        String html = htmlParseData.getHtml();
        Set<WebURL> links = htmlParseData.getOutgoingUrls();
        System.out.println("Text length: " + text.length());
        System.out.println("Html length: " + html.length());
        System.out.println("Number of outgoing links: " + links.size());
    }
}