Examples with DownLoadMethod com.virjar.vscrawler.core.processor.configurableprocessor.annotiondriven.annotation.DownLoadMethod used on opensource projects

Example 1 with DownLoadMethod

use of com.virjar.vscrawler.core.processor.configurableprocessor.annotiondriven.annotation.DownLoadMethod in project vscrawler by virjar.

the class ImageDownloadProcessor method downLoadImage.

// DownLoadMethod修饰方法,当下载逻辑不是get,或者需要设置下载参数等特殊逻辑的时候,可以通过这个注解替换默认下载逻辑
@DownLoadMethod
public // 不过由DownLoadMethod修饰的方法,必须满足DownLoadMethod需要的方法签名格式(返回类型为string,有且仅包含seed,crawlerSession两个参数)
String downLoadImage(Seed seed, CrawlerSession crawlerSession) {
    Header[] headers = HeaderBuilder.create().withRefer(seed.getExt().get("fromUrl")).defaultCommonHeader().buildArray();
    byte[] entity = crawlerSession.getCrawlerHttpClient().getEntity(seed.getData(), headers);
    if (entity == null) {
        seed.retry();
        return null;
    }
    try {
        // 文件根据网站,路径,base自动计算
        Files.write(// 文件根据网站,路径,base自动计算
        entity, new File(PathResolver.sourceToUnderLine("~/Desktop/testpic", seed.getData())));
    } catch (IOException e) {
        e.printStackTrace();
    }
    return null;
}