Examples with VSCrawler com.virjar.vscrawler.core.VSCrawler used on opensource projects

Example 6 with VSCrawler

use of com.virjar.vscrawler.core.VSCrawler in project vscrawler by virjar.

the class GrabController method grab.

@RequestMapping("/grab")
@ResponseBody
public WebJsonResponse<?> grab(@RequestBody GrabRequest grabRequestBean) {
    try {
        VSCrawler vsCrawler = crawlerManager.get(grabRequestBean.getCrawlerName());
        if (vsCrawler == null) {
            return ReturnUtil.failed("no crawler defined :" + grabRequestBean.getCrawlerName());
        }
        Seed seed = new Seed(JSONObject.toJSONString(grabRequestBean));
        GrabResult crawlResult = vsCrawler.grabSync(seed);
        List<Object> strings = crawlResult.allObjectResult();
        if (strings.size() == 0 && seed.getRetry() > 0) {
            return ReturnUtil.failed("timeOut", ReturnUtil.status_timeout);
        } else {
            return ReturnUtil.success(strings);
        }
    } catch (Exception e) {
        return ReturnUtil.failed(e.getMessage());
    }
}

Example 7 with VSCrawler

use of com.virjar.vscrawler.core.VSCrawler in project vscrawler by virjar.

the class CrawlerController method start.

@RequestMapping("/startCrawler")
@ResponseBody
public WebJsonResponse<String> start(@RequestParam("crawlerName") String crawlerName) {
    VSCrawler vsCrawler = crawlerManager.get(crawlerName);
    if (vsCrawler == null) {
        return ReturnUtil.failed("not crawler defined");
    }
    vsCrawler.start();
    return ReturnUtil.success("success");
}

Example 8 with VSCrawler

use of com.virjar.vscrawler.core.VSCrawler in project vscrawler by virjar.

the class ResourceController method reloadResource.

@RequestMapping("/reloadResource")
@ResponseBody
public WebJsonResponse<String> reloadResource(@RequestParam("crawlerName") String appSource, @RequestParam("resourceName") String resourceName) {
    VSCrawler vsCrawler = crawlerManager.get(appSource);
    if (vsCrawler == null) {
        return ReturnUtil.failed("no crawler defined :" + appSource);
    }
    VSCrawlerContext vsCrawlerContext = vsCrawler.getVsCrawlerContext();
    vsCrawlerContext.getResourceManager().reloadResource(resourceName);
    return ReturnUtil.success("success");
}

Example 9 with VSCrawler

use of com.virjar.vscrawler.core.VSCrawler in project vscrawler by virjar.

the class ResourceController method reloadAccount.

@RequestMapping("/reloadAccount")
@ResponseBody
public WebJsonResponse<String> reloadAccount(@RequestParam("crawlerName") String appSource) {
    VSCrawler vsCrawler = crawlerManager.get(appSource);
    if (vsCrawler == null) {
        return ReturnUtil.failed("no crawler defined :" + appSource);
    }
    VSCrawlerContext vsCrawlerContext = vsCrawler.getVsCrawlerContext();
    vsCrawlerContext.getResourceManager().reloadResource(vsCrawlerContext.makeUserResourceTag());
    return ReturnUtil.success("success");
}

Example 10 with VSCrawler

use of com.virjar.vscrawler.core.VSCrawler in project vscrawler by virjar.

the class VSCrawlerClassLoader method loadCrawler.

/**
 * @param crawlerEntryName 爬虫入口类,应该是com.virjar.vscrawler.web.crawler.CrawlerBuilder的实现类
 * @return 由入口类构造的一个爬虫对象
 * @see CrawlerBuilder
 */
public CrawlerBean loadCrawler(String crawlerEntryName, WebApplicationContext webApplicationContext) throws InstantiationException, IllegalAccessException {
    // check
    try {
        CrawlerBuilder crawlerBuilder = (CrawlerBuilder) loadClass(crawlerEntryName).newInstance();
        if (crawlerBuilder instanceof SpringContextAware) {
            SpringContextAware springContextAware = (SpringContextAware) crawlerBuilder;
            springContextAware.init4SpringContext(webApplicationContext);
        }
        // for spring bean auto injection
        injectDependency(crawlerBuilder, true, webApplicationContext);
        VSCrawler vsCrawler = crawlerBuilder.build();
        return new CrawlerBean(vsCrawler, true, this);
    } catch (ClassNotFoundException e) {
    // this exception will not happen
    }
    return null;
}