Examples with GeccoEngine com.geccocrawler.gecco.GeccoEngine used on opensource projects

Example 1 with GeccoEngine

use of com.geccocrawler.gecco.GeccoEngine in project dq-easy-cloud by dq-open-cloud.

the class EcReptileEngineUtils method loadReptileEngine.

/**
 * <p>
 * 根据数据传输对象运行引擎 加载爬虫引擎
 * </p>
 *
 * @param reptileEngineDTO
 * @author daiqi
 * @创建时间 2018年6月11日 上午11:25:52
 */
public static synchronized GeccoEngine loadReptileEngine(EcReptileEngineDTO reptileEngineDTO, EcReptileDynamicBeanDTO reptileDynamicBeanDTO) {
    EcAssert.verifyObjNull(reptileEngineDTO, "reptileEngineDTO");
    GeccoEngine geccoEngineFromCache = EcReptileConfig.getReptileEngineBeanClassDTO().getGeccoEngine();
    GeccoEngine geccoEngine = convertFromEnginDTO(reptileEngineDTO, geccoEngineFromCache);
    if (EcBaseUtils.isNull(geccoEngineFromCache)) {
        geccoEngine.run();
    }
    geccoEngine.loop(false);
    EcReptileConfig.loadReptileEngineBeanClassDTO(reptileDynamicBeanDTO, geccoEngine);
    // 注册ruleClass到爬虫引擎
    registerRuleClassToReptileEngine();
    return geccoEngine;
}

Example 2 with GeccoEngine

use of com.geccocrawler.gecco.GeccoEngine in project dq-easy-cloud by dq-open-cloud.

the class EcReptileDynamicBeanServiceImpl method reptileData.

@Override
public EcBaseServiceResult reptileData(EcReptileDataDTO reptileDataDTO) {
    EcReptileEngineBeanClassDTO reptileEngineBeanClassDTOFromCache = EcReptileConfig.getReptileEngineBeanClassDTO();
    EcAssert.verifyObjNull(reptileEngineBeanClassDTOFromCache, "缓存reptileEngineBeanClassDTOFromCache");
    EcReptileDynamicBeanDTO dynamicBeanDTO = reptileEngineBeanClassDTOFromCache.getReptileDynamicBeanDTO();
    // 获取爬虫引擎
    GeccoEngine geccoEngine = reptileEngineBeanClassDTOFromCache.getGeccoEngine();
    EcAssert.verifyDataNotExistent(geccoEngine, "缓存中geccoEngine");
    EcReptileUtils.intoScheduler(geccoEngine, dynamicBeanDTO, reptileDataDTO);
    return EcBaseServiceResult.newInstanceOfSuccess();
}

Example 3 with GeccoEngine

use of com.geccocrawler.gecco.GeccoEngine in project dq-easy-cloud by dq-open-cloud.

the class EcReptileEngineUtils method registerRuleClassToReptileEngine.

/**
 * <p>
 * 注册规则class到爬虫引擎中
 * </p>
 * <pre>
 *     所需参数示例及其说明
 *     参数名称 : 示例值 : 说明 : 是否必须
 * </pre>
 * @author daiqi
 * @date 2018/6/13 17:20
 * @param
 * @return void
 */
private static void registerRuleClassToReptileEngine() {
    GeccoEngine geccoEngine = EcReptileConfig.getReptileEngineBeanClassDTO().getGeccoEngine();
    Map<String, Class<?>> ruleBeanClazzs = EcReptileConfig.getReptileEngineBeanClassDTO().getRuleBeanClazzs();
    try {
        geccoEngine.beginUpdateRule();
        for (String ruleBeanClazzName : ruleBeanClazzs.keySet()) {
            geccoEngine.register(ruleBeanClazzs.get(ruleBeanClazzName));
        }
    } finally {
        geccoEngine.endUpdateRule();
    }
}