Examples with HdfsSourceDTO com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO used on opensource projects

Example 1 with HdfsSourceDTO

use of com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO in project Taier by DTStack.

the class LogPluginDownload method init.

private void init() throws Exception {
    Object kerberosConfig = hdfsConf.get("kerberosConfig");
    Map<String, Object> kerberosConfMap = null;
    if (Objects.nonNull(kerberosConfig)) {
        if (kerberosConfig instanceof String) {
            kerberosConfMap = PublicUtil.objectToMap(kerberosConfig);
        } else if (kerberosConfig instanceof Map) {
            kerberosConfMap = (Map<String, Object>) kerberosConfig;
        }
    }
    HdfsSourceDTO sourceDTO = HdfsSourceDTO.builder().config(PublicUtil.objectToStr(hdfsConf)).defaultFS(hdfsConf.getOrDefault("fs.defaultFS", "").toString()).kerberosConfig(kerberosConfMap).yarnConf(yarnConf).appIdStr(applicationStr).readLimit(readLimit).user(user).build();
    IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
    SqlQueryDTO sqlQueryDTO = SqlQueryDTO.builder().build();
    hdfsLogDownloader = hdfsClient.getLogDownloader(sourceDTO, sqlQueryDTO);
}

Example 2 with HdfsSourceDTO

use of com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO in project Taier by DTStack.

the class HdfsOperator method isDirExist.

@SuppressWarnings("resource")
public static boolean isDirExist(Map<String, Object> conf, Map<String, Object> kerberos, String dir) {
    dir = uri(dir);
    HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
    IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
    return hdfsClient.isDirExist(sourceDTO, dir);
}

Example 3 with HdfsSourceDTO

use of com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO in project Taier by DTStack.

the class HdfsOperator method checkConnection.

public static boolean checkConnection(Map<String, Object> conf, Map<String, Object> kerberos, String defaultFs) {
    HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
    if (StringUtils.isNotEmpty(defaultFs)) {
        sourceDTO.setDefaultFS(defaultFs);
    }
    IClient hdfsClient = ClientCache.getClient(DataSourceType.HDFS.getVal());
    return hdfsClient.testCon(sourceDTO);
}

Example 4 with HdfsSourceDTO

use of com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO in project Taier by DTStack.

the class HdfsOperator method getFileStatus.

public static FileStatus getFileStatus(Map<String, Object> conf, Map<String, Object> kerberos, String dir) {
    HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
    IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
    if (isFileExist(conf, kerberos, dir)) {
        return hdfsClient.getStatus(sourceDTO, uri(dir));
    } else {
        throw new DtCenterDefException(dir + " 文件不存在");
    }
}

Example 5 with HdfsSourceDTO

use of com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO in project Taier by DTStack.

the class HdfsOperator method listStatus.

/**
 * 获取 HDFS 文件或者文件夹状态
 *
 * @param conf
 * @param kerberos
 * @param parentSrc
 * @return
 * @throws Exception
 */
public static List<FileStatus> listStatus(Map<String, Object> conf, Map<String, Object> kerberos, String parentSrc) {
    HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
    IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getPluginName());
    return hdfsClient.listStatus(sourceDTO, parentSrc);
}