Examples with HdfsSourceDTO com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO used on opensource projects

Example 11 with HdfsSourceDTO

use of com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO in project Taier by DTStack.

the class HdfsOperator method listAllFilePath.

public static List<String> listAllFilePath(Map<String, Object> conf, Map<String, Object> kerberos, String parentSrc) {
    HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
    IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
    return hdfsClient.listAllFilePath(sourceDTO, parentSrc);
}

Example 12 with HdfsSourceDTO

use of com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO in project Taier by DTStack.

the class HdfsOperator method delete.

/**
 * 直接删除hdfs上文件
 * @param conf hadoop文件
 * @param kerberos kerberos文件
 * @param remotePath hdfs路径
 * @param recursive 是否递归删除
 * @return 删除结果
 * @throws Exception 异常
 */
public static boolean delete(Map<String, Object> conf, Map<String, Object> kerberos, String remotePath, boolean recursive) {
    HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
    IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
    return hdfsClient.delete(sourceDTO, remotePath, recursive);
}

Example 13 with HdfsSourceDTO

use of com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO in project Taier by DTStack.

the class HdfsOperator method checkAndDele.

/**
 * 文件检测并删除,递归删除
 *
 * @return
 */
public static boolean checkAndDele(Map<String, Object> conf, Map<String, Object> kerberos, String fileUri) {
    HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
    IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
    return hdfsClient.checkAndDelete(sourceDTO, fileUri);
}

Example 14 with HdfsSourceDTO

use of com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO in project Taier by DTStack.

the class HdfsOperator method createDir.

public static boolean createDir(Map<String, Object> conf, Map<String, Object> kerberos, String dir, short permission) {
    HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
    IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
    dir = uri(dir);
    return hdfsClient.createDir(sourceDTO, dir, permission);
}

Example 15 with HdfsSourceDTO

use of com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO in project Taier by DTStack.

the class HdfsOperator method getContentSummary.

/**
 * 获取hdfs上指定路径集合的ContentSummary
 *
 * @param conf
 * @param kerberos
 * @param targetPaths
 * @return
 */
public static List<HDFSContentSummary> getContentSummary(Map<String, Object> conf, Map<String, Object> kerberos, List<String> targetPaths) {
    try {
        HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
        IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
        return hdfsClient.getContentSummary(sourceDTO, targetPaths);
    } catch (Exception e) {
        throw new DtCenterDefException(String.format("获取hdfs文件内容摘要异常!原因是：%s", e.getMessage()));
    }
}