Examples with HdfsSourceDTO com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO used on opensource projects

Example 16 with HdfsSourceDTO

use of com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO in project Taier by DTStack.

the class HdfsOperator method deleteFiles.

public static boolean deleteFiles(Map<String, Object> conf, Map<String, Object> kerberos, List<String> fileNames) {
    HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
    IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
    return hdfsClient.deleteFiles(sourceDTO, fileNames);
}

Example 17 with HdfsSourceDTO

use of com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO in project Taier by DTStack.

the class HdfsOperator method isFileExist.

public static boolean isFileExist(Map<String, Object> conf, Map<String, Object> kerberos, String dir) {
    HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
    IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
    dir = uri(dir);
    return hdfsClient.isFileExist(sourceDTO, dir);
}

Example 18 with HdfsSourceDTO

use of com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO in project Taier by DTStack.

the class HdfsOperator method uploadInputStreamToHdfs.

public static boolean uploadInputStreamToHdfs(Map<String, Object> conf, Map<String, Object> kerberos, byte[] bytes, String hdfsPath) {
    HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
    IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
    return hdfsClient.uploadInputStreamToHdfs(sourceDTO, bytes, hdfsPath);
}

Example 19 with HdfsSourceDTO

use of com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO in project Taier by DTStack.

the class HdfsOperator method createDir.

public static boolean createDir(Map<String, Object> conf, Map<String, Object> kerberos, String dir) {
    dir = uri(dir);
    HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
    IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
    return hdfsClient.createDir(sourceDTO, dir, null);
}

Example 20 with HdfsSourceDTO

use of com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO in project Taier by DTStack.

the class HdfsOperator method fileMerge.

/**
 * 获取hdfs上指定路径集合的ContentSummary
 *
 * @param conf hadoop配置
 * @param kerberos kerberos配置
 * @param src 原路径
 * @param mergePath 合并临时目录
 * @param fileFormat 文件存储格式
 * @param needCombineFileSizeLimit 需要合并的文件大小阈值
 * @param maxCombinedFileSize 合并后的最大大小
 */
public static void fileMerge(Map<String, Object> conf, Map<String, Object> kerberos, String src, String mergePath, FileFormat fileFormat, Long maxCombinedFileSize, Long needCombineFileSizeLimit) {
    try {
        HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
        IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
        hdfsClient.fileMerge(sourceDTO, src, mergePath, fileFormat, maxCombinedFileSize, needCombineFileSizeLimit);
    } catch (Exception e) {
        throw new DtCenterDefException(String.format("文件合并异常!原因是：%s", e.getMessage()));
    }
}