Examples with HdfsSourceDTO com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO used on opensource projects

Example 6 with HdfsSourceDTO

use of com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO in project Taier by DTStack.

the class HdfsOperator method getDirSize.

public static long getDirSize(Map<String, Object> conf, Map<String, Object> kerberos, String dir) {
    long size = 0;
    try {
        HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
        IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
        size = hdfsClient.getDirSize(sourceDTO, dir);
    } catch (Exception e) {
        logger.error("get dir size error:", e);
    }
    return size;
}

Example 7 with HdfsSourceDTO

use of com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO in project Taier by DTStack.

the class HdfsOperator method setPermission.

public static void setPermission(Map<String, Object> conf, Map<String, Object> kerberos, String path, String mode) {
    HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
    IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
    hdfsClient.setPermission(sourceDTO, path, mode);
}

Example 8 with HdfsSourceDTO

use of com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO in project Taier by DTStack.

the class HdfsOperator method getSourceDTO.

private static HdfsSourceDTO getSourceDTO(Map<String, Object> conf, Map<String, Object> kerberos) {
    conf = Objects.isNull(conf) ? Maps.newHashMap() : conf;
    HdfsSourceDTO hdfsSourceDTO = HdfsSourceDTO.builder().defaultFS(conf.getOrDefault("fs.defaultFS", "").toString()).kerberosConfig(kerberos).config(JSONObject.toJSONString(conf)).build();
    return hdfsSourceDTO;
}

Example 9 with HdfsSourceDTO

use of com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO in project Taier by DTStack.

the class HdfsOperator method downloadFileFromHDFS.

/**
 * 从HDFS上下载文件或文件夹到本地
 *
 * @throws IOException
 */
public static void downloadFileFromHDFS(Map<String, Object> conf, Map<String, Object> kerberos, String remotePath, String localDir) {
    HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
    IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getPluginName());
    hdfsClient.downloadFileFromHdfs(sourceDTO, remotePath, localDir);
}

Example 10 with HdfsSourceDTO

use of com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO in project Taier by DTStack.

the class HdfsOperator method copyFile.

public static void copyFile(Map<String, Object> conf, Map<String, Object> kerberos, String src, String dist, boolean overwrite) {
    HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
    IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
    hdfsClient.copyFile(sourceDTO, src, dist, overwrite);
}