Examples with IHdfsFile com.dtstack.dtcenter.loader.client.IHdfsFile used on opensource projects

Example 21 with IHdfsFile

use of com.dtstack.dtcenter.loader.client.IHdfsFile in project Taier by DTStack.

the class HdfsOperator method copyFromLocal.

public static void copyFromLocal(Map<String, Object> conf, Map<String, Object> kerberos, String srcPath, String dstPath, boolean overwrite) {
    HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
    IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
    hdfsClient.copyFromLocal(sourceDTO, srcPath, dstPath, overwrite);
}

Example 22 with IHdfsFile

use of com.dtstack.dtcenter.loader.client.IHdfsFile in project Taier by DTStack.

the class HdfsOperator method getContentSummary.

/**
 * 获取hdfs上指定路径的ContentSummary
 *
 * @param conf
 * @param kerberos
 * @param targetPath
 * @return
 */
public static HDFSContentSummary getContentSummary(Map<String, Object> conf, Map<String, Object> kerberos, String targetPath) {
    try {
        HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
        IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
        return hdfsClient.getContentSummary(sourceDTO, targetPath);
    } catch (Exception e) {
        throw new DtCenterDefException(String.format("获取hdfs文件内容摘要异常!原因是：%s", e.getMessage()));
    }
}

Example 23 with IHdfsFile

use of com.dtstack.dtcenter.loader.client.IHdfsFile in project Taier by DTStack.

the class HdfsOperator method copyDirector.

/**
 * 获取hdfs上指定路径集合的ContentSummary
 *
 * @param conf hadoop配置
 * @param kerberos kerberos配置
 * @param src 原路径
 * @param dist 复制路径
 */
public static void copyDirector(Map<String, Object> conf, Map<String, Object> kerberos, String src, String dist) {
    try {
        HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
        IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
        hdfsClient.copyDirector(sourceDTO, src, dist);
    } catch (Exception e) {
        throw new DtCenterDefException(String.format("hdfs内复制文件夹异常!原因是：%s", e.getMessage()));
    }
}

Example 24 with IHdfsFile

use of com.dtstack.dtcenter.loader.client.IHdfsFile in project Taier by DTStack.

the class HdfsOperator method listFiles.

public static List<FileStatus> listFiles(Map<String, Object> conf, Map<String, Object> kerberos, String parentSrc, boolean isIterate) {
    HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
    IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
    return hdfsClient.listAllFiles(sourceDTO, parentSrc, isIterate);
}

Example 25 with IHdfsFile

use of com.dtstack.dtcenter.loader.client.IHdfsFile in project Taier by DTStack.

the class HdfsOperator method uploadLocalFileToHdfs.

public static void uploadLocalFileToHdfs(Map<String, Object> conf, Map<String, Object> kerberos, String localFilePath, String hdfsDir) {
    HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
    IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
    hdfsClient.uploadLocalFileToHdfs(sourceDTO, localFilePath, hdfsDir);
}