Search in sources :

Example 16 with IHdfsFile

use of com.dtstack.dtcenter.loader.client.IHdfsFile in project Taier by DTStack.

the class HdfsOperator method uploadInputStreamToHdfs.

public static boolean uploadInputStreamToHdfs(Map<String, Object> conf, Map<String, Object> kerberos, byte[] bytes, String hdfsPath) {
    HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
    IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
    return hdfsClient.uploadInputStreamToHdfs(sourceDTO, bytes, hdfsPath);
}
Also used : IHdfsFile(com.dtstack.dtcenter.loader.client.IHdfsFile) HdfsSourceDTO(com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO)

Example 17 with IHdfsFile

use of com.dtstack.dtcenter.loader.client.IHdfsFile in project Taier by DTStack.

the class HdfsOperator method createDir.

public static boolean createDir(Map<String, Object> conf, Map<String, Object> kerberos, String dir) {
    dir = uri(dir);
    HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
    IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
    return hdfsClient.createDir(sourceDTO, dir, null);
}
Also used : IHdfsFile(com.dtstack.dtcenter.loader.client.IHdfsFile) HdfsSourceDTO(com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO)

Example 18 with IHdfsFile

use of com.dtstack.dtcenter.loader.client.IHdfsFile in project Taier by DTStack.

the class HdfsOperator method fileMerge.

/**
 * 获取hdfs上指定路径集合的ContentSummary
 *
 * @param conf hadoop配置
 * @param kerberos kerberos配置
 * @param src 原路径
 * @param mergePath 合并临时目录
 * @param fileFormat 文件存储格式
 * @param needCombineFileSizeLimit 需要合并的文件大小阈值
 * @param maxCombinedFileSize 合并后的最大大小
 */
public static void fileMerge(Map<String, Object> conf, Map<String, Object> kerberos, String src, String mergePath, FileFormat fileFormat, Long maxCombinedFileSize, Long needCombineFileSizeLimit) {
    try {
        HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
        IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
        hdfsClient.fileMerge(sourceDTO, src, mergePath, fileFormat, maxCombinedFileSize, needCombineFileSizeLimit);
    } catch (Exception e) {
        throw new DtCenterDefException(String.format("文件合并异常!原因是:%s", e.getMessage()));
    }
}
Also used : IHdfsFile(com.dtstack.dtcenter.loader.client.IHdfsFile) DtCenterDefException(com.dtstack.taier.common.exception.DtCenterDefException) HdfsSourceDTO(com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO) DtCenterDefException(com.dtstack.taier.common.exception.DtCenterDefException) IOException(java.io.IOException) UnknownHostException(java.net.UnknownHostException)

Example 19 with IHdfsFile

use of com.dtstack.dtcenter.loader.client.IHdfsFile in project Taier by DTStack.

the class HdfsOperator method getHdfsFileClient.

public static IHdfsFile getHdfsFileClient(Map<String, Object> conf, Map<String, Object> kerberos) {
    HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
    IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
    return hdfsClient;
}
Also used : IHdfsFile(com.dtstack.dtcenter.loader.client.IHdfsFile) HdfsSourceDTO(com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO)

Example 20 with IHdfsFile

use of com.dtstack.dtcenter.loader.client.IHdfsFile in project Taier by DTStack.

the class HdfsOperator method copyToLocal.

public static void copyToLocal(Map<String, Object> conf, Map<String, Object> kerberos, String srcPath, String dstPath) {
    HdfsSourceDTO sourceDTO = getSourceDTO(conf, kerberos);
    IHdfsFile hdfsClient = ClientCache.getHdfs(DataSourceType.HDFS.getVal());
    hdfsClient.copyToLocal(sourceDTO, srcPath, dstPath);
}
Also used : IHdfsFile(com.dtstack.dtcenter.loader.client.IHdfsFile) HdfsSourceDTO(com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO)

Aggregations

IHdfsFile (com.dtstack.dtcenter.loader.client.IHdfsFile)26 HdfsSourceDTO (com.dtstack.dtcenter.loader.dto.source.HdfsSourceDTO)26 DtCenterDefException (com.dtstack.taier.common.exception.DtCenterDefException)6 IOException (java.io.IOException)5 UnknownHostException (java.net.UnknownHostException)5 SqlQueryDTO (com.dtstack.dtcenter.loader.dto.SqlQueryDTO)1 Map (java.util.Map)1