Search in sources :

Example 26 with MetaDataAcquireFailedException

use of com.webank.wedatasphere.qualitis.metadata.exception.MetaDataAcquireFailedException in project Qualitis by WeBankFinTech.

the class MetaDataClientImpl method getDbByUserAndCluster.

@Override
public DataInfo<DbInfoDetail> getDbByUserAndCluster(GetDbByUserAndClusterRequest request) throws UnExpectedRequestException, MetaDataAcquireFailedException {
    // Check existence of cluster name
    ClusterInfo clusterInfo = checkClusterNameExists(request.getClusterName());
    String authUser = request.getLoginUser();
    // send request to get dbs
    String url = getPath(clusterInfo.getLinkisAddress()).path(linkisConfig.getDbPath()).toString();
    HttpHeaders headers = new HttpHeaders();
    headers.setContentType(MediaType.APPLICATION_JSON);
    headers.add("Token-User", authUser);
    headers.add("Token-Code", clusterInfo.getLinkisToken());
    HttpEntity<Object> entity = new HttpEntity<>(headers);
    LOGGER.info("Start to get db by user and cluster by linkis. url: {}, method: {}, body: {}", url, javax.ws.rs.HttpMethod.GET, entity);
    Map<String, Object> response = restTemplate.exchange(url, HttpMethod.GET, entity, Map.class).getBody();
    LOGGER.info("Start to get db by user and cluster by linkis. response: {}", response);
    if (!checkResponse(response)) {
        String message = (String) response.get("message");
        LOGGER.error("Error! Can not get meta data from linkis, message: " + message);
        throw new MetaDataAcquireFailedException("Error! Can not get meta data from linkis, message: " + message);
    }
    List<String> allDbs = ((List<Map<String, String>>) ((Map<String, Object>) response.get("data")).get("dbs")).stream().map(o -> o.get("dbName")).collect(Collectors.toList());
    DataInfo<DbInfoDetail> dataInfo = new DataInfo<>(allDbs.size());
    if (CollectionUtils.isEmpty(allDbs)) {
        return dataInfo;
    }
    List<DbInfoDetail> details = new ArrayList<>();
    for (String data : allDbs) {
        DbInfoDetail detail = new DbInfoDetail(data);
        details.add(detail);
    }
    dataInfo.setContent(details);
    return dataInfo;
}
Also used : ClusterInfoDetail(com.webank.wedatasphere.qualitis.metadata.response.cluster.ClusterInfoDetail) StringUtils(org.apache.commons.lang.StringUtils) GetTableByUserAndDbRequest(com.webank.wedatasphere.qualitis.metadata.request.GetTableByUserAndDbRequest) URLDecoder(java.net.URLDecoder) SpecCharEnum(com.webank.wedatasphere.qualitis.constant.SpecCharEnum) LinkisConfig(com.webank.wedatasphere.qualitis.config.LinkisConfig) LoggerFactory(org.slf4j.LoggerFactory) GetUserColumnByCsRequest(com.webank.wedatasphere.qualitis.metadata.request.GetUserColumnByCsRequest) Autowired(org.springframework.beans.factory.annotation.Autowired) ArrayList(java.util.ArrayList) GeneralResponse(com.webank.wedatasphere.qualitis.response.GeneralResponse) ClusterInfoDao(com.webank.wedatasphere.qualitis.dao.ClusterInfoDao) JSONException(org.json.JSONException) JSONObject(org.json.JSONObject) CollectionUtils(org.apache.commons.collections.CollectionUtils) Map(java.util.Map) GetUserTableByCsIdRequest(com.webank.wedatasphere.qualitis.metadata.request.GetUserTableByCsIdRequest) GetClusterByUserRequest(com.webank.wedatasphere.qualitis.metadata.request.GetClusterByUserRequest) UriBuilder(javax.ws.rs.core.UriBuilder) RestTemplate(org.springframework.web.client.RestTemplate) CsTableInfoDetail(com.webank.wedatasphere.qualitis.metadata.response.table.CsTableInfoDetail) RestClientException(org.springframework.web.client.RestClientException) PartitionStatisticsInfo(com.webank.wedatasphere.qualitis.metadata.response.table.PartitionStatisticsInfo) Logger(org.slf4j.Logger) HttpHeaders(org.springframework.http.HttpHeaders) MediaType(org.springframework.http.MediaType) HttpMethod(org.springframework.http.HttpMethod) TableInfoDetail(com.webank.wedatasphere.qualitis.metadata.response.table.TableInfoDetail) ResourceAccessException(org.springframework.web.client.ResourceAccessException) GetColumnByUserAndTableRequest(com.webank.wedatasphere.qualitis.metadata.request.GetColumnByUserAndTableRequest) ColumnInfoDetail(com.webank.wedatasphere.qualitis.metadata.response.column.ColumnInfoDetail) Collectors(java.util.stream.Collectors) LinkisResponseKeyEnum(com.webank.wedatasphere.qualitis.constant.LinkisResponseKeyEnum) DbInfoDetail(com.webank.wedatasphere.qualitis.metadata.response.db.DbInfoDetail) TableStatisticsInfo(com.webank.wedatasphere.qualitis.metadata.response.table.TableStatisticsInfo) MetaDataAcquireFailedException(com.webank.wedatasphere.qualitis.metadata.exception.MetaDataAcquireFailedException) GetDbByUserAndClusterRequest(com.webank.wedatasphere.qualitis.metadata.request.GetDbByUserAndClusterRequest) DataInfo(com.webank.wedatasphere.qualitis.metadata.response.DataInfo) List(java.util.List) HttpEntity(org.springframework.http.HttpEntity) Component(org.springframework.stereotype.Component) ClusterInfo(com.webank.wedatasphere.qualitis.entity.ClusterInfo) MetaDataClient(com.webank.wedatasphere.qualitis.metadata.client.MetaDataClient) UnExpectedRequestException(com.webank.wedatasphere.qualitis.exception.UnExpectedRequestException) UnsupportedEncodingException(java.io.UnsupportedEncodingException) DataInfo(com.webank.wedatasphere.qualitis.metadata.response.DataInfo) HttpHeaders(org.springframework.http.HttpHeaders) HttpEntity(org.springframework.http.HttpEntity) ArrayList(java.util.ArrayList) ClusterInfo(com.webank.wedatasphere.qualitis.entity.ClusterInfo) DbInfoDetail(com.webank.wedatasphere.qualitis.metadata.response.db.DbInfoDetail) MetaDataAcquireFailedException(com.webank.wedatasphere.qualitis.metadata.exception.MetaDataAcquireFailedException) JSONObject(org.json.JSONObject) ArrayList(java.util.ArrayList) List(java.util.List) Map(java.util.Map)

Example 27 with MetaDataAcquireFailedException

use of com.webank.wedatasphere.qualitis.metadata.exception.MetaDataAcquireFailedException in project Qualitis by WeBankFinTech.

the class MetaDataClientImpl method getDataSourceEnv.

@Override
public GeneralResponse<Map> getDataSourceEnv(String clusterName, String authUser) throws UnExpectedRequestException, MetaDataAcquireFailedException {
    // Check existence of cluster name
    ClusterInfo clusterInfo = checkClusterNameExists(clusterName);
    // send request to get dbs
    String url = getPath(clusterInfo.getLinkisAddress()).path(linkisConfig.getDatasourceEnv()).toString();
    HttpHeaders headers = new HttpHeaders();
    headers.setContentType(MediaType.APPLICATION_JSON);
    headers.add("Token-User", authUser);
    headers.add("Token-Code", clusterInfo.getLinkisToken());
    HttpEntity<Object> entity = new HttpEntity<>(headers);
    LOGGER.info("Start to get data source env by user and cluster by linkis. url: {}, method: {}, body: {}", url, javax.ws.rs.HttpMethod.GET, entity);
    Map<String, Object> response = restTemplate.exchange(url, HttpMethod.GET, entity, Map.class).getBody();
    LOGGER.info("Finish to get data source env by user and cluster by linkis. response: {}", response);
    if (!checkResponse(response)) {
        String message = (String) response.get("message");
        LOGGER.error("Error! Can not get meta data from linkis, message: " + message);
        throw new MetaDataAcquireFailedException("Error! Can not get meta data from linkis, exception: " + message);
    }
    Map data = (Map) response.get(LinkisResponseKeyEnum.DATA.getKey());
    List<Map> types = (List<Map>) data.get("query_list");
    return new GeneralResponse<>("200", "Success to get datasource env", data);
}
Also used : GeneralResponse(com.webank.wedatasphere.qualitis.response.GeneralResponse) HttpHeaders(org.springframework.http.HttpHeaders) ClusterInfo(com.webank.wedatasphere.qualitis.entity.ClusterInfo) HttpEntity(org.springframework.http.HttpEntity) MetaDataAcquireFailedException(com.webank.wedatasphere.qualitis.metadata.exception.MetaDataAcquireFailedException) JSONObject(org.json.JSONObject) ArrayList(java.util.ArrayList) List(java.util.List) Map(java.util.Map)

Example 28 with MetaDataAcquireFailedException

use of com.webank.wedatasphere.qualitis.metadata.exception.MetaDataAcquireFailedException in project Qualitis by WeBankFinTech.

the class MetaDataClientImpl method getTableByUserAndDb.

@Override
public DataInfo<TableInfoDetail> getTableByUserAndDb(GetTableByUserAndDbRequest request) throws UnExpectedRequestException, MetaDataAcquireFailedException {
    ClusterInfo clusterInfo = checkClusterNameExists(request.getClusterName());
    String authUser = request.getLoginUser();
    // send request to get dbs
    String url = getPath(clusterInfo.getLinkisAddress()).path(linkisConfig.getTablePath()).queryParam("database", request.getDbName()).toString();
    HttpHeaders headers = new HttpHeaders();
    headers.setContentType(MediaType.APPLICATION_JSON);
    headers.add("Token-User", authUser);
    headers.add("Token-Code", clusterInfo.getLinkisToken());
    HttpEntity<Object> entity = new HttpEntity<>(headers);
    LOGGER.info("Start to get table by user and cluster and db by linkis. url: {}, method: {}, body: {}", url, javax.ws.rs.HttpMethod.GET, entity);
    Map<String, Object> response = restTemplate.exchange(url, HttpMethod.GET, entity, Map.class).getBody();
    LOGGER.info("Finished to get table by user and cluster and db by linkis. response: {}", response);
    if (!checkResponse(response)) {
        String message = (String) response.get("message");
        LOGGER.error("Error! Can not get meta data from linkis, message: " + message);
        throw new MetaDataAcquireFailedException("Error! Can not get meta data from linkis, exception: " + message);
    }
    List<String> allTables = ((List<Map<String, String>>) ((Map<String, Object>) response.get("data")).get("tables")).stream().map(o -> o.get("tableName")).collect(Collectors.toList());
    DataInfo<TableInfoDetail> dataInfo = new DataInfo<>(allTables.size());
    if (CollectionUtils.isEmpty(allTables)) {
        return dataInfo;
    }
    List<TableInfoDetail> details = new ArrayList<>();
    for (String data : allTables) {
        TableInfoDetail detail = new TableInfoDetail(data);
        details.add(detail);
    }
    dataInfo.setContent(details);
    return dataInfo;
}
Also used : ClusterInfoDetail(com.webank.wedatasphere.qualitis.metadata.response.cluster.ClusterInfoDetail) StringUtils(org.apache.commons.lang.StringUtils) GetTableByUserAndDbRequest(com.webank.wedatasphere.qualitis.metadata.request.GetTableByUserAndDbRequest) URLDecoder(java.net.URLDecoder) SpecCharEnum(com.webank.wedatasphere.qualitis.constant.SpecCharEnum) LinkisConfig(com.webank.wedatasphere.qualitis.config.LinkisConfig) LoggerFactory(org.slf4j.LoggerFactory) GetUserColumnByCsRequest(com.webank.wedatasphere.qualitis.metadata.request.GetUserColumnByCsRequest) Autowired(org.springframework.beans.factory.annotation.Autowired) ArrayList(java.util.ArrayList) GeneralResponse(com.webank.wedatasphere.qualitis.response.GeneralResponse) ClusterInfoDao(com.webank.wedatasphere.qualitis.dao.ClusterInfoDao) JSONException(org.json.JSONException) JSONObject(org.json.JSONObject) CollectionUtils(org.apache.commons.collections.CollectionUtils) Map(java.util.Map) GetUserTableByCsIdRequest(com.webank.wedatasphere.qualitis.metadata.request.GetUserTableByCsIdRequest) GetClusterByUserRequest(com.webank.wedatasphere.qualitis.metadata.request.GetClusterByUserRequest) UriBuilder(javax.ws.rs.core.UriBuilder) RestTemplate(org.springframework.web.client.RestTemplate) CsTableInfoDetail(com.webank.wedatasphere.qualitis.metadata.response.table.CsTableInfoDetail) RestClientException(org.springframework.web.client.RestClientException) PartitionStatisticsInfo(com.webank.wedatasphere.qualitis.metadata.response.table.PartitionStatisticsInfo) Logger(org.slf4j.Logger) HttpHeaders(org.springframework.http.HttpHeaders) MediaType(org.springframework.http.MediaType) HttpMethod(org.springframework.http.HttpMethod) TableInfoDetail(com.webank.wedatasphere.qualitis.metadata.response.table.TableInfoDetail) ResourceAccessException(org.springframework.web.client.ResourceAccessException) GetColumnByUserAndTableRequest(com.webank.wedatasphere.qualitis.metadata.request.GetColumnByUserAndTableRequest) ColumnInfoDetail(com.webank.wedatasphere.qualitis.metadata.response.column.ColumnInfoDetail) Collectors(java.util.stream.Collectors) LinkisResponseKeyEnum(com.webank.wedatasphere.qualitis.constant.LinkisResponseKeyEnum) DbInfoDetail(com.webank.wedatasphere.qualitis.metadata.response.db.DbInfoDetail) TableStatisticsInfo(com.webank.wedatasphere.qualitis.metadata.response.table.TableStatisticsInfo) MetaDataAcquireFailedException(com.webank.wedatasphere.qualitis.metadata.exception.MetaDataAcquireFailedException) GetDbByUserAndClusterRequest(com.webank.wedatasphere.qualitis.metadata.request.GetDbByUserAndClusterRequest) DataInfo(com.webank.wedatasphere.qualitis.metadata.response.DataInfo) List(java.util.List) HttpEntity(org.springframework.http.HttpEntity) Component(org.springframework.stereotype.Component) ClusterInfo(com.webank.wedatasphere.qualitis.entity.ClusterInfo) MetaDataClient(com.webank.wedatasphere.qualitis.metadata.client.MetaDataClient) UnExpectedRequestException(com.webank.wedatasphere.qualitis.exception.UnExpectedRequestException) UnsupportedEncodingException(java.io.UnsupportedEncodingException) DataInfo(com.webank.wedatasphere.qualitis.metadata.response.DataInfo) HttpHeaders(org.springframework.http.HttpHeaders) HttpEntity(org.springframework.http.HttpEntity) ArrayList(java.util.ArrayList) ClusterInfo(com.webank.wedatasphere.qualitis.entity.ClusterInfo) MetaDataAcquireFailedException(com.webank.wedatasphere.qualitis.metadata.exception.MetaDataAcquireFailedException) JSONObject(org.json.JSONObject) ArrayList(java.util.ArrayList) List(java.util.List) Map(java.util.Map) CsTableInfoDetail(com.webank.wedatasphere.qualitis.metadata.response.table.CsTableInfoDetail) TableInfoDetail(com.webank.wedatasphere.qualitis.metadata.response.table.TableInfoDetail)

Example 29 with MetaDataAcquireFailedException

use of com.webank.wedatasphere.qualitis.metadata.exception.MetaDataAcquireFailedException in project Qualitis by WeBankFinTech.

the class MetaDataClientImpl method getColumnByUserAndTable.

@Override
public DataInfo<ColumnInfoDetail> getColumnByUserAndTable(GetColumnByUserAndTableRequest request) throws UnExpectedRequestException, MetaDataAcquireFailedException {
    ClusterInfo clusterInfo = checkClusterNameExists(request.getClusterName());
    String authUser = request.getLoginUser();
    // send request to get dbs
    String url = getPath(clusterInfo.getLinkisAddress()).path(linkisConfig.getColumnPath()).queryParam("database", request.getDbName()).queryParam("table", request.getTableName()).toString();
    HttpHeaders headers = new HttpHeaders();
    headers.setContentType(MediaType.APPLICATION_JSON);
    headers.add("Token-User", authUser);
    headers.add("Token-Code", clusterInfo.getLinkisToken());
    HttpEntity<Object> entity = new HttpEntity<>(headers);
    LOGGER.info("Start to get column by user and cluster and db and table by linkis. url: {}, method: {}, body: {}", url, javax.ws.rs.HttpMethod.GET, entity);
    Map<String, Object> response = restTemplate.exchange(url, HttpMethod.GET, entity, Map.class).getBody();
    LOGGER.info("Finished to get table by user and cluster and and table by linkis. response: {}", response);
    if (!checkResponse(response)) {
        String message = (String) response.get("message");
        LOGGER.error("Error! Can not get meta data from linkis, message: " + message);
        throw new MetaDataAcquireFailedException("Error! Can not get meta data from linkis, exception: " + message);
    }
    List<Map<String, String>> allTables = ((List<Map<String, String>>) ((Map<String, Object>) response.get("data")).get("columns"));
    DataInfo<ColumnInfoDetail> dataInfo = new DataInfo<>(allTables.size());
    if (CollectionUtils.isEmpty(allTables)) {
        return dataInfo;
    }
    List<ColumnInfoDetail> details = new ArrayList<>();
    for (Map<String, String> table : allTables) {
        ColumnInfoDetail detail = new ColumnInfoDetail(table.get("columnName"), table.get("columnType"));
        details.add(detail);
    }
    dataInfo.setContent(details);
    return dataInfo;
}
Also used : DataInfo(com.webank.wedatasphere.qualitis.metadata.response.DataInfo) HttpHeaders(org.springframework.http.HttpHeaders) HttpEntity(org.springframework.http.HttpEntity) ArrayList(java.util.ArrayList) ColumnInfoDetail(com.webank.wedatasphere.qualitis.metadata.response.column.ColumnInfoDetail) ClusterInfo(com.webank.wedatasphere.qualitis.entity.ClusterInfo) MetaDataAcquireFailedException(com.webank.wedatasphere.qualitis.metadata.exception.MetaDataAcquireFailedException) JSONObject(org.json.JSONObject) Map(java.util.Map)

Example 30 with MetaDataAcquireFailedException

use of com.webank.wedatasphere.qualitis.metadata.exception.MetaDataAcquireFailedException in project Qualitis by WeBankFinTech.

the class MetaDataClientImpl method getDataSourceConnectParams.

@Override
public GeneralResponse<Map> getDataSourceConnectParams(String clusterName, String authUser, Long dataSourceId, Long versionId) throws UnExpectedRequestException, MetaDataAcquireFailedException {
    // Check existence of cluster name
    ClusterInfo clusterInfo = checkClusterNameExists(clusterName);
    // send request to get dbs
    UriBuilder uriBuilder = getPath(clusterInfo.getLinkisAddress()).path(linkisConfig.getDatasourceConnectParam());
    if (versionId != null) {
        uriBuilder.path(versionId.toString());
    }
    String url = uriBuilder.toString().replace("{DATA_SOURCE_ID}", dataSourceId.toString());
    HttpHeaders headers = new HttpHeaders();
    headers.setContentType(MediaType.APPLICATION_JSON);
    headers.add("Token-User", authUser);
    headers.add("Token-Code", clusterInfo.getLinkisToken());
    HttpEntity<Object> entity = new HttpEntity<>(headers);
    LOGGER.info("Start to get data source connect params by user and cluster by linkis. url: {}, method: {}, body: {}", url, javax.ws.rs.HttpMethod.GET, entity);
    Map<String, Object> response = restTemplate.exchange(url, HttpMethod.GET, entity, Map.class).getBody();
    LOGGER.info("Finish to get data source connect params by user and cluster by linkis. response: {}", response);
    if (!checkResponse(response)) {
        String message = (String) response.get("message");
        LOGGER.error("Error! Can not get meta data from linkis, message: " + message);
        throw new MetaDataAcquireFailedException("Error! Can not get meta data from linkis, exception: " + message);
    }
    Map data = (Map) response.get(LinkisResponseKeyEnum.DATA.getKey());
    return new GeneralResponse<>("200", "Success to get datasource connect params", data);
}
Also used : GeneralResponse(com.webank.wedatasphere.qualitis.response.GeneralResponse) HttpHeaders(org.springframework.http.HttpHeaders) ClusterInfo(com.webank.wedatasphere.qualitis.entity.ClusterInfo) HttpEntity(org.springframework.http.HttpEntity) MetaDataAcquireFailedException(com.webank.wedatasphere.qualitis.metadata.exception.MetaDataAcquireFailedException) JSONObject(org.json.JSONObject) UriBuilder(javax.ws.rs.core.UriBuilder) Map(java.util.Map)

Aggregations

MetaDataAcquireFailedException (com.webank.wedatasphere.qualitis.metadata.exception.MetaDataAcquireFailedException)33 Map (java.util.Map)31 ClusterInfo (com.webank.wedatasphere.qualitis.entity.ClusterInfo)28 JSONObject (org.json.JSONObject)26 HttpEntity (org.springframework.http.HttpEntity)26 HttpHeaders (org.springframework.http.HttpHeaders)26 GeneralResponse (com.webank.wedatasphere.qualitis.response.GeneralResponse)21 ArrayList (java.util.ArrayList)18 List (java.util.List)16 UnExpectedRequestException (com.webank.wedatasphere.qualitis.exception.UnExpectedRequestException)11 ColumnInfoDetail (com.webank.wedatasphere.qualitis.metadata.response.column.ColumnInfoDetail)8 DataInfo (com.webank.wedatasphere.qualitis.metadata.response.DataInfo)7 ResourceAccessException (org.springframework.web.client.ResourceAccessException)7 CsTableInfoDetail (com.webank.wedatasphere.qualitis.metadata.response.table.CsTableInfoDetail)6 UriBuilder (javax.ws.rs.core.UriBuilder)6 SpecCharEnum (com.webank.wedatasphere.qualitis.constant.SpecCharEnum)4 ClusterInfoDao (com.webank.wedatasphere.qualitis.dao.ClusterInfoDao)4 MetaDataClient (com.webank.wedatasphere.qualitis.metadata.client.MetaDataClient)4 PartitionStatisticsInfo (com.webank.wedatasphere.qualitis.metadata.response.table.PartitionStatisticsInfo)4 TableInfoDetail (com.webank.wedatasphere.qualitis.metadata.response.table.TableInfoDetail)4