Search in sources :

Example 1 with OTSConf

use of com.alibaba.datax.plugin.reader.otsreader.model.OTSConf in project DataX by alibaba.

the class OtsReaderSlaveProxy method read.

public void read(RecordSender sender, Configuration configuration) throws Exception {
    LOG.info("read begin.");
    OTSConf conf = GsonParser.jsonToConf(configuration.getString(OTSConst.OTS_CONF));
    OTSRange range = GsonParser.jsonToRange(configuration.getString(OTSConst.OTS_RANGE));
    Direction direction = GsonParser.jsonToDirection(configuration.getString(OTSConst.OTS_DIRECTION));
    OTSServiceConfiguration configure = new OTSServiceConfiguration();
    configure.setRetryStrategy(new DefaultNoRetry());
    OTSClientAsync ots = new OTSClientAsync(conf.getEndpoint(), conf.getAccessId(), conf.getAccesskey(), conf.getInstanceName(), null, configure, null);
    RowPrimaryKey token = range.getBegin();
    List<String> columns = Common.getNormalColumnNameList(conf.getColumns());
    RequestItem request = null;
    do {
        LOG.debug("Next token : {}", GsonParser.rowPrimaryKeyToJson(token));
        if (request == null) {
            request = generateRequestItem(ots, conf, token, range.getEnd(), direction, columns);
        } else {
            RequestItem req = request;
            GetRangeResult result = RetryHelper.executeWithRetry(new GetRangeCallable(ots, req.getCriteria(), req.getFuture()), conf.getRetry(), conf.getSleepInMilliSecond());
            if ((token = result.getNextStartPrimaryKey()) != null) {
                request = generateRequestItem(ots, conf, token, range.getEnd(), direction, columns);
            }
            rowsToSender(result.getRows(), sender, conf.getColumns());
        }
    } while (token != null);
    ots.shutdown();
    LOG.info("read end.");
}
Also used : OTSServiceConfiguration(com.aliyun.openservices.ots.OTSServiceConfiguration) DefaultNoRetry(com.alibaba.datax.plugin.reader.otsreader.utils.DefaultNoRetry) OTSClientAsync(com.aliyun.openservices.ots.OTSClientAsync) OTSRange(com.alibaba.datax.plugin.reader.otsreader.model.OTSRange) GetRangeResult(com.aliyun.openservices.ots.model.GetRangeResult) RowPrimaryKey(com.aliyun.openservices.ots.model.RowPrimaryKey) OTSConf(com.alibaba.datax.plugin.reader.otsreader.model.OTSConf) GetRangeCallable(com.alibaba.datax.plugin.reader.otsreader.callable.GetRangeCallable) Direction(com.aliyun.openservices.ots.model.Direction)

Aggregations

GetRangeCallable (com.alibaba.datax.plugin.reader.otsreader.callable.GetRangeCallable)1 OTSConf (com.alibaba.datax.plugin.reader.otsreader.model.OTSConf)1 OTSRange (com.alibaba.datax.plugin.reader.otsreader.model.OTSRange)1 DefaultNoRetry (com.alibaba.datax.plugin.reader.otsreader.utils.DefaultNoRetry)1 OTSClientAsync (com.aliyun.openservices.ots.OTSClientAsync)1 OTSServiceConfiguration (com.aliyun.openservices.ots.OTSServiceConfiguration)1 Direction (com.aliyun.openservices.ots.model.Direction)1 GetRangeResult (com.aliyun.openservices.ots.model.GetRangeResult)1 RowPrimaryKey (com.aliyun.openservices.ots.model.RowPrimaryKey)1