Search in sources :

Example 26 with ActionInfo

use of org.smartdata.model.ActionInfo in project SSM by Intel-bigdata.

the class CmdletManager method submitCmdlet.

public long submitCmdlet(CmdletDescriptor cmdletDescriptor) throws IOException {
    // with the same rule id and cmdlet string, return -1.
    if (tracker.contains(cmdletDescriptor)) {
        LOG.debug("Refuse to repeatedly submit Cmdlet for {}", cmdletDescriptor);
        return -1;
    }
    if (LOG.isDebugEnabled()) {
        LOG.debug(String.format("Received Cmdlet -> [ %s ]", cmdletDescriptor.getCmdletString()));
    }
    if (maxNumPendingCmdlets <= pendingCmdlet.size() + schedulingCmdlet.size()) {
        throw new QueueFullException("Pending cmdlets exceeds value specified by key '" + SmartConfKeys.SMART_CMDLET_MAX_NUM_PENDING_KEY + "' = " + maxNumPendingCmdlets);
    }
    long submitTime = System.currentTimeMillis();
    CmdletInfo cmdletInfo = new CmdletInfo(maxCmdletId.getAndIncrement(), cmdletDescriptor.getRuleId(), CmdletState.PENDING, cmdletDescriptor.getCmdletString(), submitTime, submitTime, submitTime + cmdletDescriptor.getDeferIntervalMs());
    List<ActionInfo> actionInfos = createActionInfos(cmdletDescriptor, cmdletInfo.getCid());
    // Check action names
    checkActionNames(cmdletDescriptor);
    // Check if action path is in whitelist
    if (WhitelistHelper.isEnabled(getContext().getConf())) {
        if (!WhitelistHelper.isCmdletInWhitelist(cmdletDescriptor)) {
            throw new IOException("This path is not in the whitelist.");
        }
    }
    // Let Scheduler check actioninfo onsubmit and add them to cmdletinfo
    checkActionsOnSubmit(cmdletInfo, actionInfos);
    // Insert cmdletinfo and actionInfos to metastore and cache.
    syncCmdAction(cmdletInfo, actionInfos);
    // Track in the submission portal. For cmdlets recovered from DB
    // (see #recover), they will be not be tracked.
    tracker.track(cmdletInfo.getCid(), cmdletDescriptor);
    return cmdletInfo.getCid();
}
Also used : QueueFullException(org.smartdata.exception.QueueFullException) ActionInfo(org.smartdata.model.ActionInfo) IOException(java.io.IOException) CmdletInfo(org.smartdata.model.CmdletInfo)

Example 27 with ActionInfo

use of org.smartdata.model.ActionInfo in project SSM by Intel-bigdata.

the class CmdletManager method listActions.

public ActionGroup listActions(long pageIndex, long numPerPage, List<String> orderBy, List<Boolean> isDesc) throws IOException, MetaStoreException {
    if (pageIndex == Long.parseLong("0")) {
        if (tmpActions.totalNumOfActions != 0) {
            return tmpActions;
        } else {
            pageIndex = 1;
        }
    }
    List<ActionInfo> infos = metaStore.listPageAction((pageIndex - 1) * numPerPage, numPerPage, orderBy, isDesc);
    for (ActionInfo info : infos) {
        ActionInfo memInfo = idToActions.get(info.getActionId());
        if (memInfo != null) {
            info.setCreateTime(memInfo.getCreateTime());
            info.setProgress(memInfo.getProgress());
        }
    }
    tmpActions = new ActionGroup(infos, metaStore.getCountOfAllAction());
    return tmpActions;
}
Also used : ActionInfo(org.smartdata.model.ActionInfo)

Example 28 with ActionInfo

use of org.smartdata.model.ActionInfo in project SSM by Intel-bigdata.

the class CmdletManager method cmdletFinishedInternal.

private void cmdletFinishedInternal(CmdletInfo cmdletInfo, boolean success) throws IOException {
    numCmdletsFinished.incrementAndGet();
    ActionInfo actionInfo;
    for (Long aid : cmdletInfo.getAids()) {
        actionInfo = idToActions.get(aid);
        synchronized (actionInfo) {
            // Set all action as finished
            actionInfo.setProgress(1.0F);
            actionInfo.setFinished(true);
            actionInfo.setCreateTime(cmdletInfo.getStateChangedTime());
            actionInfo.setFinishTime(cmdletInfo.getStateChangedTime());
            actionInfo.setExecHost(ActiveServerInfo.getInstance().getId());
            actionInfo.setSuccessful(success);
        }
    }
}
Also used : AtomicLong(java.util.concurrent.atomic.AtomicLong) ActionInfo(org.smartdata.model.ActionInfo)

Example 29 with ActionInfo

use of org.smartdata.model.ActionInfo in project SSM by Intel-bigdata.

the class CmdletManager method updateCmdletExecHost.

public void updateCmdletExecHost(long cmdletId, String host) throws IOException {
    CmdletInfo cmdlet = getCmdletInfo(cmdletId);
    if (cmdlet == null) {
        return;
    }
    ActionInfo action;
    for (long id : cmdlet.getAids()) {
        action = getActionInfo(id);
        if (action != null) {
            action.setExecHost(host);
        }
    }
}
Also used : ActionInfo(org.smartdata.model.ActionInfo) CmdletInfo(org.smartdata.model.CmdletInfo)

Example 30 with ActionInfo

use of org.smartdata.model.ActionInfo in project SSM by Intel-bigdata.

the class AdminProtocolClientSideTranslator method listActionInfoOfLastActions.

@Override
public List<ActionInfo> listActionInfoOfLastActions(int maxNumActions) throws IOException {
    ListActionInfoOfLastActionsRequestProto req = ListActionInfoOfLastActionsRequestProto.newBuilder().setMaxNumActions(maxNumActions).build();
    try {
        List<ActionInfoProto> protoslist = rpcProxy.listActionInfoOfLastActions(null, req).getActionInfoListList();
        if (protoslist == null) {
            return new ArrayList<>();
        }
        List<ActionInfo> list = new ArrayList<>();
        for (ActionInfoProto infoProto : protoslist) {
            list.add(ProtoBufferHelper.convert(infoProto));
        }
        return list;
    } catch (ServiceException e) {
        throw ProtoBufferHelper.getRemoteException(e);
    }
}
Also used : ListActionInfoOfLastActionsRequestProto(org.smartdata.protocol.AdminServerProto.ListActionInfoOfLastActionsRequestProto) ActionInfoProto(org.smartdata.protocol.AdminServerProto.ActionInfoProto) ServiceException(com.google.protobuf.ServiceException) ArrayList(java.util.ArrayList) ActionInfo(org.smartdata.model.ActionInfo)

Aggregations

ActionInfo (org.smartdata.model.ActionInfo)48 Test (org.junit.Test)23 HashMap (java.util.HashMap)16 CmdletInfo (org.smartdata.model.CmdletInfo)16 ArrayList (java.util.ArrayList)12 IOException (java.io.IOException)7 AtomicLong (java.util.concurrent.atomic.AtomicLong)5 SmartAdmin (org.smartdata.admin.SmartAdmin)5 MetaStore (org.smartdata.metastore.MetaStore)5 Path (org.apache.hadoop.fs.Path)4 DistributedFileSystem (org.apache.hadoop.hdfs.DistributedFileSystem)4 MetaStoreException (org.smartdata.metastore.MetaStoreException)4 FileInfo (org.smartdata.model.FileInfo)4 CmdletDescriptor (org.smartdata.model.CmdletDescriptor)3 ActionScheduler (org.smartdata.model.action.ActionScheduler)3 ServiceException (com.google.protobuf.ServiceException)2 ParseException (java.text.ParseException)2 ConcurrentHashMap (java.util.concurrent.ConcurrentHashMap)2 QueueFullException (org.smartdata.exception.QueueFullException)2 DetailedFileAction (org.smartdata.model.DetailedFileAction)2