Search in sources :

Example 6 with TException

use of org.apache.thrift.TException in project storm by apache.

the class Nimbus method downloadBlobChunk.

@SuppressWarnings("deprecation")
@Override
public ByteBuffer downloadBlobChunk(String session) throws AuthorizationException, TException {
    try {
        BufferInputStream is = blobDownloaders.get(session);
        if (is == null) {
            throw new RuntimeException("Blob for session " + session + " does not exist (or timed out)");
        }
        byte[] ret = is.read();
        if (ret.length == 0) {
            is.close();
            blobDownloaders.remove(session);
        } else {
            blobDownloaders.put(session, is);
        }
        LOG.debug("Sending {} bytes", ret.length);
        return ByteBuffer.wrap(ret);
    } catch (Exception e) {
        LOG.warn("download blob chunk exception.", e);
        if (e instanceof TException) {
            throw (TException) e;
        }
        throw new RuntimeException(e);
    }
}
Also used : BufferInputStream(org.apache.storm.utils.BufferInputStream) TException(org.apache.thrift.TException) AuthorizationException(org.apache.storm.generated.AuthorizationException) NotAliveException(org.apache.storm.generated.NotAliveException) InterruptedIOException(java.io.InterruptedIOException) TException(org.apache.thrift.TException) IOException(java.io.IOException) AlreadyAliveException(org.apache.storm.generated.AlreadyAliveException) KeyAlreadyExistsException(org.apache.storm.generated.KeyAlreadyExistsException) KeyNotFoundException(org.apache.storm.generated.KeyNotFoundException) InvalidTopologyException(org.apache.storm.generated.InvalidTopologyException) BindException(java.net.BindException)

Example 7 with TException

use of org.apache.thrift.TException in project storm by apache.

the class Nimbus method finishBlobUpload.

@SuppressWarnings("deprecation")
@Override
public void finishBlobUpload(String session) throws AuthorizationException, TException {
    try {
        OutputStream os = blobUploaders.get(session);
        if (os == null) {
            throw new RuntimeException("Blob for session " + session + " does not exist (or timed out)");
        }
        os.close();
        LOG.info("Finished uploading blob for session {}. Closing session.", session);
        blobUploaders.remove(session);
    } catch (Exception e) {
        LOG.warn("finish blob upload exception.", e);
        if (e instanceof TException) {
            throw (TException) e;
        }
        throw new RuntimeException(e);
    }
}
Also used : TException(org.apache.thrift.TException) FileOutputStream(java.io.FileOutputStream) AtomicOutputStream(org.apache.storm.blobstore.AtomicOutputStream) OutputStream(java.io.OutputStream) AuthorizationException(org.apache.storm.generated.AuthorizationException) NotAliveException(org.apache.storm.generated.NotAliveException) InterruptedIOException(java.io.InterruptedIOException) TException(org.apache.thrift.TException) IOException(java.io.IOException) AlreadyAliveException(org.apache.storm.generated.AlreadyAliveException) KeyAlreadyExistsException(org.apache.storm.generated.KeyAlreadyExistsException) KeyNotFoundException(org.apache.storm.generated.KeyNotFoundException) InvalidTopologyException(org.apache.storm.generated.InvalidTopologyException) BindException(java.net.BindException)

Example 8 with TException

use of org.apache.thrift.TException in project storm by apache.

the class Nimbus method beginBlobDownload.

@SuppressWarnings("deprecation")
@Override
public BeginDownloadResult beginBlobDownload(String key) throws AuthorizationException, KeyNotFoundException, TException {
    try {
        InputStreamWithMeta is = blobStore.getBlob(key, getSubject());
        String sessionId = Utils.uuid();
        BeginDownloadResult ret = new BeginDownloadResult(is.getVersion(), sessionId);
        ret.set_data_size(is.getFileLength());
        blobDownloaders.put(sessionId, new BufferInputStream(is, (int) conf.getOrDefault(Config.STORM_BLOBSTORE_INPUTSTREAM_BUFFER_SIZE_BYTES, 65536)));
        LOG.info("Created download session for {}", key);
        return ret;
    } catch (Exception e) {
        LOG.warn("begin blob download exception.", e);
        if (e instanceof TException) {
            throw (TException) e;
        }
        throw new RuntimeException(e);
    }
}
Also used : BufferInputStream(org.apache.storm.utils.BufferInputStream) TException(org.apache.thrift.TException) InputStreamWithMeta(org.apache.storm.blobstore.InputStreamWithMeta) BeginDownloadResult(org.apache.storm.generated.BeginDownloadResult) AuthorizationException(org.apache.storm.generated.AuthorizationException) NotAliveException(org.apache.storm.generated.NotAliveException) InterruptedIOException(java.io.InterruptedIOException) TException(org.apache.thrift.TException) IOException(java.io.IOException) AlreadyAliveException(org.apache.storm.generated.AlreadyAliveException) KeyAlreadyExistsException(org.apache.storm.generated.KeyAlreadyExistsException) KeyNotFoundException(org.apache.storm.generated.KeyNotFoundException) InvalidTopologyException(org.apache.storm.generated.InvalidTopologyException) BindException(java.net.BindException)

Example 9 with TException

use of org.apache.thrift.TException in project storm by apache.

the class Nimbus method getComponentPageInfo.

@Override
public ComponentPageInfo getComponentPageInfo(String topoId, String componentId, String window, boolean includeSys) throws NotAliveException, AuthorizationException, TException {
    try {
        getComponentPageInfoCalls.mark();
        CommonTopoInfo info = getCommonTopoInfo(topoId, "getComponentPageInfo");
        if (info.base == null) {
            throw new NotAliveException(topoId);
        }
        StormTopology topology = info.topology;
        Map<String, Object> topoConf = info.topoConf;
        Assignment assignment = info.assignment;
        Map<List<Long>, List<Object>> exec2NodePort = new HashMap<>();
        Map<String, String> nodeToHost;
        Map<List<Long>, List<Object>> exec2HostPort = new HashMap<>();
        if (assignment != null) {
            Map<List<Long>, NodeInfo> execToNodeInfo = assignment.get_executor_node_port();
            nodeToHost = assignment.get_node_host();
            for (Entry<List<Long>, NodeInfo> entry : execToNodeInfo.entrySet()) {
                NodeInfo ni = entry.getValue();
                List<Object> nodePort = Arrays.asList(ni.get_node(), ni.get_port_iterator().next());
                List<Object> hostPort = Arrays.asList(nodeToHost.get(ni.get_node()), ni.get_port_iterator().next());
                exec2NodePort.put(entry.getKey(), nodePort);
                exec2HostPort.put(entry.getKey(), hostPort);
            }
        } else {
            nodeToHost = Collections.emptyMap();
        }
        ComponentPageInfo compPageInfo = StatsUtil.aggCompExecsStats(exec2HostPort, info.taskToComponent, info.beats, window, includeSys, topoId, topology, componentId);
        if (compPageInfo.get_component_type() == ComponentType.SPOUT) {
            compPageInfo.set_resources_map(setResourcesDefaultIfNotSet(ResourceUtils.getSpoutsResources(topology, topoConf), componentId, topoConf));
        } else {
            //bolt
            compPageInfo.set_resources_map(setResourcesDefaultIfNotSet(ResourceUtils.getBoltsResources(topology, topoConf), componentId, topoConf));
        }
        compPageInfo.set_topology_name(info.topoName);
        compPageInfo.set_errors(stormClusterState.errors(topoId, componentId));
        compPageInfo.set_topology_status(extractStatusStr(info.base));
        if (info.base.is_set_component_debug()) {
            DebugOptions debug = info.base.get_component_debug().get(componentId);
            if (debug != null) {
                compPageInfo.set_debug_options(debug);
            }
        }
        // Add the event logger details.
        Map<String, List<Integer>> compToTasks = Utils.reverseMap(info.taskToComponent);
        if (compToTasks.containsKey(StormCommon.EVENTLOGGER_COMPONENT_ID)) {
            List<Integer> tasks = compToTasks.get(StormCommon.EVENTLOGGER_COMPONENT_ID);
            tasks.sort(null);
            // Find the task the events from this component route to.
            int taskIndex = TupleUtils.chooseTaskIndex(Collections.singletonList(componentId), tasks.size());
            int taskId = tasks.get(taskIndex);
            String host = null;
            Integer port = null;
            for (Entry<List<Long>, List<Object>> entry : exec2HostPort.entrySet()) {
                int start = entry.getKey().get(0).intValue();
                int end = entry.getKey().get(1).intValue();
                if (taskId >= start && taskId <= end) {
                    host = (String) entry.getValue().get(0);
                    port = ((Number) entry.getValue().get(1)).intValue();
                    break;
                }
            }
            if (host != null && port != null) {
                compPageInfo.set_eventlog_host(host);
                compPageInfo.set_eventlog_port(port);
            }
        }
        return compPageInfo;
    } catch (Exception e) {
        LOG.warn("getComponentPageInfo exception. (topo id='{}')", topoId, e);
        if (e instanceof TException) {
            throw (TException) e;
        }
        throw new RuntimeException(e);
    }
}
Also used : TException(org.apache.thrift.TException) HashMap(java.util.HashMap) StormTopology(org.apache.storm.generated.StormTopology) DebugOptions(org.apache.storm.generated.DebugOptions) Assignment(org.apache.storm.generated.Assignment) SchedulerAssignment(org.apache.storm.scheduler.SchedulerAssignment) NotAliveException(org.apache.storm.generated.NotAliveException) ArrayList(java.util.ArrayList) List(java.util.List) DataPoint(org.apache.storm.metric.api.DataPoint) AuthorizationException(org.apache.storm.generated.AuthorizationException) NotAliveException(org.apache.storm.generated.NotAliveException) InterruptedIOException(java.io.InterruptedIOException) TException(org.apache.thrift.TException) IOException(java.io.IOException) AlreadyAliveException(org.apache.storm.generated.AlreadyAliveException) KeyAlreadyExistsException(org.apache.storm.generated.KeyAlreadyExistsException) KeyNotFoundException(org.apache.storm.generated.KeyNotFoundException) InvalidTopologyException(org.apache.storm.generated.InvalidTopologyException) BindException(java.net.BindException) ComponentPageInfo(org.apache.storm.generated.ComponentPageInfo) NodeInfo(org.apache.storm.generated.NodeInfo)

Example 10 with TException

use of org.apache.thrift.TException in project storm by apache.

the class Nimbus method getComponentPendingProfileActions.

@Override
public List<ProfileRequest> getComponentPendingProfileActions(String id, String componentId, ProfileAction action) throws TException {
    try {
        getComponentPendingProfileActionsCalls.mark();
        CommonTopoInfo info = getCommonTopoInfo(id, "getComponentPendingProfileActions");
        Map<String, String> nodeToHost = info.assignment.get_node_host();
        Map<List<? extends Number>, List<Object>> exec2hostPort = new HashMap<>();
        for (Entry<List<Long>, NodeInfo> entry : info.assignment.get_executor_node_port().entrySet()) {
            NodeInfo ni = entry.getValue();
            List<Object> hostPort = Arrays.asList(nodeToHost.get(ni.get_node()), ni.get_port_iterator().next().intValue());
            exec2hostPort.put(entry.getKey(), hostPort);
        }
        List<Map<String, Object>> nodeInfos = StatsUtil.extractNodeInfosFromHbForComp(exec2hostPort, info.taskToComponent, false, componentId);
        List<ProfileRequest> ret = new ArrayList<>();
        for (Map<String, Object> ni : nodeInfos) {
            String niHost = (String) ni.get("host");
            int niPort = ((Integer) ni.get("port")).intValue();
            ProfileRequest newestMatch = null;
            long reqTime = -1;
            for (ProfileRequest req : stormClusterState.getTopologyProfileRequests(id)) {
                String expectedHost = req.get_nodeInfo().get_node();
                int expectedPort = req.get_nodeInfo().get_port_iterator().next().intValue();
                ProfileAction expectedAction = req.get_action();
                if (niHost.equals(expectedHost) && niPort == expectedPort && action == expectedAction) {
                    long time = req.get_time_stamp();
                    if (time > reqTime) {
                        reqTime = time;
                        newestMatch = req;
                    }
                }
            }
            if (newestMatch != null) {
                ret.add(newestMatch);
            }
        }
        LOG.info("Latest profile actions for topology {} component {} {}", id, componentId, ret);
        return ret;
    } catch (Exception e) {
        LOG.warn("Get comp actions topology exception. (topology id='{}')", id, e);
        if (e instanceof TException) {
            throw (TException) e;
        }
        throw new RuntimeException(e);
    }
}
Also used : TException(org.apache.thrift.TException) HashMap(java.util.HashMap) ArrayList(java.util.ArrayList) ProfileAction(org.apache.storm.generated.ProfileAction) ProfileRequest(org.apache.storm.generated.ProfileRequest) DataPoint(org.apache.storm.metric.api.DataPoint) AuthorizationException(org.apache.storm.generated.AuthorizationException) NotAliveException(org.apache.storm.generated.NotAliveException) InterruptedIOException(java.io.InterruptedIOException) TException(org.apache.thrift.TException) IOException(java.io.IOException) AlreadyAliveException(org.apache.storm.generated.AlreadyAliveException) KeyAlreadyExistsException(org.apache.storm.generated.KeyAlreadyExistsException) KeyNotFoundException(org.apache.storm.generated.KeyNotFoundException) InvalidTopologyException(org.apache.storm.generated.InvalidTopologyException) BindException(java.net.BindException) KeySequenceNumber(org.apache.storm.blobstore.KeySequenceNumber) NodeInfo(org.apache.storm.generated.NodeInfo) ArrayList(java.util.ArrayList) List(java.util.List) Map(java.util.Map) TimeCacheMap(org.apache.storm.utils.TimeCacheMap) ImmutableMap(com.google.common.collect.ImmutableMap) HashMap(java.util.HashMap)

Aggregations

TException (org.apache.thrift.TException)325 IOException (java.io.IOException)135 MetaException (org.apache.hadoop.hive.metastore.api.MetaException)41 HashMap (java.util.HashMap)36 NoSuchObjectException (org.apache.hadoop.hive.metastore.api.NoSuchObjectException)35 ArrayList (java.util.ArrayList)32 Table (org.apache.hadoop.hive.metastore.api.Table)27 AuthorizationException (org.apache.storm.generated.AuthorizationException)27 Map (java.util.Map)26 InterruptedIOException (java.io.InterruptedIOException)24 BindException (java.net.BindException)24 UnknownHostException (java.net.UnknownHostException)22 TProtocol (org.apache.thrift.protocol.TProtocol)21 FileNotFoundException (java.io.FileNotFoundException)20 List (java.util.List)20 LoginException (javax.security.auth.login.LoginException)20 HiveSQLException (org.apache.hive.service.cli.HiveSQLException)20 TTransportException (org.apache.thrift.transport.TTransportException)19 ServiceException (org.apache.hive.service.ServiceException)18 Test (org.junit.Test)18