Examples with WorkerTopologyContext backtype.storm.task.WorkerTopologyContext used on opensource projects

Example 1 with WorkerTopologyContext

use of backtype.storm.task.WorkerTopologyContext in project jstorm by alibaba.

the class WorkerData method setSerializeThreads.

protected List<AsyncLoopThread> setSerializeThreads() {
    WorkerTopologyContext workerTopologyContext = contextMaker.makeWorkerTopologyContext(sysTopology);
    int tasksNum = shutdownTasks.size();
    double workerRatio = ConfigExtension.getWorkerSerializeThreadRatio(stormConf);
    int workerSerialThreadNum = Utils.getInt(Math.ceil(workerRatio * tasksNum));
    if (workerSerialThreadNum > 0 && tasksNum > 0) {
        double average = tasksNum / (double) workerSerialThreadNum;
        for (int i = 0; i < workerSerialThreadNum; i++) {
            int startRunTaskIndex = Utils.getInt(Math.rint(average * i));
            serializeThreads.add(new AsyncLoopThread(new WorkerSerializeRunnable(shutdownTasks, stormConf, workerTopologyContext, startRunTaskIndex, i)));
        }
    }
    return serializeThreads;
}

Example 2 with WorkerTopologyContext

use of backtype.storm.task.WorkerTopologyContext in project jstorm by alibaba.

the class WorkerData method updateKryoSerializer.

// create kryo serializer
public void updateKryoSerializer() {
    WorkerTopologyContext workerTopologyContext = contextMaker.makeWorkerTopologyContext(sysTopology);
    KryoTupleDeserializer kryoTupleDeserializer = new KryoTupleDeserializer(stormConf, workerTopologyContext, workerTopologyContext.getRawTopology());
    KryoTupleSerializer kryoTupleSerializer = new KryoTupleSerializer(stormConf, workerTopologyContext.getRawTopology());
    atomKryoDeserializer.getAndSet(kryoTupleDeserializer);
    atomKryoSerializer.getAndSet(kryoTupleSerializer);
}

Example 3 with WorkerTopologyContext

use of backtype.storm.task.WorkerTopologyContext in project jstorm by alibaba.

the class ContextMaker method makeWorkerTopologyContext.

public WorkerTopologyContext makeWorkerTopologyContext(StormTopology topology) {
    Map stormConf = workerData.getStormConf();
    String topologyId = workerData.getTopologyId();
    HashMap<String, Map<String, Fields>> componentToStreamToFields = workerData.generateComponentToStreamToFields(topology);
    return new WorkerTopologyContext(topology, stormConf, workerData.getTasksToComponent(), workerData.getComponentToSortedTasks(), componentToStreamToFields, topologyId, resourcePath, workerId, workerData.getPort(), workerTasks, workerData.getDefaultResources(), workerData.getUserResources());
}

Example 4 with WorkerTopologyContext

use of backtype.storm.task.WorkerTopologyContext in project jstorm by alibaba.

the class WorkerData method setDeserializeThreads.

protected List<AsyncLoopThread> setDeserializeThreads() {
    WorkerTopologyContext workerTopologyContext = contextMaker.makeWorkerTopologyContext(sysTopology);
    int tasksNum = shutdownTasks.size();
    double workerRatio = ConfigExtension.getWorkerDeserializeThreadRatio(stormConf);
    int workerDeseriaThreadNum = Utils.getInt(Math.ceil(workerRatio * tasksNum));
    if (workerDeseriaThreadNum > 0 && tasksNum > 0) {
        double average = tasksNum / (double) workerDeseriaThreadNum;
        for (int i = 0; i < workerDeseriaThreadNum; i++) {
            int startRunTaskIndex = Utils.getInt(Math.rint(average * i));
            deserializeThreads.add(new AsyncLoopThread(new WorkerDeserializeRunnable(shutdownTasks, stormConf, workerTopologyContext, startRunTaskIndex, i)));
        }
    }
    return deserializeThreads;
}