Search in sources :

Example 16 with JobConfig

use of org.apache.helix.task.JobConfig in project helix by apache.

the class TaskDataCache method refresh.

/**
 * This refreshes the cluster data by re-fetching the data from zookeeper in an efficient way
 *
 * @param accessor
 *
 * @return
 */
public synchronized boolean refresh(HelixDataAccessor accessor, Map<String, ResourceConfig> resourceConfigMap) {
    refreshJobContexts(accessor);
    // update workflow and job configs.
    _workflowConfigMap.clear();
    _jobConfigMap.clear();
    for (Map.Entry<String, ResourceConfig> entry : resourceConfigMap.entrySet()) {
        if (entry.getValue().getRecord().getSimpleFields().containsKey(WorkflowConfig.WorkflowConfigProperty.Dag.name())) {
            _workflowConfigMap.put(entry.getKey(), new WorkflowConfig(entry.getValue()));
        } else if (entry.getValue().getRecord().getSimpleFields().containsKey(WorkflowConfig.WorkflowConfigProperty.WorkflowID.name())) {
            _jobConfigMap.put(entry.getKey(), new JobConfig(entry.getValue()));
        }
    }
    return true;
}
Also used : WorkflowConfig(org.apache.helix.task.WorkflowConfig) ResourceConfig(org.apache.helix.model.ResourceConfig) HashMap(java.util.HashMap) Map(java.util.Map) JobConfig(org.apache.helix.task.JobConfig)

Aggregations

JobConfig (org.apache.helix.task.JobConfig)16 Test (org.testng.annotations.Test)10 WorkflowConfig (org.apache.helix.task.WorkflowConfig)9 JobContext (org.apache.helix.task.JobContext)7 TaskDriver (org.apache.helix.task.TaskDriver)5 Workflow (org.apache.helix.task.Workflow)5 WorkflowContext (org.apache.helix.task.WorkflowContext)5 HashMap (java.util.HashMap)3 Path (javax.ws.rs.Path)3 JobQueue (org.apache.helix.task.JobQueue)3 ArrayList (java.util.ArrayList)2 Map (java.util.Map)2 GET (javax.ws.rs.GET)2 ZNRecord (org.apache.helix.ZNRecord)2 TaskConfig (org.apache.helix.task.TaskConfig)2 IOException (java.io.IOException)1 HashSet (java.util.HashSet)1 PUT (javax.ws.rs.PUT)1 Entity (javax.ws.rs.client.Entity)1 MultiWorkUnit (org.apache.gobblin.source.workunit.MultiWorkUnit)1