Search in sources :

Example 11 with HyracksConnection

use of org.apache.hyracks.api.client.HyracksConnection in project asterixdb by apache.

the class InsertPipelineExample method main.

public static void main(String[] args) throws Exception {
    Options options = new Options();
    CmdLineParser parser = new CmdLineParser(options);
    parser.parseArgument(args);
    IHyracksClientConnection hcc = new HyracksConnection(options.host, options.port);
    JobSpecification job = createJob(options);
    long start = System.currentTimeMillis();
    JobId jobId = hcc.startJob(job);
    hcc.waitForCompletion(jobId);
    long end = System.currentTimeMillis();
    System.err.println(start + " " + end + " " + (end - start));
}
Also used : IHyracksClientConnection(org.apache.hyracks.api.client.IHyracksClientConnection) CmdLineParser(org.kohsuke.args4j.CmdLineParser) JobSpecification(org.apache.hyracks.api.job.JobSpecification) HyracksConnection(org.apache.hyracks.api.client.HyracksConnection) JobId(org.apache.hyracks.api.job.JobId)

Example 12 with HyracksConnection

use of org.apache.hyracks.api.client.HyracksConnection in project asterixdb by apache.

the class AsterixHyracksIntegrationUtil method init.

public void init(boolean deleteOldInstanceData) throws Exception {
    final ICCApplication ccApplication = createCCApplication();
    configManager = new ConfigManager();
    ccApplication.registerConfig(configManager);
    final CCConfig ccConfig = createCCConfig(configManager);
    cc = new ClusterControllerService(ccConfig, ccApplication);
    nodeNames = ccConfig.getConfigManager().getNodeNames();
    if (deleteOldInstanceData) {
        deleteTransactionLogs();
        removeTestStorageFiles();
    }
    final List<NodeControllerService> nodeControllers = new ArrayList<>();
    for (String nodeId : nodeNames) {
        // mark this NC as virtual in the CC's config manager, so he doesn't try to contact NCService...
        configManager.set(nodeId, NCConfig.Option.VIRTUAL_NC, true);
        final INCApplication ncApplication = createNCApplication();
        ConfigManager ncConfigManager = new ConfigManager();
        ncApplication.registerConfig(ncConfigManager);
        nodeControllers.add(new NodeControllerService(fixupIODevices(createNCConfig(nodeId, ncConfigManager)), ncApplication));
    }
    ;
    cc.start();
    // Starts ncs.
    nodeNames = ccConfig.getConfigManager().getNodeNames();
    List<Thread> startupThreads = new ArrayList<>();
    for (NodeControllerService nc : nodeControllers) {
        Thread ncStartThread = new Thread("IntegrationUtil-" + nc.getId()) {

            @Override
            public void run() {
                try {
                    nc.start();
                } catch (Exception e) {
                    LOGGER.log(Level.SEVERE, e.getMessage(), e);
                }
            }
        };
        ncStartThread.start();
        startupThreads.add(ncStartThread);
    }
    //wait until all NCs complete their startup
    for (Thread thread : startupThreads) {
        thread.join();
    }
    // Wait until cluster becomes active
    ClusterStateManager.INSTANCE.waitForState(ClusterState.ACTIVE);
    hcc = new HyracksConnection(cc.getConfig().getClientListenAddress(), cc.getConfig().getClientListenPort());
    this.ncs = nodeControllers.toArray(new NodeControllerService[nodeControllers.size()]);
}
Also used : NodeControllerService(org.apache.hyracks.control.nc.NodeControllerService) INCApplication(org.apache.hyracks.api.application.INCApplication) ArrayList(java.util.ArrayList) ConfigManager(org.apache.hyracks.control.common.config.ConfigManager) AsterixException(org.apache.asterix.common.exceptions.AsterixException) IOException(java.io.IOException) CmdLineException(org.kohsuke.args4j.CmdLineException) ICCApplication(org.apache.hyracks.api.application.ICCApplication) CCConfig(org.apache.hyracks.control.common.controllers.CCConfig) HyracksConnection(org.apache.hyracks.api.client.HyracksConnection) ClusterControllerService(org.apache.hyracks.control.cc.ClusterControllerService)

Example 13 with HyracksConnection

use of org.apache.hyracks.api.client.HyracksConnection in project asterixdb by apache.

the class NCApplication method getHcc.

protected IHyracksClientConnection getHcc() throws Exception {
    NodeControllerService ncSrv = (NodeControllerService) ncServiceCtx.getControllerService();
    ClusterControllerInfo ccInfo = ncSrv.getNodeParameters().getClusterControllerInfo();
    return new HyracksConnection(ccInfo.getClientNetAddress(), ccInfo.getClientNetPort());
}
Also used : NodeControllerService(org.apache.hyracks.control.nc.NodeControllerService) ClusterControllerInfo(org.apache.hyracks.api.client.ClusterControllerInfo) HyracksConnection(org.apache.hyracks.api.client.HyracksConnection)

Example 14 with HyracksConnection

use of org.apache.hyracks.api.client.HyracksConnection in project asterixdb by apache.

the class SecondaryIndexSearchExample method main.

public static void main(String[] args) throws Exception {
    Options options = new Options();
    CmdLineParser parser = new CmdLineParser(options);
    parser.parseArgument(args);
    IHyracksClientConnection hcc = new HyracksConnection(options.host, options.port);
    JobSpecification job = createJob(options);
    long start = System.currentTimeMillis();
    JobId jobId = hcc.startJob(job);
    hcc.waitForCompletion(jobId);
    long end = System.currentTimeMillis();
    System.err.println(start + " " + end + " " + (end - start));
}
Also used : IHyracksClientConnection(org.apache.hyracks.api.client.IHyracksClientConnection) CmdLineParser(org.kohsuke.args4j.CmdLineParser) JobSpecification(org.apache.hyracks.api.job.JobSpecification) HyracksConnection(org.apache.hyracks.api.client.HyracksConnection) JobId(org.apache.hyracks.api.job.JobId)

Example 15 with HyracksConnection

use of org.apache.hyracks.api.client.HyracksConnection in project asterixdb by apache.

the class PrimaryIndexBulkLoadExample method main.

public static void main(String[] args) throws Exception {
    Options options = new Options();
    CmdLineParser parser = new CmdLineParser(options);
    parser.parseArgument(args);
    IHyracksClientConnection hcc = new HyracksConnection(options.host, options.port);
    JobSpecification job = createJob(options);
    long start = System.currentTimeMillis();
    JobId jobId = hcc.startJob(job);
    hcc.waitForCompletion(jobId);
    long end = System.currentTimeMillis();
    System.err.println(start + " " + end + " " + (end - start));
}
Also used : IHyracksClientConnection(org.apache.hyracks.api.client.IHyracksClientConnection) CmdLineParser(org.kohsuke.args4j.CmdLineParser) JobSpecification(org.apache.hyracks.api.job.JobSpecification) HyracksConnection(org.apache.hyracks.api.client.HyracksConnection) JobId(org.apache.hyracks.api.job.JobId)

Aggregations

HyracksConnection (org.apache.hyracks.api.client.HyracksConnection)22 IHyracksClientConnection (org.apache.hyracks.api.client.IHyracksClientConnection)13 JobId (org.apache.hyracks.api.job.JobId)11 JobSpecification (org.apache.hyracks.api.job.JobSpecification)11 CmdLineParser (org.kohsuke.args4j.CmdLineParser)10 NodeControllerService (org.apache.hyracks.control.nc.NodeControllerService)8 ClusterControllerService (org.apache.hyracks.control.cc.ClusterControllerService)7 CCConfig (org.apache.hyracks.control.common.controllers.CCConfig)7 NCConfig (org.apache.hyracks.control.common.controllers.NCConfig)6 File (java.io.File)4 BeforeClass (org.junit.BeforeClass)3 Path (org.apache.hadoop.fs.Path)2 IBinaryComparatorFactory (org.apache.hyracks.api.dataflow.value.IBinaryComparatorFactory)2 RecordDescriptor (org.apache.hyracks.api.dataflow.value.RecordDescriptor)2 UTF8StringSerializerDeserializer (org.apache.hyracks.dataflow.common.data.marshalling.UTF8StringSerializerDeserializer)2 FieldHashPartitionComputerFactory (org.apache.hyracks.dataflow.common.data.partition.FieldHashPartitionComputerFactory)2 MToNPartitioningMergingConnectorDescriptor (org.apache.hyracks.dataflow.std.connectors.MToNPartitioningMergingConnectorDescriptor)2 OneToOneConnectorDescriptor (org.apache.hyracks.dataflow.std.connectors.OneToOneConnectorDescriptor)2 ExternalSortOperatorDescriptor (org.apache.hyracks.dataflow.std.sort.ExternalSortOperatorDescriptor)2 TextKeyValueParserFactory (org.apache.hyracks.hdfs.lib.TextKeyValueParserFactory)2