Search in sources :

Example 1 with IHyracksDatasetReader

use of org.apache.hyracks.api.dataset.IHyracksDatasetReader in project asterixdb by apache.

the class AbstractIntegrationTest method readResults.

protected List<String> readResults(JobSpecification spec, JobId jobId, ResultSetId resultSetId) throws Exception {
    int nReaders = 1;
    IFrameTupleAccessor frameTupleAccessor = new ResultFrameTupleAccessor();
    IHyracksDataset hyracksDataset = new HyracksDataset(hcc, spec.getFrameSize(), nReaders);
    IHyracksDatasetReader reader = hyracksDataset.createReader(jobId, resultSetId);
    List<String> resultRecords = new ArrayList<>();
    ByteBufferInputStream bbis = new ByteBufferInputStream();
    FrameManager resultDisplayFrameMgr = new FrameManager(spec.getFrameSize());
    VSizeFrame frame = new VSizeFrame(resultDisplayFrameMgr);
    int readSize = reader.read(frame);
    while (readSize > 0) {
        try {
            frameTupleAccessor.reset(frame.getBuffer());
            for (int tIndex = 0; tIndex < frameTupleAccessor.getTupleCount(); tIndex++) {
                int start = frameTupleAccessor.getTupleStartOffset(tIndex);
                int length = frameTupleAccessor.getTupleEndOffset(tIndex) - start;
                bbis.setByteBuffer(frame.getBuffer(), start);
                byte[] recordBytes = new byte[length];
                bbis.read(recordBytes, 0, length);
                resultRecords.add(new String(recordBytes, 0, length));
            }
        } finally {
            bbis.close();
        }
        readSize = reader.read(frame);
    }
    return resultRecords;
}
Also used : ArrayList(java.util.ArrayList) ByteBufferInputStream(org.apache.hyracks.dataflow.common.comm.util.ByteBufferInputStream) VSizeFrame(org.apache.hyracks.api.comm.VSizeFrame) HyracksDataset(org.apache.hyracks.client.dataset.HyracksDataset) IHyracksDataset(org.apache.hyracks.api.dataset.IHyracksDataset) IFrameTupleAccessor(org.apache.hyracks.api.comm.IFrameTupleAccessor) FrameManager(org.apache.hyracks.control.nc.resources.memory.FrameManager) ResultFrameTupleAccessor(org.apache.hyracks.dataflow.common.comm.io.ResultFrameTupleAccessor) IHyracksDatasetReader(org.apache.hyracks.api.dataset.IHyracksDatasetReader) IHyracksDataset(org.apache.hyracks.api.dataset.IHyracksDataset)

Example 2 with IHyracksDatasetReader

use of org.apache.hyracks.api.dataset.IHyracksDatasetReader in project asterixdb by apache.

the class AbstractMultiNCIntegrationTest method runTest.

protected void runTest(JobSpecification spec) throws Exception {
    if (LOGGER.isLoggable(Level.INFO)) {
        LOGGER.info(spec.toJSON().asText());
    }
    JobId jobId = hcc.startJob(spec, EnumSet.of(JobFlag.PROFILE_RUNTIME));
    if (LOGGER.isLoggable(Level.INFO)) {
        LOGGER.info(jobId.toString());
    }
    int nReaders = 1;
    FrameManager resultDisplayFrameMgr = new FrameManager(spec.getFrameSize());
    VSizeFrame resultFrame = new VSizeFrame(resultDisplayFrameMgr);
    IFrameTupleAccessor frameTupleAccessor = new ResultFrameTupleAccessor();
    if (!spec.getResultSetIds().isEmpty()) {
        IHyracksDataset hyracksDataset = new HyracksDataset(hcc, spec.getFrameSize(), nReaders);
        IHyracksDatasetReader reader = hyracksDataset.createReader(jobId, spec.getResultSetIds().get(0));
        ObjectMapper om = new ObjectMapper();
        ArrayNode resultRecords = om.createArrayNode();
        ByteBufferInputStream bbis = new ByteBufferInputStream();
        int readSize = reader.read(resultFrame);
        while (readSize > 0) {
            try {
                frameTupleAccessor.reset(resultFrame.getBuffer());
                for (int tIndex = 0; tIndex < frameTupleAccessor.getTupleCount(); tIndex++) {
                    int start = frameTupleAccessor.getTupleStartOffset(tIndex);
                    int length = frameTupleAccessor.getTupleEndOffset(tIndex) - start;
                    bbis.setByteBuffer(resultFrame.getBuffer(), start);
                    byte[] recordBytes = new byte[length];
                    bbis.read(recordBytes, 0, length);
                    resultRecords.add(new String(recordBytes, 0, length));
                }
            } finally {
                try {
                    bbis.close();
                } catch (IOException e) {
                    throw new HyracksDataException(e);
                }
            }
            readSize = reader.read(resultFrame);
        }
    }
    hcc.waitForCompletion(jobId);
    dumpOutputFiles();
}
Also used : ByteBufferInputStream(org.apache.hyracks.dataflow.common.comm.util.ByteBufferInputStream) IOException(java.io.IOException) VSizeFrame(org.apache.hyracks.api.comm.VSizeFrame) HyracksDataException(org.apache.hyracks.api.exceptions.HyracksDataException) HyracksDataset(org.apache.hyracks.client.dataset.HyracksDataset) IHyracksDataset(org.apache.hyracks.api.dataset.IHyracksDataset) IFrameTupleAccessor(org.apache.hyracks.api.comm.IFrameTupleAccessor) FrameManager(org.apache.hyracks.control.nc.resources.memory.FrameManager) ResultFrameTupleAccessor(org.apache.hyracks.dataflow.common.comm.io.ResultFrameTupleAccessor) IHyracksDatasetReader(org.apache.hyracks.api.dataset.IHyracksDatasetReader) ArrayNode(com.fasterxml.jackson.databind.node.ArrayNode) IHyracksDataset(org.apache.hyracks.api.dataset.IHyracksDataset) JobId(org.apache.hyracks.api.job.JobId) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper)

Aggregations

IFrameTupleAccessor (org.apache.hyracks.api.comm.IFrameTupleAccessor)2 VSizeFrame (org.apache.hyracks.api.comm.VSizeFrame)2 IHyracksDataset (org.apache.hyracks.api.dataset.IHyracksDataset)2 IHyracksDatasetReader (org.apache.hyracks.api.dataset.IHyracksDatasetReader)2 HyracksDataset (org.apache.hyracks.client.dataset.HyracksDataset)2 FrameManager (org.apache.hyracks.control.nc.resources.memory.FrameManager)2 ResultFrameTupleAccessor (org.apache.hyracks.dataflow.common.comm.io.ResultFrameTupleAccessor)2 ByteBufferInputStream (org.apache.hyracks.dataflow.common.comm.util.ByteBufferInputStream)2 ObjectMapper (com.fasterxml.jackson.databind.ObjectMapper)1 ArrayNode (com.fasterxml.jackson.databind.node.ArrayNode)1 IOException (java.io.IOException)1 ArrayList (java.util.ArrayList)1 HyracksDataException (org.apache.hyracks.api.exceptions.HyracksDataException)1 JobId (org.apache.hyracks.api.job.JobId)1