Search in sources :

Example 1 with RAMOutputStream

use of org.apache.lucene.store.RAMOutputStream in project elasticsearch by elastic.

the class TransportSearchHelper method buildScrollId.

static String buildScrollId(AtomicArray<? extends SearchPhaseResult> searchPhaseResults) throws IOException {
    try (RAMOutputStream out = new RAMOutputStream()) {
        out.writeString(searchPhaseResults.length() == 1 ? ParsedScrollId.QUERY_AND_FETCH_TYPE : ParsedScrollId.QUERY_THEN_FETCH_TYPE);
        out.writeVInt(searchPhaseResults.asList().size());
        for (AtomicArray.Entry<? extends SearchPhaseResult> entry : searchPhaseResults.asList()) {
            SearchPhaseResult searchPhaseResult = entry.value;
            out.writeLong(searchPhaseResult.id());
            out.writeString(searchPhaseResult.shardTarget().getNodeId());
        }
        byte[] bytes = new byte[(int) out.getFilePointer()];
        out.writeTo(bytes, 0);
        return Base64.getUrlEncoder().encodeToString(bytes);
    }
}
Also used : AtomicArray(org.elasticsearch.common.util.concurrent.AtomicArray) RAMOutputStream(org.apache.lucene.store.RAMOutputStream) SearchPhaseResult(org.elasticsearch.search.SearchPhaseResult)

Example 2 with RAMOutputStream

use of org.apache.lucene.store.RAMOutputStream in project lucene-solr by apache.

the class TestLucene54DocValuesFormat method testSortedNumericAroundBlockSize.

@Slow
public void testSortedNumericAroundBlockSize() throws IOException {
    final int frontier = 1 << Lucene54DocValuesFormat.DIRECT_MONOTONIC_BLOCK_SHIFT;
    for (int maxDoc = frontier - 1; maxDoc <= frontier + 1; ++maxDoc) {
        final Directory dir = newDirectory();
        IndexWriter w = new IndexWriter(dir, newIndexWriterConfig().setMergePolicy(newLogMergePolicy()));
        RAMFile buffer = new RAMFile();
        RAMOutputStream out = new RAMOutputStream(buffer, false);
        Document doc = new Document();
        SortedNumericDocValuesField field1 = new SortedNumericDocValuesField("snum", 0L);
        doc.add(field1);
        SortedNumericDocValuesField field2 = new SortedNumericDocValuesField("snum", 0L);
        doc.add(field2);
        for (int i = 0; i < maxDoc; ++i) {
            long s1 = random().nextInt(100);
            long s2 = random().nextInt(100);
            field1.setLongValue(s1);
            field2.setLongValue(s2);
            w.addDocument(doc);
            out.writeVLong(Math.min(s1, s2));
            out.writeVLong(Math.max(s1, s2));
        }
        out.close();
        w.forceMerge(1);
        DirectoryReader r = DirectoryReader.open(w);
        w.close();
        LeafReader sr = getOnlyLeafReader(r);
        assertEquals(maxDoc, sr.maxDoc());
        SortedNumericDocValues values = sr.getSortedNumericDocValues("snum");
        assertNotNull(values);
        RAMInputStream in = new RAMInputStream("", buffer);
        for (int i = 0; i < maxDoc; ++i) {
            assertEquals(i, values.nextDoc());
            assertEquals(2, values.docValueCount());
            assertEquals(in.readVLong(), values.nextValue());
            assertEquals(in.readVLong(), values.nextValue());
        }
        r.close();
        dir.close();
    }
}
Also used : RAMFile(org.apache.lucene.store.RAMFile) SortedNumericDocValuesField(org.apache.lucene.document.SortedNumericDocValuesField) SortedNumericDocValues(org.apache.lucene.index.SortedNumericDocValues) LeafReader(org.apache.lucene.index.LeafReader) IndexWriter(org.apache.lucene.index.IndexWriter) RandomIndexWriter(org.apache.lucene.index.RandomIndexWriter) DirectoryReader(org.apache.lucene.index.DirectoryReader) RAMInputStream(org.apache.lucene.store.RAMInputStream) RAMOutputStream(org.apache.lucene.store.RAMOutputStream) Document(org.apache.lucene.document.Document) Directory(org.apache.lucene.store.Directory)

Example 3 with RAMOutputStream

use of org.apache.lucene.store.RAMOutputStream in project lucene-solr by apache.

the class TestLucene54DocValuesFormat method testSortedSetAroundBlockSize.

@Slow
public void testSortedSetAroundBlockSize() throws IOException {
    final int frontier = 1 << Lucene54DocValuesFormat.DIRECT_MONOTONIC_BLOCK_SHIFT;
    for (int maxDoc = frontier - 1; maxDoc <= frontier + 1; ++maxDoc) {
        final Directory dir = newDirectory();
        IndexWriter w = new IndexWriter(dir, newIndexWriterConfig().setMergePolicy(newLogMergePolicy()));
        RAMFile buffer = new RAMFile();
        RAMOutputStream out = new RAMOutputStream(buffer, false);
        Document doc = new Document();
        SortedSetDocValuesField field1 = new SortedSetDocValuesField("sset", new BytesRef());
        doc.add(field1);
        SortedSetDocValuesField field2 = new SortedSetDocValuesField("sset", new BytesRef());
        doc.add(field2);
        for (int i = 0; i < maxDoc; ++i) {
            BytesRef s1 = new BytesRef(TestUtil.randomSimpleString(random(), 2));
            BytesRef s2 = new BytesRef(TestUtil.randomSimpleString(random(), 2));
            field1.setBytesValue(s1);
            field2.setBytesValue(s2);
            w.addDocument(doc);
            Set<BytesRef> set = new TreeSet<>(Arrays.asList(s1, s2));
            out.writeVInt(set.size());
            for (BytesRef ref : set) {
                out.writeVInt(ref.length);
                out.writeBytes(ref.bytes, ref.offset, ref.length);
            }
        }
        out.close();
        w.forceMerge(1);
        DirectoryReader r = DirectoryReader.open(w);
        w.close();
        LeafReader sr = getOnlyLeafReader(r);
        assertEquals(maxDoc, sr.maxDoc());
        SortedSetDocValues values = sr.getSortedSetDocValues("sset");
        assertNotNull(values);
        RAMInputStream in = new RAMInputStream("", buffer);
        BytesRefBuilder b = new BytesRefBuilder();
        for (int i = 0; i < maxDoc; ++i) {
            assertEquals(i, values.nextDoc());
            final int numValues = in.readVInt();
            for (int j = 0; j < numValues; ++j) {
                b.setLength(in.readVInt());
                b.grow(b.length());
                in.readBytes(b.bytes(), 0, b.length());
                assertEquals(b.get(), values.lookupOrd(values.nextOrd()));
            }
            assertEquals(SortedSetDocValues.NO_MORE_ORDS, values.nextOrd());
        }
        r.close();
        dir.close();
    }
}
Also used : BytesRefBuilder(org.apache.lucene.util.BytesRefBuilder) LeafReader(org.apache.lucene.index.LeafReader) DirectoryReader(org.apache.lucene.index.DirectoryReader) RAMInputStream(org.apache.lucene.store.RAMInputStream) Document(org.apache.lucene.document.Document) RAMFile(org.apache.lucene.store.RAMFile) SortedSetDocValues(org.apache.lucene.index.SortedSetDocValues) IndexWriter(org.apache.lucene.index.IndexWriter) RandomIndexWriter(org.apache.lucene.index.RandomIndexWriter) TreeSet(java.util.TreeSet) RAMOutputStream(org.apache.lucene.store.RAMOutputStream) SortedSetDocValuesField(org.apache.lucene.document.SortedSetDocValuesField) BytesRef(org.apache.lucene.util.BytesRef) Directory(org.apache.lucene.store.Directory)

Example 4 with RAMOutputStream

use of org.apache.lucene.store.RAMOutputStream in project lucene-solr by apache.

the class TestLucene70DocValuesFormat method testSortedSetAroundBlockSize.

@Slow
public void testSortedSetAroundBlockSize() throws IOException {
    final int frontier = 1 << Lucene70DocValuesFormat.DIRECT_MONOTONIC_BLOCK_SHIFT;
    for (int maxDoc = frontier - 1; maxDoc <= frontier + 1; ++maxDoc) {
        final Directory dir = newDirectory();
        IndexWriter w = new IndexWriter(dir, newIndexWriterConfig().setMergePolicy(newLogMergePolicy()));
        RAMFile buffer = new RAMFile();
        RAMOutputStream out = new RAMOutputStream(buffer, false);
        Document doc = new Document();
        SortedSetDocValuesField field1 = new SortedSetDocValuesField("sset", new BytesRef());
        doc.add(field1);
        SortedSetDocValuesField field2 = new SortedSetDocValuesField("sset", new BytesRef());
        doc.add(field2);
        for (int i = 0; i < maxDoc; ++i) {
            BytesRef s1 = new BytesRef(TestUtil.randomSimpleString(random(), 2));
            BytesRef s2 = new BytesRef(TestUtil.randomSimpleString(random(), 2));
            field1.setBytesValue(s1);
            field2.setBytesValue(s2);
            w.addDocument(doc);
            Set<BytesRef> set = new TreeSet<>(Arrays.asList(s1, s2));
            out.writeVInt(set.size());
            for (BytesRef ref : set) {
                out.writeVInt(ref.length);
                out.writeBytes(ref.bytes, ref.offset, ref.length);
            }
        }
        out.close();
        w.forceMerge(1);
        DirectoryReader r = DirectoryReader.open(w);
        w.close();
        LeafReader sr = getOnlyLeafReader(r);
        assertEquals(maxDoc, sr.maxDoc());
        SortedSetDocValues values = sr.getSortedSetDocValues("sset");
        assertNotNull(values);
        RAMInputStream in = new RAMInputStream("", buffer);
        BytesRefBuilder b = new BytesRefBuilder();
        for (int i = 0; i < maxDoc; ++i) {
            assertEquals(i, values.nextDoc());
            final int numValues = in.readVInt();
            for (int j = 0; j < numValues; ++j) {
                b.setLength(in.readVInt());
                b.grow(b.length());
                in.readBytes(b.bytes(), 0, b.length());
                assertEquals(b.get(), values.lookupOrd(values.nextOrd()));
            }
            assertEquals(SortedSetDocValues.NO_MORE_ORDS, values.nextOrd());
        }
        r.close();
        dir.close();
    }
}
Also used : BytesRefBuilder(org.apache.lucene.util.BytesRefBuilder) LeafReader(org.apache.lucene.index.LeafReader) DirectoryReader(org.apache.lucene.index.DirectoryReader) RAMInputStream(org.apache.lucene.store.RAMInputStream) Document(org.apache.lucene.document.Document) RAMFile(org.apache.lucene.store.RAMFile) SortedSetDocValues(org.apache.lucene.index.SortedSetDocValues) IndexWriter(org.apache.lucene.index.IndexWriter) RandomIndexWriter(org.apache.lucene.index.RandomIndexWriter) TreeSet(java.util.TreeSet) RAMOutputStream(org.apache.lucene.store.RAMOutputStream) SortedSetDocValuesField(org.apache.lucene.document.SortedSetDocValuesField) BytesRef(org.apache.lucene.util.BytesRef) Directory(org.apache.lucene.store.Directory)

Example 5 with RAMOutputStream

use of org.apache.lucene.store.RAMOutputStream in project lucene-solr by apache.

the class TestCodecUtil method testSegmentHeaderLength.

public void testSegmentHeaderLength() throws Exception {
    RAMFile file = new RAMFile();
    IndexOutput output = new RAMOutputStream(file, true);
    CodecUtil.writeIndexHeader(output, "FooBar", 5, StringHelper.randomId(), "xyz");
    output.writeString("this is the data");
    output.close();
    IndexInput input = new RAMInputStream("file", file);
    input.seek(CodecUtil.indexHeaderLength("FooBar", "xyz"));
    assertEquals("this is the data", input.readString());
    input.close();
}
Also used : RAMFile(org.apache.lucene.store.RAMFile) RAMInputStream(org.apache.lucene.store.RAMInputStream) RAMOutputStream(org.apache.lucene.store.RAMOutputStream) IndexInput(org.apache.lucene.store.IndexInput) ChecksumIndexInput(org.apache.lucene.store.ChecksumIndexInput) BufferedChecksumIndexInput(org.apache.lucene.store.BufferedChecksumIndexInput) IndexOutput(org.apache.lucene.store.IndexOutput)

Aggregations

RAMOutputStream (org.apache.lucene.store.RAMOutputStream)27 RAMFile (org.apache.lucene.store.RAMFile)21 IndexOutput (org.apache.lucene.store.IndexOutput)16 RAMInputStream (org.apache.lucene.store.RAMInputStream)15 BufferedChecksumIndexInput (org.apache.lucene.store.BufferedChecksumIndexInput)11 ChecksumIndexInput (org.apache.lucene.store.ChecksumIndexInput)11 IndexInput (org.apache.lucene.store.IndexInput)7 CorruptIndexException (org.apache.lucene.index.CorruptIndexException)5 BytesRef (org.apache.lucene.util.BytesRef)5 BytesRefBuilder (org.apache.lucene.util.BytesRefBuilder)5 IOException (java.io.IOException)4 Document (org.apache.lucene.document.Document)4 DirectoryReader (org.apache.lucene.index.DirectoryReader)4 IndexWriter (org.apache.lucene.index.IndexWriter)4 LeafReader (org.apache.lucene.index.LeafReader)4 RandomIndexWriter (org.apache.lucene.index.RandomIndexWriter)4 Directory (org.apache.lucene.store.Directory)4 TreeSet (java.util.TreeSet)2 SortedNumericDocValuesField (org.apache.lucene.document.SortedNumericDocValuesField)2 SortedSetDocValuesField (org.apache.lucene.document.SortedSetDocValuesField)2