Examples with IndexOutputOutputStream org.elasticsearch.common.lucene.store.IndexOutputOutputStream used on opensource projects

Example 6 with IndexOutputOutputStream

use of org.elasticsearch.common.lucene.store.IndexOutputOutputStream in project elasticsearch by elastic.

the class MetaDataStateFormat method write.

/**
     * Writes the given state to the given directories. The state is written to a
     * state directory ({@value #STATE_DIR_NAME}) underneath each of the given file locations and is created if it
     * doesn't exist. The state is serialized to a temporary file in that directory and is then atomically moved to
     * it's target filename of the pattern <tt>{prefix}{version}.st</tt>.
     *
     * @param state the state object to write
     * @param locations the locations where the state should be written to.
     * @throws IOException if an IOException occurs
     */
public final void write(final T state, final Path... locations) throws IOException {
    if (locations == null) {
        throw new IllegalArgumentException("Locations must not be null");
    }
    if (locations.length <= 0) {
        throw new IllegalArgumentException("One or more locations required");
    }
    final long maxStateId = findMaxStateId(prefix, locations) + 1;
    assert maxStateId >= 0 : "maxStateId must be positive but was: [" + maxStateId + "]";
    final String fileName = prefix + maxStateId + STATE_FILE_EXTENSION;
    Path stateLocation = locations[0].resolve(STATE_DIR_NAME);
    Files.createDirectories(stateLocation);
    final Path tmpStatePath = stateLocation.resolve(fileName + ".tmp");
    final Path finalStatePath = stateLocation.resolve(fileName);
    try {
        final String resourceDesc = "MetaDataStateFormat.write(path=\"" + tmpStatePath + "\")";
        try (OutputStreamIndexOutput out = new OutputStreamIndexOutput(resourceDesc, fileName, Files.newOutputStream(tmpStatePath), BUFFER_SIZE)) {
            CodecUtil.writeHeader(out, STATE_FILE_CODEC, STATE_FILE_VERSION);
            out.writeInt(format.index());
            try (XContentBuilder builder = newXContentBuilder(format, new IndexOutputOutputStream(out) {

                @Override
                public void close() throws IOException {
                // this is important since some of the XContentBuilders write bytes on close.
                // in order to write the footer we need to prevent closing the actual index input.
                }
            })) {
                builder.startObject();
                {
                    toXContent(builder, state);
                }
                builder.endObject();
            }
            CodecUtil.writeFooter(out);
        }
        // fsync the state file
        IOUtils.fsync(tmpStatePath, false);
        Files.move(tmpStatePath, finalStatePath, StandardCopyOption.ATOMIC_MOVE);
        IOUtils.fsync(stateLocation, true);
        for (int i = 1; i < locations.length; i++) {
            stateLocation = locations[i].resolve(STATE_DIR_NAME);
            Files.createDirectories(stateLocation);
            Path tmpPath = stateLocation.resolve(fileName + ".tmp");
            Path finalPath = stateLocation.resolve(fileName);
            try {
                Files.copy(finalStatePath, tmpPath);
                // we are on the same FileSystem / Partition here we can do an atomic move
                Files.move(tmpPath, finalPath, StandardCopyOption.ATOMIC_MOVE);
                // we just fsync the dir here..
                IOUtils.fsync(stateLocation, true);
            } finally {
                Files.deleteIfExists(tmpPath);
            }
        }
    } finally {
        Files.deleteIfExists(tmpStatePath);
    }
    cleanupOldFiles(prefix, fileName, locations);
}

Example 7 with IndexOutputOutputStream

use of org.elasticsearch.common.lucene.store.IndexOutputOutputStream in project crate by crate.

the class ChecksumBlobStoreFormat method writeTo.

private void writeTo(final T obj, final String blobName, final CheckedConsumer<BytesArray, IOException> consumer) throws IOException {
    final BytesReference bytes;
    try (BytesStreamOutput bytesStreamOutput = new BytesStreamOutput()) {
        if (compress) {
            try (StreamOutput compressedStreamOutput = CompressorFactory.COMPRESSOR.streamOutput(bytesStreamOutput)) {
                write(obj, compressedStreamOutput);
            }
        } else {
            write(obj, bytesStreamOutput);
        }
        bytes = bytesStreamOutput.bytes();
    }
    try (ByteArrayOutputStream outputStream = new ByteArrayOutputStream()) {
        final String resourceDesc = "ChecksumBlobStoreFormat.writeBlob(blob=\"" + blobName + "\")";
        try (OutputStreamIndexOutput indexOutput = new OutputStreamIndexOutput(resourceDesc, blobName, outputStream, BUFFER_SIZE)) {
            CodecUtil.writeHeader(indexOutput, codec, VERSION);
            try (OutputStream indexOutputOutputStream = new IndexOutputOutputStream(indexOutput) {

                @Override
                public void close() {
                // this is important since some of the XContentBuilders write bytes on close.
                // in order to write the footer we need to prevent closing the actual index input.
                }
            }) {
                bytes.writeTo(indexOutputOutputStream);
            }
            CodecUtil.writeFooter(indexOutput);
        }
        consumer.accept(new BytesArray(outputStream.toByteArray()));
    }
}