Search in sources :

Example 6 with DataOutputViewStreamWrapper

use of org.apache.flink.core.memory.DataOutputViewStreamWrapper in project flink by apache.

the class TypeSerializerSerializationProxyTest method testStateSerializerSerializationProxyClassNotFound.

@Test
public void testStateSerializerSerializationProxyClassNotFound() throws Exception {
    TypeSerializer<?> serializer = IntSerializer.INSTANCE;
    TypeSerializerSerializationProxy<?> proxy = new TypeSerializerSerializationProxy<>(serializer);
    byte[] serialized;
    try (ByteArrayOutputStreamWithPos out = new ByteArrayOutputStreamWithPos()) {
        proxy.write(new DataOutputViewStreamWrapper(out));
        serialized = out.toByteArray();
    }
    proxy = new TypeSerializerSerializationProxy<>(new URLClassLoader(new URL[0], null));
    try (ByteArrayInputStreamWithPos in = new ByteArrayInputStreamWithPos(serialized)) {
        proxy.read(new DataInputViewStreamWrapper(in));
        fail("ClassNotFoundException expected, leading to IOException");
    } catch (IOException expected) {
    }
    proxy = new TypeSerializerSerializationProxy<>(new URLClassLoader(new URL[0], null), true);
    try (ByteArrayInputStreamWithPos in = new ByteArrayInputStreamWithPos(serialized)) {
        proxy.read(new DataInputViewStreamWrapper(in));
    }
    Assert.assertTrue(proxy.getTypeSerializer() instanceof TypeSerializerSerializationProxy.ClassNotFoundDummyTypeSerializer);
    Assert.assertArrayEquals(InstantiationUtil.serializeObject(serializer), ((TypeSerializerSerializationProxy.ClassNotFoundDummyTypeSerializer<?>) proxy.getTypeSerializer()).getActualBytes());
}
Also used : DataOutputViewStreamWrapper(org.apache.flink.core.memory.DataOutputViewStreamWrapper) URLClassLoader(java.net.URLClassLoader) ByteArrayInputStreamWithPos(org.apache.flink.core.memory.ByteArrayInputStreamWithPos) IOException(java.io.IOException) ByteArrayOutputStreamWithPos(org.apache.flink.core.memory.ByteArrayOutputStreamWithPos) DataInputViewStreamWrapper(org.apache.flink.core.memory.DataInputViewStreamWrapper) Test(org.junit.Test)

Example 7 with DataOutputViewStreamWrapper

use of org.apache.flink.core.memory.DataOutputViewStreamWrapper in project flink by apache.

the class TypeSerializerSerializationProxyTest method testStateSerializerSerializationProxy.

@Test
public void testStateSerializerSerializationProxy() throws Exception {
    TypeSerializer<?> serializer = IntSerializer.INSTANCE;
    TypeSerializerSerializationProxy<?> proxy = new TypeSerializerSerializationProxy<>(serializer);
    byte[] serialized;
    try (ByteArrayOutputStreamWithPos out = new ByteArrayOutputStreamWithPos()) {
        proxy.write(new DataOutputViewStreamWrapper(out));
        serialized = out.toByteArray();
    }
    proxy = new TypeSerializerSerializationProxy<>(Thread.currentThread().getContextClassLoader());
    try (ByteArrayInputStreamWithPos in = new ByteArrayInputStreamWithPos(serialized)) {
        proxy.read(new DataInputViewStreamWrapper(in));
    }
    Assert.assertEquals(serializer, proxy.getTypeSerializer());
}
Also used : DataOutputViewStreamWrapper(org.apache.flink.core.memory.DataOutputViewStreamWrapper) ByteArrayInputStreamWithPos(org.apache.flink.core.memory.ByteArrayInputStreamWithPos) ByteArrayOutputStreamWithPos(org.apache.flink.core.memory.ByteArrayOutputStreamWithPos) DataInputViewStreamWrapper(org.apache.flink.core.memory.DataInputViewStreamWrapper) Test(org.junit.Test)

Example 8 with DataOutputViewStreamWrapper

use of org.apache.flink.core.memory.DataOutputViewStreamWrapper in project flink by apache.

the class RocksDBReducingState method add.

@Override
public void add(V value) throws IOException {
    try {
        writeCurrentKeyWithGroupAndNamespace();
        byte[] key = keySerializationStream.toByteArray();
        byte[] valueBytes = backend.db.get(columnFamily, key);
        DataOutputViewStreamWrapper out = new DataOutputViewStreamWrapper(keySerializationStream);
        if (valueBytes == null) {
            keySerializationStream.reset();
            valueSerializer.serialize(value, out);
            backend.db.put(columnFamily, writeOptions, key, keySerializationStream.toByteArray());
        } else {
            V oldValue = valueSerializer.deserialize(new DataInputViewStreamWrapper(new ByteArrayInputStream(valueBytes)));
            V newValue = reduceFunction.reduce(oldValue, value);
            keySerializationStream.reset();
            valueSerializer.serialize(newValue, out);
            backend.db.put(columnFamily, writeOptions, key, keySerializationStream.toByteArray());
        }
    } catch (Exception e) {
        throw new RuntimeException("Error while adding data to RocksDB", e);
    }
}
Also used : DataOutputViewStreamWrapper(org.apache.flink.core.memory.DataOutputViewStreamWrapper) ByteArrayInputStream(java.io.ByteArrayInputStream) DataInputViewStreamWrapper(org.apache.flink.core.memory.DataInputViewStreamWrapper) IOException(java.io.IOException) RocksDBException(org.rocksdb.RocksDBException)

Example 9 with DataOutputViewStreamWrapper

use of org.apache.flink.core.memory.DataOutputViewStreamWrapper in project flink by apache.

the class CollectSink method open.

/**
	 * Initialize the connection with the Socket in the server.
	 * @param parameters Configuration.
	 */
@Override
public void open(Configuration parameters) throws Exception {
    try {
        client = new Socket(hostIp, port);
        outputStream = client.getOutputStream();
        streamWriter = new DataOutputViewStreamWrapper(outputStream);
    } catch (IOException e) {
        throw new IOException("Cannot connect to the client to send back the stream", e);
    }
}
Also used : DataOutputViewStreamWrapper(org.apache.flink.core.memory.DataOutputViewStreamWrapper) IOException(java.io.IOException) Socket(java.net.Socket)

Example 10 with DataOutputViewStreamWrapper

use of org.apache.flink.core.memory.DataOutputViewStreamWrapper in project flink by apache.

the class AbstractStreamOperator method snapshotState.

/**
	 * Stream operators with state, which want to participate in a snapshot need to override this hook method.
	 *
	 * @param context context that provides information and means required for taking a snapshot
	 */
public void snapshotState(StateSnapshotContext context) throws Exception {
    if (getKeyedStateBackend() != null) {
        KeyedStateCheckpointOutputStream out;
        try {
            out = context.getRawKeyedOperatorStateOutput();
        } catch (Exception exception) {
            throw new Exception("Could not open raw keyed operator state stream for " + getOperatorName() + '.', exception);
        }
        try {
            KeyGroupsList allKeyGroups = out.getKeyGroupList();
            for (int keyGroupIdx : allKeyGroups) {
                out.startNewKeyGroup(keyGroupIdx);
                timeServiceManager.snapshotStateForKeyGroup(new DataOutputViewStreamWrapper(out), keyGroupIdx);
            }
        } catch (Exception exception) {
            throw new Exception("Could not write timer service of " + getOperatorName() + " to checkpoint state stream.", exception);
        } finally {
            try {
                out.close();
            } catch (Exception closeException) {
                LOG.warn("Could not close raw keyed operator state stream for {}. This " + "might have prevented deleting some state data.", getOperatorName(), closeException);
            }
        }
    }
}
Also used : DataOutputViewStreamWrapper(org.apache.flink.core.memory.DataOutputViewStreamWrapper) KeyGroupsList(org.apache.flink.runtime.state.KeyGroupsList) ConcurrentModificationException(java.util.ConcurrentModificationException) IOException(java.io.IOException) KeyedStateCheckpointOutputStream(org.apache.flink.runtime.state.KeyedStateCheckpointOutputStream)

Aggregations

DataOutputViewStreamWrapper (org.apache.flink.core.memory.DataOutputViewStreamWrapper)123 DataInputViewStreamWrapper (org.apache.flink.core.memory.DataInputViewStreamWrapper)55 ByteArrayOutputStream (java.io.ByteArrayOutputStream)49 Test (org.junit.Test)43 ByteArrayOutputStreamWithPos (org.apache.flink.core.memory.ByteArrayOutputStreamWithPos)35 IOException (java.io.IOException)28 ByteArrayInputStream (java.io.ByteArrayInputStream)26 ByteArrayInputStreamWithPos (org.apache.flink.core.memory.ByteArrayInputStreamWithPos)23 DataOutputView (org.apache.flink.core.memory.DataOutputView)18 HashMap (java.util.HashMap)13 ArrayList (java.util.ArrayList)12 ExecutionConfig (org.apache.flink.api.common.ExecutionConfig)11 Map (java.util.Map)10 TypeSerializerSnapshot (org.apache.flink.api.common.typeutils.TypeSerializerSnapshot)7 StateMetaInfoSnapshot (org.apache.flink.runtime.state.metainfo.StateMetaInfoSnapshot)7 Before (org.junit.Before)6 Socket (java.net.Socket)5 PipedInputStream (java.io.PipedInputStream)4 PipedOutputStream (java.io.PipedOutputStream)4 List (java.util.List)4