Examples with UTF8CodePointDecoder org.antlr.v4.runtime.UTF8CodePointDecoder used on opensource projects

Example 6 with UTF8CodePointDecoder

use of org.antlr.v4.runtime.UTF8CodePointDecoder in project antlr4 by antlr.

the class TestUTF8CodePointDecoder method decodeEmptyByteBufferWritesNothing.

@Test
public void decodeEmptyByteBufferWritesNothing() throws Exception {
    UTF8CodePointDecoder decoder = new UTF8CodePointDecoder(CodingErrorAction.REPLACE);
    ByteBuffer utf8BytesIn = ByteBuffer.allocate(0);
    IntBuffer codePointsOut = IntBuffer.allocate(0);
    IntBuffer result = decoder.decodeCodePointsFromBuffer(utf8BytesIn, codePointsOut, true);
    result.flip();
    assertEquals(0, result.remaining());
}

Example 7 with UTF8CodePointDecoder

use of org.antlr.v4.runtime.UTF8CodePointDecoder in project antlr4 by antlr.

the class TestUTF8CodePointDecoder method decodingInvalidLeadInReportModeThrows.

@Test
public void decodingInvalidLeadInReportModeThrows() throws Exception {
    UTF8CodePointDecoder decoder = new UTF8CodePointDecoder(CodingErrorAction.REPORT);
    ByteBuffer utf8BytesIn = ByteBuffer.wrap(new byte[] { (byte) 0xF8 });
    IntBuffer codePointsOut = IntBuffer.allocate(1);
    thrown.expect(CharacterCodingException.class);
    thrown.expectMessage("Invalid UTF-8 leading byte 0xF8");
    decoder.decodeCodePointsFromBuffer(utf8BytesIn, codePointsOut, true);
}

Example 8 with UTF8CodePointDecoder

use of org.antlr.v4.runtime.UTF8CodePointDecoder in project antlr4 by antlr.

the class TestUTF8CodePointDecoder method decodeCyrillicByteBufferWritesCodePoint.

@Test
public void decodeCyrillicByteBufferWritesCodePoint() throws Exception {
    UTF8CodePointDecoder decoder = new UTF8CodePointDecoder(CodingErrorAction.REPLACE);
    ByteBuffer utf8BytesIn = StandardCharsets.UTF_8.encode("Я");
    IntBuffer codePointsOut = IntBuffer.allocate(1);
    IntBuffer result = decoder.decodeCodePointsFromBuffer(utf8BytesIn, codePointsOut, true);
    result.flip();
    assertEquals(1, result.remaining());
    assertEquals(0x042F, result.get(0));
}

Example 9 with UTF8CodePointDecoder

use of org.antlr.v4.runtime.UTF8CodePointDecoder in project antlr4 by antlr.

the class TestUTF8CodePointDecoder method decodingInvalidTrailInReplaceModeWritesSubstitutionCharacter.

@Test
public void decodingInvalidTrailInReplaceModeWritesSubstitutionCharacter() throws Exception {
    UTF8CodePointDecoder decoder = new UTF8CodePointDecoder(CodingErrorAction.REPLACE);
    ByteBuffer utf8BytesIn = ByteBuffer.wrap(new byte[] { (byte) 0xC0, (byte) 0xC0 });
    IntBuffer codePointsOut = IntBuffer.allocate(1);
    IntBuffer result = decoder.decodeCodePointsFromBuffer(utf8BytesIn, codePointsOut, true);
    result.flip();
    assertEquals(1, result.remaining());
    assertEquals(0xFFFD, result.get(0));
}

Example 10 with UTF8CodePointDecoder

use of org.antlr.v4.runtime.UTF8CodePointDecoder in project antlr4 by antlr.

the class TestUTF8CodePointDecoder method decodeCJKByteBufferWritesCodePoint.

@Test
public void decodeCJKByteBufferWritesCodePoint() throws Exception {
    UTF8CodePointDecoder decoder = new UTF8CodePointDecoder(CodingErrorAction.REPLACE);
    ByteBuffer utf8BytesIn = StandardCharsets.UTF_8.encode("愛");
    IntBuffer codePointsOut = IntBuffer.allocate(1);
    IntBuffer result = decoder.decodeCodePointsFromBuffer(utf8BytesIn, codePointsOut, true);
    result.flip();
    assertEquals(1, result.remaining());
    assertEquals(0x611B, result.get(0));
}