Examples with UTF8CodePointDecoder org.antlr.v4.runtime.UTF8CodePointDecoder used on opensource projects

Example 1 with UTF8CodePointDecoder

use of org.antlr.v4.runtime.UTF8CodePointDecoder in project antlr4 by antlr.

the class TestUTF8CodePointDecoder method decodeLatinByteBufferWritesCodePoint.

@Test
public void decodeLatinByteBufferWritesCodePoint() throws Exception {
    UTF8CodePointDecoder decoder = new UTF8CodePointDecoder(CodingErrorAction.REPLACE);
    ByteBuffer utf8BytesIn = StandardCharsets.UTF_8.encode("X");
    IntBuffer codePointsOut = IntBuffer.allocate(1);
    IntBuffer result = decoder.decodeCodePointsFromBuffer(utf8BytesIn, codePointsOut, true);
    result.flip();
    assertEquals(1, result.remaining());
    assertEquals('X', result.get(0));
}

Example 2 with UTF8CodePointDecoder

use of org.antlr.v4.runtime.UTF8CodePointDecoder in project antlr4 by antlr.

the class TestUTF8CodePointDecoder method decodingInvalidLeadInReplaceModeWritesSubstitutionCharacter.

@Test
public void decodingInvalidLeadInReplaceModeWritesSubstitutionCharacter() throws Exception {
    UTF8CodePointDecoder decoder = new UTF8CodePointDecoder(CodingErrorAction.REPLACE);
    ByteBuffer utf8BytesIn = ByteBuffer.wrap(new byte[] { (byte) 0xF8 });
    IntBuffer codePointsOut = IntBuffer.allocate(1);
    IntBuffer result = decoder.decodeCodePointsFromBuffer(utf8BytesIn, codePointsOut, true);
    result.flip();
    assertEquals(1, result.remaining());
    assertEquals(0xFFFD, result.get(0));
}

Example 3 with UTF8CodePointDecoder

use of org.antlr.v4.runtime.UTF8CodePointDecoder in project antlr4 by antlr.

the class TestUTF8CodePointDecoder method decodeEmojiByteBufferWritesCodePoint.

@Test
public void decodeEmojiByteBufferWritesCodePoint() throws Exception {
    UTF8CodePointDecoder decoder = new UTF8CodePointDecoder(CodingErrorAction.REPLACE);
    ByteBuffer utf8BytesIn = StandardCharsets.UTF_8.encode(new StringBuilder().appendCodePoint(0x1F4A9).toString());
    IntBuffer codePointsOut = IntBuffer.allocate(1);
    IntBuffer result = decoder.decodeCodePointsFromBuffer(utf8BytesIn, codePointsOut, true);
    result.flip();
    assertEquals(1, result.remaining());
    assertEquals(0x1F4A9, result.get(0));
}

Example 4 with UTF8CodePointDecoder

use of org.antlr.v4.runtime.UTF8CodePointDecoder in project antlr4 by antlr.

the class TestUTF8CodePointDecoder method decodingInvalidTrailInReportModeThrows.

@Test
public void decodingInvalidTrailInReportModeThrows() throws Exception {
    UTF8CodePointDecoder decoder = new UTF8CodePointDecoder(CodingErrorAction.REPORT);
    ByteBuffer utf8BytesIn = ByteBuffer.wrap(new byte[] { (byte) 0xC0, (byte) 0xC0 });
    IntBuffer codePointsOut = IntBuffer.allocate(1);
    thrown.expect(CharacterCodingException.class);
    thrown.expectMessage("Invalid UTF-8 trailing byte 0xC0");
    decoder.decodeCodePointsFromBuffer(utf8BytesIn, codePointsOut, true);
}

Example 5 with UTF8CodePointDecoder

use of org.antlr.v4.runtime.UTF8CodePointDecoder in project antlr4 by antlr.

the class TestUTF8CodePointDecoder method decodingNonShortestFormInReportModeThrows.

@Test
public void decodingNonShortestFormInReportModeThrows() throws Exception {
    UTF8CodePointDecoder decoder = new UTF8CodePointDecoder(CodingErrorAction.REPORT);
    // 0xC1 0x9C would decode to \ (U+005C) if we didn't have this check
    ByteBuffer utf8BytesIn = ByteBuffer.wrap(new byte[] { (byte) 0xC1, (byte) 0x9C });
    IntBuffer codePointsOut = IntBuffer.allocate(1);
    thrown.expect(CharacterCodingException.class);
    thrown.expectMessage("Code point 92 is out of expected range 128..2047");
    decoder.decodeCodePointsFromBuffer(utf8BytesIn, codePointsOut, true);
}