Examples with JsonRecordSetWriter org.apache.nifi.json.JsonRecordSetWriter used on opensource projects

Search in sources :

Example 1 with JsonRecordSetWriter

use of org.apache.nifi.json.JsonRecordSetWriter in project nifi by apache.

the class TestUpdateRecord method testSetRootPathRelativeWithSingleValue.

@Test
public void testSetRootPathRelativeWithSingleValue() throws InitializationException, IOException {
    final JsonTreeReader jsonReader = new JsonTreeReader();
    runner.addControllerService("reader", jsonReader);
    final String inputSchemaText = new String(Files.readAllBytes(Paths.get("src/test/resources/TestUpdateRecord/schema/person-with-name-record.avsc")));
    final String outputSchemaText = new String(Files.readAllBytes(Paths.get("src/test/resources/TestUpdateRecord/schema/name-fields-only.avsc")));
    runner.setProperty(jsonReader, SchemaAccessUtils.SCHEMA_ACCESS_STRATEGY, SchemaAccessUtils.SCHEMA_TEXT_PROPERTY);
    runner.setProperty(jsonReader, SchemaAccessUtils.SCHEMA_TEXT, inputSchemaText);
    runner.enableControllerService(jsonReader);
    final JsonRecordSetWriter jsonWriter = new JsonRecordSetWriter();
    runner.addControllerService("writer", jsonWriter);
    runner.setProperty(jsonWriter, SchemaAccessUtils.SCHEMA_ACCESS_STRATEGY, SchemaAccessUtils.SCHEMA_TEXT_PROPERTY);
    runner.setProperty(jsonWriter, SchemaAccessUtils.SCHEMA_TEXT, outputSchemaText);
    runner.setProperty(jsonWriter, "Pretty Print JSON", "true");
    runner.setProperty(jsonWriter, "Schema Write Strategy", "full-schema-attribute");
    runner.enableControllerService(jsonWriter);
    runner.enqueue(Paths.get("src/test/resources/TestUpdateRecord/input/person.json"));
    runner.setProperty("/name/..", "/name");
    runner.setProperty(UpdateRecord.REPLACEMENT_VALUE_STRATEGY, UpdateRecord.RECORD_PATH_VALUES);
    runner.run();
    runner.assertAllFlowFilesTransferred(UpdateRecord.REL_SUCCESS, 1);
    final String expectedOutput = new String(Files.readAllBytes(Paths.get("src/test/resources/TestUpdateRecord/output/name-fields-only.json")));
    runner.getFlowFilesForRelationship(UpdateRecord.REL_SUCCESS).get(0).assertContentEquals(expectedOutput);
}
Also used : JsonTreeReader(org.apache.nifi.json.JsonTreeReader) JsonRecordSetWriter(org.apache.nifi.json.JsonRecordSetWriter) Test(org.junit.Test)

Example 2 with JsonRecordSetWriter

use of org.apache.nifi.json.JsonRecordSetWriter in project nifi by apache.

the class TestUpdateRecord method testUpdateComplexArrays.

@Test
public void testUpdateComplexArrays() throws InitializationException, IOException {
    final JsonTreeReader jsonReader = new JsonTreeReader();
    runner.addControllerService("reader", jsonReader);
    runner.setValidateExpressionUsage(false);
    final String inputSchemaText = new String(Files.readAllBytes(Paths.get("src/test/resources/TestUpdateRecord/schema/multi-arrays.avsc")));
    final String outputSchemaText = new String(Files.readAllBytes(Paths.get("src/test/resources/TestUpdateRecord/schema/multi-arrays.avsc")));
    runner.setProperty(jsonReader, SchemaAccessUtils.SCHEMA_ACCESS_STRATEGY, SchemaAccessUtils.SCHEMA_TEXT_PROPERTY);
    runner.setProperty(jsonReader, SchemaAccessUtils.SCHEMA_TEXT, inputSchemaText);
    runner.enableControllerService(jsonReader);
    final JsonRecordSetWriter jsonWriter = new JsonRecordSetWriter();
    runner.addControllerService("writer", jsonWriter);
    runner.setProperty(jsonWriter, SchemaAccessUtils.SCHEMA_ACCESS_STRATEGY, SchemaAccessUtils.SCHEMA_TEXT_PROPERTY);
    runner.setProperty(jsonWriter, SchemaAccessUtils.SCHEMA_TEXT, outputSchemaText);
    runner.setProperty(jsonWriter, "Pretty Print JSON", "true");
    runner.setProperty(jsonWriter, "Schema Write Strategy", "full-schema-attribute");
    runner.setProperty(UpdateRecord.REPLACEMENT_VALUE_STRATEGY, UpdateRecord.RECORD_PATH_VALUES);
    runner.enableControllerService(jsonWriter);
    runner.enqueue(Paths.get("src/test/resources/TestUpdateRecord/input/multi-arrays.json"));
    runner.setProperty("/peoples[*]", "/peoples[3]");
    runner.run();
    runner.assertAllFlowFilesTransferred(UpdateRecord.REL_SUCCESS, 1);
    String content = new String(runner.getFlowFilesForRelationship(UpdateRecord.REL_SUCCESS).get(0).toByteArray());
    int count = StringUtils.countMatches(content, "Mary Doe");
    assertEquals(4, count);
    runner.removeProperty("/peoples[*]");
    runner.clearTransferState();
    runner.enqueue(Paths.get("src/test/resources/TestUpdateRecord/input/multi-arrays.json"));
    runner.setProperty("/peoples[1]", "/peoples[3]");
    runner.run();
    runner.assertAllFlowFilesTransferred(UpdateRecord.REL_SUCCESS, 1);
    content = new String(runner.getFlowFilesForRelationship(UpdateRecord.REL_SUCCESS).get(0).toByteArray());
    count = StringUtils.countMatches(content, "Mary Doe");
    assertEquals(2, count);
    runner.removeProperty("/peoples[1]");
    runner.clearTransferState();
    runner.enqueue(Paths.get("src/test/resources/TestUpdateRecord/input/multi-arrays.json"));
    runner.setProperty("/peoples[0..1]", "/peoples[3]");
    runner.run();
    runner.assertAllFlowFilesTransferred(UpdateRecord.REL_SUCCESS, 1);
    String expectedOutput = new String(Files.readAllBytes(Paths.get("src/test/resources/TestUpdateRecord/output/updateArrays/multi-arrays-0and1.json")));
    runner.getFlowFilesForRelationship(UpdateRecord.REL_SUCCESS).get(0).assertContentEquals(expectedOutput);
    runner.removeProperty("/peoples[0..1]");
    runner.clearTransferState();
    runner.enqueue(Paths.get("src/test/resources/TestUpdateRecord/input/multi-arrays.json"));
    runner.setProperty("/peoples[0,2]", "/peoples[3]");
    runner.run();
    runner.assertAllFlowFilesTransferred(UpdateRecord.REL_SUCCESS, 1);
    expectedOutput = new String(Files.readAllBytes(Paths.get("src/test/resources/TestUpdateRecord/output/updateArrays/multi-arrays-0and2.json")));
    runner.getFlowFilesForRelationship(UpdateRecord.REL_SUCCESS).get(0).assertContentEquals(expectedOutput);
    runner.removeProperty("/peoples[0,2]");
    runner.clearTransferState();
    runner.enqueue(Paths.get("src/test/resources/TestUpdateRecord/input/multi-arrays.json"));
    runner.setProperty("/peoples[0,1..2]", "/peoples[3]");
    runner.run();
    runner.assertAllFlowFilesTransferred(UpdateRecord.REL_SUCCESS, 1);
    content = new String(runner.getFlowFilesForRelationship(UpdateRecord.REL_SUCCESS).get(0).toByteArray());
    count = StringUtils.countMatches(content, "Mary Doe");
    assertEquals(4, count);
    runner.removeProperty("/peoples[0,1..2]");
    runner.clearTransferState();
    runner.enqueue(Paths.get("src/test/resources/TestUpdateRecord/input/multi-arrays.json"));
    runner.setProperty("/peoples[0..-1][./name != 'Mary Doe']", "/peoples[3]");
    runner.run();
    runner.assertAllFlowFilesTransferred(UpdateRecord.REL_SUCCESS, 1);
    content = new String(runner.getFlowFilesForRelationship(UpdateRecord.REL_SUCCESS).get(0).toByteArray());
    count = StringUtils.countMatches(content, "Mary Doe");
    assertEquals(4, count);
    runner.removeProperty("/peoples[0..-1][./name != 'Mary Doe']");
    runner.clearTransferState();
    runner.enqueue(Paths.get("src/test/resources/TestUpdateRecord/input/multi-arrays.json"));
    runner.setProperty("/peoples[0..-1][./name != 'Mary Doe']/addresses[*]", "/peoples[3]/addresses[0]");
    runner.run();
    runner.assertAllFlowFilesTransferred(UpdateRecord.REL_SUCCESS, 1);
    content = new String(runner.getFlowFilesForRelationship(UpdateRecord.REL_SUCCESS).get(0).toByteArray());
    count = StringUtils.countMatches(content, "1 nifi road");
    assertEquals(13, count);
    runner.removeProperty("/peoples[0..-1][./name != 'Mary Doe']/addresses[*]");
    runner.clearTransferState();
    runner.enqueue(Paths.get("src/test/resources/TestUpdateRecord/input/multi-arrays.json"));
    runner.setProperty("/peoples[0..-1][./name != 'Mary Doe']/addresses[0,1..2]", "/peoples[3]/addresses[0]");
    runner.run();
    runner.assertAllFlowFilesTransferred(UpdateRecord.REL_SUCCESS, 1);
    expectedOutput = new String(Files.readAllBytes(Paths.get("src/test/resources/TestUpdateRecord/output/updateArrays/multi-arrays-streets.json")));
    runner.getFlowFilesForRelationship(UpdateRecord.REL_SUCCESS).get(0).assertContentEquals(expectedOutput);
    runner.removeProperty("/peoples[0..-1][./name != 'Mary Doe']/addresses[0,1..2]");
    runner.clearTransferState();
    runner.enqueue(Paths.get("src/test/resources/TestUpdateRecord/input/multi-arrays.json"));
    runner.setProperty("/peoples[0..-1][./name != 'Mary Doe']/addresses[0,1..2]/city", "newCity");
    runner.setProperty(UpdateRecord.REPLACEMENT_VALUE_STRATEGY, UpdateRecord.LITERAL_VALUES);
    runner.run();
    runner.assertAllFlowFilesTransferred(UpdateRecord.REL_SUCCESS, 1);
    content = new String(runner.getFlowFilesForRelationship(UpdateRecord.REL_SUCCESS).get(0).toByteArray());
    count = StringUtils.countMatches(content, "newCity");
    assertEquals(9, count);
    runner.removeProperty("/peoples[0..-1][./name != 'Mary Doe']/addresses[0,1..2]/city");
}
Also used : JsonTreeReader(org.apache.nifi.json.JsonTreeReader) JsonRecordSetWriter(org.apache.nifi.json.JsonRecordSetWriter) Test(org.junit.Test)

Example 3 with JsonRecordSetWriter

use of org.apache.nifi.json.JsonRecordSetWriter in project nifi by apache.

the class TestUpdateRecord method testSetAbsolutePathWithAnotherRecord.

@Test
public void testSetAbsolutePathWithAnotherRecord() throws InitializationException, IOException {
    final JsonTreeReader jsonReader = new JsonTreeReader();
    runner.addControllerService("reader", jsonReader);
    final String inputSchemaText = new String(Files.readAllBytes(Paths.get("src/test/resources/TestUpdateRecord/schema/person-with-name-and-mother.avsc")));
    final String outputSchemaText = new String(Files.readAllBytes(Paths.get("src/test/resources/TestUpdateRecord/schema/person-with-name-and-mother.avsc")));
    runner.setProperty(jsonReader, SchemaAccessUtils.SCHEMA_ACCESS_STRATEGY, SchemaAccessUtils.SCHEMA_TEXT_PROPERTY);
    runner.setProperty(jsonReader, SchemaAccessUtils.SCHEMA_TEXT, inputSchemaText);
    runner.enableControllerService(jsonReader);
    final JsonRecordSetWriter jsonWriter = new JsonRecordSetWriter();
    runner.addControllerService("writer", jsonWriter);
    runner.setProperty(jsonWriter, SchemaAccessUtils.SCHEMA_ACCESS_STRATEGY, SchemaAccessUtils.SCHEMA_TEXT_PROPERTY);
    runner.setProperty(jsonWriter, SchemaAccessUtils.SCHEMA_TEXT, outputSchemaText);
    runner.setProperty(jsonWriter, "Pretty Print JSON", "true");
    runner.setProperty(jsonWriter, "Schema Write Strategy", "full-schema-attribute");
    runner.enableControllerService(jsonWriter);
    runner.enqueue(Paths.get("src/test/resources/TestUpdateRecord/input/person.json"));
    runner.setProperty("/name", "/mother");
    runner.setProperty(UpdateRecord.REPLACEMENT_VALUE_STRATEGY, UpdateRecord.RECORD_PATH_VALUES);
    runner.run();
    runner.assertAllFlowFilesTransferred(UpdateRecord.REL_SUCCESS, 1);
    final String expectedOutput = new String(Files.readAllBytes(Paths.get("src/test/resources/TestUpdateRecord/output/name-and-mother-same.json")));
    runner.getFlowFilesForRelationship(UpdateRecord.REL_SUCCESS).get(0).assertContentEquals(expectedOutput);
}
Also used : JsonTreeReader(org.apache.nifi.json.JsonTreeReader) JsonRecordSetWriter(org.apache.nifi.json.JsonRecordSetWriter) Test(org.junit.Test)

Example 4 with JsonRecordSetWriter

use of org.apache.nifi.json.JsonRecordSetWriter in project nifi by apache.

the class TestUpdateRecord method testSetRootPathAbsoluteWithSingleValue.

@Test
public void testSetRootPathAbsoluteWithSingleValue() throws InitializationException, IOException {
    final JsonTreeReader jsonReader = new JsonTreeReader();
    runner.addControllerService("reader", jsonReader);
    final String inputSchemaText = new String(Files.readAllBytes(Paths.get("src/test/resources/TestUpdateRecord/schema/person-with-name-record.avsc")));
    final String outputSchemaText = new String(Files.readAllBytes(Paths.get("src/test/resources/TestUpdateRecord/schema/name-fields-only.avsc")));
    runner.setProperty(jsonReader, SchemaAccessUtils.SCHEMA_ACCESS_STRATEGY, SchemaAccessUtils.SCHEMA_TEXT_PROPERTY);
    runner.setProperty(jsonReader, SchemaAccessUtils.SCHEMA_TEXT, inputSchemaText);
    runner.enableControllerService(jsonReader);
    final JsonRecordSetWriter jsonWriter = new JsonRecordSetWriter();
    runner.addControllerService("writer", jsonWriter);
    runner.setProperty(jsonWriter, SchemaAccessUtils.SCHEMA_ACCESS_STRATEGY, SchemaAccessUtils.SCHEMA_TEXT_PROPERTY);
    runner.setProperty(jsonWriter, SchemaAccessUtils.SCHEMA_TEXT, outputSchemaText);
    runner.setProperty(jsonWriter, "Pretty Print JSON", "true");
    runner.setProperty(jsonWriter, "Schema Write Strategy", "full-schema-attribute");
    runner.enableControllerService(jsonWriter);
    runner.enqueue(Paths.get("src/test/resources/TestUpdateRecord/input/person.json"));
    runner.setProperty("/", "/name");
    runner.setProperty(UpdateRecord.REPLACEMENT_VALUE_STRATEGY, UpdateRecord.RECORD_PATH_VALUES);
    runner.run();
    runner.assertAllFlowFilesTransferred(UpdateRecord.REL_SUCCESS, 1);
    final String expectedOutput = new String(Files.readAllBytes(Paths.get("src/test/resources/TestUpdateRecord/output/name-fields-only.json")));
    runner.getFlowFilesForRelationship(UpdateRecord.REL_SUCCESS).get(0).assertContentEquals(expectedOutput);
}
Also used : JsonTreeReader(org.apache.nifi.json.JsonTreeReader) JsonRecordSetWriter(org.apache.nifi.json.JsonRecordSetWriter) Test(org.junit.Test)

Example 5 with JsonRecordSetWriter

use of org.apache.nifi.json.JsonRecordSetWriter in project nifi by apache.

the class TestGetSolr method testRecordWriter.

@Test
public void testRecordWriter() throws IOException, SolrServerException, InitializationException {
    final org.apache.nifi.processors.solr.TestGetSolr.TestableProcessor proc = new org.apache.nifi.processors.solr.TestGetSolr.TestableProcessor(solrClient);
    TestRunner runner = createDefaultTestRunner(proc);
    runner.setProperty(GetSolr.RETURN_TYPE, GetSolr.MODE_REC.getValue());
    runner.setProperty(GetSolr.RETURN_FIELDS, "id,created,integer_single");
    runner.setProperty(GetSolr.BATCH_SIZE, "10");
    final String outputSchemaText = new String(Files.readAllBytes(Paths.get("src/test/resources/test-schema.avsc")));
    final JsonRecordSetWriter jsonWriter = new JsonRecordSetWriter();
    runner.addControllerService("writer", jsonWriter);
    runner.setProperty(jsonWriter, SchemaAccessUtils.SCHEMA_ACCESS_STRATEGY, SchemaAccessUtils.SCHEMA_TEXT_PROPERTY);
    runner.setProperty(jsonWriter, SchemaAccessUtils.SCHEMA_TEXT, outputSchemaText);
    runner.setProperty(jsonWriter, "Pretty Print JSON", "true");
    runner.setProperty(jsonWriter, "Schema Write Strategy", "full-schema-attribute");
    runner.enableControllerService(jsonWriter);
    runner.setProperty(GetSolr.RECORD_WRITER, "writer");
    runner.run(1, true, true);
    runner.assertQueueEmpty();
    runner.assertAllFlowFilesTransferred(GetSolr.REL_SUCCESS, 1);
    runner.assertAllFlowFilesContainAttribute(CoreAttributes.MIME_TYPE.key());
    // Check for valid json
    JsonReader reader = new JsonReader(new InputStreamReader(new ByteArrayInputStream(runner.getContentAsByteArray(runner.getFlowFilesForRelationship(GetSolr.REL_SUCCESS).get(0)))));
    reader.beginArray();
    int controlScore = 0;
    while (reader.hasNext()) {
        reader.beginObject();
        while (reader.hasNext()) {
            if (reader.nextName().equals("integer_single"))
                controlScore += reader.nextInt();
            else
                reader.skipValue();
        }
        reader.endObject();
    }
    assertEquals(controlScore, 45);
}
Also used : InputStreamReader(java.io.InputStreamReader) TestRunner(org.apache.nifi.util.TestRunner) ByteArrayInputStream(java.io.ByteArrayInputStream) JsonReader(com.google.gson.stream.JsonReader) JsonRecordSetWriter(org.apache.nifi.json.JsonRecordSetWriter) Test(org.junit.Test)

Aggregations

JsonRecordSetWriter (org.apache.nifi.json.JsonRecordSetWriter)13 Test (org.junit.Test)13 JsonTreeReader (org.apache.nifi.json.JsonTreeReader)12 JsonReader (com.google.gson.stream.JsonReader)1 ByteArrayInputStream (java.io.ByteArrayInputStream)1 InputStreamReader (java.io.InputStreamReader)1 TestRunner (org.apache.nifi.util.TestRunner)1
About Me
UseOf

Java Tutorials :

Simple Java RabbitMQ Example with Spring
Simple Springboot JPA Eclipeslink Example
Simple SpringBoot JPA Hibernate Example
Jackson JSON @JsonIgnore and @JsonIgnoreProperties Examples
Java Infinite recursion (StackOverflowError) Jackson solutions
Simple Java Jackson Serialize Objects to JSON and convert them back To Object
ElasticSearch 5 - Upload files using Java API in binary field Example
Java JAXB marshall unmarshall Examples from String and Stream
Java 8 forEach List and Map examples
ElasticSearch 5 Java API SearchRequestBuilder examples
Java ElasticSearch 5 examples with node index and mapping - running local
Convert String to int or Integer Java 8
Java 9 in action - JShell - Ubuntu
Java - removing invalid characters from a file name
Hello world!? or Hello Tutorial