ElasticSearch 5 - Upload files using Java API in binary field Example

Today We will try to upload files to elasticsearch to see if is hard or not ,im writing this example while i'm testing elasticearch.

Now I will use PreBuiltTransportClient instead of starting the node in my local computer (i'm lucky i have my own cluster:) )

        Settings settings = Settings.builder()
                .put("cluster.name", "production")
//                .put("client.transport.sniff", true)
                .put("client.transport.ping_timeout", 10, TimeUnit.SECONDS)
                .build();
        Client client= new PreBuiltTransportClient(settings)
                .addTransportAddress(new InetSocketTransportAddress(Inet4Address.getByName("my.own.clutster".change this), 9300));

i commented the line  .put("client.transport.sniff", true) because i moved data between nodes and somehow is a bug in elasticsearch and I got Exception in thread "main" NoNodeAvailableException[None of the configured nodes are available:

whitout this line everything works perfect for me , but you should use it if you are not doing crazy things with the server/cluster.

 String indexName = "binary_index2";
        String documentsType = "binary_doc";
        ActionFuture<IndicesExistsResponse> indicesExistsResponseAction = client.admin().indices().exists(new IndicesExistsRequest(indexName));
        IndicesExistsResponse indicesExistsResponse = indicesExistsResponseAction.actionGet();
        if (!indicesExistsResponse.isExists())
        {
            ActionFuture<CreateIndexResponse> createIndexResponseAction = client.admin().indices().create(new CreateIndexRequest(indexName));
            CreateIndexResponse createIndexResponse = createIndexResponseAction.get();
            if (!createIndexResponse.isAcknowledged()) {
                throw new IllegalStateException("Failed to create index " + indexName);
            }
            putMapping(indexName,client
                    , documentsType
                    , "{\"properties\":{\"name\":{\"type\":\"string\"},\"data\":{\"type\":\"binary\",\"store\":\"false\"}}}");

        }

the index should be created now , with property data as binary.

next we have to send files to server for this i created a method to create the document :

    private static XContentBuilder createDocumentXContent(String name, String data) throws IOException {
        return XContentFactory.jsonBuilder()
                .startObject()
                .field("name", name)
                .field("data", data)
                .endObject();
    }

Finally we can send files to the server :

        Files.walk(new File("/path/to/files". you shoudl change this).toPath())

                .map(Path::toFile)
                .forEach(file -> {
                    if (!file.isFile()) return;
                    System.out.println("encoding :" + file.getAbsoluteFile());
                    try {
                        String base64 = Base64.getEncoder().encodeToString(Files.readAllBytes(file.toPath()));
                        if (base64.length()>0)
                        {
                            XContentBuilder document = createDocumentXContent(file.getName(), base64);
                            IndexResponse response = client.prepareIndex(indexName, documentsType)
                                    .setSource(document).get();
                        }
                        System.out.println(base64);

                    } catch (IOException e) {
                        e.printStackTrace();
                    }
                });
    }

We aresearching all files from /path/to/files , encode them and sending them to ElasticSearch

just now I saw that putMapping method is missing :) , here it is :

    private static void putMapping(String index, Client client, String type, String mappingSource) throws NodeValidationException {
        PutMappingResponse mappingResponse = client.admin().indices().preparePutMapping(index)
                .setType(type)
                .setSource(mappingSource)
                .get();
        if (!mappingResponse.isAcknowledged()) {
            throw new IllegalStateException("Failed to create type " + index);
        }
    }

good luck !

Git sources : Download Sources