Search in sources :

Example 1 with FileInput

use of io.crate.execution.engine.collect.files.FileInput in project crate by crate.

the class S3FileReadingCollectorTest method createBatchIterator.

private BatchIterator<Row> createBatchIterator(Collection<String> fileUris, String compression, final S3ObjectInputStream s3InputStream, boolean collectSourceUriFailure) {
    InputFactory.Context<LineCollectorExpression<?>> ctx = inputFactory.ctxForRefs(txnCtx, FileLineReferenceResolver::getImplementation);
    List<Input<?>> inputs = new ArrayList<>(2);
    Reference raw = createReference(SourceLineExpression.COLUMN_NAME, DataTypes.STRING);
    inputs.add(ctx.add(raw));
    if (collectSourceUriFailure) {
        Reference sourceUriFailure = createReference(SourceUriFailureExpression.COLUMN_NAME, DataTypes.STRING);
        // noinspection unchecked
        sourceUriFailureInput = (Input<String>) ctx.add(sourceUriFailure);
        inputs.add(sourceUriFailureInput);
    }
    return FileReadingIterator.newInstance(fileUris, inputs, ctx.expressions(), compression, Map.of(S3FileInputFactory.NAME, new FileInputFactory() {

        @Override
        public FileInput create(URI uri, Settings withClauseOptions) throws IOException {
            return new S3FileInput(new S3ClientHelper() {

                @Override
                protected AmazonS3 initClient(String accessKey, String secretKey, String endpoint, String protocol) throws IOException {
                    AmazonS3 client = mock(AmazonS3Client.class);
                    ObjectListing objectListing = mock(ObjectListing.class);
                    S3ObjectSummary summary = mock(S3ObjectSummary.class);
                    S3Object s3Object = mock(S3Object.class);
                    when(client.listObjects(anyString(), anyString())).thenReturn(objectListing);
                    when(objectListing.getObjectSummaries()).thenReturn(Collections.singletonList(summary));
                    when(summary.getKey()).thenReturn("foo");
                    when(client.getObject("fakebucket", "foo")).thenReturn(s3Object);
                    when(s3Object.getObjectContent()).thenReturn(s3InputStream);
                    when(client.listNextBatchOfObjects(any(ObjectListing.class))).thenReturn(objectListing);
                    when(objectListing.isTruncated()).thenReturn(false);
                    return client;
                }
            }, uri, "https");
        }
    }), false, 1, 0, CopyFromParserProperties.DEFAULT, FileUriCollectPhase.InputFormat.JSON, Settings.EMPTY);
}
Also used : FileInputFactory(io.crate.execution.engine.collect.files.FileInputFactory) InputFactory(io.crate.expression.InputFactory) AmazonS3(com.amazonaws.services.s3.AmazonS3) LineCollectorExpression(io.crate.execution.engine.collect.files.LineCollectorExpression) TestingHelpers.createReference(io.crate.testing.TestingHelpers.createReference) Reference(io.crate.metadata.Reference) ArrayList(java.util.ArrayList) ObjectListing(com.amazonaws.services.s3.model.ObjectListing) S3ObjectSummary(com.amazonaws.services.s3.model.S3ObjectSummary) ArgumentMatchers.anyString(org.mockito.ArgumentMatchers.anyString) IOException(java.io.IOException) URI(java.net.URI) Input(io.crate.data.Input) FileInput(io.crate.execution.engine.collect.files.FileInput) AmazonS3Client(com.amazonaws.services.s3.AmazonS3Client) FileLineReferenceResolver(io.crate.expression.reference.file.FileLineReferenceResolver) S3ClientHelper(io.crate.copy.s3.common.S3ClientHelper) FileInputFactory(io.crate.execution.engine.collect.files.FileInputFactory) S3Object(com.amazonaws.services.s3.model.S3Object) Settings(org.elasticsearch.common.settings.Settings)

Aggregations

AmazonS3 (com.amazonaws.services.s3.AmazonS3)1 AmazonS3Client (com.amazonaws.services.s3.AmazonS3Client)1 ObjectListing (com.amazonaws.services.s3.model.ObjectListing)1 S3Object (com.amazonaws.services.s3.model.S3Object)1 S3ObjectSummary (com.amazonaws.services.s3.model.S3ObjectSummary)1 S3ClientHelper (io.crate.copy.s3.common.S3ClientHelper)1 Input (io.crate.data.Input)1 FileInput (io.crate.execution.engine.collect.files.FileInput)1 FileInputFactory (io.crate.execution.engine.collect.files.FileInputFactory)1 LineCollectorExpression (io.crate.execution.engine.collect.files.LineCollectorExpression)1 InputFactory (io.crate.expression.InputFactory)1 FileLineReferenceResolver (io.crate.expression.reference.file.FileLineReferenceResolver)1 Reference (io.crate.metadata.Reference)1 TestingHelpers.createReference (io.crate.testing.TestingHelpers.createReference)1 IOException (java.io.IOException)1 URI (java.net.URI)1 ArrayList (java.util.ArrayList)1 Settings (org.elasticsearch.common.settings.Settings)1 ArgumentMatchers.anyString (org.mockito.ArgumentMatchers.anyString)1