Examples with S3URI io.crate.copy.s3.common.S3URI used on opensource projects

Example 1 with S3URI

use of io.crate.copy.s3.common.S3URI in project crate by crate.

the class S3FileInput method getStream.

@Override
public InputStream getStream(URI uri) throws IOException {
    S3URI s3URI = S3URI.toS3URI(uri);
    if (client == null) {
        client = clientBuilder.client(s3URI, protocolSetting);
    }
    S3Object object = client.getObject(s3URI.bucket(), s3URI.key());
    if (object != null) {
        return object.getObjectContent();
    }
    throw new IOException("Failed to load S3 URI: " + uri.toString());
}

Example 2 with S3URI

use of io.crate.copy.s3.common.S3URI in project crate by crate.

the class S3URITest method assertValidS3URI.

private void assertValidS3URI(String toBeParsed, String accessKey, String secretKey, String host, int port, String bucketName, String key) {
    S3URI s3URI = S3URI.toS3URI(URI.create(toBeParsed));
    URI uri = s3URI.uri();
    if (accessKey != null) {
        assertNotNull(secretKey);
        assertThat(s3URI.accessKey() + ":" + s3URI.secretKey(), is(accessKey + ":" + secretKey));
        assertThat(getUserInfo(uri), is(accessKey + ":" + secretKey));
    }
    if (host != null) {
        assertThat(uri.getHost(), is(host));
        assertThat(s3URI.endpoint(), startsWith(uri.getHost() + ":"));
        assertThat(s3URI.endpoint(), endsWith(":" + uri.getPort()));
    }
    assertThat(uri.getPort(), is(port));
    if (bucketName != null) {
        assertTrue(uri.getPath().startsWith("/" + bucketName));
        assertEquals(bucketName, s3URI.bucket());
    }
    if (key != null) {
        assertTrue(uri.getPath().endsWith(key));
        assertEquals(key, s3URI.key());
    }
}

Example 3 with S3URI

use of io.crate.copy.s3.common.S3URI in project crate by crate.

the class S3URITest method testReplacePathMethod.

@Test
public void testReplacePathMethod() {
    S3URI s3URI = S3URI.toS3URI(URI.create("s3://minioadmin:minio%2Fadmin@mj.myb/localhost:9000/"));
    S3URI replacedURI = s3URI.replacePath("new.Bucket", "newKey");
    assertThat(replacedURI.uri().toString(), is("s3://minioadmin:minio%2Fadmin@/new.Bucket/newKey"));
    s3URI = S3URI.toS3URI(URI.create("s3://host:123/myb"));
    replacedURI = s3URI.replacePath("new.Bucket", "newKey");
    assertThat(replacedURI.uri().toString(), is("s3://host:123/new.Bucket/newKey"));
}

Example 4 with S3URI

use of io.crate.copy.s3.common.S3URI in project crate by crate.

the class S3FileInput method toPreGlobUri.

@VisibleForTesting
@Nullable
static S3URI toPreGlobUri(S3URI uri) {
    Matcher hasGlobMatcher = HAS_GLOBS_PATTERN.matcher(uri.toString());
    S3URI preGlobUri = null;
    if (hasGlobMatcher.matches()) {
        preGlobUri = S3URI.toS3URI(URI.create(hasGlobMatcher.group(1)));
    }
    return preGlobUri;
}