Examples with GSBlobIdentifier org.apache.flink.fs.gs.storage.GSBlobIdentifier used on opensource projects

Example 21 with GSBlobIdentifier

use of org.apache.flink.fs.gs.storage.GSBlobIdentifier in project flink by apache.

the class GSResumeRecoverableTest method data.

@Parameterized.Parameters(name = "position={0}, closed={1}, componentObjectIds={2}, temporaryBucketName={3}")
public static Collection<Object[]> data() {
    ArrayList<UUID> emptyComponentObjectIds = new ArrayList<>();
    ArrayList<UUID> populatedComponentObjectIds = new ArrayList<>();
    for (int i = 0; i < 2; i++) {
        populatedComponentObjectIds.add(UUID.randomUUID());
    }
    GSBlobIdentifier blobIdentifier = new GSBlobIdentifier("foo", "bar");
    return Arrays.asList(new Object[][] { // position=0, closed, no component ids, no explicit temporary bucket name
    { 0, true, emptyComponentObjectIds, null }, // name
    { 1024, false, emptyComponentObjectIds, null }, // name
    { 0, true, populatedComponentObjectIds, null }, // bucket name
    { 1024, false, populatedComponentObjectIds, null }, // position=0, closed, populated component ids, explicit temporary bucket name
    { 0, true, populatedComponentObjectIds, "temporary-bucket" }, // bucket name
    { 1024, false, populatedComponentObjectIds, "temporary-bucket" } });
}

Example 22 with GSBlobIdentifier

use of org.apache.flink.fs.gs.storage.GSBlobIdentifier in project flink by apache.

the class GSRecoverableWriter method open.

@Override
public RecoverableFsDataOutputStream open(Path path) throws IOException {
    LOGGER.trace("Opening output stream for path {}", path);
    Preconditions.checkNotNull(path);
    GSBlobIdentifier finalBlobIdentifier = BlobUtils.parseUri(path.toUri());
    return new GSRecoverableFsDataOutputStream(storage, options, finalBlobIdentifier);
}

Example 23 with GSBlobIdentifier

use of org.apache.flink.fs.gs.storage.GSBlobIdentifier in project flink by apache.

the class BlobUtilsTest method shouldProperlyConstructTemporaryObjectName.

@Test
public void shouldProperlyConstructTemporaryObjectName() {
    GSBlobIdentifier identifier = new GSBlobIdentifier("foo", "bar");
    UUID temporaryObjectId = UUID.fromString("f09c43e5-ea49-4537-a406-0586f8f09d47");
    String partialName = BlobUtils.getTemporaryObjectName(identifier, temporaryObjectId);
    assertEquals(".inprogress/foo/bar/f09c43e5-ea49-4537-a406-0586f8f09d47", partialName);
}

Example 24 with GSBlobIdentifier

use of org.apache.flink.fs.gs.storage.GSBlobIdentifier in project flink by apache.

the class BlobUtilsTest method shouldParseValidUri.

@Test
public void shouldParseValidUri() {
    GSBlobIdentifier blobIdentifier = BlobUtils.parseUri(URI.create("gs://bucket/foo/bar"));
    assertEquals("bucket", blobIdentifier.bucketName);
    assertEquals("foo/bar", blobIdentifier.objectName);
}

Example 25 with GSBlobIdentifier

use of org.apache.flink.fs.gs.storage.GSBlobIdentifier in project flink by apache.

the class BlobUtilsTest method shouldProperlyConstructTemporaryBlobIdentifierWithTemporaryBucket.

@Test
public void shouldProperlyConstructTemporaryBlobIdentifierWithTemporaryBucket() {
    Configuration flinkConfig = new Configuration();
    flinkConfig.set(GSFileSystemOptions.WRITER_TEMPORARY_BUCKET_NAME, "temp");
    GSFileSystemOptions options = new GSFileSystemOptions(flinkConfig);
    GSBlobIdentifier identifier = new GSBlobIdentifier("foo", "bar");
    UUID temporaryObjectId = UUID.fromString("f09c43e5-ea49-4537-a406-0586f8f09d47");
    GSBlobIdentifier temporaryBlobIdentifier = BlobUtils.getTemporaryBlobIdentifier(identifier, temporaryObjectId, options);
    assertEquals("temp", temporaryBlobIdentifier.bucketName);
    assertEquals(".inprogress/foo/bar/f09c43e5-ea49-4537-a406-0586f8f09d47", temporaryBlobIdentifier.objectName);
}