Search in sources :

Example 26 with S3FileSystem

use of org.apache.hop.vfs.s3.s3.vfs.S3FileSystem in project hop by apache.

the class S3FileSystemTest method getTestInstance.

public S3FileSystem getTestInstance() throws Exception {
    FileName rootName = mock(FileName.class);
    FileSystemOptions fileSystemOptions = new FileSystemOptions();
    return new S3FileSystem(rootName, fileSystemOptions);
}
Also used : S3FileName(org.apache.hop.vfs.s3.s3.vfs.S3FileName) S3FileSystem(org.apache.hop.vfs.s3.s3.vfs.S3FileSystem)

Example 27 with S3FileSystem

use of org.apache.hop.vfs.s3.s3.vfs.S3FileSystem in project hop by apache.

the class S3FileSystemTest method getPartSize.

@Test
public void getPartSize() throws Exception {
    S3FileSystem s3FileSystem = getTestInstance();
    s3FileSystem.storageUnitConverter = new StorageUnitConverter();
    S3HopProperty s3HopProperty = mock(S3HopProperty.class);
    when(s3HopProperty.getPartSize()).thenReturn("6MB");
    s3FileSystem.s3HopProperty = s3HopProperty;
    // TEST 1: Below max
    assertEquals(6 * 1024 * 1024, s3FileSystem.getPartSize());
    // TEst 2: above max
    when(s3HopProperty.getPartSize()).thenReturn("600GB");
    assertEquals(Integer.MAX_VALUE, s3FileSystem.getPartSize());
}
Also used : StorageUnitConverter(org.apache.hop.core.util.StorageUnitConverter) S3HopProperty(org.apache.hop.vfs.s3.s3common.S3HopProperty) S3FileSystem(org.apache.hop.vfs.s3.s3.vfs.S3FileSystem) Test(org.junit.Test)

Example 28 with S3FileSystem

use of org.apache.hop.vfs.s3.s3.vfs.S3FileSystem in project hop by apache.

the class S3FileSystemTest method testConvertToInt.

@Test
public void testConvertToInt() throws Exception {
    S3FileSystem s3FileSystem = getTestInstance();
    // TEST 1: below int max
    assertEquals(10, s3FileSystem.convertToInt(10L));
    // TEST 2: at int max
    assertEquals(Integer.MAX_VALUE, s3FileSystem.convertToInt((long) Integer.MAX_VALUE));
    // TEST 3: above int max
    assertEquals(Integer.MAX_VALUE, s3FileSystem.convertToInt(5L * 1024L * 1024L * 1024L));
}
Also used : S3FileSystem(org.apache.hop.vfs.s3.s3.vfs.S3FileSystem) Test(org.junit.Test)

Aggregations

Test (org.junit.Test)25 S3TestUtils.buildMockedS3FileSystem (org.apache.beam.sdk.io.aws2.s3.S3TestUtils.buildMockedS3FileSystem)20 HeadObjectResponse (software.amazon.awssdk.services.s3.model.HeadObjectResponse)9 MatchResult (org.apache.beam.sdk.io.fs.MatchResult)8 S3FileSystem (org.apache.hop.vfs.s3.s3.vfs.S3FileSystem)8 SdkServiceException (software.amazon.awssdk.core.exception.SdkServiceException)5 StorageUnitConverter (org.apache.hop.core.util.StorageUnitConverter)4 FileNotFoundException (java.io.FileNotFoundException)3 IOException (java.io.IOException)3 ArrayList (java.util.ArrayList)3 S3FileName (org.apache.hop.vfs.s3.s3.vfs.S3FileName)3 HeadObjectRequest (software.amazon.awssdk.services.s3.model.HeadObjectRequest)3 ListObjectsV2Response (software.amazon.awssdk.services.s3.model.ListObjectsV2Response)3 S3Object (software.amazon.awssdk.services.s3.model.S3Object)3 URL (java.net.URL)2 S3HopProperty (org.apache.hop.vfs.s3.s3common.S3HopProperty)2 Before (org.junit.Before)2 DeleteObjectsRequest (software.amazon.awssdk.services.s3.model.DeleteObjectsRequest)2 ListObjectsV2Request (software.amazon.awssdk.services.s3.model.ListObjectsV2Request)2 AmazonS3 (com.amazonaws.services.s3.AmazonS3)1