Search in sources :

Example 1 with MinioStorage

use of io.trino.plugin.exchange.containers.MinioStorage in project trino by trinodb.

the class TestHiveFaultTolerantExecutionConnectorTest method createQueryRunner.

@Override
protected QueryRunner createQueryRunner() throws Exception {
    this.minioStorage = new MinioStorage("test-exchange-spooling-" + randomTableSuffix());
    minioStorage.start();
    return BaseHiveConnectorTest.createHiveQueryRunner(getExtraProperties(), getExchangeManagerProperties(minioStorage));
}
Also used : MinioStorage(io.trino.plugin.exchange.containers.MinioStorage)

Example 2 with MinioStorage

use of io.trino.plugin.exchange.containers.MinioStorage in project trino by trinodb.

the class TestHiveFaultTolerantExecutionWindowQueries method createQueryRunner.

@Override
protected QueryRunner createQueryRunner(Map<String, String> extraProperties) throws Exception {
    this.minioStorage = new MinioStorage("test-exchange-spooling-" + randomTableSuffix());
    minioStorage.start();
    return HiveQueryRunner.builder().setExtraProperties(extraProperties).setAdditionalSetup(runner -> {
        runner.installPlugin(new FileSystemExchangePlugin());
        runner.loadExchangeManager("filesystem", getExchangeManagerProperties(minioStorage));
    }).setInitialTables(getTables()).build();
}
Also used : MinioStorage(io.trino.plugin.exchange.containers.MinioStorage) FileSystemExchangePlugin(io.trino.plugin.exchange.FileSystemExchangePlugin)

Example 3 with MinioStorage

use of io.trino.plugin.exchange.containers.MinioStorage in project trino by trinodb.

the class TestHiveQueryFailureRecoveryTest method createQueryRunner.

@Override
protected QueryRunner createQueryRunner(List<TpchTable<?>> requiredTpchTables, Map<String, String> configProperties, Map<String, String> coordinatorProperties) throws Exception {
    // randomizing bucket name to ensure cached TrinoS3FileSystem objects are not reused
    String bucketName = "test-hive-insert-overwrite-" + randomTableSuffix();
    this.dockerizedS3DataLake = new HiveMinioDataLake(bucketName, ImmutableMap.of(), HiveHadoop.DEFAULT_IMAGE);
    dockerizedS3DataLake.start();
    this.minioStorage = new MinioStorage("test-exchange-spooling-" + randomTableSuffix());
    minioStorage.start();
    return S3HiveQueryRunner.builder(dockerizedS3DataLake).setInitialTables(requiredTpchTables).setExtraProperties(configProperties).setCoordinatorProperties(coordinatorProperties).setAdditionalSetup(runner -> {
        runner.installPlugin(new FileSystemExchangePlugin());
        runner.loadExchangeManager("filesystem", getExchangeManagerProperties(minioStorage));
    }).setHiveProperties(ImmutableMap.<String, String>builder().put("hive.s3.streaming.enabled", "false").buildOrThrow()).build();
}
Also used : MinioStorage(io.trino.plugin.exchange.containers.MinioStorage) FileSystemExchangePlugin(io.trino.plugin.exchange.FileSystemExchangePlugin) HiveMinioDataLake(io.trino.plugin.hive.containers.HiveMinioDataLake)

Example 4 with MinioStorage

use of io.trino.plugin.exchange.containers.MinioStorage in project trino by trinodb.

the class TestIcebergQueryFailureRecoveryTest method createQueryRunner.

@Override
protected QueryRunner createQueryRunner(List<TpchTable<?>> requiredTpchTables, Map<String, String> configProperties, Map<String, String> coordinatorProperties) throws Exception {
    this.minioStorage = new MinioStorage("test-exchange-spooling-" + randomTableSuffix());
    minioStorage.start();
    return IcebergQueryRunner.builder().setInitialTables(requiredTpchTables).setCoordinatorProperties(coordinatorProperties).setExtraProperties(configProperties).setAdditionalSetup(runner -> {
        runner.installPlugin(new FileSystemExchangePlugin());
        runner.loadExchangeManager("filesystem", getExchangeManagerProperties(minioStorage));
    }).build();
}
Also used : TpchTable(io.trino.tpch.TpchTable) AfterClass(org.testng.annotations.AfterClass) List(java.util.List) MinioStorage(io.trino.plugin.exchange.containers.MinioStorage) FileSystemExchangePlugin(io.trino.plugin.exchange.FileSystemExchangePlugin) QueryRunner(io.trino.testing.QueryRunner) Map(java.util.Map) RetryPolicy(io.trino.operator.RetryPolicy) TestTable.randomTableSuffix(io.trino.testing.sql.TestTable.randomTableSuffix) MinioStorage.getExchangeManagerProperties(io.trino.plugin.exchange.containers.MinioStorage.getExchangeManagerProperties) MinioStorage(io.trino.plugin.exchange.containers.MinioStorage) FileSystemExchangePlugin(io.trino.plugin.exchange.FileSystemExchangePlugin)

Example 5 with MinioStorage

use of io.trino.plugin.exchange.containers.MinioStorage in project trino by trinodb.

the class TestHiveTaskFailureRecoveryTest method createQueryRunner.

@Override
protected QueryRunner createQueryRunner(List<TpchTable<?>> requiredTpchTables, Map<String, String> configProperties, Map<String, String> coordinatorProperties) throws Exception {
    // randomizing bucket name to ensure cached TrinoS3FileSystem objects are not reused
    String bucketName = "test-hive-insert-overwrite-" + randomTableSuffix();
    this.dockerizedS3DataLake = new HiveMinioDataLake(bucketName, ImmutableMap.of(), HiveHadoop.DEFAULT_IMAGE);
    dockerizedS3DataLake.start();
    this.minioStorage = new MinioStorage("test-exchange-spooling-" + randomTableSuffix());
    minioStorage.start();
    return S3HiveQueryRunner.builder(dockerizedS3DataLake).setInitialTables(requiredTpchTables).setExtraProperties(ImmutableMap.<String, String>builder().putAll(configProperties).put("enable-dynamic-filtering", "false").buildOrThrow()).setCoordinatorProperties(coordinatorProperties).setAdditionalSetup(runner -> {
        runner.installPlugin(new FileSystemExchangePlugin());
        runner.loadExchangeManager("filesystem", getExchangeManagerProperties(minioStorage));
    }).setHiveProperties(ImmutableMap.<String, String>builder().put("hive.s3.streaming.enabled", "false").buildOrThrow()).build();
}
Also used : MinioStorage(io.trino.plugin.exchange.containers.MinioStorage) FileSystemExchangePlugin(io.trino.plugin.exchange.FileSystemExchangePlugin) HiveMinioDataLake(io.trino.plugin.hive.containers.HiveMinioDataLake)

Aggregations

MinioStorage (io.trino.plugin.exchange.containers.MinioStorage)10 FileSystemExchangePlugin (io.trino.plugin.exchange.FileSystemExchangePlugin)8 RetryPolicy (io.trino.operator.RetryPolicy)2 MinioStorage.getExchangeManagerProperties (io.trino.plugin.exchange.containers.MinioStorage.getExchangeManagerProperties)2 HiveMinioDataLake (io.trino.plugin.hive.containers.HiveMinioDataLake)2 QueryRunner (io.trino.testing.QueryRunner)2 TestTable.randomTableSuffix (io.trino.testing.sql.TestTable.randomTableSuffix)2 TpchTable (io.trino.tpch.TpchTable)2 List (java.util.List)2 Map (java.util.Map)2 AfterClass (org.testng.annotations.AfterClass)2 ImmutableMap (com.google.common.collect.ImmutableMap)1 FileSystemExchangeManagerFactory (io.trino.plugin.exchange.FileSystemExchangeManagerFactory)1 Assertions.assertThatThrownBy (org.assertj.core.api.Assertions.assertThatThrownBy)1