Examples with HetuLocalFileSystemClient io.hetu.core.filesystem.HetuLocalFileSystemClient used on opensource projects

Example 1 with HetuLocalFileSystemClient

use of io.hetu.core.filesystem.HetuLocalFileSystemClient in project hetu-core by openlookeng.

the class TestOrderByOperator method setUp.

@BeforeMethod
public void setUp() throws IOException {
    executor = newCachedThreadPool(daemonThreadsNamed("test-executor-%s"));
    scheduledExecutor = newScheduledThreadPool(2, daemonThreadsNamed("test-scheduledExecutor-%s"));
    spillerFactory = new DummySpillerFactory();
    when(fileSystemClientManager.getFileSystemClient(any(Path.class))).thenReturn(new HetuLocalFileSystemClient(new LocalConfig(new Properties()), Paths.get("/tmp/hetu/snapshot/")));
}

Example 2 with HetuLocalFileSystemClient

use of io.hetu.core.filesystem.HetuLocalFileSystemClient in project hetu-core by openlookeng.

the class TestWindowOperator method setUp.

@BeforeMethod
public void setUp() throws IOException {
    executor = newCachedThreadPool(daemonThreadsNamed("test-executor-%s"));
    scheduledExecutor = newScheduledThreadPool(2, daemonThreadsNamed("test-scheduledExecutor-%s"));
    spillerFactory = new DummySpillerFactory();
    when(fileSystemClientManager.getFileSystemClient(any(Path.class))).thenReturn(new HetuLocalFileSystemClient(new LocalConfig(new Properties()), Paths.get("/tmp/hetu/snapshot/")));
}

Example 3 with HetuLocalFileSystemClient

use of io.hetu.core.filesystem.HetuLocalFileSystemClient in project hetu-core by openlookeng.

the class TestFileSingleStreamSpillerFactory method setUp.

@BeforeMethod
public void setUp() throws IOException {
    closer = Closer.create();
    executor = MoreExecutors.listeningDecorator(Executors.newCachedThreadPool());
    closer.register(() -> executor.shutdownNow());
    spillPath1 = createTempDirectory(getClass().getName()).toFile();
    closer.register(() -> deleteRecursively(spillPath1.toPath(), ALLOW_INSECURE));
    spillPath2 = createTempDirectory(getClass().getName()).toFile();
    closer.register(() -> deleteRecursively(spillPath2.toPath(), ALLOW_INSECURE));
    when(fileSystemClientManager.getFileSystemClient(spillPath1.toPath())).thenReturn(new HetuLocalFileSystemClient(new LocalConfig(new Properties()), Paths.get(spillPath1.getCanonicalPath())));
    when(fileSystemClientManager.getFileSystemClient(spillPath2.toPath())).thenReturn(new HetuLocalFileSystemClient(new LocalConfig(new Properties()), Paths.get(spillPath2.getCanonicalPath())));
    when(fileSystemClientManager.getFileSystemClient("hdfs", spillPath1.toPath())).thenReturn(getLocalHdfs(spillPath1.toString()));
    when(fileSystemClientManager.getFileSystemClient("hdfs", spillPath2.toPath())).thenReturn(getLocalHdfs(spillPath2.toString()));
}

Example 4 with HetuLocalFileSystemClient

use of io.hetu.core.filesystem.HetuLocalFileSystemClient in project hetu-core by openlookeng.

the class TestGenericPartitioningSpiller method setUp.

@BeforeClass
public void setUp() throws Exception {
    tempDirectory = createTempDirectory(getClass().getSimpleName());
    when(fileSystemClientManager.getFileSystemClient(tempDirectory)).thenReturn(new HetuLocalFileSystemClient(new LocalConfig(new Properties()), tempDirectory));
    FeaturesConfig featuresConfig = new FeaturesConfig();
    featuresConfig.setSpillerSpillPaths(tempDirectory.toString());
    featuresConfig.setSpillerThreads(8);
    featuresConfig.setSpillMaxUsedSpaceThreshold(1.0);
    SingleStreamSpillerFactory singleStreamSpillerFactory = new FileSingleStreamSpillerFactory(createTestMetadataManager(), new SpillerStats(), featuresConfig, new NodeSpillConfig(), fileSystemClientManager);
    factory = new GenericPartitioningSpillerFactory(singleStreamSpillerFactory);
    scheduledExecutor = newSingleThreadScheduledExecutor();
}

Example 5 with HetuLocalFileSystemClient

use of io.hetu.core.filesystem.HetuLocalFileSystemClient in project hetu-core by openlookeng.

the class TestTaskSnapshotManager method setup.

@BeforeMethod
public void setup() throws Exception {
    // Set up snapshot config
    snapshotConfig = new SnapshotConfig();
    // Set up mock file system client manager
    fileSystemClientManager = mock(FileSystemClientManager.class);
    when(fileSystemClientManager.getFileSystemClient(any(Path.class))).thenReturn(new HetuLocalFileSystemClient(new LocalConfig(new Properties()), Paths.get(SNAPSHOT_FILE_SYSTEM_DIR)));
    snapshotUtils = new SnapshotUtils(fileSystemClientManager, snapshotConfig, new InMemoryNodeManager());
    snapshotUtils.rootPath = SNAPSHOT_FILE_SYSTEM_DIR;
    snapshotUtils.initialize();
}