Search in sources :

Example 16 with FileBasedLock

use of io.prestosql.spi.filesystem.FileBasedLock in project hetu-core by openlookeng.

the class FileBasedSeedStoreOnYarn method remove.

@Override
public Set<Seed> remove(Collection<Seed> seeds) throws IOException {
    LOG.debug("FileBasedOnYarnSeedStore::remove() invoked.");
    Path lockPath = seedFileFolder.resolve(name);
    checkArgument(!lockPath.toString().contains("../"), "Lock path must be absolute and at user workspace " + SecurePathWhiteList.getSecurePathWhiteList().toString());
    checkArgument(SecurePathWhiteList.isSecurePath(lockPath.toString()), "Lock path must be at user workspace " + SecurePathWhiteList.getSecurePathWhiteList().toString());
    Lock lock = new FileBasedLock(fs, lockPath);
    try {
        lock.lock();
        Set<Seed> outputs = new HashSet<>();
        if (fs.exists(seedFilePath)) {
            String json = loadFromFile(seedFilePath);
            List<FileBasedSeedOnYarn> existingSeeds = LIST_FILE_BASED_SEED_CODEC.fromJson(json);
            Set<FileBasedSeedOnYarn> latestSeeds = new HashSet<>(existingSeeds);
            latestSeeds.removeAll(seeds.stream().filter(s -> (s instanceof FileBasedSeedOnYarn)).map(s -> (FileBasedSeedOnYarn) s).collect(Collectors.toList()));
            String output = LIST_FILE_BASED_SEED_CODEC.toJson(ImmutableList.copyOf(latestSeeds));
            writeToFile(seedFilePath, output, true);
            outputs.addAll(latestSeeds);
        }
        return outputs;
    } catch (UncheckedIOException e) {
        throw new IOException(e);
    } finally {
        lock.unlock();
    }
}
Also used : Path(java.nio.file.Path) Logger(io.airlift.log.Logger) HetuFileSystemClient(io.prestosql.spi.filesystem.HetuFileSystemClient) FileBasedLock(io.prestosql.spi.filesystem.FileBasedLock) HashSet(java.util.HashSet) Preconditions.checkArgument(com.google.common.base.Preconditions.checkArgument) ImmutableList(com.google.common.collect.ImmutableList) Map(java.util.Map) Path(java.nio.file.Path) OutputStream(java.io.OutputStream) SeedStore(io.prestosql.spi.seedstore.SeedStore) Collection(java.util.Collection) Seed(io.prestosql.spi.seedstore.Seed) Set(java.util.Set) IOException(java.io.IOException) InputStreamReader(java.io.InputStreamReader) Collectors(java.util.stream.Collectors) StandardCharsets(java.nio.charset.StandardCharsets) UncheckedIOException(java.io.UncheckedIOException) List(java.util.List) Lock(java.util.concurrent.locks.Lock) Paths(java.nio.file.Paths) JsonCodec.listJsonCodec(io.airlift.json.JsonCodec.listJsonCodec) BufferedReader(java.io.BufferedReader) CREATE_NEW(java.nio.file.StandardOpenOption.CREATE_NEW) SecurePathWhiteList(io.hetu.core.common.util.SecurePathWhiteList) JsonCodec(io.airlift.json.JsonCodec) FileBasedLock(io.prestosql.spi.filesystem.FileBasedLock) UncheckedIOException(java.io.UncheckedIOException) IOException(java.io.IOException) UncheckedIOException(java.io.UncheckedIOException) FileBasedLock(io.prestosql.spi.filesystem.FileBasedLock) Lock(java.util.concurrent.locks.Lock) Seed(io.prestosql.spi.seedstore.Seed) HashSet(java.util.HashSet)

Example 17 with FileBasedLock

use of io.prestosql.spi.filesystem.FileBasedLock in project hetu-core by openlookeng.

the class HetuFsMetastore method runTransaction.

public synchronized void runTransaction(HetuFsMetadataHandler<PrestoException> handler) {
    try {
        Lock transactionLock;
        transactionLock = new FileBasedLock(client, Paths.get(metadataPath));
        transactionLock.lock();
        try {
            handler.handle();
        } finally {
            transactionLock.unlock();
        }
    } catch (IOException e) {
        throw new PrestoException(HETU_METASTORE_CODE, "Create transaction lock failed", e);
    }
}
Also used : FileBasedLock(io.prestosql.spi.filesystem.FileBasedLock) PrestoException(io.prestosql.spi.PrestoException) IOException(java.io.IOException) FileBasedLock(io.prestosql.spi.filesystem.FileBasedLock) Lock(java.util.concurrent.locks.Lock)

Aggregations

FileBasedLock (io.prestosql.spi.filesystem.FileBasedLock)17 Path (java.nio.file.Path)13 AfterTest (org.testng.annotations.AfterTest)10 Test (org.testng.annotations.Test)10 Lock (java.util.concurrent.locks.Lock)7 IOException (java.io.IOException)6 OutputStream (java.io.OutputStream)6 UncheckedIOException (java.io.UncheckedIOException)5 HashSet (java.util.HashSet)5 BeforeTest (org.testng.annotations.BeforeTest)5 Seed (io.prestosql.spi.seedstore.Seed)4 Preconditions.checkArgument (com.google.common.base.Preconditions.checkArgument)3 ImmutableList (com.google.common.collect.ImmutableList)3 Logger (io.airlift.log.Logger)3 SecurePathWhiteList (io.hetu.core.common.util.SecurePathWhiteList)3 HetuFileSystemClient (io.prestosql.spi.filesystem.HetuFileSystemClient)3 Paths (java.nio.file.Paths)3 List (java.util.List)3 Map (java.util.Map)3 Collectors (java.util.stream.Collectors)3