Search in sources :

Example 1 with MountPOptions

use of alluxio.grpc.MountPOptions in project alluxio by Alluxio.

the class DefaultFileSystemMaster method getUfsInfo.

@Override
public UfsInfo getUfsInfo(long mountId) {
    MountInfo info = mMountTable.getMountInfo(mountId);
    if (info == null) {
        return new UfsInfo();
    }
    MountPOptions options = info.getOptions();
    return new UfsInfo().setUri(info.getUfsUri()).setMountOptions(MountContext.mergeFrom(MountPOptions.newBuilder().putAllProperties(options.getPropertiesMap()).setReadOnly(options.getReadOnly()).setShared(options.getShared())).getOptions().build());
}
Also used : MountPOptions(alluxio.grpc.MountPOptions) UfsInfo(alluxio.wire.UfsInfo) MountInfo(alluxio.master.file.meta.options.MountInfo)

Example 2 with MountPOptions

use of alluxio.grpc.MountPOptions in project alluxio by Alluxio.

the class DefaultFileSystemMaster method getRootMountInfo.

private static MountInfo getRootMountInfo(MasterUfsManager ufsManager) {
    try (CloseableResource<UnderFileSystem> resource = ufsManager.getRoot().acquireUfsResource()) {
        boolean shared = resource.get().isObjectStorage() && ServerConfiguration.getBoolean(PropertyKey.UNDERFS_OBJECT_STORE_MOUNT_SHARED_PUBLICLY);
        boolean readonly = ServerConfiguration.getBoolean(PropertyKey.MASTER_MOUNT_TABLE_ROOT_READONLY);
        String rootUfsUri = PathUtils.normalizePath(ServerConfiguration.getString(PropertyKey.MASTER_MOUNT_TABLE_ROOT_UFS), AlluxioURI.SEPARATOR);
        Map<String, String> rootUfsConf = ServerConfiguration.getNestedProperties(PropertyKey.MASTER_MOUNT_TABLE_ROOT_OPTION).entrySet().stream().filter(entry -> entry.getValue() != null).collect(Collectors.toMap(Map.Entry::getKey, entry -> String.valueOf(entry.getValue())));
        MountPOptions mountOptions = MountContext.mergeFrom(MountPOptions.newBuilder().setShared(shared).setReadOnly(readonly).putAllProperties(rootUfsConf)).getOptions().build();
        return new MountInfo(new AlluxioURI(MountTable.ROOT), new AlluxioURI(rootUfsUri), IdUtils.ROOT_MOUNT_ID, mountOptions);
    }
}
Also used : SystemClock(alluxio.clock.SystemClock) OK(alluxio.master.file.InodeSyncStream.SyncStatus.OK) Server(alluxio.Server) PropertyKey(alluxio.conf.PropertyKey) StringUtils(org.apache.commons.lang3.StringUtils) CloseableResource(alluxio.resource.CloseableResource) Map(java.util.Map) LockedInodePathList(alluxio.master.file.meta.LockedInodePathList) WorkerHeartbeatContext(alluxio.master.file.contexts.WorkerHeartbeatContext) InodeDirectory(alluxio.master.file.meta.InodeDirectory) ClientContext(alluxio.ClientContext) ReadOnlyInodeStore(alluxio.master.metastore.ReadOnlyInodeStore) UfsSyncPathCache(alluxio.master.file.meta.UfsSyncPathCache) ConnectionFailedException(alluxio.exception.ConnectionFailedException) ProtobufUtils(alluxio.master.ProtobufUtils) UpdateInodeFileEntry(alluxio.proto.journal.File.UpdateInodeFileEntry) Stream(java.util.stream.Stream) InodeTree(alluxio.master.file.meta.InodeTree) OperationContext(alluxio.master.file.contexts.OperationContext) SetAclAction(alluxio.grpc.SetAclAction) GetStatusPOptions(alluxio.grpc.GetStatusPOptions) InternalOperationContext(alluxio.master.file.contexts.InternalOperationContext) InodeFile(alluxio.master.file.meta.InodeFile) PersistConfig(alluxio.job.plan.persist.PersistConfig) AuthType(alluxio.security.authentication.AuthType) Supplier(java.util.function.Supplier) UnderFileSystemUtils(alluxio.util.UnderFileSystemUtils) LockedInodePath(alluxio.master.file.meta.LockedInodePath) GrpcUtils(alluxio.grpc.GrpcUtils) MountInfo(alluxio.master.file.meta.options.MountInfo) AlluxioURI(alluxio.AlluxioURI) MountPOptions(alluxio.grpc.MountPOptions) MetricsSystem(alluxio.metrics.MetricsSystem) UpdateInodeEntry(alluxio.proto.journal.File.UpdateInodeEntry) RetryPolicy(alluxio.retry.RetryPolicy) PersistFile(alluxio.wire.PersistFile) IdUtils(alluxio.util.IdUtils) PersistenceState(alluxio.master.file.meta.PersistenceState) IOException(java.io.IOException) UfsAbsentPathCache(alluxio.master.file.meta.UfsAbsentPathCache) TreeMap(java.util.TreeMap) UfsStatus(alluxio.underfs.UfsStatus) UfsFileStatus(alluxio.underfs.UfsFileStatus) Preconditions(com.google.common.base.Preconditions) FreeContext(alluxio.master.file.contexts.FreeContext) AclEntry(alluxio.security.authorization.AclEntry) Reconfigurable(alluxio.conf.Reconfigurable) MountTable(alluxio.master.file.meta.MountTable) CommonUtils(alluxio.util.CommonUtils) InodeLockManager(alluxio.master.file.meta.InodeLockManager) InodePathPair(alluxio.master.file.meta.InodePathPair) FailedPreconditionException(alluxio.exception.status.FailedPreconditionException) Fingerprint(alluxio.underfs.Fingerprint) AclEntryType(alluxio.security.authorization.AclEntryType) DeleteContext(alluxio.master.file.contexts.DeleteContext) LoadMetadataContext(alluxio.master.file.contexts.LoadMetadataContext) RenameEntry(alluxio.proto.journal.File.RenameEntry) HeartbeatThread(alluxio.heartbeat.HeartbeatThread) InvalidPathException(alluxio.exception.InvalidPathException) ListStatusContext(alluxio.master.file.contexts.ListStatusContext) MetricKey(alluxio.metrics.MetricKey) SyncPointInfo(alluxio.wire.SyncPointInfo) FileSystemMasterCommonPOptions(alluxio.grpc.FileSystemMasterCommonPOptions) Journaled(alluxio.master.journal.Journaled) ImmutableSet(com.google.common.collect.ImmutableSet) Collection(java.util.Collection) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) CheckpointName(alluxio.master.journal.checkpoint.CheckpointName) Collectors(java.util.stream.Collectors) MkdirsOptions(alluxio.underfs.options.MkdirsOptions) InodeStore(alluxio.master.metastore.InodeStore) SecurityUtils(alluxio.util.SecurityUtils) CoreMasterContext(alluxio.master.CoreMasterContext) CommandType(alluxio.wire.CommandType) SetAttributeContext(alluxio.master.file.contexts.SetAttributeContext) FileAlreadyCompletedException(alluxio.exception.FileAlreadyCompletedException) Function(java.util.function.Function) Stack(java.util.Stack) MetricInfo(alluxio.metrics.MetricInfo) ExistsContext(alluxio.master.file.contexts.ExistsContext) HashSet(java.util.HashSet) Constants(alluxio.Constants) FileSystemMasterView(alluxio.master.file.meta.FileSystemMasterView) NOT_NEEDED(alluxio.master.file.InodeSyncStream.SyncStatus.NOT_NEEDED) ExecutorService(java.util.concurrent.ExecutorService) AuthenticatedClientUser(alluxio.security.authentication.AuthenticatedClientUser) SetAclEntry(alluxio.proto.journal.File.SetAclEntry) Logger(org.slf4j.Logger) Pair(alluxio.collections.Pair) AsyncUserAccessAuditLogWriter(alluxio.master.audit.AsyncUserAccessAuditLogWriter) NotFoundException(alluxio.exception.status.NotFoundException) LockResource(alluxio.resource.LockResource) UnexpectedAlluxioException(alluxio.exception.UnexpectedAlluxioException) CallTracker(alluxio.master.file.contexts.CallTracker) UfsBlockLocationCache(alluxio.master.file.meta.UfsBlockLocationCache) CreateDirectoryContext(alluxio.master.file.contexts.CreateDirectoryContext) CreateFileContext(alluxio.master.file.contexts.CreateFileContext) Arrays(java.util.Arrays) BlockInfo(alluxio.wire.BlockInfo) PersistCommandOptions(alluxio.wire.PersistCommandOptions) GrpcService(alluxio.grpc.GrpcService) TimeSeries(alluxio.metrics.TimeSeries) WorkerInfo(alluxio.wire.WorkerInfo) DelegatingJournaled(alluxio.master.journal.DelegatingJournaled) PreconditionMessage(alluxio.exception.PreconditionMessage) AuditContext(alluxio.master.audit.AuditContext) DescendantType(alluxio.file.options.DescendantType) ActiveSyncManager(alluxio.master.file.activesync.ActiveSyncManager) Set(java.util.Set) AlluxioException(alluxio.exception.AlluxioException) GetStatusContext(alluxio.master.file.contexts.GetStatusContext) UnderFileSystem(alluxio.underfs.UnderFileSystem) JobInfo(alluxio.job.wire.JobInfo) ServiceType(alluxio.grpc.ServiceType) Iterables(com.google.common.collect.Iterables) CountingRetry(alluxio.retry.CountingRetry) UnderFileSystemConfiguration(alluxio.underfs.UnderFileSystemConfiguration) Callable(java.util.concurrent.Callable) InodeDirectoryView(alluxio.master.file.meta.InodeDirectoryView) Mode(alluxio.security.authorization.Mode) Metric(alluxio.metrics.Metric) ArrayList(java.util.ArrayList) ReconfigurableRegistry(alluxio.conf.ReconfigurableRegistry) BlockInfoException(alluxio.exception.BlockInfoException) InodeDirectoryIdGenerator(alluxio.master.file.meta.InodeDirectoryIdGenerator) Builder(alluxio.proto.journal.File.UpdateInodeFileEntry.Builder) Nullable(javax.annotation.Nullable) LoadDescendantPType(alluxio.grpc.LoadDescendantPType) MountContext(alluxio.master.file.contexts.MountContext) MetricRegistry(com.codahale.metrics.MetricRegistry) FileAlreadyExistsException(alluxio.exception.FileAlreadyExistsException) RenameContext(alluxio.master.file.contexts.RenameContext) FileSystemCommandOptions(alluxio.wire.FileSystemCommandOptions) JobMasterClientPool(alluxio.client.job.JobMasterClientPool) FileSystemCommand(alluxio.wire.FileSystemCommand) ExecutorServiceFactory(alluxio.util.executor.ExecutorServiceFactory) UfsInfo(alluxio.wire.UfsInfo) JournaledGroup(alluxio.master.journal.JournaledGroup) ExecutorServiceFactories(alluxio.util.executor.ExecutorServiceFactories) NotThreadSafe(javax.annotation.concurrent.NotThreadSafe) ServerInterceptors(io.grpc.ServerInterceptors) MasterUfsManager(alluxio.underfs.MasterUfsManager) ScheduleAsyncPersistenceContext(alluxio.master.file.contexts.ScheduleAsyncPersistenceContext) LoggerFactory(org.slf4j.LoggerFactory) FileBlockInfo(alluxio.wire.FileBlockInfo) NewBlockEntry(alluxio.proto.journal.File.NewBlockEntry) LogUtils(alluxio.util.LogUtils) ResourceExhaustedException(alluxio.exception.status.ResourceExhaustedException) Counter(com.codahale.metrics.Counter) JobMasterClient(alluxio.client.job.JobMasterClient) CheckConsistencyContext(alluxio.master.file.contexts.CheckConsistencyContext) InvalidArgumentException(alluxio.exception.status.InvalidArgumentException) LockPattern(alluxio.master.file.meta.InodeTree.LockPattern) TtlAction(alluxio.grpc.TtlAction) ServerConfiguration(alluxio.conf.ServerConfiguration) ImmutableMap(com.google.common.collect.ImmutableMap) CompleteFileContext(alluxio.master.file.contexts.CompleteFileContext) TimeSeriesStore(alluxio.master.metrics.TimeSeriesStore) PermissionDeniedException(alluxio.exception.status.PermissionDeniedException) LoadMetadataPOptions(alluxio.grpc.LoadMetadataPOptions) Streams(com.google.common.collect.Streams) LinkedBlockingQueue(java.util.concurrent.LinkedBlockingQueue) Sets(com.google.common.collect.Sets) FileNotFoundException(java.io.FileNotFoundException) DelegatingReadOnlyInodeStore(alluxio.master.metastore.DelegatingReadOnlyInodeStore) AccessControlException(alluxio.exception.AccessControlException) File(alluxio.proto.journal.File) List(java.util.List) InvalidFileSizeException(alluxio.exception.InvalidFileSizeException) FileDoesNotExistException(alluxio.exception.FileDoesNotExistException) LoadMetadataPType(alluxio.grpc.LoadMetadataPType) ModeUtils(alluxio.util.ModeUtils) Gauge(com.codahale.metrics.Gauge) UFS_OP_SAVED_PREFIX(alluxio.metrics.MetricInfo.UFS_OP_SAVED_PREFIX) UfsManager(alluxio.underfs.UfsManager) JournalContext(alluxio.master.journal.JournalContext) SortedMap(java.util.SortedMap) UnavailableException(alluxio.exception.status.UnavailableException) MountPointInfo(alluxio.wire.MountPointInfo) JobMasterClientContext(alluxio.worker.job.JobMasterClientContext) ThreadPoolExecutor(java.util.concurrent.ThreadPoolExecutor) HashMap(java.util.HashMap) SetAttributePOptions(alluxio.grpc.SetAttributePOptions) BlockId(alluxio.master.block.BlockId) BlockMaster(alluxio.master.block.BlockMaster) CheckAccessContext(alluxio.master.file.contexts.CheckAccessContext) JournalEntry(alluxio.proto.journal.Journal.JournalEntry) DeletePOptions(alluxio.grpc.DeletePOptions) PathUtils(alluxio.util.io.PathUtils) DirectoryNotEmptyException(alluxio.exception.DirectoryNotEmptyException) ClientIpAddressInjector(alluxio.security.authentication.ClientIpAddressInjector) PrefixList(alluxio.collections.PrefixList) SetAclContext(alluxio.master.file.contexts.SetAclContext) CoreMaster(alluxio.master.CoreMaster) HeartbeatContext(alluxio.heartbeat.HeartbeatContext) Iterator(java.util.Iterator) ExceptionMessage(alluxio.exception.ExceptionMessage) ThreadFactoryUtils(alluxio.util.ThreadFactoryUtils) ProtoUtils(alluxio.util.proto.ProtoUtils) TimeUnit(java.util.concurrent.TimeUnit) BlockLocation(alluxio.wire.BlockLocation) FAILED(alluxio.master.file.InodeSyncStream.SyncStatus.FAILED) LockingScheme(alluxio.master.file.meta.LockingScheme) FileInfo(alluxio.wire.FileInfo) VisibleForTesting(com.google.common.annotations.VisibleForTesting) UfsMode(alluxio.underfs.UfsMode) Collections(java.util.Collections) Inode(alluxio.master.file.meta.Inode) MountPOptions(alluxio.grpc.MountPOptions) UnderFileSystem(alluxio.underfs.UnderFileSystem) MountInfo(alluxio.master.file.meta.options.MountInfo) Map(java.util.Map) TreeMap(java.util.TreeMap) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) ImmutableMap(com.google.common.collect.ImmutableMap) SortedMap(java.util.SortedMap) HashMap(java.util.HashMap) AlluxioURI(alluxio.AlluxioURI)

Example 3 with MountPOptions

use of alluxio.grpc.MountPOptions in project alluxio by Alluxio.

the class BaseFileSystem method mount.

@Override
public void mount(AlluxioURI alluxioPath, AlluxioURI ufsPath, final MountPOptions options) throws IOException, AlluxioException {
    checkUri(alluxioPath);
    rpc(client -> {
        MountPOptions mergedOptions = FileSystemOptions.mountDefaults(mFsContext.getPathConf(alluxioPath)).toBuilder().mergeFrom(options).build();
        // TODO(calvin): Make this fail on the master side
        client.mount(alluxioPath, ufsPath, mergedOptions);
        LOG.debug("Mount {} to {}", ufsPath, alluxioPath.getPath());
        return null;
    });
}
Also used : MountPOptions(alluxio.grpc.MountPOptions)

Example 4 with MountPOptions

use of alluxio.grpc.MountPOptions in project alluxio by Alluxio.

the class MountTable method add.

/**
 * Mounts the given UFS path at the given Alluxio path. The Alluxio path should not be nested
 * under an existing mount point.
 *
 * @param journalContext the journal context
 * @param alluxioUri an Alluxio path URI
 * @param ufsUri a UFS path URI
 * @param mountId the mount id
 * @param options the mount options
 * @throws FileAlreadyExistsException if the mount point already exists
 * @throws InvalidPathException if an invalid path is encountered
 */
public void add(Supplier<JournalContext> journalContext, AlluxioURI alluxioUri, AlluxioURI ufsUri, long mountId, MountPOptions options) throws FileAlreadyExistsException, InvalidPathException {
    String alluxioPath = alluxioUri.getPath().isEmpty() ? "/" : alluxioUri.getPath();
    LOG.info("Mounting {} at {}", ufsUri, alluxioPath);
    try (LockResource r = new LockResource(mWriteLock)) {
        if (mState.getMountTable().containsKey(alluxioPath)) {
            throw new FileAlreadyExistsException(ExceptionMessage.MOUNT_POINT_ALREADY_EXISTS.getMessage(alluxioPath));
        }
        // or suffix of any existing mount path.
        for (Map.Entry<String, MountInfo> entry : mState.getMountTable().entrySet()) {
            AlluxioURI mountedUfsUri = entry.getValue().getUfsUri();
            if ((ufsUri.getScheme() == null || ufsUri.getScheme().equals(mountedUfsUri.getScheme())) && (ufsUri.getAuthority().toString().equals(mountedUfsUri.getAuthority().toString()))) {
                String ufsPath = ufsUri.getPath().isEmpty() ? "/" : ufsUri.getPath();
                String mountedUfsPath = mountedUfsUri.getPath().isEmpty() ? "/" : mountedUfsUri.getPath();
                if (PathUtils.hasPrefix(ufsPath, mountedUfsPath)) {
                    throw new InvalidPathException(ExceptionMessage.MOUNT_POINT_PREFIX_OF_ANOTHER.getMessage(mountedUfsUri.toString(), ufsUri.toString()));
                }
                if (PathUtils.hasPrefix(mountedUfsPath, ufsPath)) {
                    throw new InvalidPathException(ExceptionMessage.MOUNT_POINT_PREFIX_OF_ANOTHER.getMessage(ufsUri.toString(), mountedUfsUri.toString()));
                }
            }
        }
        Map<String, String> properties = options.getPropertiesMap();
        mState.applyAndJournal(journalContext, AddMountPointEntry.newBuilder().addAllProperties(properties.entrySet().stream().map(entry -> StringPairEntry.newBuilder().setKey(entry.getKey()).setValue(entry.getValue()).build()).collect(Collectors.toList())).setAlluxioPath(alluxioPath).setMountId(mountId).setReadOnly(options.getReadOnly()).setShared(options.getShared()).setUfsPath(ufsUri.toString()).build());
    }
}
Also used : CloseableIterator(alluxio.resource.CloseableIterator) LoggerFactory(org.slf4j.LoggerFactory) HashMap(java.util.HashMap) ReentrantReadWriteLock(java.util.concurrent.locks.ReentrantReadWriteLock) DeleteMountPointEntry(alluxio.proto.journal.File.DeleteMountPointEntry) Supplier(java.util.function.Supplier) ArrayList(java.util.ArrayList) JournalEntry(alluxio.proto.journal.Journal.JournalEntry) PathUtils(alluxio.util.io.PathUtils) InvalidPathException(alluxio.exception.InvalidPathException) CloseableResource(alluxio.resource.CloseableResource) GrpcUtils(alluxio.grpc.GrpcUtils) AddMountPointEntry(alluxio.proto.journal.File.AddMountPointEntry) MountInfo(alluxio.master.file.meta.options.MountInfo) AlluxioURI(alluxio.AlluxioURI) Map(java.util.Map) DelegatingJournaled(alluxio.master.journal.DelegatingJournaled) MountPOptions(alluxio.grpc.MountPOptions) NoSuchElementException(java.util.NoSuchElementException) Nullable(javax.annotation.Nullable) Journaled(alluxio.master.journal.Journaled) Logger(org.slf4j.Logger) Iterator(java.util.Iterator) IdUtils(alluxio.util.IdUtils) ExceptionMessage(alluxio.exception.ExceptionMessage) FileAlreadyExistsException(alluxio.exception.FileAlreadyExistsException) Throwables(com.google.common.base.Throwables) ThreadSafe(javax.annotation.concurrent.ThreadSafe) CheckpointName(alluxio.master.journal.checkpoint.CheckpointName) GuardedBy(javax.annotation.concurrent.GuardedBy) NotFoundException(alluxio.exception.status.NotFoundException) LockResource(alluxio.resource.LockResource) Collectors(java.util.stream.Collectors) AccessControlException(alluxio.exception.AccessControlException) File(alluxio.proto.journal.File) StringPairEntry(alluxio.proto.journal.File.StringPairEntry) List(java.util.List) Lock(java.util.concurrent.locks.Lock) UnderFileSystem(alluxio.underfs.UnderFileSystem) Journal(alluxio.proto.journal.Journal) UfsManager(alluxio.underfs.UfsManager) Collections(java.util.Collections) JournalContext(alluxio.master.journal.JournalContext) UnavailableException(alluxio.exception.status.UnavailableException) FileAlreadyExistsException(alluxio.exception.FileAlreadyExistsException) LockResource(alluxio.resource.LockResource) MountInfo(alluxio.master.file.meta.options.MountInfo) HashMap(java.util.HashMap) Map(java.util.Map) InvalidPathException(alluxio.exception.InvalidPathException) AlluxioURI(alluxio.AlluxioURI)

Example 5 with MountPOptions

use of alluxio.grpc.MountPOptions in project alluxio by Alluxio.

the class MountContext method mergeFrom.

/**
 * Merges and embeds the given {@link MountPOptions} with the corresponding master options.
 *
 * @param optionsBuilder Builder for proto {@link MountPOptions} to embed
 * @return the instance of {@link MountContext} with default values for master
 */
public static MountContext mergeFrom(MountPOptions.Builder optionsBuilder) {
    MountPOptions masterOptions = FileSystemOptions.mountDefaults(ServerConfiguration.global());
    MountPOptions.Builder mergedOptionsBuilder = masterOptions.toBuilder().mergeFrom(optionsBuilder.build());
    return create(mergedOptionsBuilder);
}
Also used : MountPOptions(alluxio.grpc.MountPOptions)

Aggregations

MountPOptions (alluxio.grpc.MountPOptions)19 AlluxioURI (alluxio.AlluxioURI)14 Test (org.junit.Test)10 MountInfo (alluxio.master.file.meta.options.MountInfo)5 MountPointInfo (alluxio.wire.MountPointInfo)4 AccessControlException (alluxio.exception.AccessControlException)3 BaseIntegrationTest (alluxio.testutils.BaseIntegrationTest)3 ConfExpectingUnderFileSystemFactory (alluxio.testutils.underfs.ConfExpectingUnderFileSystemFactory)3 MasterUfsManager (alluxio.underfs.MasterUfsManager)3 ExceptionMessage (alluxio.exception.ExceptionMessage)2 FileAlreadyExistsException (alluxio.exception.FileAlreadyExistsException)2 InvalidPathException (alluxio.exception.InvalidPathException)2 NotFoundException (alluxio.exception.status.NotFoundException)2 UnavailableException (alluxio.exception.status.UnavailableException)2 GrpcUtils (alluxio.grpc.GrpcUtils)2 DelegatingJournaled (alluxio.master.journal.DelegatingJournaled)2 JournalContext (alluxio.master.journal.JournalContext)2 Journaled (alluxio.master.journal.Journaled)2 CheckpointName (alluxio.master.journal.checkpoint.CheckpointName)2 File (alluxio.proto.journal.File)2