Search in sources :

Example 1 with UnderFileSystem

use of alluxio.underfs.UnderFileSystem in project alluxio by Alluxio.

the class InodeTree method createPath.

/**
   * Creates a file or directory at path.
   *
   * @param inodePath the path
   * @param options method options
   * @return a {@link CreatePathResult} representing the modified inodes and created inodes during
   *         path creation
   * @throws FileAlreadyExistsException when there is already a file at path if we want to create a
   *         directory there
   * @throws BlockInfoException when blockSizeBytes is invalid
   * @throws InvalidPathException when path is invalid, for example, (1) when there is nonexistent
   *         necessary parent directories and recursive is false, (2) when one of the necessary
   *         parent directories is actually a file
   * @throws IOException if creating the path fails
   * @throws FileDoesNotExistException if the parent of the path does not exist and the recursive
   *         option is false
   */
public CreatePathResult createPath(LockedInodePath inodePath, CreatePathOptions<?> options) throws FileAlreadyExistsException, BlockInfoException, InvalidPathException, IOException, FileDoesNotExistException {
    AlluxioURI path = inodePath.getUri();
    if (path.isRoot()) {
        String errorMessage = ExceptionMessage.FILE_ALREADY_EXISTS.getMessage(path);
        LOG.error(errorMessage);
        throw new FileAlreadyExistsException(errorMessage);
    }
    if (options instanceof CreateFileOptions) {
        CreateFileOptions fileOptions = (CreateFileOptions) options;
        if (fileOptions.getBlockSizeBytes() < 1) {
            throw new BlockInfoException("Invalid block size " + fileOptions.getBlockSizeBytes());
        }
    }
    if (!(inodePath instanceof MutableLockedInodePath)) {
        throw new InvalidPathException(ExceptionMessage.NOT_MUTABLE_INODE_PATH.getMessage(inodePath.getUri()));
    }
    LOG.debug("createPath {}", path);
    TraversalResult traversalResult = traverseToInode(inodePath, LockMode.WRITE_PARENT);
    InodeLockList lockList = traversalResult.getInodeLockList();
    MutableLockedInodePath extensibleInodePath = (MutableLockedInodePath) inodePath;
    String[] pathComponents = extensibleInodePath.getPathComponents();
    String name = path.getName();
    // pathIndex is the index into pathComponents where we start filling in the path from the inode.
    int pathIndex = extensibleInodePath.getInodes().size();
    if (pathIndex < pathComponents.length - 1) {
        // Otherwise we add the remaining path components to the list of components to create.
        if (!options.isRecursive()) {
            final String msg = new StringBuilder().append("File ").append(path).append(" creation failed. Component ").append(pathIndex).append("(").append(pathComponents[pathIndex]).append(") does not exist").toString();
            LOG.error("FileDoesNotExistException: {}", msg);
            throw new FileDoesNotExistException(msg);
        }
    }
    // The ancestor inode (parent or ancestor) of the target path.
    Inode<?> ancestorInode = extensibleInodePath.getAncestorInode();
    if (!ancestorInode.isDirectory()) {
        throw new InvalidPathException("Could not traverse to parent directory of path " + path + ". Component " + pathComponents[pathIndex - 1] + " is not a directory.");
    }
    InodeDirectory currentInodeDirectory = (InodeDirectory) ancestorInode;
    List<Inode<?>> createdInodes = new ArrayList<>();
    List<Inode<?>> modifiedInodes = new ArrayList<>();
    // These are inodes that already exist, that should be journaled as persisted.
    List<Inode<?>> existingNonPersisted = new ArrayList<>();
    // These are inodes to mark as persisted at the end of this method.
    List<Inode<?>> toPersistDirectories = new ArrayList<>();
    if (options.isPersisted()) {
        // Directory persistence will not happen until the end of this method.
        toPersistDirectories.addAll(traversalResult.getNonPersisted());
        existingNonPersisted.addAll(traversalResult.getNonPersisted());
    }
    if (pathIndex < (pathComponents.length - 1) || currentInodeDirectory.getChild(name) == null) {
        // (1) There are components in parent paths that need to be created. Or
        // (2) The last component of the path needs to be created.
        // In these two cases, the last traversed Inode will be modified.
        modifiedInodes.add(currentInodeDirectory);
    }
    // TODO(gpang): We may not have to lock the newly created inodes if the last inode is write
    // locked. This could improve performance. Further investigation is needed.
    // Fill in the ancestor directories that were missing.
    // NOTE, we set the mode of missing ancestor directories to be the default value, rather
    // than inheriting the option of the final file to create, because it may not have
    // "execute" permission.
    CreateDirectoryOptions missingDirOptions = CreateDirectoryOptions.defaults().setMountPoint(false).setPersisted(options.isPersisted()).setOwner(options.getOwner()).setGroup(options.getGroup());
    for (int k = pathIndex; k < (pathComponents.length - 1); k++) {
        InodeDirectory dir = InodeDirectory.create(mDirectoryIdGenerator.getNewDirectoryId(), currentInodeDirectory.getId(), pathComponents[k], missingDirOptions);
        // Lock the newly created inode before subsequent operations, and add it to the lock group.
        lockList.lockWriteAndCheckNameAndParent(dir, currentInodeDirectory, pathComponents[k]);
        dir.setPinned(currentInodeDirectory.isPinned());
        currentInodeDirectory.addChild(dir);
        currentInodeDirectory.setLastModificationTimeMs(options.getOperationTimeMs());
        if (options.isPersisted()) {
            toPersistDirectories.add(dir);
        }
        createdInodes.add(dir);
        mInodes.add(dir);
        currentInodeDirectory = dir;
    }
    // Create the final path component. First we need to make sure that there isn't already a file
    // here with that name. If there is an existing file that is a directory and we're creating a
    // directory, update persistence property of the directories if needed, otherwise, throw
    // FileAlreadyExistsException unless options.allowExists is true.
    Inode<?> lastInode = currentInodeDirectory.getChild(name);
    if (lastInode != null) {
        // Lock the last inode before subsequent operations, and add it to the lock group.
        lockList.lockWriteAndCheckNameAndParent(lastInode, currentInodeDirectory, name);
        if (lastInode.isDirectory() && options instanceof CreateDirectoryOptions && !lastInode.isPersisted() && options.isPersisted()) {
            // The final path component already exists and is not persisted, so it should be added
            // to the non-persisted Inodes of traversalResult.
            existingNonPersisted.add(lastInode);
            toPersistDirectories.add(lastInode);
        } else if (!lastInode.isDirectory() || !(options instanceof CreateDirectoryOptions && ((CreateDirectoryOptions) options).isAllowExists())) {
            String errorMessage = ExceptionMessage.FILE_ALREADY_EXISTS.getMessage(path);
            LOG.error(errorMessage);
            throw new FileAlreadyExistsException(errorMessage);
        }
    } else {
        if (options instanceof CreateDirectoryOptions) {
            CreateDirectoryOptions directoryOptions = (CreateDirectoryOptions) options;
            lastInode = InodeDirectory.create(mDirectoryIdGenerator.getNewDirectoryId(), currentInodeDirectory.getId(), name, directoryOptions);
            // Lock the created inode before subsequent operations, and add it to the lock group.
            lockList.lockWriteAndCheckNameAndParent(lastInode, currentInodeDirectory, name);
            if (directoryOptions.isPersisted()) {
                toPersistDirectories.add(lastInode);
            }
            lastInode.setPinned(currentInodeDirectory.isPinned());
        } else if (options instanceof CreateFileOptions) {
            CreateFileOptions fileOptions = (CreateFileOptions) options;
            lastInode = InodeFile.create(mContainerIdGenerator.getNewContainerId(), currentInodeDirectory.getId(), name, System.currentTimeMillis(), fileOptions);
            // Lock the created inode before subsequent operations, and add it to the lock group.
            lockList.lockWriteAndCheckNameAndParent(lastInode, currentInodeDirectory, name);
            if (currentInodeDirectory.isPinned()) {
                // Update set of pinned file ids.
                mPinnedInodeFileIds.add(lastInode.getId());
            }
            lastInode.setPinned(currentInodeDirectory.isPinned());
        }
        createdInodes.add(lastInode);
        mInodes.add(lastInode);
        currentInodeDirectory.addChild(lastInode);
        currentInodeDirectory.setLastModificationTimeMs(options.getOperationTimeMs());
    }
    // we mark it as persisted.
    for (Inode<?> inode : toPersistDirectories) {
        MountTable.Resolution resolution = mMountTable.resolve(getPath(inode));
        String ufsUri = resolution.getUri().toString();
        UnderFileSystem ufs = resolution.getUfs();
        MkdirsOptions mkdirsOptions = MkdirsOptions.defaults().setCreateParent(false).setOwner(inode.getOwner()).setGroup(inode.getGroup()).setMode(new Mode(inode.getMode()));
        if (ufs.isDirectory(ufsUri) || ufs.mkdirs(ufsUri, mkdirsOptions)) {
            inode.setPersistenceState(PersistenceState.PERSISTED);
        }
    }
    // Extend the inodePath with the created inodes.
    extensibleInodePath.getInodes().addAll(createdInodes);
    LOG.debug("createFile: File Created: {} parent: {}", lastInode, currentInodeDirectory);
    return new CreatePathResult(modifiedInodes, createdInodes, existingNonPersisted);
}
Also used : FileDoesNotExistException(alluxio.exception.FileDoesNotExistException) FileAlreadyExistsException(alluxio.exception.FileAlreadyExistsException) MkdirsOptions(alluxio.underfs.options.MkdirsOptions) CreateDirectoryOptions(alluxio.master.file.options.CreateDirectoryOptions) ArrayList(java.util.ArrayList) InvalidPathException(alluxio.exception.InvalidPathException) CreateFileOptions(alluxio.master.file.options.CreateFileOptions) UnderFileSystem(alluxio.underfs.UnderFileSystem) Mode(alluxio.security.authorization.Mode) BlockInfoException(alluxio.exception.BlockInfoException) AlluxioURI(alluxio.AlluxioURI)

Example 2 with UnderFileSystem

use of alluxio.underfs.UnderFileSystem in project alluxio by Alluxio.

the class WebInterfaceGeneralServlet method populateValues.

/**
   * Populates key, value pairs for UI display.
   *
   * @param request The {@link HttpServletRequest} object
   * @throws IOException if an I/O error occurs
   */
private void populateValues(HttpServletRequest request) throws IOException {
    request.setAttribute("debug", Configuration.getBoolean(PropertyKey.DEBUG));
    request.setAttribute("masterNodeAddress", mMaster.getRpcAddress().toString());
    request.setAttribute("uptime", WebUtils.convertMsToClockTime(System.currentTimeMillis() - mMaster.getStartTimeMs()));
    request.setAttribute("startTime", WebUtils.convertMsToDate(mMaster.getStartTimeMs()));
    request.setAttribute("version", RuntimeConstants.VERSION);
    request.setAttribute("liveWorkerNodes", Integer.toString(mMaster.getBlockMaster().getWorkerCount()));
    request.setAttribute("capacity", FormatUtils.getSizeFromBytes(mMaster.getBlockMaster().getCapacityBytes()));
    request.setAttribute("usedCapacity", FormatUtils.getSizeFromBytes(mMaster.getBlockMaster().getUsedBytes()));
    request.setAttribute("freeCapacity", FormatUtils.getSizeFromBytes(mMaster.getBlockMaster().getCapacityBytes() - mMaster.getBlockMaster().getUsedBytes()));
    FileSystemMaster.StartupConsistencyCheck check = mMaster.getFileSystemMaster().getStartupConsistencyCheck();
    request.setAttribute("consistencyCheckStatus", check.getStatus());
    if (check.getStatus() == FileSystemMaster.StartupConsistencyCheck.Status.COMPLETE) {
        request.setAttribute("inconsistentPaths", check.getInconsistentUris().size());
        request.setAttribute("inconsistentPathItems", check.getInconsistentUris());
    } else {
        request.setAttribute("inconsistentPaths", 0);
    }
    String ufsRoot = Configuration.get(PropertyKey.UNDERFS_ADDRESS);
    UnderFileSystem ufs = UnderFileSystem.Factory.get(ufsRoot);
    long sizeBytes = ufs.getSpace(ufsRoot, UnderFileSystem.SpaceType.SPACE_TOTAL);
    if (sizeBytes >= 0) {
        request.setAttribute("diskCapacity", FormatUtils.getSizeFromBytes(sizeBytes));
    } else {
        request.setAttribute("diskCapacity", "UNKNOWN");
    }
    sizeBytes = ufs.getSpace(ufsRoot, UnderFileSystem.SpaceType.SPACE_USED);
    if (sizeBytes >= 0) {
        request.setAttribute("diskUsedCapacity", FormatUtils.getSizeFromBytes(sizeBytes));
    } else {
        request.setAttribute("diskUsedCapacity", "UNKNOWN");
    }
    sizeBytes = ufs.getSpace(ufsRoot, UnderFileSystem.SpaceType.SPACE_FREE);
    if (sizeBytes >= 0) {
        request.setAttribute("diskFreeCapacity", FormatUtils.getSizeFromBytes(sizeBytes));
    } else {
        request.setAttribute("diskFreeCapacity", "UNKNOWN");
    }
    StorageTierInfo[] infos = generateOrderedStorageTierInfo();
    request.setAttribute("storageTierInfos", infos);
}
Also used : FileSystemMaster(alluxio.master.file.FileSystemMaster) UnderFileSystem(alluxio.underfs.UnderFileSystem)

Example 3 with UnderFileSystem

use of alluxio.underfs.UnderFileSystem in project alluxio by Alluxio.

the class AlluxioMasterRestServiceHandlerTest method registerFileSystemMock.

private void registerFileSystemMock() throws IOException {
    Configuration.set(PropertyKey.UNDERFS_ADDRESS, TEST_PATH);
    UnderFileSystemFactory underFileSystemFactoryMock = mock(UnderFileSystemFactory.class);
    when(underFileSystemFactoryMock.supportsPath(anyString())).thenReturn(Boolean.FALSE);
    when(underFileSystemFactoryMock.supportsPath(TEST_PATH)).thenReturn(Boolean.TRUE);
    UnderFileSystem underFileSystemMock = mock(UnderFileSystem.class);
    when(underFileSystemMock.getSpace(TEST_PATH, UnderFileSystem.SpaceType.SPACE_FREE)).thenReturn(UFS_SPACE_FREE);
    when(underFileSystemMock.getSpace(TEST_PATH, UnderFileSystem.SpaceType.SPACE_TOTAL)).thenReturn(UFS_SPACE_TOTAL);
    when(underFileSystemMock.getSpace(TEST_PATH, UnderFileSystem.SpaceType.SPACE_USED)).thenReturn(UFS_SPACE_USED);
    when(underFileSystemFactoryMock.create(eq(TEST_PATH), anyObject())).thenReturn(underFileSystemMock);
    UnderFileSystemRegistry.register(underFileSystemFactoryMock);
}
Also used : UnderFileSystem(alluxio.underfs.UnderFileSystem) UnderFileSystemFactory(alluxio.underfs.UnderFileSystemFactory)

Example 4 with UnderFileSystem

use of alluxio.underfs.UnderFileSystem in project alluxio by Alluxio.

the class FileInStreamTest method failToUnderFs.

/**
   * Tests the capability to fall back to a ufs stream when getting an alluxio stream fails.
   */
@Test
public void failToUnderFs() throws AlluxioException, IOException {
    mInfo.setPersisted(true).setUfsPath("testUfsPath");
    mStatus = new URIStatus(mInfo);
    mTestStream = new FileInStream(mStatus, InStreamOptions.defaults().setCachePartiallyReadBlock(false), mContext);
    Mockito.when(mBlockStore.getInStream(Mockito.eq(1L), Mockito.any(InStreamOptions.class))).thenThrow(new IOException("test IOException"));
    if (mDelegateUfsOps) {
        UnderFileSystemBlockInStream inStream = PowerMockito.mock(UnderFileSystemBlockInStream.class);
        PowerMockito.when(StreamFactory.createUfsBlockInStream(Mockito.any(FileSystemContext.class), Mockito.anyString(), Mockito.anyLong(), Mockito.anyLong(), Mockito.anyLong(), Mockito.any(WorkerNetAddress.class), Mockito.any(InStreamOptions.class))).thenReturn(inStream);
        mTestStream.seek(BLOCK_LENGTH + (BLOCK_LENGTH / 2));
    } else {
        UnderFileSystem ufs = ClientMockUtils.mockUnderFileSystem(Mockito.eq("testUfsPath"));
        InputStream stream = Mockito.mock(InputStream.class);
        Mockito.when(ufs.open("testUfsPath")).thenReturn(stream);
        Mockito.when(stream.skip(BLOCK_LENGTH)).thenReturn(BLOCK_LENGTH);
        Mockito.when(stream.skip(BLOCK_LENGTH / 2)).thenReturn(BLOCK_LENGTH / 2);
        mTestStream.seek(BLOCK_LENGTH + (BLOCK_LENGTH / 2));
        Mockito.verify(ufs).open("testUfsPath");
        Mockito.verify(stream).skip(100);
        Mockito.verify(stream).skip(50);
    }
}
Also used : WorkerNetAddress(alluxio.wire.WorkerNetAddress) InputStream(java.io.InputStream) UnderFileSystemBlockInStream(alluxio.client.block.UnderFileSystemBlockInStream) IOException(java.io.IOException) UnderFileSystem(alluxio.underfs.UnderFileSystem) InStreamOptions(alluxio.client.file.options.InStreamOptions) PrepareForTest(org.powermock.core.classloader.annotations.PrepareForTest) Test(org.junit.Test)

Example 5 with UnderFileSystem

use of alluxio.underfs.UnderFileSystem in project alluxio by Alluxio.

the class ClientMockUtils method mockUnderFileSystem.

/**
   * When {@link UnderFileSystem#get(String)} is called to get an {@link UnderFileSystem}, it
   * will instead return the mock returned by this method, as long as `filename` matches
   * `ufsPathMatcher`.
   *
   * Because this method needs to mock a static method from {@link UnderFileSystem}, calling tests
   * should make sure to annotate their classes with `@PrepareForTesting(UnderFileSystem.class)`.
   *
   * @param ufsPathMatcher a {@link Mockito} String Matcher
   * @return the {@link UnderFileSystem} mock
   */
public static UnderFileSystem mockUnderFileSystem(String ufsPathMatcher) {
    UnderFileSystem ufs = PowerMockito.mock(UnderFileSystem.class);
    PowerMockito.mockStatic(UnderFileSystem.Factory.class);
    PowerMockito.when(UnderFileSystem.Factory.get(ufsPathMatcher)).thenReturn(ufs);
    return ufs;
}
Also used : UnderFileSystem(alluxio.underfs.UnderFileSystem)

Aggregations

UnderFileSystem (alluxio.underfs.UnderFileSystem)123 AlluxioURI (alluxio.AlluxioURI)59 Test (org.junit.Test)44 IOException (java.io.IOException)37 MountTable (alluxio.master.file.meta.MountTable)24 URIStatus (alluxio.client.file.URIStatus)17 Mode (alluxio.security.authorization.Mode)15 UfsManager (alluxio.underfs.UfsManager)13 UfsStatus (alluxio.underfs.UfsStatus)13 InvalidPathException (alluxio.exception.InvalidPathException)12 Inode (alluxio.master.file.meta.Inode)12 OutputStream (java.io.OutputStream)12 ArrayList (java.util.ArrayList)12 BaseIntegrationTest (alluxio.testutils.BaseIntegrationTest)11 FileAlreadyExistsException (alluxio.exception.FileAlreadyExistsException)9 AccessControlException (alluxio.exception.AccessControlException)8 BlockInfoException (alluxio.exception.BlockInfoException)7 FileDoesNotExistException (alluxio.exception.FileDoesNotExistException)7 InodeDirectory (alluxio.master.file.meta.InodeDirectory)7 InodeFile (alluxio.master.file.meta.InodeFile)7