Search in sources :

Example 6 with Tuple2

use of com.evolvedbinary.j8fu.tuple.Tuple2 in project exist by eXist-db.

the class BlobStoreImplTest method addUnique.

@Test
public void addUnique() throws IOException {
    final Path blobDbx = temporaryFolder.getRoot().toPath().resolve("blob.dbx");
    final Path blobDir = temporaryFolder.newFolder("blob").toPath();
    final List<Tuple2<byte[], MessageDigest>> testFiles = Arrays.asList(generateTestFile(), generateTestFile(), generateTestFile(), generateTestFile(), generateTestFile());
    try (final BlobStore blobStore = newBlobStore(blobDbx, blobDir)) {
        blobStore.open();
        for (final Tuple2<byte[], MessageDigest> testFile : testFiles) {
            addAndVerify(blobStore, testFile);
        }
    }
    // should be 1 entry per unique test file in the blob.dbx, each entry is the digest and then the reference count
    final long expectedBlobDbxLen = calculateBlobStoreSize(testFiles.size());
    final long actualBlobDbxLen = Files.size(blobDbx);
    assertEquals(expectedBlobDbxLen, actualBlobDbxLen);
}
Also used : Path(java.nio.file.Path) Tuple2(com.evolvedbinary.j8fu.tuple.Tuple2) MessageDigest(org.exist.util.crypto.digest.MessageDigest) Test(org.junit.Test)

Example 7 with Tuple2

use of com.evolvedbinary.j8fu.tuple.Tuple2 in project exist by eXist-db.

the class FluentBrokerAPITest method collectionThenCollectionAndDoc.

@Test
public void collectionThenCollectionAndDoc() throws PermissionDeniedException, EXistException, LockException {
    final XmldbURI docUri = uri("all-test.xml");
    final long collectionCreated = 1234;
    final IMocksControl ctrl = createStrictControl();
    ctrl.checkOrder(true);
    final BrokerPool mockBrokerPool = ctrl.createMock(BrokerPool.class);
    final DBBroker mockBroker = ctrl.createMock(DBBroker.class);
    final Collection mockCollection = ctrl.createMock(Collection.class);
    final LockedDocument mockLockedDocument = ctrl.createMock(LockedDocument.class);
    final DocumentImpl mockDocument = ctrl.createMock(DocumentImpl.class);
    expect(mockBrokerPool.getBroker()).andReturn(mockBroker);
    expect(mockBroker.openCollection(TEST_COLLECTION_URI, READ_LOCK)).andReturn(mockCollection);
    expect(mockCollection.getCreated()).andReturn(collectionCreated);
    expect(mockCollection.getDocumentWithLock(mockBroker, docUri, READ_LOCK)).andReturn(mockLockedDocument);
    expect(mockLockedDocument.getDocument()).andReturn(mockDocument);
    expect(mockCollection.getURI()).andReturn(TEST_COLLECTION_URI);
    expect(mockDocument.getFileURI()).andReturn(docUri);
    // NOTE: checks that Collection lock is release before Document lock
    mockCollection.close();
    mockLockedDocument.close();
    mockBroker.close();
    ctrl.replay();
    final Function<Collection, Long> collectionOp = collection -> collection.getCreated();
    final BiFunction<Collection, DocumentImpl, String> collectionDocOp = (collection, doc) -> collection.getURI().append(doc.getFileURI()).toString();
    final Tuple2<Long, String> result = FluentBrokerAPI.builder(mockBrokerPool).withCollection(TEST_COLLECTION_URI, READ_LOCK).execute(collectionOp).withDocument(collection -> new Tuple2<>(docUri, READ_LOCK)).execute(collectionDocOp).doAll();
    assertEquals(collectionCreated, result._1.longValue());
    assertEquals(TEST_COLLECTION_URI.append(docUri), result._2);
    ctrl.verify();
}
Also used : Tuple3(com.evolvedbinary.j8fu.tuple.Tuple3) Tuple2(com.evolvedbinary.j8fu.tuple.Tuple2) IMocksControl(org.easymock.IMocksControl) LockMode(org.exist.storage.lock.Lock.LockMode) LockedDocument(org.exist.dom.persistent.LockedDocument) EasyMock.createStrictControl(org.easymock.EasyMock.createStrictControl) BiFunction(java.util.function.BiFunction) Test(org.junit.Test) PermissionDeniedException(org.exist.security.PermissionDeniedException) EasyMock.expect(org.easymock.EasyMock.expect) Function(java.util.function.Function) FluentBrokerAPI.uri(org.exist.storage.FluentBrokerAPI.uri) LockException(org.exist.util.LockException) Collection(org.exist.collections.Collection) XmldbURI(org.exist.xmldb.XmldbURI) DocumentImpl(org.exist.dom.persistent.DocumentImpl) EXistException(org.exist.EXistException) Lock(org.exist.storage.lock.Lock) Assert.assertEquals(org.junit.Assert.assertEquals) DocumentImpl(org.exist.dom.persistent.DocumentImpl) IMocksControl(org.easymock.IMocksControl) LockedDocument(org.exist.dom.persistent.LockedDocument) Collection(org.exist.collections.Collection) XmldbURI(org.exist.xmldb.XmldbURI) Test(org.junit.Test)

Example 8 with Tuple2

use of com.evolvedbinary.j8fu.tuple.Tuple2 in project exist by eXist-db.

the class RpcConnection method summarise.

private Tuple2<java.util.Collection<NodeCount>, java.util.Collection<DoctypeCount>> summarise(final Sequence results) throws XPathException {
    final Map<String, NodeCount> nodeCounts = new HashMap<>();
    final Map<String, DoctypeCount> doctypeCounts = new HashMap<>();
    NodeCount counter;
    DoctypeCount doctypeCounter;
    for (final SequenceIterator i = results.iterate(); i.hasNext(); ) {
        final Item item = i.nextItem();
        if (Type.subTypeOf(item.getType(), Type.NODE)) {
            final NodeValue nv = (NodeValue) item;
            if (nv.getImplementationType() == NodeValue.PERSISTENT_NODE) {
                final NodeProxy p = (NodeProxy) nv;
                final String docName = p.getOwnerDocument().getURI().toString();
                final DocumentType doctype = p.getOwnerDocument().getDoctype();
                if (nodeCounts.containsKey(docName)) {
                    counter = nodeCounts.get(docName);
                    counter.inc();
                } else {
                    counter = new NodeCount(p.getOwnerDocument());
                    nodeCounts.put(docName, counter);
                }
                if (doctype == null) {
                    continue;
                }
                if (doctypeCounts.containsKey(doctype.getName())) {
                    doctypeCounter = doctypeCounts.get(doctype.getName());
                    doctypeCounter.inc();
                } else {
                    doctypeCounter = new DoctypeCount(doctype);
                    doctypeCounts.put(doctype.getName(), doctypeCounter);
                }
            }
        }
    }
    return new Tuple2<>(nodeCounts.values(), doctypeCounts.values());
}
Also used : Tuple2(com.evolvedbinary.j8fu.tuple.Tuple2) DocumentType(org.w3c.dom.DocumentType)

Example 9 with Tuple2

use of com.evolvedbinary.j8fu.tuple.Tuple2 in project exist by eXist-db.

the class NativeBroker method createCollectionObject.

/**
 * NOTE - When this is called there must be a WRITE_LOCK on collectionUri
 * and at least a READ_LOCK on parentCollection (if it is not null)
 */
private Collection createCollectionObject(final Txn transaction, @Nullable @EnsureLocked(mode = LockMode.READ_LOCK) final Collection parentCollection, @EnsureLocked(mode = LockMode.WRITE_LOCK, type = LockType.COLLECTION) final XmldbURI collectionUri, final Optional<Tuple2<Permission, Long>> creationAttributes) throws ReadOnlyException, PermissionDeniedException, LockException {
    final int collectionId = getNextCollectionId(transaction);
    final Collection collection = creationAttributes.map(attrs -> new MutableCollection(this, collectionId, collectionUri, attrs._1, attrs._2)).orElseGet(() -> new MutableCollection(this, collectionId, collectionUri));
    // inherit the group to collection if parent-collection is setGid
    if (parentCollection != null) {
        final Permission parentPermissions = parentCollection.getPermissionsNoLock();
        if (parentPermissions.isSetGid()) {
            final Permission collectionPermissions = collection.getPermissionsNoLock();
            // inherit group
            collectionPermissions.setGroupFrom(parentPermissions);
            // inherit setGid bit
            collectionPermissions.setSetGid(true);
        }
    }
    return collection;
}
Also used : INodeIterator(org.exist.storage.dom.INodeIterator) Tuple2(com.evolvedbinary.j8fu.tuple.Tuple2) BasePooledObjectFactory(org.apache.commons.pool2.BasePooledObjectFactory) LockMode(org.exist.storage.lock.Lock.LockMode) XmlSerializerPool(org.exist.storage.serializers.XmlSerializerPool) Txn(org.exist.storage.txn.Txn) DEFAULT_TEMPORARY_COLLECTION_PERM(org.exist.security.Permission.DEFAULT_TEMPORARY_COLLECTION_PERM) PooledObject(org.apache.commons.pool2.PooledObject) RawDataBackup(org.exist.backup.RawDataBackup) org.exist.security(org.exist.security) VariableByteOutputStream(org.exist.storage.io.VariableByteOutputStream) IEmbeddedXMLStreamReader(org.exist.stax.IEmbeddedXMLStreamReader) Matcher(java.util.regex.Matcher) TerminatedException(org.exist.xquery.TerminatedException) Document(org.w3c.dom.Document) FunctionE(com.evolvedbinary.j8fu.function.FunctionE) DigestType(org.exist.util.crypto.digest.DigestType) XMLStreamException(javax.xml.stream.XMLStreamException) Collection(org.exist.collections.Collection) LockType(org.exist.storage.lock.Lock.LockType) BFile(org.exist.storage.index.BFile) URI(java.net.URI) Path(java.nio.file.Path) ACEAider(org.exist.security.internal.aider.ACEAider) CollectionStore(org.exist.storage.index.CollectionStore) org.exist.storage.lock(org.exist.storage.lock) GenericObjectPool(org.apache.commons.pool2.impl.GenericObjectPool) Type(org.exist.xquery.value.Type) org.exist.collections.triggers(org.exist.collections.triggers) Logger(org.apache.logging.log4j.Logger) StructuralIndex(org.exist.indexing.StructuralIndex) SAXException(org.xml.sax.SAXException) DOMIndexer(org.exist.dom.memtree.DOMIndexer) org.exist.storage.journal(org.exist.storage.journal) Pattern(java.util.regex.Pattern) java.util(java.util) QName(org.exist.dom.QName) SubCollectionEntry(org.exist.collections.Collection.SubCollectionEntry) NodeIterator(org.exist.storage.dom.NodeIterator) DOMFile(org.exist.storage.dom.DOMFile) org.exist.dom.persistent(org.exist.dom.persistent) Function(java.util.function.Function) DefaultPooledObject(org.apache.commons.pool2.impl.DefaultPooledObject) NumberFormat(java.text.NumberFormat) BlobStore(org.exist.storage.blob.BlobStore) GenericObjectPoolConfig(org.apache.commons.pool2.impl.GenericObjectPoolConfig) XMLReader(org.xml.sax.XMLReader) UnsynchronizedByteArrayOutputStream(org.apache.commons.io.output.UnsynchronizedByteArrayOutputStream) NativeSerializer(org.exist.storage.serializers.NativeSerializer) RawNodeIterator(org.exist.storage.dom.RawNodeIterator) VariableByteInput(org.exist.storage.io.VariableByteInput) Node(org.w3c.dom.Node) XmldbURI(org.exist.xmldb.XmldbURI) EXistException(org.exist.EXistException) Indexer(org.exist.Indexer) Nullable(javax.annotation.Nullable) InputSource(org.xml.sax.InputSource) org.exist.collections(org.exist.collections) NodeList(org.w3c.dom.NodeList) ReentrantLock(java.util.concurrent.locks.ReentrantLock) UnsynchronizedByteArrayInputStream(org.apache.commons.io.input.UnsynchronizedByteArrayInputStream) Files(java.nio.file.Files) UTF_8(java.nio.charset.StandardCharsets.UTF_8) Sync(org.exist.storage.sync.Sync) TransactionException(org.exist.storage.txn.TransactionException) DOMTransaction(org.exist.storage.dom.DOMTransaction) MessageDigest(org.exist.util.crypto.digest.MessageDigest) DocumentType(org.w3c.dom.DocumentType) BlobId(org.exist.storage.blob.BlobId) TransactionManager(org.exist.storage.txn.TransactionManager) StreamListener(org.exist.indexing.StreamListener) InputStreamUtil.copy(org.exist.util.io.InputStreamUtil.copy) java.io(java.io) NodeId(org.exist.numbering.NodeId) Paths(java.nio.file.Paths) ReindexMode(org.exist.indexing.StreamListener.ReindexMode) EmbeddedXMLStreamReader(org.exist.stax.EmbeddedXMLStreamReader) org.exist.util(org.exist.util) InputStreamUtil(org.exist.util.io.InputStreamUtil) Serializer(org.exist.storage.serializers.Serializer) LogManager(org.apache.logging.log4j.LogManager) org.exist.storage.btree(org.exist.storage.btree) Collection(org.exist.collections.Collection)

Example 10 with Tuple2

use of com.evolvedbinary.j8fu.tuple.Tuple2 in project exist by eXist-db.

the class NativeBroker method doCopyCollection.

/**
 * Copy a collection and all its sub-Collections.
 *
 * @param transaction The current transaction
 * @param documentTrigger The trigger to use for document events
 * @param sourceCollection The Collection to copy
 * @param destinationCollectionUri The destination Collection URI for the sourceCollection copy
 * @param copyCollectionMode false on the first call, true on recursive calls
 *
 * @return A reference to the Collection, no additional locks are held on the Collection
 *
 * @throws PermissionDeniedException If the current user does not have appropriate permissions
 * @throws LockException If an exception occurs whilst acquiring locks
 * @throws IOException If an error occurs whilst copying the Collection on disk
 * @throws TriggerException If a CollectionTrigger throws an exception
 * @throws EXistException If no more Document IDs are available
 */
private Collection doCopyCollection(final Txn transaction, final DocumentTrigger documentTrigger, @EnsureLocked(mode = LockMode.READ_LOCK) final Collection sourceCollection, @EnsureLocked(mode = LockMode.WRITE_LOCK) final Collection destinationParentCollection, @EnsureLocked(mode = LockMode.WRITE_LOCK, type = LockType.COLLECTION) final XmldbURI destinationCollectionUri, final boolean copyCollectionMode, final PreserveType preserve) throws PermissionDeniedException, IOException, EXistException, TriggerException, LockException {
    if (LOG.isDebugEnabled()) {
        LOG.debug("Copying collection to '{}'", destinationCollectionUri);
    }
    // permissions and attributes for the destCollection (if we have to create it)
    final Permission createCollectionPerms = PermissionFactory.getDefaultCollectionPermission(getBrokerPool().getSecurityManager());
    copyModeAndAcl(sourceCollection.getPermissions(), createCollectionPerms);
    final long created;
    if (preserveOnCopy(preserve)) {
        // only copy the owner and group from the source if we are creating a new collection and we are the DBA
        if (getCurrentSubject().hasDbaRole()) {
            PermissionFactory.chown(this, createCollectionPerms, Optional.of(sourceCollection.getPermissions().getOwner().getName()), Optional.of(sourceCollection.getPermissions().getGroup().getName()));
        }
        created = sourceCollection.getCreated();
    } else {
        created = 0;
    }
    final Tuple2<Boolean, Collection> destinationCollection = getOrCreateCollectionExplicit(transaction, destinationCollectionUri, Optional.of(new Tuple2<>(createCollectionPerms, created)));
    // if we didn't create destCollection but we need to preserve the attributes
    if ((!destinationCollection._1) && preserveOnCopy(preserve)) {
        copyModeAndAcl(sourceCollection.getPermissions(), destinationCollection._2.getPermissions());
    }
    // inherit the group to the destinationCollection if parent is setGid
    if (destinationParentCollection != null && destinationParentCollection.getPermissions().isSetGid()) {
        // inherit group
        destinationCollection._2.getPermissions().setGroupFrom(destinationParentCollection.getPermissions());
        // inherit setGid bit
        destinationCollection._2.getPermissions().setSetGid(true);
    }
    doCopyCollectionDocuments(transaction, documentTrigger, sourceCollection, destinationCollection._2, preserve);
    final XmldbURI sourceCollectionUri = sourceCollection.getURI();
    for (final Iterator<XmldbURI> i = sourceCollection.collectionIterator(this); i.hasNext(); ) {
        final XmldbURI childName = i.next();
        final XmldbURI childUri = sourceCollectionUri.append(childName);
        try (final Collection child = getCollection(childUri)) {
            // NOTE: we already have a READ lock on child implicitly
            if (child == null) {
                throw new IOException("Child collection " + childUri + " not found");
            } else {
                doCopyCollection(transaction, documentTrigger, child, destinationCollection._2, destinationCollection._2.getURI().append(childName), true, preserve);
            }
        }
    }
    return destinationCollection._2;
}
Also used : Tuple2(com.evolvedbinary.j8fu.tuple.Tuple2) Collection(org.exist.collections.Collection) XmldbURI(org.exist.xmldb.XmldbURI)

Aggregations

Tuple2 (com.evolvedbinary.j8fu.tuple.Tuple2)17 XmldbURI (org.exist.xmldb.XmldbURI)10 Function (java.util.function.Function)8 EXistException (org.exist.EXistException)8 Collection (org.exist.collections.Collection)8 Test (org.junit.Test)8 Tuple3 (com.evolvedbinary.j8fu.tuple.Tuple3)7 DocumentImpl (org.exist.dom.persistent.DocumentImpl)7 LockedDocument (org.exist.dom.persistent.LockedDocument)7 Lock (org.exist.storage.lock.Lock)7 LockException (org.exist.util.LockException)7 Path (java.nio.file.Path)6 BiFunction (java.util.function.BiFunction)6 LockMode (org.exist.storage.lock.Lock.LockMode)6 EasyMock.createStrictControl (org.easymock.EasyMock.createStrictControl)5 EasyMock.expect (org.easymock.EasyMock.expect)5 IMocksControl (org.easymock.IMocksControl)5 PermissionDeniedException (org.exist.security.PermissionDeniedException)5 FluentBrokerAPI.uri (org.exist.storage.FluentBrokerAPI.uri)5 Assert.assertEquals (org.junit.Assert.assertEquals)5