Search in sources :

Example 1 with DatasetLocalResourceFactory

use of org.apache.asterix.transaction.management.resource.DatasetLocalResourceFactory in project asterixdb by apache.

the class MetadataBootstrap method enlistMetadataDataset.

/**
     * Enlist a metadata index so it is available for metadata operations
     * should be performed upon bootstrapping
     *
     * @param index
     * @throws HyracksDataException
     */
public static void enlistMetadataDataset(INCServiceContext ncServiceCtx, IMetadataIndex index) throws HyracksDataException {
    ClusterPartition metadataPartition = appContext.getMetadataProperties().getMetadataPartition();
    int metadataDeviceId = metadataPartition.getIODeviceNum();
    String metadataPartitionPath = StoragePathUtil.prepareStoragePartitionPath(ClusterProperties.INSTANCE.getStorageDirectoryName(), metadataPartition.getPartitionId());
    String resourceName = metadataPartitionPath + File.separator + index.getFileNameRelativePath();
    FileReference file = ioManager.getFileReference(metadataDeviceId, resourceName);
    index.setFile(file);
    ITypeTraits[] typeTraits = index.getTypeTraits();
    IBinaryComparatorFactory[] cmpFactories = index.getKeyBinaryComparatorFactory();
    int[] bloomFilterKeyFields = index.getBloomFilterKeyFields();
    // opTrackerProvider and ioOpCallbackFactory should both be acquired through IStorageManager
    // We are unable to do this since IStorageManager needs a dataset to determine the appropriate
    // objects
    ILSMOperationTrackerFactory opTrackerFactory = index.isPrimaryIndex() ? new PrimaryIndexOperationTrackerFactory(index.getDatasetId().getId()) : new SecondaryIndexOperationTrackerFactory(index.getDatasetId().getId());
    ILSMIOOperationCallbackFactory ioOpCallbackFactory = LSMBTreeIOOperationCallbackFactory.INSTANCE;
    IStorageComponentProvider storageComponentProvider = appContext.getStorageComponentProvider();
    if (isNewUniverse()) {
        LSMBTreeLocalResourceFactory lsmBtreeFactory = new LSMBTreeLocalResourceFactory(storageComponentProvider.getStorageManager(), typeTraits, cmpFactories, null, null, null, opTrackerFactory, ioOpCallbackFactory, storageComponentProvider.getMetadataPageManagerFactory(), new AsterixVirtualBufferCacheProvider(index.getDatasetId().getId()), storageComponentProvider.getIoOperationSchedulerProvider(), appContext.getMetadataMergePolicyFactory(), GlobalConfig.DEFAULT_COMPACTION_POLICY_PROPERTIES, true, bloomFilterKeyFields, appContext.getBloomFilterFalsePositiveRate(), true, null);
        DatasetLocalResourceFactory dsLocalResourceFactory = new DatasetLocalResourceFactory(index.getDatasetId().getId(), lsmBtreeFactory);
        // TODO(amoudi) Creating the index should be done through the same code path as other indexes
        // This is to be done by having a metadata dataset associated with each index
        IIndexBuilder indexBuilder = new IndexBuilder(ncServiceCtx, storageComponentProvider.getStorageManager(), index::getResourceId, file, dsLocalResourceFactory, true);
        indexBuilder.build();
    } else {
        final LocalResource resource = localResourceRepository.get(file.getRelativePath());
        if (resource == null) {
            throw new HyracksDataException("Could not find required metadata indexes. Please delete " + appContext.getMetadataProperties().getTransactionLogDirs().get(appContext.getTransactionSubsystem().getId()) + " to intialize as a new instance. (WARNING: all data will be lost.)");
        }
        // Why do we care about metadata dataset's resource ids? why not assign them ids similar to other resources?
        if (index.getResourceId() != resource.getId()) {
            throw new HyracksDataException("Resource Id doesn't match expected metadata index resource id");
        }
        IndexDataflowHelper indexHelper = new IndexDataflowHelper(ncServiceCtx, storageComponentProvider.getStorageManager(), file);
        // Opening the index through the helper will ensure it gets instantiated
        indexHelper.open();
        indexHelper.close();
    }
}
Also used : DatasetLocalResourceFactory(org.apache.asterix.transaction.management.resource.DatasetLocalResourceFactory) ILSMIOOperationCallbackFactory(org.apache.hyracks.storage.am.lsm.common.api.ILSMIOOperationCallbackFactory) IStorageComponentProvider(org.apache.asterix.common.context.IStorageComponentProvider) ITypeTraits(org.apache.hyracks.api.dataflow.value.ITypeTraits) IBinaryComparatorFactory(org.apache.hyracks.api.dataflow.value.IBinaryComparatorFactory) AsterixVirtualBufferCacheProvider(org.apache.asterix.common.context.AsterixVirtualBufferCacheProvider) ILSMOperationTrackerFactory(org.apache.hyracks.storage.am.lsm.common.api.ILSMOperationTrackerFactory) HyracksDataException(org.apache.hyracks.api.exceptions.HyracksDataException) IIndexBuilder(org.apache.hyracks.storage.am.common.api.IIndexBuilder) IndexBuilder(org.apache.hyracks.storage.am.common.build.IndexBuilder) LocalResource(org.apache.hyracks.storage.common.LocalResource) SecondaryIndexOperationTrackerFactory(org.apache.asterix.transaction.management.opcallbacks.SecondaryIndexOperationTrackerFactory) IIndexBuilder(org.apache.hyracks.storage.am.common.api.IIndexBuilder) LSMBTreeLocalResourceFactory(org.apache.hyracks.storage.am.lsm.btree.dataflow.LSMBTreeLocalResourceFactory) FileReference(org.apache.hyracks.api.io.FileReference) IndexDataflowHelper(org.apache.hyracks.storage.am.common.dataflow.IndexDataflowHelper) PrimaryIndexOperationTrackerFactory(org.apache.asterix.transaction.management.opcallbacks.PrimaryIndexOperationTrackerFactory) ClusterPartition(org.apache.asterix.common.cluster.ClusterPartition)

Example 2 with DatasetLocalResourceFactory

use of org.apache.asterix.transaction.management.resource.DatasetLocalResourceFactory in project asterixdb by apache.

the class Dataset method getResourceFactory.

/**
     * Create the index dataflow helper factory for a particular index on the dataset
     *
     * @param mdProvider
     *            metadata provider to get metadata information, components, and runtimes
     * @param index
     *            the index to get the dataflow helper factory for
     * @param recordType
     *            the record type for the dataset
     * @param metaType
     *            the meta type for the dataset
     * @param mergePolicyFactory
     *            the merge policy factory of the dataset
     * @param mergePolicyProperties
     *            the merge policy properties for the dataset
     * @return indexDataflowHelperFactory
     *         an instance of {@link org.apache.hyracks.storage.am.common.dataflow.IIndexDataflowHelperFactory}
     * @throws AlgebricksException
     *             if dataflow helper factory could not be created
     */
public IResourceFactory getResourceFactory(MetadataProvider mdProvider, Index index, ARecordType recordType, ARecordType metaType, ILSMMergePolicyFactory mergePolicyFactory, Map<String, String> mergePolicyProperties) throws AlgebricksException {
    ITypeTraits[] filterTypeTraits = DatasetUtil.computeFilterTypeTraits(this, recordType);
    IBinaryComparatorFactory[] filterCmpFactories = DatasetUtil.computeFilterBinaryComparatorFactories(this, recordType, mdProvider.getStorageComponentProvider().getComparatorFactoryProvider());
    IResourceFactory resourceFactory;
    switch(index.getIndexType()) {
        case BTREE:
            resourceFactory = bTreeResourceFactoryProvider.getResourceFactory(mdProvider, this, index, recordType, metaType, mergePolicyFactory, mergePolicyProperties, filterTypeTraits, filterCmpFactories);
            break;
        case RTREE:
            resourceFactory = rTreeResourceFactoryProvider.getResourceFactory(mdProvider, this, index, recordType, metaType, mergePolicyFactory, mergePolicyProperties, filterTypeTraits, filterCmpFactories);
            break;
        case LENGTH_PARTITIONED_NGRAM_INVIX:
        case LENGTH_PARTITIONED_WORD_INVIX:
        case SINGLE_PARTITION_NGRAM_INVIX:
        case SINGLE_PARTITION_WORD_INVIX:
            resourceFactory = invertedIndexResourceFactoryProvider.getResourceFactory(mdProvider, this, index, recordType, metaType, mergePolicyFactory, mergePolicyProperties, filterTypeTraits, filterCmpFactories);
            break;
        default:
            throw new CompilationException(ErrorCode.COMPILATION_UNKNOWN_INDEX_TYPE, index.getIndexType().toString());
    }
    return new DatasetLocalResourceFactory(datasetId, resourceFactory);
}
Also used : CompilationException(org.apache.asterix.common.exceptions.CompilationException) DatasetLocalResourceFactory(org.apache.asterix.transaction.management.resource.DatasetLocalResourceFactory) ITypeTraits(org.apache.hyracks.api.dataflow.value.ITypeTraits) IBinaryComparatorFactory(org.apache.hyracks.api.dataflow.value.IBinaryComparatorFactory) IResourceFactory(org.apache.hyracks.storage.common.IResourceFactory)

Aggregations

DatasetLocalResourceFactory (org.apache.asterix.transaction.management.resource.DatasetLocalResourceFactory)2 IBinaryComparatorFactory (org.apache.hyracks.api.dataflow.value.IBinaryComparatorFactory)2 ITypeTraits (org.apache.hyracks.api.dataflow.value.ITypeTraits)2 ClusterPartition (org.apache.asterix.common.cluster.ClusterPartition)1 AsterixVirtualBufferCacheProvider (org.apache.asterix.common.context.AsterixVirtualBufferCacheProvider)1 IStorageComponentProvider (org.apache.asterix.common.context.IStorageComponentProvider)1 CompilationException (org.apache.asterix.common.exceptions.CompilationException)1 PrimaryIndexOperationTrackerFactory (org.apache.asterix.transaction.management.opcallbacks.PrimaryIndexOperationTrackerFactory)1 SecondaryIndexOperationTrackerFactory (org.apache.asterix.transaction.management.opcallbacks.SecondaryIndexOperationTrackerFactory)1 HyracksDataException (org.apache.hyracks.api.exceptions.HyracksDataException)1 FileReference (org.apache.hyracks.api.io.FileReference)1 IIndexBuilder (org.apache.hyracks.storage.am.common.api.IIndexBuilder)1 IndexBuilder (org.apache.hyracks.storage.am.common.build.IndexBuilder)1 IndexDataflowHelper (org.apache.hyracks.storage.am.common.dataflow.IndexDataflowHelper)1 LSMBTreeLocalResourceFactory (org.apache.hyracks.storage.am.lsm.btree.dataflow.LSMBTreeLocalResourceFactory)1 ILSMIOOperationCallbackFactory (org.apache.hyracks.storage.am.lsm.common.api.ILSMIOOperationCallbackFactory)1 ILSMOperationTrackerFactory (org.apache.hyracks.storage.am.lsm.common.api.ILSMOperationTrackerFactory)1 IResourceFactory (org.apache.hyracks.storage.common.IResourceFactory)1 LocalResource (org.apache.hyracks.storage.common.LocalResource)1