Examples with LocalPoliciesResources org.apache.pulsar.broker.resources.LocalPoliciesResources used on opensource projects

Search in sources :

Example 1 with LocalPoliciesResources

use of org.apache.pulsar.broker.resources.LocalPoliciesResources in project pulsar by yahoo.

the class BrokerService method getManagedLedgerConfig.

public CompletableFuture<ManagedLedgerConfig> getManagedLedgerConfig(TopicName topicName) {
    NamespaceName namespace = topicName.getNamespaceObject();
    ServiceConfiguration serviceConfig = pulsar.getConfiguration();
    NamespaceResources nsr = pulsar.getPulsarResources().getNamespaceResources();
    LocalPoliciesResources lpr = pulsar.getPulsarResources().getLocalPolicies();
    return nsr.getPoliciesAsync(namespace).thenCombine(lpr.getLocalPoliciesAsync(namespace), (policies, localPolicies) -> {
        PersistencePolicies persistencePolicies = null;
        RetentionPolicies retentionPolicies = null;
        OffloadPoliciesImpl topicLevelOffloadPolicies = null;
        if (pulsar.getConfig().isTopicLevelPoliciesEnabled()) {
            try {
                TopicPolicies topicPolicies = pulsar.getTopicPoliciesService().getTopicPolicies(topicName);
                if (topicPolicies != null) {
                    persistencePolicies = topicPolicies.getPersistence();
                    retentionPolicies = topicPolicies.getRetentionPolicies();
                    topicLevelOffloadPolicies = topicPolicies.getOffloadPolicies();
                }
            } catch (BrokerServiceException.TopicPoliciesCacheNotInitException e) {
                log.debug("Topic {} policies have not been initialized yet.", topicName);
            }
        }
        if (persistencePolicies == null) {
            persistencePolicies = policies.map(p -> p.persistence).orElseGet(() -> new PersistencePolicies(serviceConfig.getManagedLedgerDefaultEnsembleSize(), serviceConfig.getManagedLedgerDefaultWriteQuorum(), serviceConfig.getManagedLedgerDefaultAckQuorum(), serviceConfig.getManagedLedgerDefaultMarkDeleteRateLimit()));
        }
        if (retentionPolicies == null) {
            retentionPolicies = policies.map(p -> p.retention_policies).orElseGet(() -> new RetentionPolicies(serviceConfig.getDefaultRetentionTimeInMinutes(), serviceConfig.getDefaultRetentionSizeInMB()));
        }
        ManagedLedgerConfig managedLedgerConfig = new ManagedLedgerConfig();
        managedLedgerConfig.setEnsembleSize(persistencePolicies.getBookkeeperEnsemble());
        managedLedgerConfig.setWriteQuorumSize(persistencePolicies.getBookkeeperWriteQuorum());
        managedLedgerConfig.setAckQuorumSize(persistencePolicies.getBookkeeperAckQuorum());
        if (serviceConfig.isStrictBookieAffinityEnabled()) {
            managedLedgerConfig.setBookKeeperEnsemblePlacementPolicyClassName(IsolatedBookieEnsemblePlacementPolicy.class);
            if (localPolicies.isPresent() && localPolicies.get().bookieAffinityGroup != null) {
                Map<String, Object> properties = Maps.newHashMap();
                properties.put(IsolatedBookieEnsemblePlacementPolicy.ISOLATION_BOOKIE_GROUPS, localPolicies.get().bookieAffinityGroup.getBookkeeperAffinityGroupPrimary());
                properties.put(IsolatedBookieEnsemblePlacementPolicy.SECONDARY_ISOLATION_BOOKIE_GROUPS, localPolicies.get().bookieAffinityGroup.getBookkeeperAffinityGroupSecondary());
                managedLedgerConfig.setBookKeeperEnsemblePlacementPolicyProperties(properties);
            } else if (isSystemTopic(topicName)) {
                Map<String, Object> properties = Maps.newHashMap();
                properties.put(IsolatedBookieEnsemblePlacementPolicy.ISOLATION_BOOKIE_GROUPS, "*");
                properties.put(IsolatedBookieEnsemblePlacementPolicy.SECONDARY_ISOLATION_BOOKIE_GROUPS, "*");
                managedLedgerConfig.setBookKeeperEnsemblePlacementPolicyProperties(properties);
            } else {
                Map<String, Object> properties = Maps.newHashMap();
                properties.put(IsolatedBookieEnsemblePlacementPolicy.ISOLATION_BOOKIE_GROUPS, "");
                properties.put(IsolatedBookieEnsemblePlacementPolicy.SECONDARY_ISOLATION_BOOKIE_GROUPS, "");
                managedLedgerConfig.setBookKeeperEnsemblePlacementPolicyProperties(properties);
            }
        } else {
            if (localPolicies.isPresent() && localPolicies.get().bookieAffinityGroup != null) {
                managedLedgerConfig.setBookKeeperEnsemblePlacementPolicyClassName(IsolatedBookieEnsemblePlacementPolicy.class);
                Map<String, Object> properties = Maps.newHashMap();
                properties.put(IsolatedBookieEnsemblePlacementPolicy.ISOLATION_BOOKIE_GROUPS, localPolicies.get().bookieAffinityGroup.getBookkeeperAffinityGroupPrimary());
                properties.put(IsolatedBookieEnsemblePlacementPolicy.SECONDARY_ISOLATION_BOOKIE_GROUPS, localPolicies.get().bookieAffinityGroup.getBookkeeperAffinityGroupSecondary());
                managedLedgerConfig.setBookKeeperEnsemblePlacementPolicyProperties(properties);
            }
        }
        managedLedgerConfig.setThrottleMarkDelete(persistencePolicies.getManagedLedgerMaxMarkDeleteRate());
        managedLedgerConfig.setDigestType(serviceConfig.getManagedLedgerDigestType());
        managedLedgerConfig.setPassword(serviceConfig.getManagedLedgerPassword());
        managedLedgerConfig.setMaxUnackedRangesToPersist(serviceConfig.getManagedLedgerMaxUnackedRangesToPersist());
        managedLedgerConfig.setPersistentUnackedRangesWithMultipleEntriesEnabled(serviceConfig.isPersistentUnackedRangesWithMultipleEntriesEnabled());
        managedLedgerConfig.setMaxUnackedRangesToPersistInMetadataStore(serviceConfig.getManagedLedgerMaxUnackedRangesToPersistInMetadataStore());
        managedLedgerConfig.setMaxEntriesPerLedger(serviceConfig.getManagedLedgerMaxEntriesPerLedger());
        managedLedgerConfig.setMinimumRolloverTime(serviceConfig.getManagedLedgerMinLedgerRolloverTimeMinutes(), TimeUnit.MINUTES);
        managedLedgerConfig.setMaximumRolloverTime(serviceConfig.getManagedLedgerMaxLedgerRolloverTimeMinutes(), TimeUnit.MINUTES);
        managedLedgerConfig.setMaxSizePerLedgerMb(serviceConfig.getManagedLedgerMaxSizePerLedgerMbytes());
        managedLedgerConfig.setMetadataOperationsTimeoutSeconds(serviceConfig.getManagedLedgerMetadataOperationsTimeoutSeconds());
        managedLedgerConfig.setReadEntryTimeoutSeconds(serviceConfig.getManagedLedgerReadEntryTimeoutSeconds());
        managedLedgerConfig.setAddEntryTimeoutSeconds(serviceConfig.getManagedLedgerAddEntryTimeoutSeconds());
        managedLedgerConfig.setMetadataEnsembleSize(serviceConfig.getManagedLedgerDefaultEnsembleSize());
        managedLedgerConfig.setUnackedRangesOpenCacheSetEnabled(serviceConfig.isManagedLedgerUnackedRangesOpenCacheSetEnabled());
        managedLedgerConfig.setMetadataWriteQuorumSize(serviceConfig.getManagedLedgerDefaultWriteQuorum());
        managedLedgerConfig.setMetadataAckQuorumSize(serviceConfig.getManagedLedgerDefaultAckQuorum());
        managedLedgerConfig.setMetadataMaxEntriesPerLedger(serviceConfig.getManagedLedgerCursorMaxEntriesPerLedger());
        managedLedgerConfig.setLedgerRolloverTimeout(serviceConfig.getManagedLedgerCursorRolloverTimeInSeconds());
        managedLedgerConfig.setRetentionTime(retentionPolicies.getRetentionTimeInMinutes(), TimeUnit.MINUTES);
        managedLedgerConfig.setRetentionSizeInMB(retentionPolicies.getRetentionSizeInMB());
        managedLedgerConfig.setAutoSkipNonRecoverableData(serviceConfig.isAutoSkipNonRecoverableData());
        managedLedgerConfig.setLazyCursorRecovery(serviceConfig.isLazyCursorRecovery());
        managedLedgerConfig.setInactiveLedgerRollOverTime(serviceConfig.getManagedLedgerInactiveLedgerRolloverTimeSeconds(), TimeUnit.SECONDS);
        managedLedgerConfig.setCacheEvictionByMarkDeletedPosition(serviceConfig.isCacheEvictionByMarkDeletedPosition());
        OffloadPoliciesImpl nsLevelOffloadPolicies = (OffloadPoliciesImpl) policies.map(p -> p.offload_policies).orElse(null);
        OffloadPoliciesImpl offloadPolicies = OffloadPoliciesImpl.mergeConfiguration(topicLevelOffloadPolicies, OffloadPoliciesImpl.oldPoliciesCompatible(nsLevelOffloadPolicies, policies.orElse(null)), getPulsar().getConfig().getProperties());
        if (NamespaceService.isSystemServiceNamespace(namespace.toString())) {
            managedLedgerConfig.setLedgerOffloader(NullLedgerOffloader.INSTANCE);
        } else {
            if (topicLevelOffloadPolicies != null) {
                try {
                    LedgerOffloader topicLevelLedgerOffLoader = pulsar().createManagedLedgerOffloader(offloadPolicies);
                    managedLedgerConfig.setLedgerOffloader(topicLevelLedgerOffLoader);
                } catch (PulsarServerException e) {
                    throw new RuntimeException(e);
                }
            } else {
                // If the topic level policy is null, use the namespace level
                managedLedgerConfig.setLedgerOffloader(pulsar.getManagedLedgerOffloader(namespace, offloadPolicies));
            }
        }
        managedLedgerConfig.setDeletionAtBatchIndexLevelEnabled(serviceConfig.isAcknowledgmentAtBatchIndexLevelEnabled());
        managedLedgerConfig.setNewEntriesCheckDelayInMillis(serviceConfig.getManagedLedgerNewEntriesCheckDelayInMillis());
        return managedLedgerConfig;
    });
}
Also used : NamespaceResources(org.apache.pulsar.broker.resources.NamespaceResources) RetentionPolicies(org.apache.pulsar.common.policies.data.RetentionPolicies) PulsarServerException(org.apache.pulsar.broker.PulsarServerException) PersistencePolicies(org.apache.pulsar.common.policies.data.PersistencePolicies) TopicPolicies(org.apache.pulsar.common.policies.data.TopicPolicies) OffloadPoliciesImpl(org.apache.pulsar.common.policies.data.OffloadPoliciesImpl) NamespaceName(org.apache.pulsar.common.naming.NamespaceName) ServiceConfiguration(org.apache.pulsar.broker.ServiceConfiguration) LocalPoliciesResources(org.apache.pulsar.broker.resources.LocalPoliciesResources) LedgerOffloader(org.apache.bookkeeper.mledger.LedgerOffloader) NullLedgerOffloader(org.apache.bookkeeper.mledger.impl.NullLedgerOffloader) ManagedLedgerConfig(org.apache.bookkeeper.mledger.ManagedLedgerConfig) Map(java.util.Map) ConcurrentOpenHashMap(org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap) ImmutableMap(com.google.common.collect.ImmutableMap) HashMap(java.util.HashMap)

Example 2 with LocalPoliciesResources

use of org.apache.pulsar.broker.resources.LocalPoliciesResources in project incubator-pulsar by apache.

the class BrokerService method getManagedLedgerConfig.

public CompletableFuture<ManagedLedgerConfig> getManagedLedgerConfig(TopicName topicName) {
    NamespaceName namespace = topicName.getNamespaceObject();
    ServiceConfiguration serviceConfig = pulsar.getConfiguration();
    NamespaceResources nsr = pulsar.getPulsarResources().getNamespaceResources();
    LocalPoliciesResources lpr = pulsar.getPulsarResources().getLocalPolicies();
    return nsr.getPoliciesAsync(namespace).thenCombine(lpr.getLocalPoliciesAsync(namespace), (policies, localPolicies) -> {
        PersistencePolicies persistencePolicies = null;
        RetentionPolicies retentionPolicies = null;
        OffloadPoliciesImpl topicLevelOffloadPolicies = null;
        if (pulsar.getConfig().isTopicLevelPoliciesEnabled()) {
            try {
                TopicPolicies topicPolicies = pulsar.getTopicPoliciesService().getTopicPolicies(topicName);
                if (topicPolicies != null) {
                    persistencePolicies = topicPolicies.getPersistence();
                    retentionPolicies = topicPolicies.getRetentionPolicies();
                    topicLevelOffloadPolicies = topicPolicies.getOffloadPolicies();
                }
            } catch (BrokerServiceException.TopicPoliciesCacheNotInitException e) {
                log.debug("Topic {} policies have not been initialized yet.", topicName);
            }
        }
        if (persistencePolicies == null) {
            persistencePolicies = policies.map(p -> p.persistence).orElseGet(() -> new PersistencePolicies(serviceConfig.getManagedLedgerDefaultEnsembleSize(), serviceConfig.getManagedLedgerDefaultWriteQuorum(), serviceConfig.getManagedLedgerDefaultAckQuorum(), serviceConfig.getManagedLedgerDefaultMarkDeleteRateLimit()));
        }
        if (retentionPolicies == null) {
            retentionPolicies = policies.map(p -> p.retention_policies).orElseGet(() -> new RetentionPolicies(serviceConfig.getDefaultRetentionTimeInMinutes(), serviceConfig.getDefaultRetentionSizeInMB()));
        }
        ManagedLedgerConfig managedLedgerConfig = new ManagedLedgerConfig();
        managedLedgerConfig.setEnsembleSize(persistencePolicies.getBookkeeperEnsemble());
        managedLedgerConfig.setWriteQuorumSize(persistencePolicies.getBookkeeperWriteQuorum());
        managedLedgerConfig.setAckQuorumSize(persistencePolicies.getBookkeeperAckQuorum());
        if (serviceConfig.isStrictBookieAffinityEnabled()) {
            managedLedgerConfig.setBookKeeperEnsemblePlacementPolicyClassName(IsolatedBookieEnsemblePlacementPolicy.class);
            if (localPolicies.isPresent() && localPolicies.get().bookieAffinityGroup != null) {
                Map<String, Object> properties = Maps.newHashMap();
                properties.put(IsolatedBookieEnsemblePlacementPolicy.ISOLATION_BOOKIE_GROUPS, localPolicies.get().bookieAffinityGroup.getBookkeeperAffinityGroupPrimary());
                properties.put(IsolatedBookieEnsemblePlacementPolicy.SECONDARY_ISOLATION_BOOKIE_GROUPS, localPolicies.get().bookieAffinityGroup.getBookkeeperAffinityGroupSecondary());
                managedLedgerConfig.setBookKeeperEnsemblePlacementPolicyProperties(properties);
            } else if (isSystemTopic(topicName)) {
                Map<String, Object> properties = Maps.newHashMap();
                properties.put(IsolatedBookieEnsemblePlacementPolicy.ISOLATION_BOOKIE_GROUPS, "*");
                properties.put(IsolatedBookieEnsemblePlacementPolicy.SECONDARY_ISOLATION_BOOKIE_GROUPS, "*");
                managedLedgerConfig.setBookKeeperEnsemblePlacementPolicyProperties(properties);
            } else {
                Map<String, Object> properties = Maps.newHashMap();
                properties.put(IsolatedBookieEnsemblePlacementPolicy.ISOLATION_BOOKIE_GROUPS, "");
                properties.put(IsolatedBookieEnsemblePlacementPolicy.SECONDARY_ISOLATION_BOOKIE_GROUPS, "");
                managedLedgerConfig.setBookKeeperEnsemblePlacementPolicyProperties(properties);
            }
        } else {
            if (localPolicies.isPresent() && localPolicies.get().bookieAffinityGroup != null) {
                managedLedgerConfig.setBookKeeperEnsemblePlacementPolicyClassName(IsolatedBookieEnsemblePlacementPolicy.class);
                Map<String, Object> properties = Maps.newHashMap();
                properties.put(IsolatedBookieEnsemblePlacementPolicy.ISOLATION_BOOKIE_GROUPS, localPolicies.get().bookieAffinityGroup.getBookkeeperAffinityGroupPrimary());
                properties.put(IsolatedBookieEnsemblePlacementPolicy.SECONDARY_ISOLATION_BOOKIE_GROUPS, localPolicies.get().bookieAffinityGroup.getBookkeeperAffinityGroupSecondary());
                managedLedgerConfig.setBookKeeperEnsemblePlacementPolicyProperties(properties);
            }
        }
        managedLedgerConfig.setThrottleMarkDelete(persistencePolicies.getManagedLedgerMaxMarkDeleteRate());
        managedLedgerConfig.setDigestType(serviceConfig.getManagedLedgerDigestType());
        managedLedgerConfig.setPassword(serviceConfig.getManagedLedgerPassword());
        managedLedgerConfig.setMaxUnackedRangesToPersist(serviceConfig.getManagedLedgerMaxUnackedRangesToPersist());
        managedLedgerConfig.setPersistentUnackedRangesWithMultipleEntriesEnabled(serviceConfig.isPersistentUnackedRangesWithMultipleEntriesEnabled());
        managedLedgerConfig.setMaxUnackedRangesToPersistInMetadataStore(serviceConfig.getManagedLedgerMaxUnackedRangesToPersistInMetadataStore());
        managedLedgerConfig.setMaxEntriesPerLedger(serviceConfig.getManagedLedgerMaxEntriesPerLedger());
        managedLedgerConfig.setMinimumRolloverTime(serviceConfig.getManagedLedgerMinLedgerRolloverTimeMinutes(), TimeUnit.MINUTES);
        managedLedgerConfig.setMaximumRolloverTime(serviceConfig.getManagedLedgerMaxLedgerRolloverTimeMinutes(), TimeUnit.MINUTES);
        managedLedgerConfig.setMaxSizePerLedgerMb(serviceConfig.getManagedLedgerMaxSizePerLedgerMbytes());
        managedLedgerConfig.setMetadataOperationsTimeoutSeconds(serviceConfig.getManagedLedgerMetadataOperationsTimeoutSeconds());
        managedLedgerConfig.setReadEntryTimeoutSeconds(serviceConfig.getManagedLedgerReadEntryTimeoutSeconds());
        managedLedgerConfig.setAddEntryTimeoutSeconds(serviceConfig.getManagedLedgerAddEntryTimeoutSeconds());
        managedLedgerConfig.setMetadataEnsembleSize(serviceConfig.getManagedLedgerDefaultEnsembleSize());
        managedLedgerConfig.setUnackedRangesOpenCacheSetEnabled(serviceConfig.isManagedLedgerUnackedRangesOpenCacheSetEnabled());
        managedLedgerConfig.setMetadataWriteQuorumSize(serviceConfig.getManagedLedgerDefaultWriteQuorum());
        managedLedgerConfig.setMetadataAckQuorumSize(serviceConfig.getManagedLedgerDefaultAckQuorum());
        managedLedgerConfig.setMetadataMaxEntriesPerLedger(serviceConfig.getManagedLedgerCursorMaxEntriesPerLedger());
        managedLedgerConfig.setLedgerRolloverTimeout(serviceConfig.getManagedLedgerCursorRolloverTimeInSeconds());
        managedLedgerConfig.setRetentionTime(retentionPolicies.getRetentionTimeInMinutes(), TimeUnit.MINUTES);
        managedLedgerConfig.setRetentionSizeInMB(retentionPolicies.getRetentionSizeInMB());
        managedLedgerConfig.setAutoSkipNonRecoverableData(serviceConfig.isAutoSkipNonRecoverableData());
        managedLedgerConfig.setLazyCursorRecovery(serviceConfig.isLazyCursorRecovery());
        managedLedgerConfig.setInactiveLedgerRollOverTime(serviceConfig.getManagedLedgerInactiveLedgerRolloverTimeSeconds(), TimeUnit.SECONDS);
        managedLedgerConfig.setCacheEvictionByMarkDeletedPosition(serviceConfig.isCacheEvictionByMarkDeletedPosition());
        OffloadPoliciesImpl nsLevelOffloadPolicies = (OffloadPoliciesImpl) policies.map(p -> p.offload_policies).orElse(null);
        OffloadPoliciesImpl offloadPolicies = OffloadPoliciesImpl.mergeConfiguration(topicLevelOffloadPolicies, OffloadPoliciesImpl.oldPoliciesCompatible(nsLevelOffloadPolicies, policies.orElse(null)), getPulsar().getConfig().getProperties());
        if (NamespaceService.isSystemServiceNamespace(namespace.toString())) {
            managedLedgerConfig.setLedgerOffloader(NullLedgerOffloader.INSTANCE);
        } else {
            if (topicLevelOffloadPolicies != null) {
                try {
                    LedgerOffloader topicLevelLedgerOffLoader = pulsar().createManagedLedgerOffloader(offloadPolicies);
                    managedLedgerConfig.setLedgerOffloader(topicLevelLedgerOffLoader);
                } catch (PulsarServerException e) {
                    throw new RuntimeException(e);
                }
            } else {
                // If the topic level policy is null, use the namespace level
                managedLedgerConfig.setLedgerOffloader(pulsar.getManagedLedgerOffloader(namespace, offloadPolicies));
            }
        }
        managedLedgerConfig.setDeletionAtBatchIndexLevelEnabled(serviceConfig.isAcknowledgmentAtBatchIndexLevelEnabled());
        managedLedgerConfig.setNewEntriesCheckDelayInMillis(serviceConfig.getManagedLedgerNewEntriesCheckDelayInMillis());
        return managedLedgerConfig;
    });
}
Also used : NamespaceResources(org.apache.pulsar.broker.resources.NamespaceResources) RetentionPolicies(org.apache.pulsar.common.policies.data.RetentionPolicies) PulsarServerException(org.apache.pulsar.broker.PulsarServerException) PersistencePolicies(org.apache.pulsar.common.policies.data.PersistencePolicies) TopicPolicies(org.apache.pulsar.common.policies.data.TopicPolicies) OffloadPoliciesImpl(org.apache.pulsar.common.policies.data.OffloadPoliciesImpl) NamespaceName(org.apache.pulsar.common.naming.NamespaceName) ServiceConfiguration(org.apache.pulsar.broker.ServiceConfiguration) LocalPoliciesResources(org.apache.pulsar.broker.resources.LocalPoliciesResources) LedgerOffloader(org.apache.bookkeeper.mledger.LedgerOffloader) NullLedgerOffloader(org.apache.bookkeeper.mledger.impl.NullLedgerOffloader) ManagedLedgerConfig(org.apache.bookkeeper.mledger.ManagedLedgerConfig) Map(java.util.Map) ConcurrentOpenHashMap(org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap) ImmutableMap(com.google.common.collect.ImmutableMap) HashMap(java.util.HashMap)

Example 3 with LocalPoliciesResources

use of org.apache.pulsar.broker.resources.LocalPoliciesResources in project pulsar by apache.

the class BrokerService method getManagedLedgerConfig.

public CompletableFuture<ManagedLedgerConfig> getManagedLedgerConfig(TopicName topicName) {
    NamespaceName namespace = topicName.getNamespaceObject();
    ServiceConfiguration serviceConfig = pulsar.getConfiguration();
    NamespaceResources nsr = pulsar.getPulsarResources().getNamespaceResources();
    LocalPoliciesResources lpr = pulsar.getPulsarResources().getLocalPolicies();
    return nsr.getPoliciesAsync(namespace).thenCombine(lpr.getLocalPoliciesAsync(namespace), (policies, localPolicies) -> {
        PersistencePolicies persistencePolicies = null;
        RetentionPolicies retentionPolicies = null;
        OffloadPoliciesImpl topicLevelOffloadPolicies = null;
        if (pulsar.getConfig().isTopicLevelPoliciesEnabled()) {
            try {
                TopicPolicies topicPolicies = pulsar.getTopicPoliciesService().getTopicPolicies(topicName);
                if (topicPolicies != null) {
                    persistencePolicies = topicPolicies.getPersistence();
                    retentionPolicies = topicPolicies.getRetentionPolicies();
                    topicLevelOffloadPolicies = topicPolicies.getOffloadPolicies();
                }
            } catch (BrokerServiceException.TopicPoliciesCacheNotInitException e) {
                log.debug("Topic {} policies have not been initialized yet.", topicName);
            }
        }
        if (persistencePolicies == null) {
            persistencePolicies = policies.map(p -> p.persistence).orElseGet(() -> new PersistencePolicies(serviceConfig.getManagedLedgerDefaultEnsembleSize(), serviceConfig.getManagedLedgerDefaultWriteQuorum(), serviceConfig.getManagedLedgerDefaultAckQuorum(), serviceConfig.getManagedLedgerDefaultMarkDeleteRateLimit()));
        }
        if (retentionPolicies == null) {
            retentionPolicies = policies.map(p -> p.retention_policies).orElseGet(() -> new RetentionPolicies(serviceConfig.getDefaultRetentionTimeInMinutes(), serviceConfig.getDefaultRetentionSizeInMB()));
        }
        ManagedLedgerConfig managedLedgerConfig = new ManagedLedgerConfig();
        managedLedgerConfig.setEnsembleSize(persistencePolicies.getBookkeeperEnsemble());
        managedLedgerConfig.setWriteQuorumSize(persistencePolicies.getBookkeeperWriteQuorum());
        managedLedgerConfig.setAckQuorumSize(persistencePolicies.getBookkeeperAckQuorum());
        if (localPolicies.isPresent() && localPolicies.get().bookieAffinityGroup != null) {
            managedLedgerConfig.setBookKeeperEnsemblePlacementPolicyClassName(IsolatedBookieEnsemblePlacementPolicy.class);
            Map<String, Object> properties = Maps.newHashMap();
            properties.put(IsolatedBookieEnsemblePlacementPolicy.ISOLATION_BOOKIE_GROUPS, localPolicies.get().bookieAffinityGroup.getBookkeeperAffinityGroupPrimary());
            properties.put(IsolatedBookieEnsemblePlacementPolicy.SECONDARY_ISOLATION_BOOKIE_GROUPS, localPolicies.get().bookieAffinityGroup.getBookkeeperAffinityGroupSecondary());
            managedLedgerConfig.setBookKeeperEnsemblePlacementPolicyProperties(properties);
        }
        managedLedgerConfig.setThrottleMarkDelete(persistencePolicies.getManagedLedgerMaxMarkDeleteRate());
        managedLedgerConfig.setDigestType(serviceConfig.getManagedLedgerDigestType());
        managedLedgerConfig.setPassword(serviceConfig.getManagedLedgerPassword());
        managedLedgerConfig.setMaxUnackedRangesToPersist(serviceConfig.getManagedLedgerMaxUnackedRangesToPersist());
        managedLedgerConfig.setMaxUnackedRangesToPersistInZk(serviceConfig.getManagedLedgerMaxUnackedRangesToPersistInZooKeeper());
        managedLedgerConfig.setMaxEntriesPerLedger(serviceConfig.getManagedLedgerMaxEntriesPerLedger());
        managedLedgerConfig.setMinimumRolloverTime(serviceConfig.getManagedLedgerMinLedgerRolloverTimeMinutes(), TimeUnit.MINUTES);
        managedLedgerConfig.setMaximumRolloverTime(serviceConfig.getManagedLedgerMaxLedgerRolloverTimeMinutes(), TimeUnit.MINUTES);
        managedLedgerConfig.setMaxSizePerLedgerMb(serviceConfig.getManagedLedgerMaxSizePerLedgerMbytes());
        managedLedgerConfig.setMetadataOperationsTimeoutSeconds(serviceConfig.getManagedLedgerMetadataOperationsTimeoutSeconds());
        managedLedgerConfig.setReadEntryTimeoutSeconds(serviceConfig.getManagedLedgerReadEntryTimeoutSeconds());
        managedLedgerConfig.setAddEntryTimeoutSeconds(serviceConfig.getManagedLedgerAddEntryTimeoutSeconds());
        managedLedgerConfig.setMetadataEnsembleSize(serviceConfig.getManagedLedgerDefaultEnsembleSize());
        managedLedgerConfig.setUnackedRangesOpenCacheSetEnabled(serviceConfig.isManagedLedgerUnackedRangesOpenCacheSetEnabled());
        managedLedgerConfig.setMetadataWriteQuorumSize(serviceConfig.getManagedLedgerDefaultWriteQuorum());
        managedLedgerConfig.setMetadataAckQuorumSize(serviceConfig.getManagedLedgerDefaultAckQuorum());
        managedLedgerConfig.setMetadataMaxEntriesPerLedger(serviceConfig.getManagedLedgerCursorMaxEntriesPerLedger());
        managedLedgerConfig.setLedgerRolloverTimeout(serviceConfig.getManagedLedgerCursorRolloverTimeInSeconds());
        managedLedgerConfig.setRetentionTime(retentionPolicies.getRetentionTimeInMinutes(), TimeUnit.MINUTES);
        managedLedgerConfig.setRetentionSizeInMB(retentionPolicies.getRetentionSizeInMB());
        managedLedgerConfig.setAutoSkipNonRecoverableData(serviceConfig.isAutoSkipNonRecoverableData());
        managedLedgerConfig.setLazyCursorRecovery(serviceConfig.isLazyCursorRecovery());
        managedLedgerConfig.setInactiveLedgerRollOverTime(serviceConfig.getManagedLedgerInactiveLedgerRolloverTimeSeconds(), TimeUnit.SECONDS);
        OffloadPoliciesImpl nsLevelOffloadPolicies = (OffloadPoliciesImpl) policies.map(p -> p.offload_policies).orElse(null);
        OffloadPoliciesImpl offloadPolicies = OffloadPoliciesImpl.mergeConfiguration(topicLevelOffloadPolicies, OffloadPoliciesImpl.oldPoliciesCompatible(nsLevelOffloadPolicies, policies.orElse(null)), getPulsar().getConfig().getProperties());
        if (topicLevelOffloadPolicies != null) {
            try {
                LedgerOffloader topicLevelLedgerOffLoader = pulsar().createManagedLedgerOffloader(offloadPolicies);
                managedLedgerConfig.setLedgerOffloader(topicLevelLedgerOffLoader);
            } catch (PulsarServerException e) {
                throw new RuntimeException(e);
            }
        } else {
            // If the topic level policy is null, use the namespace level
            managedLedgerConfig.setLedgerOffloader(pulsar.getManagedLedgerOffloader(namespace, offloadPolicies));
        }
        managedLedgerConfig.setDeletionAtBatchIndexLevelEnabled(serviceConfig.isAcknowledgmentAtBatchIndexLevelEnabled());
        managedLedgerConfig.setNewEntriesCheckDelayInMillis(serviceConfig.getManagedLedgerNewEntriesCheckDelayInMillis());
        return managedLedgerConfig;
    });
}
Also used : NamespaceResources(org.apache.pulsar.broker.resources.NamespaceResources) RetentionPolicies(org.apache.pulsar.common.policies.data.RetentionPolicies) PulsarServerException(org.apache.pulsar.broker.PulsarServerException) PersistencePolicies(org.apache.pulsar.common.policies.data.PersistencePolicies) TopicPolicies(org.apache.pulsar.common.policies.data.TopicPolicies) OffloadPoliciesImpl(org.apache.pulsar.common.policies.data.OffloadPoliciesImpl) NamespaceName(org.apache.pulsar.common.naming.NamespaceName) ServiceConfiguration(org.apache.pulsar.broker.ServiceConfiguration) LocalPoliciesResources(org.apache.pulsar.broker.resources.LocalPoliciesResources) LedgerOffloader(org.apache.bookkeeper.mledger.LedgerOffloader) ManagedLedgerConfig(org.apache.bookkeeper.mledger.ManagedLedgerConfig)

Aggregations

LedgerOffloader (org.apache.bookkeeper.mledger.LedgerOffloader)3 ManagedLedgerConfig (org.apache.bookkeeper.mledger.ManagedLedgerConfig)3 PulsarServerException (org.apache.pulsar.broker.PulsarServerException)3 ServiceConfiguration (org.apache.pulsar.broker.ServiceConfiguration)3 LocalPoliciesResources (org.apache.pulsar.broker.resources.LocalPoliciesResources)3 NamespaceResources (org.apache.pulsar.broker.resources.NamespaceResources)3 NamespaceName (org.apache.pulsar.common.naming.NamespaceName)3 OffloadPoliciesImpl (org.apache.pulsar.common.policies.data.OffloadPoliciesImpl)3 PersistencePolicies (org.apache.pulsar.common.policies.data.PersistencePolicies)3 RetentionPolicies (org.apache.pulsar.common.policies.data.RetentionPolicies)3 TopicPolicies (org.apache.pulsar.common.policies.data.TopicPolicies)3 ImmutableMap (com.google.common.collect.ImmutableMap)2 HashMap (java.util.HashMap)2 Map (java.util.Map)2 NullLedgerOffloader (org.apache.bookkeeper.mledger.impl.NullLedgerOffloader)2 ConcurrentOpenHashMap (org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap)2
About Me
UseOf

Java Tutorials :

Simple Java RabbitMQ Example with Spring
Simple Springboot JPA Eclipeslink Example
Simple SpringBoot JPA Hibernate Example
Jackson JSON @JsonIgnore and @JsonIgnoreProperties Examples
Java Infinite recursion (StackOverflowError) Jackson solutions
Simple Java Jackson Serialize Objects to JSON and convert them back To Object
ElasticSearch 5 - Upload files using Java API in binary field Example
Java JAXB marshall unmarshall Examples from String and Stream
Java 8 forEach List and Map examples
ElasticSearch 5 Java API SearchRequestBuilder examples
Java ElasticSearch 5 examples with node index and mapping - running local
Convert String to int or Integer Java 8
Java 9 in action - JShell - Ubuntu
Java - removing invalid characters from a file name
Hello world!? or Hello Tutorial