Search in sources :

Example 1 with BatchFlow

use of com.torodb.concurrent.akka.BatchFlow in project torodb by torodb.

the class DefaultOplogApplier method createBatcherFlow.

/**
   * Creates a flow that batches and analyze a input of {@link AnalyzedOplogBatch remote jobs}.
   *
   * This flow tries to accummulate several remote jobs into a bigger one and does not emit until:
   * <ul>
   * <li>A maximum number of operations are batched</li>
   * <li>Or a maximum time has happen since the last emit</li>
   * <li>Or the recived job is not {@link AnalyzedOplogBatch#isReadyForMore()}</li>
   * </ul>
   *
   */
private Flow<OplogBatch, AnalyzedStreamElement, NotUsed> createBatcherFlow(ApplierContext context) {
    Predicate<OplogBatch> finishBatchPredicate = (OplogBatch rawBatch) -> !rawBatch.isReadyForMore();
    ToIntFunction<OplogBatch> costFunction = (rawBatch) -> rawBatch.count();
    Supplier<RawStreamElement> zeroFun = () -> RawStreamElement.INITIAL_ELEMENT;
    BiFunction<RawStreamElement, OplogBatch, RawStreamElement> acumFun = (streamElem, newBatch) -> streamElem.concat(newBatch);
    BatchAnalyzer batchAnalyzer = batchAnalyzerFactory.createBatchAnalyzer(context);
    return Flow.of(OplogBatch.class).via(new BatchFlow<>(batchLimits.maxSize, batchLimits.maxPeriod, finishBatchPredicate, costFunction, zeroFun, acumFun)).filter(rawElem -> rawElem.rawBatch != null && !rawElem.rawBatch.isEmpty()).map(rawElem -> {
        List<OplogOperation> rawOps = rawElem.rawBatch.getOps();
        List<AnalyzedOplogBatch> analyzed = batchAnalyzer.apply(rawOps);
        return new AnalyzedStreamElement(rawElem, analyzed);
    });
}
Also used : BatchAnalyzerFactory(com.torodb.mongodb.repl.oplogreplier.batch.BatchAnalyzer.BatchAnalyzerFactory) BiFunction(java.util.function.BiFunction) Flow(akka.stream.javadsl.Flow) Source(akka.stream.javadsl.Source) Supplier(com.google.common.base.Supplier) KillSwitch(akka.stream.KillSwitch) Materializer(akka.stream.Materializer) CompletableFuture(java.util.concurrent.CompletableFuture) UniqueKillSwitch(akka.stream.UniqueKillSwitch) OplogManager(com.torodb.mongodb.repl.OplogManager) BatchAnalyzer(com.torodb.mongodb.repl.oplogreplier.batch.BatchAnalyzer) Inject(javax.inject.Inject) ActorMaterializer(akka.stream.ActorMaterializer) Keep(akka.stream.javadsl.Keep) RunnableGraph(akka.stream.javadsl.RunnableGraph) OplogOperation(com.eightkdata.mongowp.server.api.oplog.OplogOperation) Empty(com.eightkdata.mongowp.server.api.tools.Empty) OplogManagerPersistException(com.torodb.mongodb.repl.OplogManager.OplogManagerPersistException) AnalyzedOplogBatchExecutor(com.torodb.mongodb.repl.oplogreplier.batch.AnalyzedOplogBatchExecutor) ConcurrentToolsFactory(com.torodb.core.concurrent.ConcurrentToolsFactory) AnalyzedOplogBatch(com.torodb.mongodb.repl.oplogreplier.batch.AnalyzedOplogBatch) OplogFetcher(com.torodb.mongodb.repl.oplogreplier.fetcher.OplogFetcher) Shutdowner(com.torodb.core.Shutdowner) Done(akka.Done) CancellationException(java.util.concurrent.CancellationException) FiniteDuration(scala.concurrent.duration.FiniteDuration) Predicate(java.util.function.Predicate) ToIntFunction(java.util.function.ToIntFunction) Sink(akka.stream.javadsl.Sink) Throwables(com.google.common.base.Throwables) CompletionException(java.util.concurrent.CompletionException) KillSwitches(akka.stream.KillSwitches) WriteOplogTransaction(com.torodb.mongodb.repl.OplogManager.WriteOplogTransaction) ExecutionContexts(akka.dispatch.ExecutionContexts) Pair(akka.japi.Pair) TimeUnit(java.util.concurrent.TimeUnit) Duration(scala.concurrent.duration.Duration) List(java.util.List) Logger(org.apache.logging.log4j.Logger) CompletionStage(java.util.concurrent.CompletionStage) NotUsed(akka.NotUsed) BatchFlow(com.torodb.concurrent.akka.BatchFlow) ActorSystem(akka.actor.ActorSystem) Optional(java.util.Optional) LogManager(org.apache.logging.log4j.LogManager) Await(scala.concurrent.Await) AnalyzedOplogBatch(com.torodb.mongodb.repl.oplogreplier.batch.AnalyzedOplogBatch) BatchAnalyzer(com.torodb.mongodb.repl.oplogreplier.batch.BatchAnalyzer) AnalyzedOplogBatch(com.torodb.mongodb.repl.oplogreplier.batch.AnalyzedOplogBatch) OplogOperation(com.eightkdata.mongowp.server.api.oplog.OplogOperation)

Aggregations

Done (akka.Done)1 NotUsed (akka.NotUsed)1 ActorSystem (akka.actor.ActorSystem)1 ExecutionContexts (akka.dispatch.ExecutionContexts)1 Pair (akka.japi.Pair)1 ActorMaterializer (akka.stream.ActorMaterializer)1 KillSwitch (akka.stream.KillSwitch)1 KillSwitches (akka.stream.KillSwitches)1 Materializer (akka.stream.Materializer)1 UniqueKillSwitch (akka.stream.UniqueKillSwitch)1 Flow (akka.stream.javadsl.Flow)1 Keep (akka.stream.javadsl.Keep)1 RunnableGraph (akka.stream.javadsl.RunnableGraph)1 Sink (akka.stream.javadsl.Sink)1 Source (akka.stream.javadsl.Source)1 OplogOperation (com.eightkdata.mongowp.server.api.oplog.OplogOperation)1 Empty (com.eightkdata.mongowp.server.api.tools.Empty)1 Supplier (com.google.common.base.Supplier)1 Throwables (com.google.common.base.Throwables)1 BatchFlow (com.torodb.concurrent.akka.BatchFlow)1