Examples with TupleCursor mondrian.calc.TupleCursor used on opensource projects

Example 1 with TupleCursor

use of mondrian.calc.TupleCursor in project mondrian by pentaho.

the class CrossJoinFunDef method nonEmptyList.

/**
 * This is the entry point to the crossjoin non-empty optimizer code.
 *
 * <p>
 * What one wants to determine is for each individual Member of the input parameter list, a 'List-Member', whether
 * across a slice there is any data.
 *
 * <p>
 * But what data?
 *
 * <p>
 * For Members other than those in the list, the 'non-List-Members', one wants to consider all data across the scope
 * of these other Members. For instance, if Time is not a List-Member, then one wants to consider data across All
 * Time. Or, if Customer is not a List-Member, then look at data across All Customers. The theory here, is if there is
 * no data for a particular Member of the list where all other Members not part of the list are span their complete
 * hierarchy, then there is certainly no data for Members of that Hierarchy at a more specific Level (more on this
 * below).
 *
 * <p>
 * When a Member that is a non-List-Member is part of a Hierarchy that has an All Member (hasAll="true"), then its
 * very easy to make sure that the All Member is used during the optimization. If a non-List-Member is part of a
 * Hierarchy that does not have an All Member, then one must, in fact, iterate over all top-level Members of the
 * Hierarchy!!! - otherwise a List-Member might be excluded because the optimization code was not looking everywhere.
 *
 * <p>
 * Concerning default Members for those Hierarchies for the non-List-Members, ignore them. What is wanted is either
 * the All Member or one must iterate across all top-level Members, what happens to be the default Member of the
 * Hierarchy is of no relevant.
 *
 * <p>
 * The Measures Hierarchy has special considerations. First, there is no All Measure. But, certainly one need only
 * involve Measures that are actually in the query... yes and no. For Calculated Measures one must also get all of the
 * non-Calculated Measures that make up each Calculated Measure. Thus, one ends up iterating across all Calculated and
 * non-Calculated Measures that are explicitly mentioned in the query as well as all Calculated and non-Calculated
 * Measures that are used to define the Calculated Measures in the query. Why all of these? because this represents
 * the total scope of possible Measures that might yield a non-null value for the List-Members and that is what we
 * what to find. It might be a super set, but thats ok; we just do not want to miss anything.
 *
 * <p>
 * For other Members, the default Member is used, but for Measures one should look for that data for all Measures
 * associated with the query, not just one Measure. For a dense dataset this may not be a problem or even apparent,
 * but for a sparse dataset, the first Measure may, in fact, have not data but other Measures associated with the
 * query might. Hence, the solution here is to identify all Measures associated with the query and then for each
 * Member of the list, determine if there is any data iterating across all Measures until non-null data is found or
 * the end of the Measures is reached.
 *
 * <p>
 * This is a non-optimistic implementation. This means that an element of the input parameter List is only not
 * included in the returned result List if for no combination of Measures, non-All Members (for Hierarchies that have
 * no All Members) and evaluator default Members did the element evaluate to non-null.
 *
 * @param evaluator
 *          Evaluator
 * @param list
 *          List of members or tuples
 * @param call
 *          Calling ResolvedFunCall used to determine what Measures to use
 * @return List of elements from the input parameter list that have evaluated to non-null.
 */
protected TupleList nonEmptyList(Evaluator evaluator, TupleList list, ResolvedFunCall call) {
    if (list.isEmpty()) {
        return list;
    }
    TupleList result = TupleCollections.createList(list.getArity(), (list.size() + 2) >> 1);
    // Get all of the Measures
    final Query query = evaluator.getQuery();
    final String measureSetKey = "MEASURE_SET-" + ctag;
    Set<Member> measureSet = Util.cast((Set) query.getEvalCache(measureSetKey));
    final String memberSetKey = "MEMBER_SET-" + ctag;
    Set<Member> memberSet = Util.cast((Set) query.getEvalCache(memberSetKey));
    // sense to create and cache it.
    if (measureSet == null || memberSet == null) {
        measureSet = new HashSet<Member>();
        memberSet = new HashSet<Member>();
        Set<Member> queryMeasureSet = query.getMeasuresMembers();
        MeasureVisitor measureVisitor = new MeasureVisitor(measureSet, call);
        // MemberExtractingVisitor will collect the dimension members
        // referenced within the measures in the query.
        // One or more measures may conflict with the members in the tuple,
        // overriding the context of the tuple member when determining
        // non-emptiness.
        MemberExtractingVisitor memVisitor = new MemberExtractingVisitor(memberSet, call, false);
        for (Member m : queryMeasureSet) {
            if (m.isCalculated()) {
                Exp exp = m.getExpression();
                exp.accept(measureVisitor);
                exp.accept(memVisitor);
            } else {
                measureSet.add(m);
            }
        }
        Formula[] formula = query.getFormulas();
        if (formula != null) {
            for (Formula f : formula) {
                if (SqlConstraintUtils.containsValidMeasure(f.getExpression())) {
                    // short circuit if VM is present.
                    return list;
                }
                f.accept(measureVisitor);
            }
        }
        query.putEvalCache(measureSetKey, measureSet);
        query.putEvalCache(memberSetKey, memberSet);
    }
    final String allMemberListKey = "ALL_MEMBER_LIST-" + ctag;
    List<Member> allMemberList = Util.cast((List) query.getEvalCache(allMemberListKey));
    final String nonAllMembersKey = "NON_ALL_MEMBERS-" + ctag;
    Member[][] nonAllMembers = (Member[][]) query.getEvalCache(nonAllMembersKey);
    if (nonAllMembers == null) {
        // 
        // Get all of the All Members and those Hierarchies that
        // do not have All Members.
        // 
        Member[] evalMembers = evaluator.getMembers().clone();
        List<Member> listMembers = list.get(0);
        // Remove listMembers from evalMembers and independentSlicerMembers
        for (Member lm : listMembers) {
            Hierarchy h = lm.getHierarchy();
            for (int i = 0; i < evalMembers.length; i++) {
                Member em = evalMembers[i];
                if ((em != null) && h.equals(em.getHierarchy())) {
                    evalMembers[i] = null;
                }
            }
        }
        Map<Hierarchy, Set<Member>> mapOfSlicerMembers = new HashMap<Hierarchy, Set<Member>>();
        if (evaluator instanceof RolapEvaluator) {
            RolapEvaluator rev = (RolapEvaluator) evaluator;
            mapOfSlicerMembers = rev.getSlicerMembersByHierarchy();
        }
        // Now we have the non-List-Members, but some of them may not be
        // All Members (default Member need not be the All Member) and
        // for some Hierarchies there may not be an All Member.
        // So we create an array of Objects some elements of which are
        // All Members and others elements will be an array of all top-level
        // Members when there is not an All Member.
        SchemaReader schemaReader = evaluator.getSchemaReader();
        allMemberList = new ArrayList<Member>();
        List<Member[]> nonAllMemberList = new ArrayList<Member[]>();
        Member em;
        boolean isSlicerMember;
        for (Member evalMember : evalMembers) {
            em = evalMember;
            if (em == null) {
                // to null. These are the CrossJoin axes.
                continue;
            }
            if (em.isMeasure()) {
                continue;
            }
            isSlicerMember = false;
            if (mapOfSlicerMembers != null) {
                Set<Member> members = mapOfSlicerMembers.get(em.getHierarchy());
                if (members != null) {
                    isSlicerMember = members.contains(em);
                }
            }
            // 
            if ((isSlicerMember && !em.isCalculated()) || (!isSlicerMember && em.isCalculated())) {
                // hierarchy, add them to nonAllMemberList
                if (isSlicerMember) {
                    Set<Member> hierarchySlicerMembers = mapOfSlicerMembers.get(em.getHierarchy());
                    if (hierarchySlicerMembers.size() > 1) {
                        nonAllMemberList.add(hierarchySlicerMembers.toArray(new Member[hierarchySlicerMembers.size()]));
                    }
                }
                continue;
            }
            // replace with the "all" member.
            if (isSlicerMember || !em.isAll()) {
                Hierarchy h = em.getHierarchy();
                final List<Member> rootMemberList = schemaReader.getHierarchyRootMembers(h);
                if (h.hasAll()) {
                    // The Hierarchy has an All member
                    boolean found = false;
                    for (Member m : rootMemberList) {
                        if (m.isAll()) {
                            allMemberList.add(m);
                            found = true;
                            break;
                        }
                    }
                    if (!found) {
                        LOGGER.warn("CrossJoinFunDef.nonEmptyListNEW: ERROR");
                    }
                } else {
                    // The Hierarchy does NOT have an All member
                    Member[] rootMembers = rootMemberList.toArray(new Member[rootMemberList.size()]);
                    nonAllMemberList.add(rootMembers);
                }
            }
        }
        nonAllMembers = nonAllMemberList.toArray(new Member[nonAllMemberList.size()][]);
        query.putEvalCache(allMemberListKey, allMemberList);
        query.putEvalCache(nonAllMembersKey, nonAllMembers);
    }
    // 
    // Determine if there is any data.
    // 
    // Put all of the All Members into Evaluator
    final int savepoint = evaluator.savepoint();
    try {
        evaluator.setContext(allMemberList);
        // Iterate over elements of the input list. If for any
        // combination of
        // Measure and non-All Members evaluation is non-null, then
        // add it to the result List.
        final TupleCursor cursor = list.tupleCursor();
        int currentIteration = 0;
        Execution execution = query.getStatement().getCurrentExecution();
        while (cursor.forward()) {
            cursor.setContext(evaluator);
            for (Member member : memberSet) {
                // memberSet contains members referenced within measures.
                // Make sure that we don't incorrectly assume a context
                // that will be changed by the measure, so conservatively
                // push context to [All] for each of the associated
                // hierarchies.
                evaluator.setContext(member.getHierarchy().getAllMember());
            }
            // Check if the MDX query was canceled.
            // Throws an exception in case of timeout is exceeded
            // see MONDRIAN-2425
            CancellationChecker.checkCancelOrTimeout(currentIteration++, execution);
            if (tupleContainsCalcs(cursor.current()) || checkData(nonAllMembers, nonAllMembers.length - 1, measureSet, evaluator)) {
                result.addCurrent(cursor);
            }
        }
        return result;
    } finally {
        evaluator.restore(savepoint);
    }
}

Example 2 with TupleCursor

use of mondrian.calc.TupleCursor in project mondrian by pentaho.

the class CrossJoinFunDef method cartesianProductRecurse.

private static void cartesianProductRecurse(int i, List<TupleList> lists, List<Member> partial, Member[] partialArray, int partialSize, List<Member> result) {
    final TupleList tupleList = lists.get(i);
    final int partialSizeNext = partialSize + tupleList.getArity();
    final int iNext = i + 1;
    final TupleCursor cursor = tupleList.tupleCursor();
    int currentIteration = 0;
    Execution execution = Locus.peek().execution;
    while (cursor.forward()) {
        CancellationChecker.checkCancelOrTimeout(currentIteration++, execution);
        cursor.currentToArray(partialArray, partialSize);
        if (i == lists.size() - 1) {
            result.addAll(partial);
        } else {
            cartesianProductRecurse(iNext, lists, partial, partialArray, partialSizeNext, result);
        }
    }
}

Example 3 with TupleCursor

use of mondrian.calc.TupleCursor in project mondrian by pentaho.

the class FunUtil method evaluateSet.

/**
 * Evaluates one or more expressions against the member list returning a SetWrapper array. Where this differs very
 * significantly from the above evaluateSet methods is how it count null values and Throwables; this method adds nulls
 * to the SetWrapper Vector rather than not adding anything - as the above method does. The impact of this is that if,
 * for example, one was creating a list of x,y values then each list will have the same number of values (though some
 * might be null) - this allows higher level code to determine how to handle the lack of data rather than having a
 * non-equal number (if one is plotting x,y values it helps to have the same number and know where a potential gap is
 * the data is.
 */
static SetWrapper[] evaluateSet(Evaluator evaluator, TupleList list, DoubleCalc[] calcs) {
    Util.assertPrecondition(calcs != null, "calcs != null");
    // todo: treat constant exps as evaluateMembers() does
    SetWrapper[] retvals = new SetWrapper[calcs.length];
    for (int i = 0; i < calcs.length; i++) {
        retvals[i] = new SetWrapper();
    }
    final TupleCursor cursor = list.tupleCursor();
    int currentIteration = 0;
    Execution execution = evaluator.getQuery().getStatement().getCurrentExecution();
    while (cursor.forward()) {
        CancellationChecker.checkCancelOrTimeout(currentIteration++, execution);
        cursor.setContext(evaluator);
        for (int i = 0; i < calcs.length; i++) {
            DoubleCalc calc = calcs[i];
            SetWrapper retval = retvals[i];
            double o = calc.evaluateDouble(evaluator);
            if (o == FunUtil.DoubleNull) {
                retval.nullCount++;
                retval.v.add(null);
            } else {
                retval.v.add(o);
            }
        // TODO: If the expression yielded an error, carry on
        // summing, so that if we are running in a
        // BatchingCellReader, we find out all the dependent cells
        // we need
        }
    }
    return retvals;
}

Example 4 with TupleCursor

use of mondrian.calc.TupleCursor in project mondrian by pentaho.

the class RolapResult method executeStripe.

private void executeStripe(int axisOrdinal, RolapEvaluator revaluator, final int[] pos) {
    if (axisOrdinal < 0) {
        RolapAxis axis = (RolapAxis) slicerAxis;
        TupleList tupleList = axis.getTupleList();
        final Iterator<List<Member>> tupleIterator = tupleList.iterator();
        if (tupleIterator.hasNext()) {
            final List<Member> members = tupleIterator.next();
            execution.checkCancelOrTimeout();
            final int savepoint = revaluator.savepoint();
            revaluator.setContext(members);
            Object o;
            try {
                o = revaluator.evaluateCurrent();
            } catch (MondrianEvaluationException e) {
                LOGGER.warn("Mondrian: exception in executeStripe.", e);
                o = e;
            } finally {
                revaluator.restore(savepoint);
            }
            CellInfo ci = null;
            // Object.
            try {
                // This code is a combination of the code found in
                // the old RolapResult
                // <code>getCellNoDefaultFormatString</code> method and
                // the old RolapCell <code>getFormattedValue</code> method.
                // Create a CellInfo object for the given position
                // integer array.
                ci = cellInfos.create(point.getOrdinals());
                String cachedFormatString = null;
                // Determine if there is a CellFormatter registered for
                // the current Cube's Measure's Dimension. If so,
                // then find or create a CellFormatterValueFormatter
                // for it. If not, then find or create a Locale based
                // FormatValueFormatter.
                final RolapCube cube = getCube();
                Hierarchy measuresHierarchy = cube.getMeasuresHierarchy();
                RolapMeasure m = (RolapMeasure) revaluator.getContext(measuresHierarchy);
                ValueFormatter valueFormatter = m.getFormatter();
                if (valueFormatter == null) {
                    cachedFormatString = revaluator.getFormatString();
                    Locale locale = statement.getMondrianConnection().getLocale();
                    valueFormatter = formatValueFormatters.get(locale);
                    if (valueFormatter == null) {
                        valueFormatter = new FormatValueFormatter(locale);
                        formatValueFormatters.put(locale, valueFormatter);
                    }
                }
                ci.formatString = cachedFormatString;
                ci.valueFormatter = valueFormatter;
            } catch (ResultLimitExceededException e) {
                // Do NOT ignore a ResultLimitExceededException!!!
                throw e;
            } catch (CellRequestQuantumExceededException e) {
                // We need to throw this so another phase happens.
                throw e;
            } catch (MondrianEvaluationException e) {
                // ignore but warn
                LOGGER.warn("Mondrian: exception in executeStripe.", e);
            } catch (Error e) {
                // Errors indicate fatal JVM problems; do not discard
                throw e;
            } catch (Throwable e) {
                LOGGER.warn("Mondrian: exception in executeStripe.", e);
                Util.discard(e);
            }
            if (o != RolapUtil.valueNotReadyException) {
                ci.value = o;
            }
        }
    } else {
        RolapAxis axis = (RolapAxis) axes[axisOrdinal];
        TupleList tupleList = axis.getTupleList();
        // force materialize
        Util.discard(tupleList.size());
        if (isAxisHighCardinality(axisOrdinal, tupleList)) {
            final int limit = MondrianProperties.instance().HighCardChunkSize.get();
            if (positionsIterators.get(axisOrdinal) == null) {
                final TupleCursor tupleCursor = tupleList.tupleCursor();
                positionsIterators.put(axisOrdinal, tupleCursor);
                positionsIndexes.put(axisOrdinal, 0);
                final List<List<Member>> subPositions = new ArrayList<List<Member>>();
                for (int i = 0; i < limit && tupleCursor.forward(); i++) {
                    subPositions.add(tupleCursor.current());
                }
                positionsCurrent.put(axisOrdinal, subPositions);
            }
            final TupleCursor tupleCursor = positionsIterators.get(axisOrdinal);
            final int positionIndex = positionsIndexes.get(axisOrdinal);
            List<List<Member>> subTuples = positionsCurrent.get(axisOrdinal);
            if (subTuples == null) {
                return;
            }
            int pi;
            if (pos[axisOrdinal] > positionIndex + subTuples.size() - 1 && subTuples.size() == limit) {
                pi = positionIndex + subTuples.size();
                positionsIndexes.put(axisOrdinal, positionIndex + subTuples.size());
                subTuples.subList(0, subTuples.size()).clear();
                for (int i = 0; i < limit && tupleCursor.forward(); i++) {
                    subTuples.add(tupleCursor.current());
                }
                positionsCurrent.put(axisOrdinal, subTuples);
            } else {
                pi = positionIndex;
            }
            for (final List<Member> tuple : subTuples) {
                point.setAxis(axisOrdinal, pi);
                final int savepoint = revaluator.savepoint();
                try {
                    revaluator.setContext(tuple);
                    execution.checkCancelOrTimeout();
                    executeStripe(axisOrdinal - 1, revaluator, pos);
                } finally {
                    revaluator.restore(savepoint);
                }
                pi++;
            }
        } else {
            for (List<Member> tuple : tupleList) {
                List<Member> measures = new ArrayList<Member>(statement.getQuery().getMeasuresMembers());
                for (Member measure : measures) {
                    if (measure instanceof RolapBaseCubeMeasure) {
                        RolapBaseCubeMeasure baseCubeMeasure = (RolapBaseCubeMeasure) measure;
                        if (baseCubeMeasure.getAggregator() == RolapAggregator.DistinctCount) {
                            processDistinctMeasureExpr(tuple, baseCubeMeasure);
                        }
                    }
                }
            }
            int tupleIndex = 0;
            for (final List<Member> tuple : tupleList) {
                point.setAxis(axisOrdinal, tupleIndex);
                final int savepoint = revaluator.savepoint();
                try {
                    revaluator.setEvalAxes(true);
                    revaluator.setContext(tuple);
                    execution.checkCancelOrTimeout();
                    executeStripe(axisOrdinal - 1, revaluator, pos);
                } finally {
                    revaluator.restore(savepoint);
                }
                tupleIndex++;
            }
        }
    }
}

Example 5 with TupleCursor

use of mondrian.calc.TupleCursor in project mondrian by pentaho.

the class RolapResult method executeAxis.

private TupleIterable executeAxis(Evaluator evaluator, QueryAxis queryAxis, Calc axisCalc, boolean construct, AxisMemberList axisMembers) {
    if (queryAxis == null) {
        // the same as an empty axis).
        return new DelegatingTupleList(0, Collections.singletonList(Collections.<Member>emptyList()));
    }
    final int savepoint = evaluator.savepoint();
    try {
        evaluator.setNonEmpty(queryAxis.isNonEmpty());
        evaluator.setEvalAxes(true);
        final TupleIterable iterable = ((IterCalc) axisCalc).evaluateIterable(evaluator);
        if (axisCalc.getClass().getName().indexOf("OrderFunDef") != -1) {
            queryAxis.setOrdered(true);
        }
        if (iterable instanceof TupleList) {
            TupleList list = (TupleList) iterable;
            if (construct) {
            } else if (axisMembers != null) {
                axisMembers.mergeTupleList(list);
            }
        } else {
            // Iterable
            TupleCursor cursor = iterable.tupleCursor();
            if (construct) {
            } else if (axisMembers != null) {
                axisMembers.mergeTupleIter(cursor);
            }
        }
        return iterable;
    } finally {
        evaluator.restore(savepoint);
    }
}