Search in sources :

Example 81 with Function

use of java.util.function.Function in project gatk-protected by broadinstitute.

the class CNLOHCaller method calcNewRhos.

private double[] calcNewRhos(final List<ACNVModeledSegment> segments, final List<double[][][]> responsibilitiesBySeg, final double lambda, final double[] rhos, final int[] mVals, final int[] nVals, final JavaSparkContext ctx) {
    // Since, we pass in the entire responsibilities matrix, we need the correct index for each rho.  That, and the
    //  fact that this is a univariate objective function, means we need to create an instance for each rho.  And
    //  then we blast across Spark.
    final List<Pair<? extends Function<Double, Double>, SearchInterval>> objectives = IntStream.range(0, rhos.length).mapToObj(i -> new Pair<>(new Function<Double, Double>() {

        @Override
        public Double apply(Double rho) {
            return calculateESmnObjective(rho, segments, responsibilitiesBySeg, mVals, nVals, lambda, i);
        }
    }, new SearchInterval(0.0, 1.0, rhos[i]))).collect(Collectors.toList());
    final JavaRDD<Pair<? extends Function<Double, Double>, SearchInterval>> objectivesRDD = ctx.parallelize(objectives);
    final List<Double> resultsAsDouble = objectivesRDD.map(objective -> optimizeIt(objective.getFirst(), objective.getSecond())).collect();
    return resultsAsDouble.stream().mapToDouble(Double::doubleValue).toArray();
}
Also used : IntStream(java.util.stream.IntStream) Arrays(java.util.Arrays) JavaSparkContext(org.apache.spark.api.java.JavaSparkContext) SearchInterval(org.apache.commons.math3.optim.univariate.SearchInterval) RealVector(org.apache.commons.math3.linear.RealVector) Function(java.util.function.Function) ParamUtils(org.broadinstitute.hellbender.utils.param.ParamUtils) GammaDistribution(org.apache.commons.math3.distribution.GammaDistribution) Gamma(org.apache.commons.math3.special.Gamma) ACNVModeledSegment(org.broadinstitute.hellbender.tools.exome.ACNVModeledSegment) BaseAbstractUnivariateIntegrator(org.apache.commons.math3.analysis.integration.BaseAbstractUnivariateIntegrator) GoalType(org.apache.commons.math3.optim.nonlinear.scalar.GoalType) MatrixUtils(org.apache.commons.math3.linear.MatrixUtils) UnivariateObjectiveFunction(org.apache.commons.math3.optim.univariate.UnivariateObjectiveFunction) ArrayRealVector(org.apache.commons.math3.linear.ArrayRealVector) SimpsonIntegrator(org.apache.commons.math3.analysis.integration.SimpsonIntegrator) JavaRDD(org.apache.spark.api.java.JavaRDD) GATKProtectedMathUtils(org.broadinstitute.hellbender.utils.GATKProtectedMathUtils) Pair(org.apache.commons.math3.util.Pair) Collectors(java.util.stream.Collectors) BrentOptimizer(org.apache.commons.math3.optim.univariate.BrentOptimizer) Serializable(java.io.Serializable) List(java.util.List) Percentile(org.apache.commons.math3.stat.descriptive.rank.Percentile) Logger(org.apache.logging.log4j.Logger) MathUtils(org.broadinstitute.hellbender.utils.MathUtils) NormalDistribution(org.apache.commons.math3.distribution.NormalDistribution) UnivariateFunction(org.apache.commons.math3.analysis.UnivariateFunction) Variance(org.apache.commons.math3.stat.descriptive.moment.Variance) Utils(org.broadinstitute.hellbender.utils.Utils) RealMatrix(org.apache.commons.math3.linear.RealMatrix) VisibleForTesting(com.google.common.annotations.VisibleForTesting) HomoSapiensConstants(org.broadinstitute.hellbender.utils.variant.HomoSapiensConstants) MaxEval(org.apache.commons.math3.optim.MaxEval) LogManager(org.apache.logging.log4j.LogManager) Function(java.util.function.Function) UnivariateObjectiveFunction(org.apache.commons.math3.optim.univariate.UnivariateObjectiveFunction) UnivariateFunction(org.apache.commons.math3.analysis.UnivariateFunction) SearchInterval(org.apache.commons.math3.optim.univariate.SearchInterval) Pair(org.apache.commons.math3.util.Pair)

Example 82 with Function

use of java.util.function.Function in project gatk by broadinstitute.

the class GatherVcfs method getReaderFromVCFUri.

private static FeatureReader<VariantContext> getReaderFromVCFUri(final Path variantPath, final int cloudPrefetchBuffer) {
    final String variantURI = variantPath.toUri().toString();
    final Function<SeekableByteChannel, SeekableByteChannel> cloudWrapper = (cloudPrefetchBuffer > 0 ? is -> SeekableByteChannelPrefetcher.addPrefetcher(cloudPrefetchBuffer, is) : Function.identity());
    return AbstractFeatureReader.getFeatureReader(variantURI, null, new VCFCodec(), false, cloudWrapper, Function.identity());
}
Also used : CloseableIterator(htsjdk.samtools.util.CloseableIterator) CommandLineProgramProperties(org.broadinstitute.barclay.argparser.CommandLineProgramProperties) java.util(java.util) ProgressLogger(org.broadinstitute.hellbender.utils.runtime.ProgressLogger) IOUtil(htsjdk.samtools.util.IOUtil) VCFHeader(htsjdk.variant.vcf.VCFHeader) Argument(org.broadinstitute.barclay.argparser.Argument) FeatureReader(htsjdk.tribble.FeatureReader) VariantContextWriterBuilder(htsjdk.variant.variantcontext.writer.VariantContextWriterBuilder) StandardArgumentDefinitions(org.broadinstitute.hellbender.cmdline.StandardArgumentDefinitions) Function(java.util.function.Function) VariantProgramGroup(org.broadinstitute.hellbender.cmdline.programgroups.VariantProgramGroup) AbstractFeatureReader(htsjdk.tribble.AbstractFeatureReader) BlockCompressedOutputStream(htsjdk.samtools.util.BlockCompressedOutputStream) RuntimeIOException(htsjdk.samtools.util.RuntimeIOException) CollectionUtil(htsjdk.samtools.util.CollectionUtil) VCFCodec(htsjdk.variant.vcf.VCFCodec) PeekableIterator(htsjdk.samtools.util.PeekableIterator) PicardCommandLineProgram(org.broadinstitute.hellbender.cmdline.PicardCommandLineProgram) Path(java.nio.file.Path) CloserUtil(htsjdk.samtools.util.CloserUtil) BlockCompressedStreamConstants(htsjdk.samtools.util.BlockCompressedStreamConstants) IOUtils(org.broadinstitute.hellbender.utils.io.IOUtils) BlockCompressedInputStream(htsjdk.samtools.util.BlockCompressedInputStream) SAMSequenceDictionary(htsjdk.samtools.SAMSequenceDictionary) Collectors(java.util.stream.Collectors) VariantContextComparator(htsjdk.variant.variantcontext.VariantContextComparator) SeekableByteChannel(java.nio.channels.SeekableByteChannel) Logger(org.apache.logging.log4j.Logger) UserException(org.broadinstitute.hellbender.exceptions.UserException) java.io(java.io) Options(htsjdk.variant.variantcontext.writer.Options) VariantContextWriter(htsjdk.variant.variantcontext.writer.VariantContextWriter) SeekableByteChannelPrefetcher(org.broadinstitute.hellbender.utils.nio.SeekableByteChannelPrefetcher) VariantContext(htsjdk.variant.variantcontext.VariantContext) Utils(org.broadinstitute.hellbender.utils.Utils) VisibleForTesting(com.google.common.annotations.VisibleForTesting) LogManager(org.apache.logging.log4j.LogManager) SeekableByteChannel(java.nio.channels.SeekableByteChannel) VCFCodec(htsjdk.variant.vcf.VCFCodec)

Example 83 with Function

use of java.util.function.Function in project Gargoyle by callakrsos.

the class FileSearcher method listClases.

public static List<ProjectInfo> listClases(String classDirName) throws Exception {
    File file = new File(classDirName);
    // 기본적인 파일의 존재유무 및 디렉토리인지 체크.
    if (!file.exists())
        throw new FileNotFoundException(file + " Not found!");
    //
    if (!file.isDirectory())
        throw new IllegalArgumentException("only directory.");
    /*
		 * 디렉토리안에 클래스패스 정보가 존재하는지 확인하고 존재한다면 classpath에 기술된 정보 기준으로 클래스 파일을
		 * 로드한다. 프로그램내에서 workspace를 선택한 경우일수있고, 프로젝트를 선택한 두가지의 경우가 있기때문에 두가지의
		 * 케이스를 고려한 로직이 들어간다.
		 */
    /*
		 * 일단 워크스페이스를 선택한경우라고 가정하고 워크스페이스내에 폴더들을 순차적으로 돌아보면서 classpath의 존재유무를 찾고
		 * 존재하는케이스는 따로 모아놓는다. 파일레벨은 워크스페이스(0레벨)-프로젝트(1레벨)로 가정하여 1레벨까지만 이동한다.
		 */
    List<File> listFiles = findClassPaths(file);
    /*
		 * classpath파일을 찾은경우 그 파일path를 기준으로 클래스들을 로딩한다.
		 */
    List<ProjectInfo> allClasses = new ArrayList<>();
    if (listFiles != null && !listFiles.isEmpty())
        LOGGER.debug(" im will working...");
    long startTime = System.currentTimeMillis();
    int searchedDirCount = 0;
    StringBuffer srchedDirNames = new StringBuffer();
    for (File f : listFiles) {
        try {
            ClassPath parsingClassPath = parsingClassPath(f.getAbsolutePath());
            // 프로젝트파일.
            File projectFile = f.getParentFile();
            // output 속성값의 존재유무만 확인하여 컴파일되는 경로를 찾는다.
            List<ProjectInfo> collect = parsingClassPath.toStream().filter(entry -> {
                boolean notEmpty = ValueUtil.isNotEmpty(entry.getOutput());
                LOGGER.debug(String.format("srch entry path : %s is Traget %b ", entry.getPath(), notEmpty));
                return notEmpty;
            }).map(pram -> pram.getOutput()).distinct().parallel().flatMap(new Function<String, Stream<ProjectInfo>>() {

                @Override
                public Stream<ProjectInfo> apply(String entry) {
                    LOGGER.debug(String.format("entry : %s", entry));
                    File compiledFilePath = new File(projectFile, entry);
                    int length = compiledFilePath.getAbsolutePath().length() + 1;
                    List<String> findClases = findClases(projectFile.getAbsolutePath(), compiledFilePath, length);
                    LOGGER.debug(compiledFilePath.toString());
                    LOGGER.debug(findClases.toString());
                    LOGGER.debug(String.valueOf(findClases.size()));
                    ProjectInfo classInfo = new ProjectInfo();
                    classInfo.setProjectName(projectFile.getName());
                    classInfo.setProjectDir(compiledFilePath.getAbsolutePath());
                    classInfo.setClasses(findClases);
                    return Stream.of(classInfo);
                }
            }).collect(Collectors.toList());
            allClasses.addAll(collect);
            searchedDirCount++;
            srchedDirNames.append(f.getAbsolutePath()).append(SystemUtils.LINE_SEPARATOR);
        } catch (SAXParseException e) {
            LOGGER.error(String.format("정상적인 XML 형태가 아님. 파일명 :  %s", f.getAbsolutePath()));
            LOGGER.error(String.format("%d 행 :: %d 열", e.getLineNumber(), e.getColumnNumber()));
        }
    }
    long endTime = System.currentTimeMillis();
    long costMillisend = endTime - startTime;
    LOGGER.debug(String.format("Total Cost time : %s (ms) searched Directory Count : %d ", costMillisend, searchedDirCount));
    LOGGER.debug(String.format("Searched Dirs info \n%s", srchedDirNames.toString()));
    return allClasses;
}
Also used : URL(java.net.URL) ZipUtil(com.kyj.fx.voeditor.visual.util.ZipUtil) LoggerFactory(org.slf4j.LoggerFactory) Function(java.util.function.Function) Supplier(java.util.function.Supplier) MimetypesFileTypeMap(javax.activation.MimetypesFileTypeMap) ArrayList(java.util.ArrayList) URLClassLoader(java.net.URLClassLoader) Document(org.w3c.dom.Document) Node(org.w3c.dom.Node) NamedNodeMap(org.w3c.dom.NamedNodeMap) Method(java.lang.reflect.Method) SystemUtils(org.apache.commons.lang.SystemUtils) ClassPathEntry(com.kyj.fx.voeditor.visual.main.model.vo.ClassPathEntry) InputSource(org.xml.sax.InputSource) Logger(org.slf4j.Logger) NodeList(org.w3c.dom.NodeList) MalformedURLException(java.net.MalformedURLException) Predicate(java.util.function.Predicate) ClassPath(com.kyj.fx.voeditor.visual.main.model.vo.ClassPath) ValueUtil(com.kyj.fx.voeditor.visual.util.ValueUtil) FileInputStream(java.io.FileInputStream) Reader(java.io.Reader) ConfigResourceLoader(com.kyj.fx.voeditor.visual.momory.ConfigResourceLoader) InputStreamReader(java.io.InputStreamReader) Collectors(java.util.stream.Collectors) File(java.io.File) FileNotFoundException(java.io.FileNotFoundException) StringUtils(org.h2.util.StringUtils) List(java.util.List) SAXParseException(org.xml.sax.SAXParseException) Stream(java.util.stream.Stream) DocumentBuilder(javax.xml.parsers.DocumentBuilder) FileTypeMap(javax.activation.FileTypeMap) DocumentBuilderFactory(javax.xml.parsers.DocumentBuilderFactory) ClassPath(com.kyj.fx.voeditor.visual.main.model.vo.ClassPath) FileNotFoundException(java.io.FileNotFoundException) ArrayList(java.util.ArrayList) Function(java.util.function.Function) SAXParseException(org.xml.sax.SAXParseException) File(java.io.File)

Example 84 with Function

use of java.util.function.Function in project gatk by broadinstitute.

the class AlleleFractionSegmenter method relearnAdditionalParameters.

@Override
protected void relearnAdditionalParameters(final ExpectationStep eStep) {
    final Function<AlleleFractionGlobalParameters, Double> emissionLogLikelihood = params -> {
        double logLikelihood = 0.0;
        for (int position = 0; position < numPositions(); position++) {
            for (int state = 0; state < numStates(); state++) {
                final double eStepPosterior = eStep.pStateAtPosition(state, position);
                logLikelihood += eStepPosterior < NEGLIGIBLE_POSTERIOR_FOR_M_STEP ? 0 : eStepPosterior * AlleleFractionHMM.logEmissionProbability(data.get(position), getState(state), params, allelicPoN);
            }
        }
        return logLikelihood;
    };
    final Function<Double, Double> meanBiasObjective = mean -> emissionLogLikelihood.apply(globalParameters.copyWithNewMeanBias(mean));
    final double newMeanBias = OptimizationUtils.argmax(meanBiasObjective, 0, AlleleFractionInitializer.MAX_REASONABLE_MEAN_BIAS, globalParameters.getMeanBias(), RELATIVE_TOLERANCE_FOR_OPTIMIZATION, ABSOLUTE_TOLERANCE_FOR_OPTIMIZATION, MAX_EVALUATIONS_FOR_OPTIMIZATION);
    final Function<Double, Double> biasVarianceObjective = variance -> emissionLogLikelihood.apply(globalParameters.copyWithNewBiasVariance(variance));
    final double newBiasVariance = OptimizationUtils.argmax(biasVarianceObjective, 0, AlleleFractionInitializer.MAX_REASONABLE_BIAS_VARIANCE, globalParameters.getBiasVariance(), RELATIVE_TOLERANCE_FOR_OPTIMIZATION, ABSOLUTE_TOLERANCE_FOR_OPTIMIZATION, MAX_EVALUATIONS_FOR_OPTIMIZATION);
    final Function<Double, Double> outlierProbabilityObjective = pOutlier -> emissionLogLikelihood.apply(globalParameters.copyWithNewOutlierProbability(pOutlier));
    final double newOutlierProbability = OptimizationUtils.argmax(outlierProbabilityObjective, 0, AlleleFractionInitializer.MAX_REASONABLE_OUTLIER_PROBABILITY, globalParameters.getOutlierProbability(), RELATIVE_TOLERANCE_FOR_OPTIMIZATION, ABSOLUTE_TOLERANCE_FOR_OPTIMIZATION, MAX_EVALUATIONS_FOR_OPTIMIZATION);
    globalParameters = new AlleleFractionGlobalParameters(newMeanBias, newBiasVariance, newOutlierProbability);
    logger.info(String.format("Global allelic bias parameters learned.  Mean allelic bias: %f, variance of allelic bias: %f, outlier probability: %f.", newMeanBias, newBiasVariance, newOutlierProbability));
}
Also used : Arrays(java.util.Arrays) GATKProtectedMathUtils(org.broadinstitute.hellbender.utils.GATKProtectedMathUtils) ModeledSegment(org.broadinstitute.hellbender.tools.exome.ModeledSegment) AllelicCount(org.broadinstitute.hellbender.tools.exome.alleliccount.AllelicCount) AlleleFractionInitializer(org.broadinstitute.hellbender.tools.exome.allelefraction.AlleleFractionInitializer) SimpleInterval(org.broadinstitute.hellbender.utils.SimpleInterval) Function(java.util.function.Function) Collectors(java.util.stream.Collectors) ParamUtils(org.broadinstitute.hellbender.utils.param.ParamUtils) OptimizationUtils(org.broadinstitute.hellbender.utils.OptimizationUtils) List(java.util.List) AlleleFractionState(org.broadinstitute.hellbender.tools.exome.allelefraction.AlleleFractionState) Doubles(com.google.cloud.dataflow.sdk.repackaged.com.google.common.primitives.Doubles) Pair(org.apache.commons.lang3.tuple.Pair) HashedListTargetCollection(org.broadinstitute.hellbender.tools.exome.HashedListTargetCollection) AllelicPanelOfNormals(org.broadinstitute.hellbender.tools.pon.allelic.AllelicPanelOfNormals) Utils(org.broadinstitute.hellbender.utils.Utils) AlleleFractionGlobalParameters(org.broadinstitute.hellbender.tools.exome.allelefraction.AlleleFractionGlobalParameters) AllelicCountCollection(org.broadinstitute.hellbender.tools.exome.alleliccount.AllelicCountCollection) TargetCollection(org.broadinstitute.hellbender.tools.exome.TargetCollection) AlleleFractionGlobalParameters(org.broadinstitute.hellbender.tools.exome.allelefraction.AlleleFractionGlobalParameters)

Example 85 with Function

use of java.util.function.Function in project gatk by broadinstitute.

the class GenomicsDBImport method getReaderFromVCFUri.

/**
     * Creates a feature reader object from a given VCF URI (can also be
     * a local file path) and returns it
     * @param variantPath  URI or file path
     * @return  Feature reader
     */
private AbstractFeatureReader<VariantContext, LineIterator> getReaderFromVCFUri(final String variantPath) {
    final String variantURI = IOUtils.getPath(variantPath).toAbsolutePath().toUri().toString();
    final Function<SeekableByteChannel, SeekableByteChannel> cloudWrapper = (cloudPrefetchBuffer > 0 ? is -> SeekableByteChannelPrefetcher.addPrefetcher(cloudPrefetchBuffer, is) : Function.identity());
    final Function<SeekableByteChannel, SeekableByteChannel> cloudIndexWrapper = (cloudIndexPrefetchBuffer > 0 ? is -> SeekableByteChannelPrefetcher.addPrefetcher(cloudIndexPrefetchBuffer, is) : Function.identity());
    return AbstractFeatureReader.getFeatureReader(variantURI, null, new VCFCodec(), true, cloudWrapper, cloudIndexWrapper);
}
Also used : VCFHeaderLine(htsjdk.variant.vcf.VCFHeaderLine) CommandLineProgramProperties(org.broadinstitute.barclay.argparser.CommandLineProgramProperties) java.util(java.util) LineIterator(htsjdk.tribble.readers.LineIterator) VCFHeader(htsjdk.variant.vcf.VCFHeader) Argument(org.broadinstitute.barclay.argparser.Argument) Advanced(org.broadinstitute.barclay.argparser.Advanced) FeatureReader(htsjdk.tribble.FeatureReader) StandardArgumentDefinitions(org.broadinstitute.hellbender.cmdline.StandardArgumentDefinitions) GATKException(org.broadinstitute.hellbender.exceptions.GATKException) Function(java.util.function.Function) VariantProgramGroup(org.broadinstitute.hellbender.cmdline.programgroups.VariantProgramGroup) AbstractFeatureReader(htsjdk.tribble.AbstractFeatureReader) GenomicsDBCallsetsMapProto(com.intel.genomicsdb.GenomicsDBCallsetsMapProto) VCFCodec(htsjdk.variant.vcf.VCFCodec) Path(java.nio.file.Path) GenomicsDBImporter(com.intel.genomicsdb.GenomicsDBImporter) IOUtils(org.broadinstitute.hellbender.utils.io.IOUtils) Files(java.nio.file.Files) SAMSequenceDictionary(htsjdk.samtools.SAMSequenceDictionary) IOException(java.io.IOException) GenomicsDBImportConfiguration(com.intel.genomicsdb.GenomicsDBImportConfiguration) SimpleInterval(org.broadinstitute.hellbender.utils.SimpleInterval) File(java.io.File) FileNotFoundException(java.io.FileNotFoundException) GATKTool(org.broadinstitute.hellbender.engine.GATKTool) SeekableByteChannel(java.nio.channels.SeekableByteChannel) UserException(org.broadinstitute.hellbender.exceptions.UserException) SeekableByteChannelPrefetcher(org.broadinstitute.hellbender.utils.nio.SeekableByteChannelPrefetcher) VariantContext(htsjdk.variant.variantcontext.VariantContext) VCFUtils(htsjdk.variant.vcf.VCFUtils) Utils(org.broadinstitute.hellbender.utils.Utils) VisibleForTesting(com.google.common.annotations.VisibleForTesting) ChromosomeInterval(com.intel.genomicsdb.ChromosomeInterval) CommandLineException(org.broadinstitute.barclay.argparser.CommandLineException) SeekableByteChannel(java.nio.channels.SeekableByteChannel) VCFCodec(htsjdk.variant.vcf.VCFCodec)

Aggregations

Function (java.util.function.Function)176 List (java.util.List)75 ArrayList (java.util.ArrayList)55 Map (java.util.Map)51 Test (org.junit.Test)47 IOException (java.io.IOException)44 HashMap (java.util.HashMap)37 Set (java.util.Set)36 Collectors (java.util.stream.Collectors)33 Arrays (java.util.Arrays)30 Collections (java.util.Collections)26 Collection (java.util.Collection)25 File (java.io.File)20 HashSet (java.util.HashSet)19 Supplier (java.util.function.Supplier)19 BiFunction (java.util.function.BiFunction)18 Consumer (java.util.function.Consumer)16 Test (org.testng.annotations.Test)16 Stream (java.util.stream.Stream)14 Assert.assertEquals (org.junit.Assert.assertEquals)13