Search in sources :

Example 11 with ProteinSequenceEncoder

use of edu.sdsc.mmtf.spark.ml.ProteinSequenceEncoder in project mm-dev by sbl-sdsc.

the class ShapeTypeDemo method main.

public static void main(String[] args) throws IOException {
    String path = MmtfReader.getMmtfReducedPath();
    if (args.length != 1) {
        System.err.println("Usage: " + ShapeTypeDemo.class.getSimpleName() + " <dataset output file");
        System.exit(1);
    }
    SparkConf conf = new SparkConf().setMaster("local[*]").setAppName(ShapeTypeDemo.class.getSimpleName());
    JavaSparkContext sc = new JavaSparkContext(conf);
    long start = System.nanoTime();
    // load a representative PDB chain from the 40% seq. identity Blast Clusters
    int sequenceIdentity = 90;
    JavaPairRDD<String, StructureDataInterface> pdb = MmtfReader.readSequenceFile(path, sc).flatMapToPair(// extract polymer chains
    new StructureToPolymerChains()).filter(// get representative subset
    new Pisces(sequenceIdentity, 2.5));
    // get a data set with sequence info
    Dataset<Row> seqData = PolymerSequenceExtractor.getDataset(pdb);
    // convert to BioJava data structure
    JavaPairRDD<String, Structure> structures = pdb.mapValues(new StructureToBioJava());
    // calculate shape data and convert to dataset
    JavaRDD<Row> rows = structures.map(t -> getShapeData(t));
    Dataset<Row> data = JavaRDDToDataset.getDataset(rows, "structureChainId", "shape");
    // there are only few symmetric chain, leave them out
    data = data.filter("shape != 'EXCLUDE'");
    // join calculated data with the sequence data
    data = seqData.join(data, "structureChainId").cache();
    data.show(10);
    // create a Word2Vector representation of the protein sequences
    ProteinSequenceEncoder encoder = new ProteinSequenceEncoder(data);
    // create 2-grams
    int n = 2;
    // 25-amino residue window size for Word2Vector
    int windowSize = 25;
    // dimension of feature vector
    int vectorSize = 50;
    data = encoder.overlappingNgramWord2VecEncode(n, windowSize, vectorSize).cache();
    // save data in .parquet file
    data.write().mode("overwrite").format("parquet").save(args[0]);
    long end = System.nanoTime();
    System.out.println((end - start) / 1E9 + " sec.");
    sc.close();
}
Also used : ProteinSequenceEncoder(edu.sdsc.mmtf.spark.ml.ProteinSequenceEncoder) StructureDataInterface(org.rcsb.mmtf.api.StructureDataInterface) Pisces(edu.sdsc.mmtf.spark.webfilters.Pisces) StructureToBioJava(edu.sdsc.mmtf.spark.mappers.StructureToBioJava) StructureToPolymerChains(edu.sdsc.mmtf.spark.mappers.StructureToPolymerChains) JavaSparkContext(org.apache.spark.api.java.JavaSparkContext) Row(org.apache.spark.sql.Row) Structure(org.biojava.nbio.structure.Structure) SparkConf(org.apache.spark.SparkConf)

Aggregations

ProteinSequenceEncoder (edu.sdsc.mmtf.spark.ml.ProteinSequenceEncoder)11 Row (org.apache.spark.sql.Row)11 StructureToPolymerChains (edu.sdsc.mmtf.spark.mappers.StructureToPolymerChains)10 Pisces (edu.sdsc.mmtf.spark.webfilters.Pisces)10 SparkConf (org.apache.spark.SparkConf)10 JavaSparkContext (org.apache.spark.api.java.JavaSparkContext)10 StructureDataInterface (org.rcsb.mmtf.api.StructureDataInterface)10 StructureToBioJava (edu.sdsc.mmtf.spark.mappers.StructureToBioJava)1 Structure (org.biojava.nbio.structure.Structure)1