Search in sources :

Example 1 with ConceptMaps

use of com.cerner.bunsen.codes.ConceptMaps in project bunsen by cerner.

the class BroadcastableMappings method broadcast.

/**
 * Broadcasts the given set of concept maps.
 *
 * @param spark the spark session.
 * @param conceptMaps the FHIR ConceptMaps to broadcast
 * @return a broadcast variable containing a mappings object usable in UDFs.
 */
public static Broadcast<BroadcastableMappings> broadcast(SparkSession spark, List<ConceptMap> conceptMaps) {
    ConceptMaps mapData = ConceptMaps.getEmpty(spark).withConceptMaps(conceptMaps);
    Map<String, String> conceptMapUris = conceptMaps.stream().collect(Collectors.toMap(ConceptMap::getUrl, ConceptMap::getVersion));
    return broadcast(mapData, conceptMapUris);
}
Also used : ConceptMaps(com.cerner.bunsen.codes.ConceptMaps)

Example 2 with ConceptMaps

use of com.cerner.bunsen.codes.ConceptMaps in project bunsen by cerner.

the class BroadcastableMappings method broadcast.

/**
 * Broadcast mappings stored in the given conceptMaps instance that match the given
 * conceptMapUris.
 *
 * @param conceptMaps the {@link ConceptMaps} instance with the content to broadcast
 * @param conceptMapUriToVersion map of the concept map URIs to broadcast to their versions.
 * @return a broadcast variable containing a mappings object usable in UDFs.
 */
public static Broadcast<BroadcastableMappings> broadcast(ConceptMaps conceptMaps, Map<String, String> conceptMapUriToVersion) {
    Map<String, ConceptMap> mapsToLoad = conceptMaps.getMaps().collectAsList().stream().filter(conceptMap -> conceptMap.getVersion().equals(conceptMapUriToVersion.get(conceptMap.getUrl()))).collect(Collectors.toMap(ConceptMap::getUrl, Function.identity()));
    // Expand the concept maps to load and sort them so dependencies are before
    // their dependents in the list.
    List<String> sortedMapsToLoad = sortMapsToLoad(conceptMapUriToVersion.keySet(), mapsToLoad);
    // Since this is used to map from one system to another, we use only targets
    // that don't introduce inaccurate meanings. (For instance, we can't map
    // general condition code to a more specific type, since that is not
    // representative of the source data.)
    Dataset<Mapping> mappings = conceptMaps.getMappings(conceptMapUriToVersion).filter("equivalence in ('equivalent', 'equals', 'wider', 'subsumes')");
    // Group mappings by their concept map URI
    Map<String, List<Mapping>> groupedMappings = mappings.collectAsList().stream().collect(Collectors.groupingBy(Mapping::getConceptMapUri));
    Map<String, BroadcastableConceptMap> broadcastableMaps = new HashMap<>();
    for (String conceptMapUri : sortedMapsToLoad) {
        ConceptMap map = mapsToLoad.get(conceptMapUri);
        Set<String> children = getMapChildren(map);
        List<BroadcastableConceptMap> childMaps = children.stream().map(child -> broadcastableMaps.get(child)).collect(Collectors.toList());
        BroadcastableConceptMap broadcastableConceptMap = new BroadcastableConceptMap(conceptMapUri, groupedMappings.getOrDefault(conceptMapUri, Collections.emptyList()), childMaps);
        broadcastableMaps.put(conceptMapUri, broadcastableConceptMap);
    }
    JavaSparkContext ctx = new JavaSparkContext(conceptMaps.getMaps().sparkSession().sparkContext());
    return ctx.broadcast(new BroadcastableMappings(broadcastableMaps));
}
Also used : Broadcast(org.apache.spark.broadcast.Broadcast) Dataset(org.apache.spark.sql.Dataset) ConceptMaps(com.cerner.bunsen.codes.ConceptMaps) JavaSparkContext(org.apache.spark.api.java.JavaSparkContext) Set(java.util.Set) HashMap(java.util.HashMap) Deque(java.util.Deque) Function(java.util.function.Function) Collectors(java.util.stream.Collectors) Serializable(java.io.Serializable) ArrayList(java.util.ArrayList) HashSet(java.util.HashSet) List(java.util.List) Map(java.util.Map) ConceptMapGroupUnmappedMode(org.hl7.fhir.dstu3.model.ConceptMap.ConceptMapGroupUnmappedMode) ArrayDeque(java.util.ArrayDeque) Collections(java.util.Collections) ConceptMap(org.hl7.fhir.dstu3.model.ConceptMap) Mapping(com.cerner.bunsen.codes.Mapping) SparkSession(org.apache.spark.sql.SparkSession) HashMap(java.util.HashMap) Mapping(com.cerner.bunsen.codes.Mapping) ArrayList(java.util.ArrayList) List(java.util.List) JavaSparkContext(org.apache.spark.api.java.JavaSparkContext) ConceptMap(org.hl7.fhir.dstu3.model.ConceptMap)

Aggregations

ConceptMaps (com.cerner.bunsen.codes.ConceptMaps)2 Mapping (com.cerner.bunsen.codes.Mapping)1 Serializable (java.io.Serializable)1 ArrayDeque (java.util.ArrayDeque)1 ArrayList (java.util.ArrayList)1 Collections (java.util.Collections)1 Deque (java.util.Deque)1 HashMap (java.util.HashMap)1 HashSet (java.util.HashSet)1 List (java.util.List)1 Map (java.util.Map)1 Set (java.util.Set)1 Function (java.util.function.Function)1 Collectors (java.util.stream.Collectors)1 JavaSparkContext (org.apache.spark.api.java.JavaSparkContext)1 Broadcast (org.apache.spark.broadcast.Broadcast)1 Dataset (org.apache.spark.sql.Dataset)1 SparkSession (org.apache.spark.sql.SparkSession)1 ConceptMap (org.hl7.fhir.dstu3.model.ConceptMap)1 ConceptMapGroupUnmappedMode (org.hl7.fhir.dstu3.model.ConceptMap.ConceptMapGroupUnmappedMode)1