Search in sources :

Example 16 with Config

use of edu.neu.ccs.pyramid.configuration.Config in project pyramid by cheng-li.

the class App1 method loadIndex.

static MultiLabelIndex loadIndex(Config config, Logger logger, String trainOrTest) throws Exception {
    MultiLabelIndex.Builder builder = new MultiLabelIndex.Builder().setIndexName(config.getString("index.indexName")).setClusterName(config.getString("index.clusterName")).setClientType(config.getString("index.clientType")).setDocumentType(config.getString("index.documentType"));
    if (trainOrTest.endsWith("train")) {
        builder.setExtMultiLabelField(config.getString("train.label.field"));
    }
    if (trainOrTest.endsWith("test")) {
        File metaDataFolder = new File(config.getString("output.folder"), "meta_data");
        Config savedConfig = new Config(new File(metaDataFolder, "saved_config_app1"));
        builder.setExtMultiLabelField(savedConfig.getString("train.label.field"));
    }
    if (config.getString("index.clientType").equals("transport")) {
        String[] hosts = config.getString("index.hosts").split(Pattern.quote(","));
        String[] ports = config.getString("index.ports").split(Pattern.quote(","));
        builder.addHostsAndPorts(hosts, ports);
    }
    MultiLabelIndex index = builder.build();
    logger.info("index loaded");
    logger.info("there are " + index.getNumDocs() + " documents in the index.");
    return index;
}
Also used : MultiLabelIndex(edu.neu.ccs.pyramid.elasticsearch.MultiLabelIndex) Config(edu.neu.ccs.pyramid.configuration.Config) File(java.io.File)

Example 17 with Config

use of edu.neu.ccs.pyramid.configuration.Config in project pyramid by cheng-li.

the class TRECFormat method parseNumClasses.

private static int parseNumClasses(File trecFile) throws IOException {
    File configFile = new File(trecFile, TREC_CONFIG_FILE_NAME);
    int numClasses;
    try (BufferedReader br = new BufferedReader(new FileReader(configFile))) {
        Config config = new Config(configFile);
        numClasses = config.getInt(TREC_CONFIG_NUM_CLASSES);
    }
    return numClasses;
}
Also used : Config(edu.neu.ccs.pyramid.configuration.Config)

Example 18 with Config

use of edu.neu.ccs.pyramid.configuration.Config in project pyramid by cheng-li.

the class TRECFormat method writeConfigFile.

private static void writeConfigFile(MultiLabelClfDataSet dataSet, File trecFile) {
    File configFile = new File(trecFile, TREC_CONFIG_FILE_NAME);
    Config config = new Config();
    config.setInt(TREC_CONFIG_NUM_DATA_POINTS, dataSet.getNumDataPoints());
    config.setInt(TREC_CONFIG_NUM_FEATURES, dataSet.getNumFeatures());
    config.setInt(TREC_CONFIG_NUM_CLASSES, dataSet.getNumClasses());
    config.setBoolean(TREC_CONFIG_MISSING_VALUE, dataSet.hasMissingValue());
    try {
        config.store(configFile);
    } catch (Exception e) {
        e.printStackTrace();
    }
}
Also used : Config(edu.neu.ccs.pyramid.configuration.Config)

Example 19 with Config

use of edu.neu.ccs.pyramid.configuration.Config in project pyramid by cheng-li.

the class TRECFormat method parseNumFeaturess.

private static int parseNumFeaturess(File trecFile) throws IOException {
    File configFile = new File(trecFile, TREC_CONFIG_FILE_NAME);
    int numFeatures;
    try (BufferedReader br = new BufferedReader(new FileReader(configFile))) {
        Config config = new Config(configFile);
        numFeatures = config.getInt(TREC_CONFIG_NUM_FEATURES);
    }
    return numFeatures;
}
Also used : Config(edu.neu.ccs.pyramid.configuration.Config)

Example 20 with Config

use of edu.neu.ccs.pyramid.configuration.Config in project pyramid by cheng-li.

the class TRECFormat method parseMissingValue.

private static boolean parseMissingValue(File trecFile) throws IOException {
    File configFile = new File(trecFile, TREC_CONFIG_FILE_NAME);
    boolean missingValue;
    try (BufferedReader br = new BufferedReader(new FileReader(configFile))) {
        Config config = new Config(configFile);
        missingValue = config.getBoolean(TREC_CONFIG_MISSING_VALUE);
    }
    return missingValue;
}
Also used : Config(edu.neu.ccs.pyramid.configuration.Config)

Aggregations

Config (edu.neu.ccs.pyramid.configuration.Config)59 File (java.io.File)35 Collectors (java.util.stream.Collectors)18 FileUtils (org.apache.commons.io.FileUtils)18 StopWatch (org.apache.commons.lang3.time.StopWatch)18 Serialization (edu.neu.ccs.pyramid.util.Serialization)17 BufferedWriter (java.io.BufferedWriter)17 FileWriter (java.io.FileWriter)17 Paths (java.nio.file.Paths)17 IntStream (java.util.stream.IntStream)16 Pair (edu.neu.ccs.pyramid.util.Pair)15 edu.neu.ccs.pyramid.dataset (edu.neu.ccs.pyramid.dataset)14 EarlyStopper (edu.neu.ccs.pyramid.optimization.EarlyStopper)14 java.util (java.util)14 MLMeasures (edu.neu.ccs.pyramid.eval.MLMeasures)13 PrintUtil (edu.neu.ccs.pyramid.util.PrintUtil)13 MultiLabelClassifier (edu.neu.ccs.pyramid.multilabel_classification.MultiLabelClassifier)12 edu.neu.ccs.pyramid.multilabel_classification.cbm (edu.neu.ccs.pyramid.multilabel_classification.cbm)12 ListUtil (edu.neu.ccs.pyramid.util.ListUtil)12 IOException (java.io.IOException)8