Examples with FastTextClassifierTrainer zemberek.classification.FastTextClassifierTrainer used on opensource projects

Example 1 with FastTextClassifierTrainer

use of zemberek.classification.FastTextClassifierTrainer in project zemberek-nlp by ahmetaa.

the class TrainClassifier method run.

@Override
public void run() throws IOException {
    Log.info("Generating classification model from %s", input);
    FastTextClassifierTrainer trainer = FastTextClassifierTrainer.builder().epochCount(epochCount).learningRate(learningRate).lossType(lossType).quantizationCutOff(cutOff).minWordCount(minWordCount).threadCount(threadCount).wordNgramOrder(wordNGrams).dimension(dimension).contextWindowSize(contextWindowSize).build();
    Log.info("Training Started.");
    trainer.getEventBus().register(this);
    FastTextClassifier classifier = trainer.train(input);
    Log.info("Saving classification model to %s", output);
    FastText fastText = classifier.getFastText();
    fastText.saveModel(output);
    if (applyQuantization) {
        Log.info("Applying quantization.");
        if (cutOff > 0) {
            Log.info("Quantization dictionary cut-off value = %d", cutOff);
        }
        Path parent = output.getParent();
        String name = output.toFile().getName() + ".q";
        Path quantizedModel = parent == null ? Paths.get(name) : parent.resolve(name);
        Log.info("Saving quantized classification model to %s", quantizedModel);
        FastText quantized = fastText.quantize(output, fastText.getArgs());
        quantized.saveModel(quantizedModel);
    }
}