Search in sources :

Example 11 with FastText

use of zemberek.core.embeddings.FastText in project zemberek-nlp by ahmetaa.

the class FastTextTest method skipgram.

/**
 * Generates word vectors using skip-gram model. run with -Xms8G or more.
 */
@Test
@Ignore("Not an actual Test.")
public void skipgram() throws Exception {
    Args argz = Args.forWordVectors(Args.model_name.skipGram);
    argz.thread = 4;
    argz.epoch = 10;
    argz.dim = 100;
    argz.bucket = 2_000_000;
    argz.minn = 3;
    argz.maxn = 6;
    argz.subWordHashProvider = new EmbeddingHashProviders.CharacterNgramHashProvider(argz.minn, argz.maxn);
    Path input = Paths.get("/home/ahmetaa/data/nlp/corpora/sentences.50k");
    Path outRoot = Paths.get("/home/ahmetaa/data/fasttext");
    FastText fastText = new FastTextTrainer(argz).train(input);
    Path vectorFile = outRoot.resolve("sentences-50k-skipgram.vec");
    Log.info("Saving vectors to %s", vectorFile);
    fastText.saveVectors(vectorFile);
}
Also used : Path(java.nio.file.Path) Args(zemberek.core.embeddings.Args) EmbeddingHashProviders(zemberek.core.embeddings.EmbeddingHashProviders) FastTextTrainer(zemberek.core.embeddings.FastTextTrainer) FastText(zemberek.core.embeddings.FastText) Ignore(org.junit.Ignore) Test(org.junit.Test)

Example 12 with FastText

use of zemberek.core.embeddings.FastText in project zemberek-nlp by ahmetaa.

the class FastTextTest method dbpediaClassificationTest.

/**
 * Runs the dbpedia classification task. run with -Xms8G or more.
 */
@Test
@Ignore("Not an actual Test.")
public void dbpediaClassificationTest() throws Exception {
    Path inputRoot = Paths.get("/media/aaa/3t/aaa/fasttext");
    Path trainFile = inputRoot.resolve("dbpedia.train");
    Path modelPath = Paths.get("/media/aaa/3t/aaa/fasttext/dbpedia.model.bin");
    FastText fastText;
    if (modelPath.toFile().exists()) {
        fastText = FastText.load(modelPath);
    } else {
        Args argz = Args.forSupervised();
        argz.thread = 4;
        argz.epoch = 5;
        argz.wordNgrams = 2;
        argz.minCount = 1;
        argz.lr = 0.1;
        argz.dim = 32;
        argz.bucket = 5_000_000;
        fastText = new FastTextTrainer(argz).train(trainFile);
        fastText.saveModel(modelPath);
    }
    Path testFile = inputRoot.resolve("dbpedia.test");
    Log.info("Testing started.");
    EvaluationResult result = fastText.test(testFile, 1);
    Log.info(result.toString());
}
Also used : Path(java.nio.file.Path) Args(zemberek.core.embeddings.Args) FastTextTrainer(zemberek.core.embeddings.FastTextTrainer) EvaluationResult(zemberek.core.embeddings.FastText.EvaluationResult) FastText(zemberek.core.embeddings.FastText) Ignore(org.junit.Ignore) Test(org.junit.Test)

Aggregations

FastText (zemberek.core.embeddings.FastText)12 Path (java.nio.file.Path)7 Args (zemberek.core.embeddings.Args)7 FastTextTrainer (zemberek.core.embeddings.FastTextTrainer)6 Ignore (org.junit.Ignore)4 Test (org.junit.Test)4 PrintWriter (java.io.PrintWriter)2 ArrayList (java.util.ArrayList)2 ScoredItem (zemberek.core.ScoredItem)2 EmbeddingHashProviders (zemberek.core.embeddings.EmbeddingHashProviders)2 EvaluationResult (zemberek.core.embeddings.FastText.EvaluationResult)2 WebDocument (zemberek.corpus.WebDocument)2 FastTextClassifier (zemberek.classification.FastTextClassifier)1 FastTextClassifierTrainer (zemberek.classification.FastTextClassifierTrainer)1 WordVectorsTrainer (zemberek.core.embeddings.WordVectorsTrainer)1 WebCorpus (zemberek.corpus.WebCorpus)1