Examples with AlluxioLineage alluxio.client.lineage.AlluxioLineage used on opensource projects

Example 1 with AlluxioLineage

use of alluxio.client.lineage.AlluxioLineage in project alluxio by Alluxio.

the class LineageMasterIntegrationTest method docExample.

/**
   * Runs code given in the docs (http://alluxio.org/documentation/Lineage-API.html) to make sure it
   * actually runs.
   *
   * If you need to update the doc-code here, make sure you also update it in the docs.
   */
@Test
public void docExample() throws Exception {
    // create input files
    FileSystem fs = FileSystem.Factory.get();
    fs.createFile(new AlluxioURI("/inputFile1")).close();
    fs.createFile(new AlluxioURI("/inputFile2")).close();
    // ------ code block from docs ------
    AlluxioLineage tl = AlluxioLineage.get();
    // input file paths
    AlluxioURI input1 = new AlluxioURI("/inputFile1");
    AlluxioURI input2 = new AlluxioURI("/inputFile2");
    ArrayList<AlluxioURI> inputFiles = new ArrayList<>();
    Collections.addAll(inputFiles, input1, input2);
    // output file paths
    AlluxioURI output = new AlluxioURI("/outputFile");
    ArrayList<AlluxioURI> outputFiles = new ArrayList<>();
    Collections.addAll(outputFiles, output);
    // command-line job
    JobConf conf = new JobConf("/tmp/recompute.log");
    CommandLineJob job = new CommandLineJob("my-spark-job.sh", conf);
    long lineageId = tl.createLineage(inputFiles, outputFiles, job);
    // ------ code block from docs ------
    DeleteLineageOptions options = DeleteLineageOptions.defaults().setCascade(true);
    tl.deleteLineage(lineageId);
    fs.delete(new AlluxioURI("/outputFile"));
    lineageId = tl.createLineage(inputFiles, outputFiles, job);
    // ------ code block from docs ------
    tl.deleteLineage(lineageId, options);
}

Example 2 with AlluxioLineage

use of alluxio.client.lineage.AlluxioLineage in project alluxio by Alluxio.

the class CreateLineageCommand method run.

@Override
public void run(CommandLine cl) throws AlluxioException, IOException {
    String[] args = cl.getArgs();
    AlluxioLineage tl = AlluxioLineage.get(LineageContext.INSTANCE);
    // TODO(yupeng) more validation
    List<AlluxioURI> inputFiles = new ArrayList<>();
    if (!args[0].equals("noInput")) {
        for (String path : args[0].split(",")) {
            inputFiles.add(new AlluxioURI(path));
        }
    }
    List<AlluxioURI> outputFiles = new ArrayList<>();
    for (String path : args[1].split(",")) {
        outputFiles.add(new AlluxioURI(path));
    }
    String cmd = "";
    for (int i = 2; i < args.length; i++) {
        cmd += args[i] + " ";
    }
    String outputPath = Configuration.get(PropertyKey.MASTER_LINEAGE_RECOMPUTE_LOG_PATH);
    if (outputPath == null) {
        throw new IOException("recompute output log is not configured");
    }
    CommandLineJob job = new CommandLineJob(cmd, new JobConf(outputPath));
    long lineageId = tl.createLineage(inputFiles, outputFiles, job);
    System.out.println("Lineage " + lineageId + " has been created.");
}

Example 3 with AlluxioLineage

use of alluxio.client.lineage.AlluxioLineage in project alluxio by Alluxio.

the class DeleteLineageCommand method run.

@Override
public void run(CommandLine cl) throws AlluxioException, IOException {
    String[] args = cl.getArgs();
    AlluxioLineage tl = AlluxioLineage.get(LineageContext.INSTANCE);
    long lineageId = Long.parseLong(args[0]);
    boolean cascade = Boolean.parseBoolean(args[1]);
    DeleteLineageOptions options = DeleteLineageOptions.defaults().setCascade(cascade);
    tl.deleteLineage(lineageId, options);
    System.out.println("Lineage " + lineageId + " has been deleted.");
}

Example 4 with AlluxioLineage

use of alluxio.client.lineage.AlluxioLineage in project alluxio by Alluxio.

the class ListLineagesCommand method run.

@Override
public void run(CommandLine cl) throws IOException {
    AlluxioLineage tl = AlluxioLineage.get(LineageContext.INSTANCE);
    List<LineageInfo> infos = tl.getLineageInfoList();
    for (LineageInfo info : infos) {
        System.out.println(info);
    }
}