Examples with Shard com.baidu.hugegraph.structure.graph.Shard used on opensource projects

Example 6 with Shard

use of com.baidu.hugegraph.structure.graph.Shard in project incubator-hugegraph-toolchain by apache.

the class BackupManager method backupEdges.

protected void backupEdges() {
    Printer.print("Edges backup started");
    Printer.printInBackward("Edges has been backup: ");
    List<Shard> shards = retry(() -> this.client.traverser().edgeShards(splitSize()), "querying shards of edges");
    this.writeShards(this.allShardsLog(HugeType.EDGE), shards);
    for (Shard shard : shards) {
        this.backupEdgeShardAsync(shard);
    }
    this.awaitTasks();
    this.postProcessFailedShard(HugeType.EDGE);
    Printer.print("%d", this.edgeCounter.get());
    Printer.print("Edges backup finished: %d", this.edgeCounter.get());
}

Example 7 with Shard

use of com.baidu.hugegraph.structure.graph.Shard in project incubator-hugegraph-toolchain by apache.

the class BackupManager method readShards.

private List<Shard> readShards(File file) {
    E.checkArgument(file.exists() && file.isFile() && file.canRead(), "Need to specify a readable filter file rather than:" + " %s", file.toString());
    List<Shard> shards = new ArrayList<>();
    try (InputStream is = new FileInputStream(file);
        InputStreamReader isr = new InputStreamReader(is, API.CHARSET);
        BufferedReader reader = new BufferedReader(isr)) {
        String line;
        while ((line = reader.readLine()) != null) {
            shards.addAll(this.readList("shards", Shard.class, line));
        }
    } catch (IOException e) {
        throw new ToolsException("IOException occur while reading %s", e, file.getName());
    }
    return shards;
}

Example 8 with Shard

use of com.baidu.hugegraph.structure.graph.Shard in project incubator-hugegraph-toolchain by apache.

the class TraverserManagerTest method testIterateEdgesByShard.

@Test
public void testIterateEdgesByShard() {
    List<Shard> shards = traverser().edgeShards(1 * 1024 * 1024);
    List<Edge> edges = new LinkedList<>();
    for (Shard shard : shards) {
        Iterator<Edge> iter = traverser().iteratorEdges(shard, 1);
        while (iter.hasNext()) {
            edges.add(iter.next());
        }
    }
    Assert.assertEquals(6, edges.size());
}

Example 9 with Shard

use of com.baidu.hugegraph.structure.graph.Shard in project incubator-hugegraph-toolchain by apache.

the class TraverserManagerTest method testIterateVerticesByShard.

@Test
public void testIterateVerticesByShard() {
    List<Shard> shards = traverser().vertexShards(1 * 1024 * 1024);
    List<Vertex> vertices = new LinkedList<>();
    for (Shard shard : shards) {
        Iterator<Vertex> iter = traverser().iteratorVertices(shard, 1);
        while (iter.hasNext()) {
            vertices.add(iter.next());
        }
    }
    Assert.assertEquals(6, vertices.size());
}

Example 10 with Shard

use of com.baidu.hugegraph.structure.graph.Shard in project hugegraph-computer by hugegraph.

the class HugeInputSplitFetcher method fetchVertexInputSplits.

@Override
public List<InputSplit> fetchVertexInputSplits() {
    long splitSize = this.config.get(ComputerOptions.INPUT_SPLITS_SIZE);
    int maxSplits = this.config.get(ComputerOptions.INPUT_MAX_SPLITS);
    List<Shard> shards = this.client.traverser().vertexShards(splitSize);
    E.checkArgument(shards.size() <= maxSplits, "Too many shards due to too small splitSize");
    List<InputSplit> splits = new ArrayList<>();
    for (Shard shard : shards) {
        InputSplit split = new InputSplit(shard.start(), shard.end());
        splits.add(split);
    }
    return splits;
}