Examples with MissingEventsException org.apache.hadoop.hdfs.inotify.MissingEventsException used on opensource projects

Example 1 with MissingEventsException

use of org.apache.hadoop.hdfs.inotify.MissingEventsException in project SSM by Intel-bigdata.

the class InotifyFetchAndApplyTask method run.

@Override
public void run() {
    try {
        EventBatch eventBatch = inotifyEventInputStream.poll();
        while (eventBatch != null) {
            this.applier.apply(eventBatch.getEvents());
            this.lastId.getAndSet(eventBatch.getTxid());
            eventBatch = inotifyEventInputStream.poll();
        }
    } catch (IOException | MissingEventsException | SQLException e) {
        e.printStackTrace();
    }
}

Example 2 with MissingEventsException

use of org.apache.hadoop.hdfs.inotify.MissingEventsException in project hadoop by apache.

the class DFSInotifyEventInputStream method poll.

/**
   * Returns the next batch of events in the stream or null if no new
   * batches are currently available.
   *
   * @throws IOException because of network error or edit log
   * corruption. Also possible if JournalNodes are unresponsive in the
   * QJM setting (even one unresponsive JournalNode is enough in rare cases),
   * so catching this exception and retrying at least a few times is
   * recommended.
   * @throws MissingEventsException if we cannot return the next batch in the
   * stream because the data for the events (and possibly some subsequent
   * events) has been deleted (generally because this stream is a very large
   * number of transactions behind the current state of the NameNode). It is
   * safe to continue reading from the stream after this exception is thrown
   * The next available batch of events will be returned.
   */
public EventBatch poll() throws IOException, MissingEventsException {
    try (TraceScope ignored = tracer.newScope("inotifyPoll")) {
        // need to keep retrying until the NN sends us the latest committed txid
        if (lastReadTxid == -1) {
            LOG.debug("poll(): lastReadTxid is -1, reading current txid from NN");
            lastReadTxid = namenode.getCurrentEditLogTxid();
            return null;
        }
        if (!it.hasNext()) {
            EventBatchList el = namenode.getEditsFromTxid(lastReadTxid + 1);
            if (el.getLastTxid() != -1) {
                // we only want to set syncTxid when we were actually able to read some
                // edits on the NN -- otherwise it will seem like edits are being
                // generated faster than we can read them when the problem is really
                // that we are temporarily unable to read edits
                syncTxid = el.getSyncTxid();
                it = el.getBatches().iterator();
                long formerLastReadTxid = lastReadTxid;
                lastReadTxid = el.getLastTxid();
                if (el.getFirstTxid() != formerLastReadTxid + 1) {
                    throw new MissingEventsException(formerLastReadTxid + 1, el.getFirstTxid());
                }
            } else {
                LOG.debug("poll(): read no edits from the NN when requesting edits " + "after txid {}", lastReadTxid);
                return null;
            }
        }
        if (it.hasNext()) {
            // newly seen edit log ops actually got converted to events
            return it.next();
        } else {
            return null;
        }
    }
}