Search in sources :

Example 1 with FetchedInputCallback

use of org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback in project tez by apache.

the class TestUnorderedKVReader method setupReader.

private void setupReader() throws IOException, InterruptedException {
    defaultConf.set(TezRuntimeConfiguration.TEZ_RUNTIME_KEY_CLASS, Text.class.getName());
    defaultConf.set(TezRuntimeConfiguration.TEZ_RUNTIME_VALUE_CLASS, Text.class.getName());
    createIFile(outputPath, 1);
    final LinkedList<LocalDiskFetchedInput> inputs = new LinkedList<LocalDiskFetchedInput>();
    LocalDiskFetchedInput realFetchedInput = new LocalDiskFetchedInput(0, compLen, new InputAttemptIdentifier(0, 0), outputPath, defaultConf, new FetchedInputCallback() {

        @Override
        public void fetchComplete(FetchedInput fetchedInput) {
        }

        @Override
        public void fetchFailed(FetchedInput fetchedInput) {
        }

        @Override
        public void freeResources(FetchedInput fetchedInput) {
        }
    });
    LocalDiskFetchedInput fetchedInput = spy(realFetchedInput);
    doNothing().when(fetchedInput).free();
    inputs.add(fetchedInput);
    TezCounters counters = new TezCounters();
    TezCounter inputRecords = counters.findCounter(TaskCounter.INPUT_RECORDS_PROCESSED);
    ShuffleManager manager = mock(ShuffleManager.class);
    doAnswer(new Answer() {

        @Override
        public Object answer(InvocationOnMock invocationOnMock) throws Throwable {
            return (inputs.isEmpty()) ? null : inputs.remove();
        }
    }).when(manager).getNextInput();
    unorderedKVReader = new UnorderedKVReader<Text, Text>(manager, defaultConf, null, false, -1, -1, inputRecords, mock(InputContext.class));
}
Also used : LocalDiskFetchedInput(org.apache.tez.runtime.library.common.shuffle.LocalDiskFetchedInput) FetchedInput(org.apache.tez.runtime.library.common.shuffle.FetchedInput) Text(org.apache.hadoop.io.Text) InputAttemptIdentifier(org.apache.tez.runtime.library.common.InputAttemptIdentifier) TezCounter(org.apache.tez.common.counters.TezCounter) LinkedList(java.util.LinkedList) TezCounters(org.apache.tez.common.counters.TezCounters) Answer(org.mockito.stubbing.Answer) Mockito.doAnswer(org.mockito.Mockito.doAnswer) FetchedInputCallback(org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback) InvocationOnMock(org.mockito.invocation.InvocationOnMock) ShuffleManager(org.apache.tez.runtime.library.common.shuffle.impl.ShuffleManager) LocalDiskFetchedInput(org.apache.tez.runtime.library.common.shuffle.LocalDiskFetchedInput)

Aggregations

LinkedList (java.util.LinkedList)1 Text (org.apache.hadoop.io.Text)1 TezCounter (org.apache.tez.common.counters.TezCounter)1 TezCounters (org.apache.tez.common.counters.TezCounters)1 InputAttemptIdentifier (org.apache.tez.runtime.library.common.InputAttemptIdentifier)1 FetchedInput (org.apache.tez.runtime.library.common.shuffle.FetchedInput)1 FetchedInputCallback (org.apache.tez.runtime.library.common.shuffle.FetchedInputCallback)1 LocalDiskFetchedInput (org.apache.tez.runtime.library.common.shuffle.LocalDiskFetchedInput)1 ShuffleManager (org.apache.tez.runtime.library.common.shuffle.impl.ShuffleManager)1 Mockito.doAnswer (org.mockito.Mockito.doAnswer)1 InvocationOnMock (org.mockito.invocation.InvocationOnMock)1 Answer (org.mockito.stubbing.Answer)1