Search in sources :

Example 1 with ClientIOTaskResult

use of alluxio.stress.client.ClientIOTaskResult in project alluxio by Alluxio.

the class StressClientIOBenchIntegrationTest method writeTypeALLTaskTest.

@Test
public void writeTypeALLTaskTest() throws Exception {
    // redirect the output stream
    ByteArrayOutputStream out = new ByteArrayOutputStream();
    PrintStream originalOut = System.out;
    System.setOut(new PrintStream(out));
    new StressClientIOBench().run(new String[] { "--in-process", "--base", sLocalAlluxioClusterResource.get().getMasterURI() + "/client/", "--operation", "Write", "--threads", "2", "--file-size", "1m", "--buffer-size", "128k", "--warmup", "0s", "--duration", "1s", "--write-type", "ALL" });
    String printOutResult = out.toString();
    List<String> resultList = getJsonResult(printOutResult);
    assertEquals(resultList.size(), 4);
    // the possible write types
    List<String> writeTypes = ImmutableList.of("MUST_CACHE", "CACHE_THROUGH", "ASYNC_THROUGH", "THROUGH");
    for (int i = 0; i < resultList.size(); i++) {
        ClientIOTaskResult summary = (ClientIOTaskResult) JsonSerializable.fromJson(resultList.get(i));
        // confirm that the task was executed with certain write type and output no errors
        assertEquals(summary.getParameters().mWriteType, writeTypes.get(i));
        assertFalse(summary.getThreadCountResults().isEmpty());
        for (ClientIOTaskResult.ThreadCountResult threadResult : summary.getThreadCountResults().values()) {
            assertTrue(threadResult.getErrors().isEmpty());
        }
    }
    // reset the output to the console
    System.setOut(originalOut);
}
Also used : PrintStream(java.io.PrintStream) ByteArrayOutputStream(java.io.ByteArrayOutputStream) ClientIOTaskResult(alluxio.stress.client.ClientIOTaskResult) StressClientIOBench(alluxio.stress.cli.client.StressClientIOBench) Test(org.junit.Test)

Example 2 with ClientIOTaskResult

use of alluxio.stress.client.ClientIOTaskResult in project alluxio by Alluxio.

the class StressClientIOBench method runLocal.

@Override
@SuppressFBWarnings("BC_UNCONFIRMED_CAST")
public ClientIOTaskResult runLocal() throws Exception {
    List<Integer> threadCounts = new ArrayList<>(mParameters.mThreads);
    threadCounts.sort(Comparator.comparingInt(i -> i));
    ClientIOTaskResult taskResult = new ClientIOTaskResult();
    taskResult.setBaseParameters(mBaseParameters);
    taskResult.setParameters(mParameters);
    for (Integer numThreads : threadCounts) {
        ClientIOTaskResult.ThreadCountResult threadCountResult = runForThreadCount(numThreads);
        if (!mBaseParameters.mProfileAgent.isEmpty()) {
            taskResult.putTimeToFirstBytePerThread(numThreads, addAdditionalResult(threadCountResult.getRecordStartMs(), threadCountResult.getEndMs()));
        }
        taskResult.addThreadCountResults(numThreads, threadCountResult);
    }
    return taskResult;
}
Also used : CreateFilePOptions(alluxio.grpc.CreateFilePOptions) AbstractStressBench(alluxio.stress.cli.AbstractStressBench) Arrays(java.util.Arrays) FileSystem(org.apache.hadoop.fs.FileSystem) SummaryStatistics(alluxio.stress.common.SummaryStatistics) LoggerFactory(org.slf4j.LoggerFactory) HashMap(java.util.HashMap) Random(java.util.Random) Callable(java.util.concurrent.Callable) StressConstants(alluxio.stress.StressConstants) FileOutStream(alluxio.client.file.FileOutStream) PropertyKey(alluxio.conf.PropertyKey) ByteBuffer(java.nio.ByteBuffer) ArrayList(java.util.ArrayList) ClientIOParameters(alluxio.stress.client.ClientIOParameters) FSDataOutputStream(org.apache.hadoop.fs.FSDataOutputStream) ClientIOOperation(alluxio.stress.client.ClientIOOperation) Constants(alluxio.Constants) AlluxioURI(alluxio.AlluxioURI) FileInStream(alluxio.client.file.FileInStream) Configuration(org.apache.hadoop.conf.Configuration) Map(java.util.Map) FormatUtils(alluxio.util.FormatUtils) Path(org.apache.hadoop.fs.Path) URI(java.net.URI) FSDataInputStream(org.apache.hadoop.fs.FSDataInputStream) ExecutorService(java.util.concurrent.ExecutorService) HadoopConfigurationUtils(alluxio.hadoop.HadoopConfigurationUtils) Logger(org.slf4j.Logger) Iterator(java.util.Iterator) FileSystemClientType(alluxio.stress.common.FileSystemClientType) AlluxioException(alluxio.exception.AlluxioException) IOException(java.io.IOException) SuppressFBWarnings(alluxio.annotation.SuppressFBWarnings) ClientIOTaskResult(alluxio.stress.client.ClientIOTaskResult) ConfigurationUtils(alluxio.util.ConfigurationUtils) TimeUnit(java.util.concurrent.TimeUnit) List(java.util.List) Source(alluxio.conf.Source) BaseParameters(alluxio.stress.BaseParameters) Comparator(java.util.Comparator) InstancedConfiguration(alluxio.conf.InstancedConfiguration) ExecutorServiceFactories(alluxio.util.executor.ExecutorServiceFactories) CommonUtils(alluxio.util.CommonUtils) ArrayList(java.util.ArrayList) ClientIOTaskResult(alluxio.stress.client.ClientIOTaskResult) SuppressFBWarnings(alluxio.annotation.SuppressFBWarnings)

Example 3 with ClientIOTaskResult

use of alluxio.stress.client.ClientIOTaskResult in project alluxio by Alluxio.

the class StressClientIOBenchIntegrationTest method validateTheResultWithWriteType.

private void validateTheResultWithWriteType(String writeType) throws Exception {
    String output = new StressClientIOBench().run(new String[] { "--in-process", "--start-ms", Long.toString(System.currentTimeMillis() + 5000), "--base", sLocalAlluxioClusterResource.get().getMasterURI() + "/client/", "--operation", "Write", "--threads", "2", "--file-size", "1m", "--buffer-size", "128k", "--warmup", "0s", "--duration", "1s", "--write-type", writeType });
    // convert the result into summary, and check whether it have errors.
    ClientIOTaskResult summary = (ClientIOTaskResult) JsonSerializable.fromJson(output);
    assertFalse(summary.getThreadCountResults().isEmpty());
    for (ClientIOTaskResult.ThreadCountResult threadResult : summary.getThreadCountResults().values()) {
        assertTrue(threadResult.getErrors().isEmpty());
    }
}
Also used : ClientIOTaskResult(alluxio.stress.client.ClientIOTaskResult) StressClientIOBench(alluxio.stress.cli.client.StressClientIOBench)

Aggregations

ClientIOTaskResult (alluxio.stress.client.ClientIOTaskResult)3 StressClientIOBench (alluxio.stress.cli.client.StressClientIOBench)2 AlluxioURI (alluxio.AlluxioURI)1 Constants (alluxio.Constants)1 SuppressFBWarnings (alluxio.annotation.SuppressFBWarnings)1 FileInStream (alluxio.client.file.FileInStream)1 FileOutStream (alluxio.client.file.FileOutStream)1 InstancedConfiguration (alluxio.conf.InstancedConfiguration)1 PropertyKey (alluxio.conf.PropertyKey)1 Source (alluxio.conf.Source)1 AlluxioException (alluxio.exception.AlluxioException)1 CreateFilePOptions (alluxio.grpc.CreateFilePOptions)1 HadoopConfigurationUtils (alluxio.hadoop.HadoopConfigurationUtils)1 BaseParameters (alluxio.stress.BaseParameters)1 StressConstants (alluxio.stress.StressConstants)1 AbstractStressBench (alluxio.stress.cli.AbstractStressBench)1 ClientIOOperation (alluxio.stress.client.ClientIOOperation)1 ClientIOParameters (alluxio.stress.client.ClientIOParameters)1 FileSystemClientType (alluxio.stress.common.FileSystemClientType)1 SummaryStatistics (alluxio.stress.common.SummaryStatistics)1