Search in sources :

Example 71 with UserGroupInformation

use of org.apache.hadoop.security.UserGroupInformation in project hadoop by apache.

the class TestAggregatedLogFormat method writeSrcFileAndALog.

private void writeSrcFileAndALog(Path srcFilePath, String fileName, final long length, Path remoteAppLogFile, Path srcFileRoot, ContainerId testContainerId) throws Exception {
    File dir = new File(srcFilePath.toString());
    if (!dir.exists()) {
        if (!dir.mkdirs()) {
            throw new IOException("Unable to create directory : " + dir);
        }
    }
    File outputFile = new File(new File(srcFilePath.toString()), fileName);
    FileOutputStream os = new FileOutputStream(outputFile);
    final OutputStreamWriter osw = new OutputStreamWriter(os, "UTF8");
    final int ch = filler;
    UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
    LogWriter logWriter = new LogWriter(new Configuration(), remoteAppLogFile, ugi);
    LogKey logKey = new LogKey(testContainerId);
    LogValue logValue = spy(new LogValue(Collections.singletonList(srcFileRoot.toString()), testContainerId, ugi.getShortUserName()));
    final CountDownLatch latch = new CountDownLatch(1);
    Thread t = new Thread() {

        public void run() {
            try {
                for (int i = 0; i < length / 3; i++) {
                    osw.write(ch);
                }
                latch.countDown();
                for (int i = 0; i < (2 * length) / 3; i++) {
                    osw.write(ch);
                }
                osw.close();
            } catch (IOException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        }
    };
    t.start();
    //Wait till the osw is partially written
    //aggregation starts once the ows has completed 1/3rd of its work
    latch.await();
    //Aggregate The Logs
    logWriter.append(logKey, logValue);
    logWriter.close();
}
Also used : Configuration(org.apache.hadoop.conf.Configuration) LogValue(org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat.LogValue) LogWriter(org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat.LogWriter) FileOutputStream(java.io.FileOutputStream) LogKey(org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat.LogKey) OutputStreamWriter(java.io.OutputStreamWriter) IOException(java.io.IOException) CountDownLatch(java.util.concurrent.CountDownLatch) File(java.io.File) UserGroupInformation(org.apache.hadoop.security.UserGroupInformation)

Example 72 with UserGroupInformation

use of org.apache.hadoop.security.UserGroupInformation in project hadoop by apache.

the class TestAggregatedLogsBlock method writeLog.

private void writeLog(Configuration configuration, String user) throws Exception {
    ApplicationId appId = ApplicationIdPBImpl.newInstance(0, 1);
    ApplicationAttemptId appAttemptId = ApplicationAttemptIdPBImpl.newInstance(appId, 1);
    ContainerId containerId = ContainerIdPBImpl.newContainerId(appAttemptId, 1);
    String path = "target/logs/" + user + "/logs/application_0_0001/localhost_1234";
    File f = new File(path);
    if (!f.getParentFile().exists()) {
        assertTrue(f.getParentFile().mkdirs());
    }
    List<String> rootLogDirs = Arrays.asList("target/logs/logs");
    UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
    AggregatedLogFormat.LogWriter writer = new AggregatedLogFormat.LogWriter(configuration, new Path(path), ugi);
    writer.writeApplicationOwner(ugi.getUserName());
    Map<ApplicationAccessType, String> appAcls = new HashMap<ApplicationAccessType, String>();
    appAcls.put(ApplicationAccessType.VIEW_APP, ugi.getUserName());
    writer.writeApplicationACLs(appAcls);
    writer.append(new AggregatedLogFormat.LogKey("container_0_0001_01_000001"), new AggregatedLogFormat.LogValue(rootLogDirs, containerId, UserGroupInformation.getCurrentUser().getShortUserName()));
    writer.close();
}
Also used : Path(org.apache.hadoop.fs.Path) HashMap(java.util.HashMap) ApplicationAttemptId(org.apache.hadoop.yarn.api.records.ApplicationAttemptId) ContainerId(org.apache.hadoop.yarn.api.records.ContainerId) ApplicationAccessType(org.apache.hadoop.yarn.api.records.ApplicationAccessType) ApplicationId(org.apache.hadoop.yarn.api.records.ApplicationId) File(java.io.File) UserGroupInformation(org.apache.hadoop.security.UserGroupInformation)

Example 73 with UserGroupInformation

use of org.apache.hadoop.security.UserGroupInformation in project hadoop by apache.

the class TestTimelineClientForATS1_5 method createTimelineClient.

private TimelineClientImpl createTimelineClient(YarnConfiguration conf) {
    TimelineClientImpl client = new TimelineClientImpl() {

        @Override
        protected TimelineWriter createTimelineWriter(Configuration conf, UserGroupInformation authUgi, Client client, URI resURI) throws IOException {
            TimelineWriter timelineWriter = new FileSystemTimelineWriter(conf, authUgi, client, resURI) {

                public ClientResponse doPostingObject(Object object, String path) {
                    ClientResponse response = mock(ClientResponse.class);
                    when(response.getStatusInfo()).thenReturn(ClientResponse.Status.OK);
                    return response;
                }
            };
            spyTimelineWriter = spy(timelineWriter);
            return spyTimelineWriter;
        }
    };
    client.init(conf);
    client.start();
    return client;
}
Also used : ClientResponse(com.sun.jersey.api.client.ClientResponse) YarnConfiguration(org.apache.hadoop.yarn.conf.YarnConfiguration) Configuration(org.apache.hadoop.conf.Configuration) Client(com.sun.jersey.api.client.Client) URI(java.net.URI) UserGroupInformation(org.apache.hadoop.security.UserGroupInformation)

Example 74 with UserGroupInformation

use of org.apache.hadoop.security.UserGroupInformation in project hadoop by apache.

the class HtmlBlock method getCallerUGI.

protected UserGroupInformation getCallerUGI() {
    // Check for the authorization.
    String remoteUser = request().getRemoteUser();
    UserGroupInformation callerUGI = null;
    if (remoteUser != null) {
        callerUGI = UserGroupInformation.createRemoteUser(remoteUser);
    }
    return callerUGI;
}
Also used : UserGroupInformation(org.apache.hadoop.security.UserGroupInformation)

Example 75 with UserGroupInformation

use of org.apache.hadoop.security.UserGroupInformation in project hadoop by apache.

the class WebServices method getApp.

public AppInfo getApp(HttpServletRequest req, HttpServletResponse res, String appId) {
    UserGroupInformation callerUGI = getUser(req);
    final ApplicationId id = parseApplicationId(appId);
    ApplicationReport app = null;
    try {
        if (callerUGI == null) {
            GetApplicationReportRequest request = GetApplicationReportRequest.newInstance(id);
            app = appBaseProt.getApplicationReport(request).getApplicationReport();
        } else {
            app = callerUGI.doAs(new PrivilegedExceptionAction<ApplicationReport>() {

                @Override
                public ApplicationReport run() throws Exception {
                    GetApplicationReportRequest request = GetApplicationReportRequest.newInstance(id);
                    return appBaseProt.getApplicationReport(request).getApplicationReport();
                }
            });
        }
    } catch (Exception e) {
        rewrapAndThrowException(e);
    }
    if (app == null) {
        throw new NotFoundException("app with id: " + appId + " not found");
    }
    return new AppInfo(app);
}
Also used : ApplicationReport(org.apache.hadoop.yarn.api.records.ApplicationReport) GetApplicationReportRequest(org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest) ContainerNotFoundException(org.apache.hadoop.yarn.exceptions.ContainerNotFoundException) NotFoundException(org.apache.hadoop.yarn.webapp.NotFoundException) ApplicationAttemptNotFoundException(org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException) ApplicationNotFoundException(org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException) PrivilegedExceptionAction(java.security.PrivilegedExceptionAction) ApplicationId(org.apache.hadoop.yarn.api.records.ApplicationId) ForbiddenException(org.apache.hadoop.yarn.webapp.ForbiddenException) AuthorizationException(org.apache.hadoop.security.authorize.AuthorizationException) ContainerNotFoundException(org.apache.hadoop.yarn.exceptions.ContainerNotFoundException) NotFoundException(org.apache.hadoop.yarn.webapp.NotFoundException) ApplicationAttemptNotFoundException(org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException) UndeclaredThrowableException(java.lang.reflect.UndeclaredThrowableException) WebApplicationException(javax.ws.rs.WebApplicationException) ApplicationNotFoundException(org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException) BadRequestException(org.apache.hadoop.yarn.webapp.BadRequestException) UserGroupInformation(org.apache.hadoop.security.UserGroupInformation) AppInfo(org.apache.hadoop.yarn.server.webapp.dao.AppInfo)

Aggregations

UserGroupInformation (org.apache.hadoop.security.UserGroupInformation)664 IOException (java.io.IOException)281 Test (org.junit.Test)242 Configuration (org.apache.hadoop.conf.Configuration)142 Path (org.apache.hadoop.fs.Path)105 FileSystem (org.apache.hadoop.fs.FileSystem)73 YarnException (org.apache.hadoop.yarn.exceptions.YarnException)57 AccessControlException (org.apache.hadoop.security.AccessControlException)54 PrivilegedExceptionAction (java.security.PrivilegedExceptionAction)51 FsPermission (org.apache.hadoop.fs.permission.FsPermission)49 Path (javax.ws.rs.Path)47 Token (org.apache.hadoop.security.token.Token)46 Produces (javax.ws.rs.Produces)45 ApplicationId (org.apache.hadoop.yarn.api.records.ApplicationId)45 RMApp (org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMApp)43 AuthorizationException (org.apache.hadoop.security.authorize.AuthorizationException)40 ArrayList (java.util.ArrayList)38 Text (org.apache.hadoop.io.Text)38 ApplicationAttemptId (org.apache.hadoop.yarn.api.records.ApplicationAttemptId)36 YarnConfiguration (org.apache.hadoop.yarn.conf.YarnConfiguration)35