Search in sources :

Example 6 with LoggingContext

use of co.cask.cdap.common.logging.LoggingContext in project cdap by caskdata.

the class LogHandler method next.

@GET
@Path("/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/logs/next")
public void next(HttpRequest request, HttpResponder responder, @PathParam("namespace-id") String namespaceId, @PathParam("app-id") String appId, @PathParam("program-type") String programType, @PathParam("program-id") String programId, @QueryParam("max") @DefaultValue("50") int maxEvents, @QueryParam("fromOffset") @DefaultValue("") String fromOffsetStr, @QueryParam("escape") @DefaultValue("true") boolean escape, @QueryParam("filter") @DefaultValue("") String filterStr) {
    LoggingContext loggingContext = LoggingContextHelper.getLoggingContext(namespaceId, appId, programId, ProgramType.valueOfCategoryName(programType));
    doNext(responder, loggingContext, maxEvents, fromOffsetStr, escape, filterStr, null, "text", ImmutableList.<String>of());
}
Also used : LoggingContext(co.cask.cdap.common.logging.LoggingContext) Path(javax.ws.rs.Path) GET(javax.ws.rs.GET)

Example 7 with LoggingContext

use of co.cask.cdap.common.logging.LoggingContext in project cdap by caskdata.

the class LogHandler method runIdNext.

@GET
@Path("/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/runs/{run-id}/logs/next")
public void runIdNext(HttpRequest request, HttpResponder responder, @PathParam("namespace-id") String namespaceId, @PathParam("app-id") String appId, @PathParam("program-type") String programType, @PathParam("program-id") String programId, @PathParam("run-id") String runId, @QueryParam("max") @DefaultValue("50") int maxEvents, @QueryParam("fromOffset") @DefaultValue("") String fromOffsetStr, @QueryParam("escape") @DefaultValue("true") boolean escape, @QueryParam("filter") @DefaultValue("") String filterStr, @QueryParam("format") @DefaultValue("text") String format, @QueryParam("suppress") List<String> suppress) throws NotFoundException {
    ProgramType type = ProgramType.valueOfCategoryName(programType);
    RunRecordMeta runRecord = getRunRecordMeta(namespaceId, appId, type, programId, runId);
    LoggingContext loggingContext = LoggingContextHelper.getLoggingContextWithRunId(namespaceId, appId, programId, type, runId, runRecord.getSystemArgs());
    doNext(responder, loggingContext, maxEvents, fromOffsetStr, escape, filterStr, runRecord, format, suppress);
}
Also used : LoggingContext(co.cask.cdap.common.logging.LoggingContext) RunRecordMeta(co.cask.cdap.internal.app.store.RunRecordMeta) ProgramType(co.cask.cdap.proto.ProgramType) Path(javax.ws.rs.Path) GET(javax.ws.rs.GET)

Example 8 with LoggingContext

use of co.cask.cdap.common.logging.LoggingContext in project cdap by caskdata.

the class LogHandler method sysPrev.

@GET
@Path("/system/{component-id}/{service-id}/logs/prev")
public void sysPrev(HttpRequest request, HttpResponder responder, @PathParam("component-id") String componentId, @PathParam("service-id") String serviceId, @QueryParam("max") @DefaultValue("50") int maxEvents, @QueryParam("fromOffset") @DefaultValue("") String fromOffsetStr, @QueryParam("escape") @DefaultValue("true") boolean escape, @QueryParam("filter") @DefaultValue("") String filterStr, @QueryParam("format") @DefaultValue("text") String format, @QueryParam("suppress") List<String> suppress) {
    LoggingContext loggingContext = LoggingContextHelper.getLoggingContext(Id.Namespace.SYSTEM.getId(), componentId, serviceId);
    doPrev(responder, loggingContext, maxEvents, fromOffsetStr, escape, filterStr, null, format, suppress);
}
Also used : LoggingContext(co.cask.cdap.common.logging.LoggingContext) Path(javax.ws.rs.Path) GET(javax.ws.rs.GET)

Example 9 with LoggingContext

use of co.cask.cdap.common.logging.LoggingContext in project cdap by caskdata.

the class LogHandler method runIdPrev.

@GET
@Path("/namespaces/{namespace-id}/apps/{app-id}/{program-type}/{program-id}/runs/{run-id}/logs/prev")
public void runIdPrev(HttpRequest request, HttpResponder responder, @PathParam("namespace-id") String namespaceId, @PathParam("app-id") String appId, @PathParam("program-type") String programType, @PathParam("program-id") String programId, @PathParam("run-id") String runId, @QueryParam("max") @DefaultValue("50") int maxEvents, @QueryParam("fromOffset") @DefaultValue("") String fromOffsetStr, @QueryParam("escape") @DefaultValue("true") boolean escape, @QueryParam("filter") @DefaultValue("") String filterStr, @QueryParam("format") @DefaultValue("text") String format, @QueryParam("suppress") List<String> suppress) throws NotFoundException {
    ProgramType type = ProgramType.valueOfCategoryName(programType);
    RunRecordMeta runRecord = getRunRecordMeta(namespaceId, appId, type, programId, runId);
    LoggingContext loggingContext = LoggingContextHelper.getLoggingContextWithRunId(namespaceId, appId, programId, type, runId, runRecord.getSystemArgs());
    doPrev(responder, loggingContext, maxEvents, fromOffsetStr, escape, filterStr, runRecord, format, suppress);
}
Also used : LoggingContext(co.cask.cdap.common.logging.LoggingContext) RunRecordMeta(co.cask.cdap.internal.app.store.RunRecordMeta) ProgramType(co.cask.cdap.proto.ProgramType) Path(javax.ws.rs.Path) GET(javax.ws.rs.GET)

Example 10 with LoggingContext

use of co.cask.cdap.common.logging.LoggingContext in project cdap by caskdata.

the class LocalLogAppenderResilientTest method testResilientLogging.

@Test
public void testResilientLogging() throws Exception {
    Configuration hConf = new Configuration();
    CConfiguration cConf = CConfiguration.create();
    File datasetDir = new File(tmpFolder.newFolder(), "datasetUser");
    //noinspection ResultOfMethodCallIgnored
    datasetDir.mkdirs();
    cConf.set(Constants.Dataset.Manager.OUTPUT_DIR, datasetDir.getAbsolutePath());
    cConf.set(Constants.Service.MASTER_SERVICES_BIND_ADDRESS, "localhost");
    cConf.set(Constants.Dataset.Executor.ADDRESS, "localhost");
    cConf.setInt(Constants.Dataset.Executor.PORT, Networks.getRandomPort());
    cConf.set(Constants.CFG_LOCAL_DATA_DIR, tmpFolder.newFolder().getAbsolutePath());
    Injector injector = Guice.createInjector(new ConfigModule(cConf, hConf), new IOModule(), new ZKClientModule(), new KafkaClientModule(), new DiscoveryRuntimeModule().getInMemoryModules(), new NonCustomLocationUnitTestModule().getModule(), new DataFabricModules().getInMemoryModules(), new DataSetsModules().getStandaloneModules(), new DataSetServiceModules().getInMemoryModules(), new TransactionMetricsModule(), new ExploreClientModule(), new LoggingModules().getInMemoryModules(), new NamespaceClientRuntimeModule().getInMemoryModules(), new AuthorizationTestModule(), new AuthorizationEnforcementModule().getInMemoryModules(), new AuthenticationContextModules().getMasterModule(), new AbstractModule() {

        @Override
        protected void configure() {
            bind(UGIProvider.class).to(UnsupportedUGIProvider.class);
            bind(OwnerAdmin.class).to(NoOpOwnerAdmin.class);
        }
    });
    TransactionManager txManager = injector.getInstance(TransactionManager.class);
    txManager.startAndWait();
    DatasetOpExecutorService opExecutorService = injector.getInstance(DatasetOpExecutorService.class);
    opExecutorService.startAndWait();
    // Start the logging before starting the service.
    LoggingContextAccessor.setLoggingContext(new FlowletLoggingContext("TRL_ACCT_1", "APP_1", "FLOW_1", "FLOWLET_1", "RUN", "INSTANCE"));
    String logBaseDir = "trl-log/log_files_" + new Random(System.currentTimeMillis()).nextLong();
    cConf.set(LoggingConfiguration.LOG_BASE_DIR, logBaseDir);
    cConf.setInt(LoggingConfiguration.LOG_MAX_FILE_SIZE_BYTES, 20 * 1024);
    final LogAppender appender = injector.getInstance(LocalLogAppender.class);
    new LogAppenderInitializer(appender).initialize("TestResilientLogging");
    int failureMsgCount = 3;
    final CountDownLatch failureLatch = new CountDownLatch(failureMsgCount);
    LoggerContext loggerContext = (LoggerContext) LoggerFactory.getILoggerFactory();
    loggerContext.getStatusManager().add(new StatusListener() {

        @Override
        public void addStatusEvent(Status status) {
            if (status.getLevel() != Status.ERROR || status.getOrigin() != appender) {
                return;
            }
            Throwable cause = status.getThrowable();
            if (cause != null) {
                Throwable rootCause = Throwables.getRootCause(cause);
                if (rootCause instanceof ServiceUnavailableException) {
                    String serviceName = ((ServiceUnavailableException) rootCause).getServiceName();
                    if (Constants.Service.DATASET_MANAGER.equals(serviceName)) {
                        failureLatch.countDown();
                    }
                }
            }
        }
    });
    Logger logger = LoggerFactory.getLogger("TestResilientLogging");
    for (int i = 0; i < failureMsgCount; ++i) {
        Exception e1 = new Exception("Test Exception1");
        Exception e2 = new Exception("Test Exception2", e1);
        logger.warn("Test log message " + i + " {} {}", "arg1", "arg2", e2);
    }
    // Wait for the three failure to append to happen
    // The wait time has to be > 3 seconds because DatasetServiceClient has 1 second timeout on discovery
    failureLatch.await(5, TimeUnit.SECONDS);
    // Start dataset service, wait for it to be discoverable
    DatasetService dsService = injector.getInstance(DatasetService.class);
    dsService.startAndWait();
    final CountDownLatch startLatch = new CountDownLatch(1);
    DiscoveryServiceClient discoveryClient = injector.getInstance(DiscoveryServiceClient.class);
    discoveryClient.discover(Constants.Service.DATASET_MANAGER).watchChanges(new ServiceDiscovered.ChangeListener() {

        @Override
        public void onChange(ServiceDiscovered serviceDiscovered) {
            if (!Iterables.isEmpty(serviceDiscovered)) {
                startLatch.countDown();
            }
        }
    }, Threads.SAME_THREAD_EXECUTOR);
    startLatch.await(5, TimeUnit.SECONDS);
    // Do some more logging after the service is started.
    for (int i = 5; i < 10; ++i) {
        Exception e1 = new Exception("Test Exception1");
        Exception e2 = new Exception("Test Exception2", e1);
        logger.warn("Test log message " + i + " {} {}", "arg1", "arg2", e2);
    }
    appender.stop();
    // Verify - we should have at least 5 events.
    LoggingContext loggingContext = new FlowletLoggingContext("TRL_ACCT_1", "APP_1", "FLOW_1", "", "RUN", "INSTANCE");
    FileLogReader logTail = injector.getInstance(FileLogReader.class);
    LoggingTester.LogCallback logCallback1 = new LoggingTester.LogCallback();
    logTail.getLogPrev(loggingContext, ReadRange.LATEST, 10, Filter.EMPTY_FILTER, logCallback1);
    List<LogEvent> allEvents = logCallback1.getEvents();
    Assert.assertTrue(allEvents.toString(), allEvents.size() >= 5);
    // Finally - stop all services
    Services.chainStop(dsService, opExecutorService, txManager);
}
Also used : NoOpOwnerAdmin(co.cask.cdap.security.impersonation.NoOpOwnerAdmin) DataSetServiceModules(co.cask.cdap.data.runtime.DataSetServiceModules) DiscoveryServiceClient(org.apache.twill.discovery.DiscoveryServiceClient) CConfiguration(co.cask.cdap.common.conf.CConfiguration) Configuration(org.apache.hadoop.conf.Configuration) LoggingConfiguration(co.cask.cdap.logging.LoggingConfiguration) LocalLogAppender(co.cask.cdap.logging.framework.local.LocalLogAppender) DatasetService(co.cask.cdap.data2.datafabric.dataset.service.DatasetService) TransactionMetricsModule(co.cask.cdap.data.runtime.TransactionMetricsModule) FileLogReader(co.cask.cdap.logging.read.FileLogReader) Random(java.util.Random) Injector(com.google.inject.Injector) KafkaClientModule(co.cask.cdap.common.guice.KafkaClientModule) FlowletLoggingContext(co.cask.cdap.logging.context.FlowletLoggingContext) DiscoveryRuntimeModule(co.cask.cdap.common.guice.DiscoveryRuntimeModule) Status(ch.qos.logback.core.status.Status) LogEvent(co.cask.cdap.logging.read.LogEvent) AuthenticationContextModules(co.cask.cdap.security.auth.context.AuthenticationContextModules) ServiceDiscovered(org.apache.twill.discovery.ServiceDiscovered) NonCustomLocationUnitTestModule(co.cask.cdap.common.guice.NonCustomLocationUnitTestModule) CConfiguration(co.cask.cdap.common.conf.CConfiguration) ExploreClientModule(co.cask.cdap.explore.guice.ExploreClientModule) TransactionManager(org.apache.tephra.TransactionManager) File(java.io.File) DataFabricModules(co.cask.cdap.data.runtime.DataFabricModules) AuthorizationEnforcementModule(co.cask.cdap.security.authorization.AuthorizationEnforcementModule) IOModule(co.cask.cdap.common.guice.IOModule) NamespaceClientRuntimeModule(co.cask.cdap.common.namespace.guice.NamespaceClientRuntimeModule) LoggingContext(co.cask.cdap.common.logging.LoggingContext) FlowletLoggingContext(co.cask.cdap.logging.context.FlowletLoggingContext) ConfigModule(co.cask.cdap.common.guice.ConfigModule) ServiceUnavailableException(co.cask.cdap.common.ServiceUnavailableException) Logger(org.slf4j.Logger) ZKClientModule(co.cask.cdap.common.guice.ZKClientModule) UnsupportedUGIProvider(co.cask.cdap.security.impersonation.UnsupportedUGIProvider) DataSetsModules(co.cask.cdap.data.runtime.DataSetsModules) CountDownLatch(java.util.concurrent.CountDownLatch) AuthorizationTestModule(co.cask.cdap.security.authorization.AuthorizationTestModule) LoggerContext(ch.qos.logback.classic.LoggerContext) ServiceUnavailableException(co.cask.cdap.common.ServiceUnavailableException) LoggingModules(co.cask.cdap.logging.guice.LoggingModules) AbstractModule(com.google.inject.AbstractModule) DatasetOpExecutorService(co.cask.cdap.data2.datafabric.dataset.service.executor.DatasetOpExecutorService) StatusListener(ch.qos.logback.core.status.StatusListener) Test(org.junit.Test)

Aggregations

LoggingContext (co.cask.cdap.common.logging.LoggingContext)30 GET (javax.ws.rs.GET)12 Path (javax.ws.rs.Path)12 Test (org.junit.Test)11 FlowletLoggingContext (co.cask.cdap.logging.context.FlowletLoggingContext)9 RunRecordMeta (co.cask.cdap.internal.app.store.RunRecordMeta)6 LoggingTester (co.cask.cdap.logging.appender.LoggingTester)5 FileLogReader (co.cask.cdap.logging.read.FileLogReader)4 LogEvent (co.cask.cdap.logging.read.LogEvent)4 CConfiguration (co.cask.cdap.common.conf.CConfiguration)3 ApplicationLoggingContext (co.cask.cdap.common.logging.ApplicationLoggingContext)3 LogPathIdentifier (co.cask.cdap.logging.appender.system.LogPathIdentifier)3 FileMetaDataReader (co.cask.cdap.logging.meta.FileMetaDataReader)3 LogLocation (co.cask.cdap.logging.write.LogLocation)3 ProgramType (co.cask.cdap.proto.ProgramType)3 Transactional (co.cask.cdap.api.Transactional)2 DatasetManager (co.cask.cdap.api.dataset.DatasetManager)2 NamespaceLoggingContext (co.cask.cdap.common.logging.NamespaceLoggingContext)2 ServiceLoggingContext (co.cask.cdap.common.logging.ServiceLoggingContext)2 SystemDatasetInstantiator (co.cask.cdap.data.dataset.SystemDatasetInstantiator)2