Examples with XLogFilter org.apache.oozie.util.XLogFilter used on opensource projects

Search in sources :

Example 1 with XLogFilter

use of org.apache.oozie.util.XLogFilter in project oozie by apache.

the class TestZKXLogStreamingService method testStreamingWithMultipleOozieServers.

public void testStreamingWithMultipleOozieServers() throws Exception {
    XLogFilter.reset();
    XLogFilter.defineParameter("USER");
    XLogFilter.defineParameter("GROUP");
    XLogFilter.defineParameter("TOKEN");
    XLogFilter.defineParameter("APP");
    XLogFilter.defineParameter("JOB");
    XLogFilter.defineParameter("ACTION");
    XLogFilter xf = new XLogFilter();
    xf.setParameter("JOB", "0000003-130610102426873-oozie-rkan-W");
    xf.setLogLevel("WARN|INFO");
    File log4jFile = new File(getTestCaseConfDir(), "test-log4j.properties");
    ClassLoader cl = Thread.currentThread().getContextClassLoader();
    InputStream is = cl.getResourceAsStream("test-no-dash-log4j.properties");
    Properties log4jProps = new Properties();
    log4jProps.load(is);
    // prevent conflicts with other tests by changing the log file location
    log4jProps.setProperty("log4j.appender.oozie.File", getTestCaseDir() + "/oozie.log");
    log4jProps.store(new FileOutputStream(log4jFile), "");
    setSystemProperty(XLogService.LOG4J_FILE, log4jFile.getName());
    assertFalse(doStreamDisabledCheck());
    File logFile = new File(Services.get().get(XLogService.class).getOozieLogPath(), Services.get().get(XLogService.class).getOozieLogName());
    logFile.getParentFile().mkdirs();
    FileWriter logWriter = new FileWriter(logFile);
    // local logs
    logWriter.append("2013-06-10 10:25:44,008 WARN HiveActionExecutor:542 SERVER[foo] USER[rkanter] GROUP[-] TOKEN[] " + "APP[hive-wf] JOB[0000003-130610102426873-oozie-rkan-W] ACTION[0000003-130610102426873-oozie-rkan-W@hive-node] " + "credentials is null for the action _L3_").append("\n").append("2013-06-10 10:26:10,008 INFO HiveActionExecutor:539 SERVER[foo] USER[rkanter] GROUP[-] TOKEN[] " + "APP[hive-wf] JOB[0000003-130610102426873-oozie-rkan-W] ACTION[0000003-130610102426873-oozie-rkan-W@hive-node] " + "action completed, external ID [job_201306101021_0005] _L4_").append("\n").append("2013-06-10 10:26:10,341 WARN ActionStartXCommand:542 USER[rkanter] GROUP[-] TOKEN[] " + "APP[hive-wf] JOB[0000003-130610102426873-oozie-rkan-W] ACTION[0000003-130610102426873-oozie-rkan-W@end] " + "[***0000003-130610102426873-oozie-rkan-W@end***]Action updated in DB! _L6_").append("\n");
    logWriter.close();
    // logs to be returned by another "Oozie server"
    DummyLogStreamingServlet.logs = "2013-06-10 10:25:43,575 WARN ActionStartXCommand:542 SERVER[foo] USER[rkanter] GROUP[-] TOKEN[] APP[hive-wf] " + "JOB[0000003-130610102426873-oozie-rkan-W] ACTION[0000003-130610102426873-oozie-rkan-W@:start:] " + "[***0000003-130610102426873-oozie-rkan-W@:start:***]Action status=DONE _L1_" + "\n" + "2013-06-10 10:25:43,575 WARN ActionStartXCommand:542 SERVER[foo] USER[rkanter] GROUP[-] TOKEN[] APP[hive-wf] " + "JOB[0000003-130610102426873-oozie-rkan-W] ACTION[0000003-130610102426873-oozie-rkan-W@:start:] " + "[***0000003-130610102426873-oozie-rkan-W@:start:***]Action updated in DB! _L2_" + "\n" + "2013-06-10 10:26:10,148 INFO HiveActionExecutor:539 SERVER[foo] USER[rkanter] GROUP[-] TOKEN[] APP[hive-wf] " + "JOB[0000003-130610102426873-oozie-rkan-W] ACTION[0000003-130610102426873-oozie-rkan-W@hive-node] action produced" + " output _L5_" + "\n" + // a multiline message with a stack trace
    "2013-06-10 10:26:30,202  WARN ActionStartXCommand:542 - SERVER[foo] USER[rkanter] GROUP[-] TOKEN[] APP[hive-wf] " + "JOB[0000003-130610102426873-oozie-rkan-W] ACTION[0000003-130610102426873-oozie-rkan-W@hive-node] Error starting " + "action [hive-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: java.io.IOException: Unknown " + "protocol to name node: org.apache.hadoop.mapred.JobSubmissionProtocol _L7_\n" + "     at org.apache.hadoop.hdfs.server.namenode.NameNode.getProtocolVersion(NameNode.java:156) _L8_\n" + "     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)_L9_\n" + "     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) _L10_\n" + "     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426) _L11_\n" + "] _L12_\n" + "org.apache.oozie.action.ActionExecutorException: JA009: java.io.IOException: Unknown protocol to name node: " + "org.apache.hadoop.mapred.JobSubmissionProtocol _L13_\n" + "     at org.apache.hadoop.hdfs.server.namenode.NameNode.getProtocolVersion(NameNode.java:156) _L14_\n" + "     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) _L15_\n" + "     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) _L16_\n";
    String out = doStreamLog(xf);
    String[] outArr = out.split("\n");
    assertEquals(3, outArr.length);
    assertFalse(out.contains("_L1_"));
    assertFalse(out.contains("_L2_"));
    assertTrue(outArr[0].contains("_L3_"));
    assertTrue(outArr[1].contains("_L4_"));
    assertFalse(out.contains("_L5_"));
    assertTrue(outArr[2].contains("_L6_"));
    // We'll use a DummyZKOozie to create an entry in ZK and then set its url to an (unrelated) servlet that will simply return
    // some log messages
    DummyZKOozie dummyOozie = null;
    EmbeddedServletContainer container = new EmbeddedServletContainer("oozie");
    container.addServletEndpoint("/other-oozie-server/*", DummyLogStreamingServlet.class);
    try {
        container.start();
        dummyOozie = new DummyZKOozie("9876", container.getServletURL("/other-oozie-server/*"));
        DummyLogStreamingServlet.lastQueryString = null;
        out = doStreamLog(xf);
        outArr = out.split("\n");
        assertEquals(16, outArr.length);
        assertTrue(outArr[0].contains("_L1_"));
        assertTrue(outArr[1].contains("_L2_"));
        assertTrue(outArr[2].contains("_L3_"));
        assertTrue(outArr[3].contains("_L4_"));
        assertTrue(outArr[4].contains("_L5_"));
        assertTrue(outArr[5].contains("_L6_"));
        assertTrue(outArr[6].contains("_L7_"));
        assertTrue(outArr[7].contains("_L8_"));
        assertTrue(outArr[8].contains("_L9_"));
        assertTrue(outArr[9].contains("_L10_"));
        assertTrue(outArr[10].contains("_L11_"));
        assertTrue(outArr[11].contains("_L12_"));
        assertTrue(outArr[12].contains("_L13_"));
        assertTrue(outArr[13].contains("_L14_"));
        assertTrue(outArr[14].contains("_L15_"));
        assertTrue(outArr[15].contains("_L16_"));
        assertEquals("show=log&allservers=false", DummyLogStreamingServlet.lastQueryString);
        // If we stop the container but leave the DummyZKOozie running, it will simulate if that server is down but still has
        // info in ZK; we should be able to get the logs from other servers (in this case, this server) and a message about
        // which servers it couldn't reach
        container.stop();
        out = doStreamLog(xf);
        outArr = out.split("\n");
        assertEquals(6, outArr.length);
        assertTrue(outArr[0].startsWith("Unable"));
        assertEquals("9876", outArr[1].trim());
        assertEquals("", outArr[2]);
        assertFalse(out.contains("_L1_"));
        assertFalse(out.contains("_L2_"));
        assertTrue(outArr[3].contains("_L3_"));
        assertTrue(outArr[4].contains("_L4_"));
        assertFalse(out.contains("_L5_"));
        assertTrue(outArr[5].contains("_L6_"));
    } finally {
        if (dummyOozie != null) {
            dummyOozie.teardown();
        }
        container.stop();
    }
}
Also used : XLogFilter(org.apache.oozie.util.XLogFilter) InputStream(java.io.InputStream) FileOutputStream(java.io.FileOutputStream) FileWriter(java.io.FileWriter) EmbeddedServletContainer(org.apache.oozie.test.EmbeddedServletContainer) Properties(java.util.Properties) File(java.io.File)

Example 2 with XLogFilter

use of org.apache.oozie.util.XLogFilter in project oozie by apache.

the class TestXLogStreamingService method testTuncateLog.

public void testTuncateLog() throws Exception {
    File log4jFile = new File(getTestCaseConfDir(), "test-log4j.properties");
    ClassLoader cl = Thread.currentThread().getContextClassLoader();
    InputStream is = cl.getResourceAsStream("test-no-dash-log4j.properties");
    Properties log4jProps = new Properties();
    log4jProps.load(is);
    log4jProps.setProperty("log4j.appender.oozie.File", getTestCaseDir() + "/oozie.log");
    log4jProps.store(new FileOutputStream(log4jFile), "");
    setSystemProperty(XLogService.LOG4J_FILE, log4jFile.getName());
    new Services().init();
    ConfigurationService.set(XLogFilter.MAX_SCAN_DURATION, "10");
    Date startDate = new Date();
    Date endDate = new Date(startDate.getTime() + FIFTEEN_HOURS);
    String log = doStreamLog(new XLogFilter(), startDate, endDate);
    assertTrue(log.contains("Truncated logs to max log scan duration"));
    log = doStreamErrorLog(new XLogFilter(), startDate, endDate);
    assertFalse(log.contains("Truncated logs to max log scan duration"));
    log = doStreamAuditLog(new XLogFilter(), startDate, endDate);
    assertFalse(log.contains("Truncated logs to max log scan duration"));
}
Also used : XLogFilter(org.apache.oozie.util.XLogFilter) InputStream(java.io.InputStream) FileOutputStream(java.io.FileOutputStream) Properties(java.util.Properties) File(java.io.File) Date(java.util.Date)

Example 3 with XLogFilter

use of org.apache.oozie.util.XLogFilter in project oozie by apache.

the class TestXLogStreamingService method testErrorLog.

public void testErrorLog() throws Exception {
    setupXLog();
    XLogFilter xf = new XLogFilter(new XLogUserFilterParam(null));
    xf.setParameter("USER", "oozie");
    xf.setLogLevel("DEBUG|INFO");
    // Previously, a dash ("-") was always required somewhere in a line in order for that line to pass the filter; this test
    // checks that this condition is no longer required for log streaming to work
    File log4jFile = new File(getTestCaseConfDir(), "test-log4j.properties");
    ClassLoader cl = Thread.currentThread().getContextClassLoader();
    InputStream is = cl.getResourceAsStream("test-no-dash-log4j.properties");
    Properties log4jProps = new Properties();
    log4jProps.load(is);
    // prevent conflicts with other tests by changing the log file location
    log4jProps.setProperty("log4j.appender.oozie.File", getTestCaseDir() + "/oozie.log");
    log4jProps.setProperty("log4j.appender.oozieError.File", getTestCaseDir() + "/oozie-error.log");
    log4jProps.store(new FileOutputStream(log4jFile), "");
    setSystemProperty(XLogService.LOG4J_FILE, log4jFile.getName());
    try {
        new Services().init();
        assertFalse(doStreamDisabledCheck());
        LogFactory.getLog("a").info("2009-06-24 02:43:14,505 INFO _L1_:317 - SERVER[foo] USER[oozie] GROUP[oozie] TOKEN[-] " + "APP[-] JOB[-] ACTION[-] Released Lock");
        LogFactory.getLog("a").warn("2009-06-24 02:44:14,505 WARN _L2_:317 - SERVER[foo] USER[oozie] GROUP[oozie] TOKEN[-] " + "APP[-] JOB[-] ACTION[-] Error while writing to file");
        LogFactory.getLog("a").info("2009-06-24 02:45:14,505 INFO _L3_:317 - SERVER[foo] USER[blah] GROUP[oozie] TOKEN[-]" + "APP[-] JOB[-] ACTION[-] Released Lock");
        LogFactory.getLog("a").error("2009-06-24 02:46:14,505 ERROR _L4_:317 - SERVER[foo] USER[oozie] GROUP[oozie] TOKEN[-] " + "APP[-] JOB[-] ACTION[-] Error while writing to DB");
        LogFactory.getLog("a").info("2009-06-24 02:47:14,505 INFO _L5_:317 SERVER[foo] USER[oozie] GROUP[oozie] TOKEN[-] " + "APP[-] JOB[-] ACTION[-] Released Lock");
        LogFactory.getLog("a").fatal("2009-06-24 02:48:14,505 FATAL _L6_:317 SERVER[foo] USER[blah] GROUP[oozie] TOKEN[-] " + "APP[-] JOB[-] ACTION[-] Released Lock");
        String out = doStreamLog(xf);
        String[] outArr = out.split("\n");
        // Lines 2 and 4 are filtered out because they have the wrong user
        assertEquals(2, outArr.length);
        assertTrue(outArr[0].contains("_L1_"));
        assertFalse(out.contains("_L3_"));
        assertTrue(outArr[1].contains("_L5_"));
        assertFalse(out.contains("_L4_"));
        XLogFilter.reset();
        XLogFilter.defineParameter("USER");
        XLogFilter.defineParameter("GROUP");
        XLogFilter.defineParameter("TOKEN");
        XLogFilter.defineParameter("APP");
        XLogFilter.defineParameter("JOB");
        XLogFilter.defineParameter("ACTION");
        xf = new XLogFilter(new XLogUserFilterParam(null));
        xf.setParameter("USER", "oozie");
        String errorOut = doStreamErrorLog(xf);
        String[] errorOutArr = errorOut.split("\n");
        // Lines 2 and 4 are filtered out because they have the wrong user
        assertEquals(2, errorOutArr.length);
        assertTrue(errorOutArr[0].contains("_L2_"));
        assertTrue(errorOutArr[1].contains("_L4_"));
        assertFalse(errorOut.contains("_L6_"));
    } finally {
        Services.get().destroy();
    }
}
Also used : XLogFilter(org.apache.oozie.util.XLogFilter) InputStream(java.io.InputStream) FileOutputStream(java.io.FileOutputStream) Properties(java.util.Properties) XLogUserFilterParam(org.apache.oozie.util.XLogUserFilterParam) File(java.io.File)

Example 4 with XLogFilter

use of org.apache.oozie.util.XLogFilter in project oozie by apache.

the class TestXLogStreamingService method testAuditLog.

public void testAuditLog() throws Exception {
    setupXLog();
    XLogFilter xf = new XLogAuditFilter(new XLogUserFilterParam(null));
    xf.setParameter("USER", "oozie");
    xf.setLogLevel("DEBUG|INFO");
    File log4jFile = new File(getTestCaseConfDir(), "test-log4j.properties");
    ClassLoader cl = Thread.currentThread().getContextClassLoader();
    InputStream is = cl.getResourceAsStream("test-no-dash-log4j.properties");
    Properties log4jProps = new Properties();
    log4jProps.load(is);
    // prevent conflicts with other tests by changing the log file location
    log4jProps.setProperty("log4j.appender.oozie.File", getTestCaseDir() + "/oozie.log");
    log4jProps.setProperty("log4j.appender.oozieaudit.File", getTestCaseDir() + "/oozie-audit.log");
    log4jProps.store(new FileOutputStream(log4jFile), "");
    setSystemProperty(XLogService.LOG4J_FILE, log4jFile.getName());
    try {
        new Services().init();
        XLog auditLog = XLog.getLog("oozieaudit");
        xf.setParameter(DagXLogInfoService.JOB, "0000000-150322000230582-oozie-puru-C");
        auditLog.info("2015-03-22 00:04:35,494  INFO oozieaudit:520 - IP [127.0.0.1], USER [purushah], GROUP [null], " + "APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [start], " + "PARAMETER [null], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [L1]");
        auditLog.info("2015-03-22 00:05:13,823  INFO oozieaudit:520 - IP [127.0.0.1], USER [purushah], GROUP [null], " + "APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [suspend], " + "PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], " + "ERRORMESSAGE [L2]");
        auditLog.info("2015-03-22 00:05:13,823  INFO oozieaudit:520 - IP [127.0.0.1], USER [purushah], GROUP [null], " + "APP [-], JOBID [0000001-150322000230582-oozie-puru-C], OPERATION [suspend], " + "PARAMETER [0000001-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], " + "ERRORMESSAGE [L3]");
        String out = doStreamAuditLog(xf);
        String[] outArr = out.split("\n");
        // Lines 2 and 4 are filtered out because they have the wrong user
        assertEquals(2, outArr.length);
        assertTrue(outArr[0].contains("L1"));
        assertTrue(out.contains("L2"));
        assertFalse(out.contains("L3"));
    } finally {
        Services.get().destroy();
    }
}
Also used : XLogAuditFilter(org.apache.oozie.util.XLogAuditFilter) XLogFilter(org.apache.oozie.util.XLogFilter) XLog(org.apache.oozie.util.XLog) InputStream(java.io.InputStream) FileOutputStream(java.io.FileOutputStream) Properties(java.util.Properties) XLogUserFilterParam(org.apache.oozie.util.XLogUserFilterParam) File(java.io.File)

Example 5 with XLogFilter

use of org.apache.oozie.util.XLogFilter in project oozie by apache.

the class TestXLogUserFilterParam method testNoException_Withrecent.

// Test log duration - in range
public void testNoException_Withrecent() throws Exception {
    File log4jFile = new File(getTestCaseConfDir(), "test-log4j.properties");
    ClassLoader cl = Thread.currentThread().getContextClassLoader();
    InputStream is = cl.getResourceAsStream("test-no-dash-log4j.properties");
    Properties log4jProps = new Properties();
    log4jProps.load(is);
    // prevent conflicts with other tests by changing the log file location
    log4jProps.setProperty("log4j.appender.oozie.File", getTestCaseDir() + "/oozie.log");
    log4jProps.store(new FileOutputStream(log4jFile), "");
    setSystemProperty(XLogService.LOG4J_FILE, log4jFile.getName());
    new Services().init();
    Services.get().getConf().setInt(XLogFilter.MAX_SCAN_DURATION, 10);
    Map<String, String[]> paramMap = new HashMap<String, String[]>();
    String param = XLogUserFilterParam.RECENT_LOG_OFFSET + "=9";
    paramMap.put(RestConstants.LOG_FILTER_OPTION, new String[] { param });
    XLogFilter filter = new XLogFilter(new XLogUserFilterParam(paramMap));
    Date startDate = new Date();
    Date endDate = new Date(startDate.getTime() + 60 * 60 * 1000 * 15);
    try {
        doStreamLog(filter, startDate, endDate);
    } catch (Exception e) {
        fail("should not throw exception");
    }
}
Also used : HashMap(java.util.HashMap) XLogFilter(org.apache.oozie.util.XLogFilter) InputStream(java.io.InputStream) Properties(java.util.Properties) Date(java.util.Date) Services(org.apache.oozie.service.Services) FileOutputStream(java.io.FileOutputStream) File(java.io.File) XLogUserFilterParam(org.apache.oozie.util.XLogUserFilterParam)

Aggregations

XLogFilter (org.apache.oozie.util.XLogFilter)23 XLogUserFilterParam (org.apache.oozie.util.XLogUserFilterParam)16 File (java.io.File)15 FileOutputStream (java.io.FileOutputStream)15 InputStream (java.io.InputStream)15 Properties (java.util.Properties)15 HashMap (java.util.HashMap)13 Date (java.util.Date)9 Services (org.apache.oozie.service.Services)6 FileWriter (java.io.FileWriter)3 EmbeddedServletContainer (org.apache.oozie.test.EmbeddedServletContainer)3 XLogStreamingService (org.apache.oozie.service.XLogStreamingService)2 StringWriter (java.io.StringWriter)1 ArrayList (java.util.ArrayList)1 Comparator (java.util.Comparator)1 LinkedHashSet (java.util.LinkedHashSet)1 List (java.util.List)1 CoordinatorAction (org.apache.oozie.client.CoordinatorAction)1 CommandException (org.apache.oozie.command.CommandException)1 DagXLogInfoService (org.apache.oozie.service.DagXLogInfoService)1
About Me
UseOf

Java Tutorials :

Simple Java RabbitMQ Example with Spring
Simple Springboot JPA Eclipeslink Example
Simple SpringBoot JPA Hibernate Example
Jackson JSON @JsonIgnore and @JsonIgnoreProperties Examples
Java Infinite recursion (StackOverflowError) Jackson solutions
Simple Java Jackson Serialize Objects to JSON and convert them back To Object
ElasticSearch 5 - Upload files using Java API in binary field Example
Java JAXB marshall unmarshall Examples from String and Stream
Java 8 forEach List and Map examples
ElasticSearch 5 Java API SearchRequestBuilder examples
Java ElasticSearch 5 examples with node index and mapping - running local
Convert String to int or Integer Java 8
Java 9 in action - JShell - Ubuntu
Java - removing invalid characters from a file name
Hello world!? or Hello Tutorial