Changes between Version 4 and Version 5 of sysadminImportantLogs


Ignore:
Timestamp:
Oct 3, 2009, 12:22:53 AM (13 years ago)
Author:
Art Rhyno
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • sysadminImportantLogs

    v4 v5  
    1 Production postgres logs for the last few days are on polaris:
    2 {{{
    3   uowadmin@polaris:/var/log/postgresql$ ls -l
    4   total 33380
    5   -rw-r----- 1 postgres adm  2932201 2009-08-31 11:55 postgresql-8.3-main.log
    6   -rw-r----- 1 postgres adm 22720479 2009-08-30 06:25 postgresql-8.3-main.log.1
    7   -rw-r----- 1 postgres adm   564383 2009-06-28 06:25 postgresql-8.3-main.log.10.gz
    8 }}}
    9 You can use ''sudo'' to hunt down a problem in a particular timeframe:
    10 {{{
    11   sudo grep '2009-08-27' postgresql-8.3-main.log.1 | more
    12   sudo grep '2009-08-27' postgresql-8.3-main.log.1 > ~/errors.txt
    13 }}}
    141The main gathering point for the logs is ''starburst''. There is a problem with wildcard expansion for sudo commands so you need to use the format shown if you want wildcards:
    152{{{
     
    185}}}
    196So in the example above, we are looking for error messages associated with 5 PM ({{{T17}}}) on Aug. 27, 2009
     7
     8This is the error you never want to find:
     9
     10{{{
     11  sudo sh -c "grep 'NOT CONNECTED' /var/log/remote/prod/2009/10/02/osrferror.log" | more
     12}}}
     13
     14Sometimes, the trick seems to be to narrow things down:
     15
     16{{{
     17  sudo sh -c "grep '2009-10-02 13:' /var/log/remote/prod/2009/10/02/*.log" > ~/stuff/work/oct.txt
     18}}}
     19
     20And then working with a smaller set that eliminates the most obvious errors:
     21
     22{{{
     23  grep '[Ee]rror' oct.txt | grep -v 'error.js' | grep -v 'does not exist' | grep -v 'unblessed reference' | more
     24}}}
     25
     26There seems to sometimes be issues with phrase searching, this kind of convoluted grep is needed to find this:
     27
     28{{{
     29  sudo sh -c "grep 'keyword:\\\\\"' /var/log/remote/prod/2009/10/02/*.log" | more
     30}}}
     31
     32Sometimes, the response time is worth looking at:
     33
     34{{{
     35  /var/log/remote/prod/2009/10/02/ap_access.log:2009-10-02 '''13:00:40''' protostar logger: 216.254.218.64 -
     36  - [02/Oct/2009:'''12:55:40''' - 0400] "POST /osrf-gateway-v1 HTTP/1.1" 404 352
     37  "http://windsor.concat.ca/opac/en-CA/skin/uwin/xml/rresult.xml?rt=keyword&tp=keyword&
     38  t=warfare%20in%20renaissance%20europe%3A%20gunpowder%2C%20technology%2C%20and%20tactics&l=106&d=1&f=&av="
     39  "Mozilla/4.0   (compatible; MSIE 8.0; Windows NT 6.0;
     40}}}