Changes between Version 4 and Version 5 of sysadminImportantLogs

Timestamp:

Oct 3, 2009, 12:22:53 AM (13 years ago)

Author:

Art Rhyno

Comment:

--

sysadminImportantLogs

@@ -1 @@
-Production postgres logs for the last few days are on polaris:
-{{{
-  uowadmin@polaris:/var/log/postgresql$ ls -l
-  total 33380
-  -rw-r----- 1 postgres adm  2932201 2009-08-31 11:55 postgresql-8.3-main.log
-  -rw-r----- 1 postgres adm 22720479 2009-08-30 06:25 postgresql-8.3-main.log.1
-  -rw-r----- 1 postgres adm   564383 2009-06-28 06:25 postgresql-8.3-main.log.10.gz
-}}}
-You can use ''sudo'' to hunt down a problem in a particular timeframe:
-{{{
-  sudo grep '2009-08-27' postgresql-8.3-main.log.1 | more
-  sudo grep '2009-08-27' postgresql-8.3-main.log.1 > ~/errors.txt
-}}}
 The main gathering point for the logs is ''starburst''. There is a problem with wildcard expansion for sudo commands so you need to use the format shown if you want wildcards:
 {{{
@@ -18 +5 @@
 }}}
 So in the example above, we are looking for error messages associated with 5 PM ({{{T17}}}) on Aug. 27, 2009
+
+This is the error you never want to find:
+
+{{{
+  sudo sh -c "grep 'NOT CONNECTED' /var/log/remote/prod/2009/10/02/osrferror.log" | more
+}}}
+
+Sometimes, the trick seems to be to narrow things down:
+
+{{{
+  sudo sh -c "grep '2009-10-02 13:' /var/log/remote/prod/2009/10/02/*.log" > ~/stuff/work/oct.txt
+}}}
+
+And then working with a smaller set that eliminates the most obvious errors:
+
+{{{
+  grep '[Ee]rror' oct.txt | grep -v 'error.js' | grep -v 'does not exist' | grep -v 'unblessed reference' | more
+}}}
+
+There seems to sometimes be issues with phrase searching, this kind of convoluted grep is needed to find this:
+
+{{{
+  sudo sh -c "grep 'keyword:\\\\\"' /var/log/remote/prod/2009/10/02/*.log" | more
+}}}
+
+Sometimes, the response time is worth looking at:
+
+{{{
+  /var/log/remote/prod/2009/10/02/ap_access.log:2009-10-02 '''13:00:40''' protostar logger: 216.254.218.64 - 
+  - [02/Oct/2009:'''12:55:40''' - 0400] "POST /osrf-gateway-v1 HTTP/1.1" 404 352 
+  "http://windsor.concat.ca/opac/en-CA/skin/uwin/xml/rresult.xml?rt=keyword&tp=keyword&
+  t=warfare%20in%20renaissance%20europe%3A%20gunpowder%2C%20technology%2C%20and%20tactics&l=106&d=1&f=&av=" 
+  "Mozilla/4.0   (compatible; MSIE 8.0; Windows NT 6.0;
+}}}