Version 6 (modified by Art Rhyno, 13 years ago) (diff)


The main gathering point for the logs is starburst. There is a problem with wildcard expansion for sudo commands so you need to use the format shown if you want wildcards:

  sudo ls -l /var/log/remote/prod/2009/08/27
  sudo sh -c "grep 27T17 /var/log/remote/prod/2009/08/27/*" | more

So in the example above, we are looking for error messages associated with 5 PM (T17) on Aug. 27, 2009

This is the error you never want to find:

  sudo sh -c "grep 'NOT CONNECTED' /var/log/remote/prod/2009/10/02/osrferror.log" | more

Sometimes, the trick seems to be to narrow things down:

  sudo sh -c "grep '2009-10-02 13:' /var/log/remote/prod/2009/10/02/*.log" > ~/stuff/work/oct.txt

And then working with a smaller set that eliminates the most obvious errors:

  grep '[Ee]rror' oct.txt | grep -v 'error.js' | grep -v 'does not exist' | grep -v 'unblessed reference' | more

There seems to sometimes be issues with phrase searching, this kind of convoluted grep is needed to find this:

  sudo sh -c "grep 'keyword:\\\\\"' /var/log/remote/prod/2009/10/02/*.log" | more

Sometimes, the response time is worth looking at:

  /var/log/remote/prod/2009/10/02/ap_access.log:2009-10-02 13:00:40 protostar logger: - 
  - [02/Oct/2009:12:55:40 - 0400] "POST /osrf-gateway-v1 HTTP/1.1" 404 352 
  "Mozilla/4.0   (compatible; MSIE 8.0; Windows NT 6.0;

The time of the log entry is 5 minutes after the posting (recorded at 13:00:40 but sent at 12:55:40)