How to Use Loggly Surround Search: A Quick Use Case

While doing a routine check on Loggly, I did a short-term search. When I filtered by my results syslog severity, I saw there were a handful of messages like this on a new box:

2014-06-12T22:49:17.000+00:00 newbox sshd[13174]: pam_limits(sshd:session): unknown limit item ‘nofiles’

I didn’t recall seeing messages like this before.  What’s causing them?

I did a broader search, removing the severity filter and searching by pid and host.  This is easy and intuitive to do in Loggly; just click on the links in the expanded log message.

2014-06-12T22:49:16.000+00:00 newbox sshd[13174]: Accepted publickey for leslie from 10.209.4.10 port 62299 ssh2

2014-06-12T22:49:16.000+00:00 newbox sshd[13174]: pam_unix(sshd:session): session opened for user leslie by (uid=0)

2014-06-12T22:49:17.000+00:00 newbox sshd[13174]: pam_limits(sshd:session): unknown limit item ‘nofiles’

Looks like it’s ssh-related.  I searched the box’s configs for “nofiles” and found it in /etc/security/limits.conf:

*hardnofiles 0

Reading the man page, I found that this setting should be nofile, singular:

nofile

maximum number of open files

But when tinkering with ssh, it’s always a good idea to leave a running shell open and then test sshing in with a second connection in case you broke something. That way you don’t lock yourself out of the box.

And, as it happens, this change broke ssh.

$ ssh newbox.loggly.com

Write failed: Broken pipe

What happened?

I checked my previous search for logins from leslie and it showed just an open and close – not so useful.

Click on Surround Search, though…

2014-06-12T23:02:42.000+00:00 newbox sshd[14369]: Accepted publickey for leslie from 10.209.4.10 port 63011 ssh2

2014-06-12T23:02:42.000+00:00 newbox sshd[14369]: pam_unix(sshd:session): session opened for user leslie by (uid=0)

2014-06-12T23:02:42.000+00:00 newbox sshd[14369]: fatal: monitor_openfds: socketpair: Too many open files

Surround Search is a neat Loggly feature that you may have missed. It grabs all log messages immediately surrounding the log event you’re interested in. All you have to do is click on those little binoculars that you see in the upper right corner of each log entry. (I outlined them in red in my screen shot above.)

In my case, Surround Search provided context by showing events that were logged on this box right around the time as the ssh open/close. My original search was too restrictive. Broadening the search shows me that 0 isn’t a supported setting for nofile, and changing that setting to “unlimited” lets me right back in.

Surround Search is far more powerful than this, though. You can use it to see what happened at the same time as an event you’re interested in, anywhere and everywhere in your environment. Combined with other Loggly search tools, such as grid searching on individual transactions, Loggly provides fast, flexible searching. DevOps can find and fix problems and then get on with our day.

If you haven’t tried Loggly for yourself, Surround Search is just one more reason to start your free trial today!

The Loggly and SolarWinds trademarks, service marks, and logos are the exclusive property of SolarWinds Worldwide, LLC or its affiliates. All other trademarks are the property of their respective owners.