October 2015 Archives

I've been working with Logstash lately, and one of the tasks I was given was attempting to improve parsing of audit.log entries. Turning things like this:

type=SYSCALL msg=audit(1445878971.457:6169): arch=c000003e syscall=59 success=yes exit=0 a0=c2c3a8 a1=c64bc8 a2=c34408 a3=7fff44e370f0 items=2 ppid=16974 pid=18771 auid=1004 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=5 comm="compiled_evil" exe="/home/justsomeuser/bin/compiled_evil" key="hinkystuff"

Into nice and indexed entries where we can make Kibana graphs of all commands caught with the hinkystuff audit ruleset.

The problem with audit.log entries is that they're not very regexible. Oh, they can be. But optional sometimes-there-sometimes-not fields suck a lot. Take for example, the SYSCALL above. Items a0 through a3 are arguments 1-3 of the syscall, and there may be 1 to 3 of them. Expressing that in regex/grok is trying.

So I made a thing:

Logstash-auditlog: Grok patterns and examples for parsing Audit settings with Logstash.

May it be useful.

I had the dubious honor of running into a brick wall that has so far resisted my skill in both search-engines and asking for help. It went so far that I asked on StackOverflow and got nothing (12 views as of this writing).

http://stackoverflow.com/questions/33088971/dealing-with-recursion-sync-loops-in-coffeescript

So far, trying to do this thing in coffee has taken two days. This is not a language set up to deal with "do this procedure one or more times until you don't get a thing, and then continue with execution," It can be done, I have no doubt about that. I have simply failed to figure out how.

Yesterday I gave up and wrote a ruby script that did what I needed. It took 6 hours to get the features I needed, including various checks to make sure I'm not breaking things by doing it. It worked the way I needed it to, and it was awesome.

And unfortunately proved that I definitely need to figure out the 'one or more' thing in coffee, going forward. My solution to that may very well be 'shell out to an external script that can do that kind of thing and accept the results.'

I think I now know why the Amazon SDK for Javascript doesn't include paging support like the CLI tools written in python do.