Reinsert Headers into Filtered Server Log Files

This post is for my own memory. Lets say you’ve combined multiple server logs from multiple folders, and used grep to pre-filter these lines to the ones you care about (whatever they are). Your file Gbot.log doesn’t have the line denoting the field headings. As a result, Screaming Frog’s log analyzer won’t open it:

2016-07-27 12_35_15-Import Error

Inserting the headers shouldn’t be too much of an issue. You have a copy of them already – you just need to open the file and add the following lines at the top:

#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status time-taken

Problem

The file is too big to open in your text editor. You know you could dump all the unfiltered files into the tool, but don’t want to spend the time waiting for the unnecessary rows to be parsed.

Solution

Using text manipulation tool sed, we can do something like:

~$ sed -i '1i the-headers-you-want-inserted' Gbot.log

This inserts the line in place as the first line of the current file. That’s it (I think it uses a temporary file in the background).

Unfortunately the sed binary included in GOW doesn’t seem to play nice – so a more bulldozerish method is needed (unless you’ve enabled bash on windows). There isn’t a neat operator which prepends information to a file. However, we can append information to a file using ‘>>’. So we can create a new file with the headers and append the contents of the old file.

“Create a file containing only the headers. Append the contents of the combined log file to this new file.”

~$ echo "headers without quotes" > SF-friendly.log
~$ cat Gbot.log >> SF-friendly.log

For example:

~$ echo #Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status time-taken > SF-friendly.log
~$ cat Gbot.log >> SF-friendly.log

If there’s a quicker OS-agnostic way to do this for large files, comment below and I’ll update/delete the post as appropriate.

Leave a Reply

Your email address will not be published. Required fields are marked *