This post is for my own memory. Lets say you’ve combined multiple server logs from multiple folders, and used grep to pre-filter these lines to the ones you care about (whatever they are). Your file Gbot.log doesn’t have the line denoting the field headings. As a result, Screaming Frog’s log analyzer won’t open it:
Inserting the headers shouldn’t be too much of an issue. You have a copy of them already – you just need to open the file and add the following lines at the top:
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status time-taken
Problem
The file is too big to open in your text editor. You know you could dump all the unfiltered files into the tool, but don’t want to spend the time waiting for the unnecessary rows to be parsed.
Solution
Using text manipulation tool sed, we can do something like:
~$ sed -i '1i the-headers-you-want-inserted' Gbot.log
This inserts the line in place as the first line of the current file. That’s it (I think it uses a temporary file in the background).
Unfortunately the sed binary included in GOW doesn’t seem to play nice – so a more bulldozerish method is needed (unless you’ve enabled bash on windows). There isn’t a neat operator which prepends information to a file. However, we can append information to a file using ‘>>’. So we can create a new file with the headers and append the contents of the old file.
“Create a file containing only the headers. Append the contents of the combined log file to this new file.”
~$ echo "headers without quotes" > SF-friendly.log ~$ cat Gbot.log >> SF-friendly.log
For example:
~$ echo #Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status time-taken > SF-friendly.log ~$ cat Gbot.log >> SF-friendly.log
If there’s a quicker OS-agnostic way to do this for large files, comment below and I’ll update/delete the post as appropriate.