Log File Analysis

Struggling to get your site to rank, even after putting serious work into SEO-optimized content? In some cases, the answer is hiding in your website’s log file. One of Topdog’s services is carrying out log file analyses for companies that want to take their digital marketing to the next level.

• A process shaped around your circumstances and business goals
• Better control over your website’s technical search engine optimization
• Clear improvements in your visitor statistics

What is a log file?

A log file is a record of the traffic on your website. It logs every single request made to your server. That means it shows exactly how both human visitors and search engines have interacted with the site. In other words, it works as a kind of answer key for whether your website is actually doing the job you want it to do.

The log file is split into sections, with each individual visit getting its own entry. Here, you’ll find information such as:

• The URL of the page requested
• HTTP status code
• HTTP method (GET/POST)
• Time and date
• User agent (Google, for example)
• The visitor’s IP address
• The referring URL

What is a log file analysis?

Analysing a log file, or log file analysis, means working through the data it holds to understand how your website actually performs in practice. Beyond helping you spot and fix any problems, it can also give you the tools to shape your SEO strategy going forward. If you’d rather not go through the data by hand, you can use a digital tool such as Semrush or Screaming Frog, or bring in a specialist SEO agency.

How is log file analysis used in search engine optimization?

Analysing a website’s log file is a core part of the technical SEO audit, because it delivers real insight into both user behaviour and the site’s technical structure. Log file analysis serves two purposes: building a better understanding of the website, and identifying the technical faults holding it back. One particularly important angle here is watching how search engines’ bots, known as crawlers, behave as they work through the site, since that ultimately determines how well your website ranks in the search results.

Some of the questions a log file analysis answers include:

• How often do search engines crawl the site, and which pages get crawled the most?
• Which are the website’s most important pages?
• Which pages get crawled the least?
• Are there any errors in the site’s status codes?
• Is JavaScript working as it should?
• How much of your website’s crawl budget is being spent on irrelevant pages?
• How do the site’s redirects work, and do they preserve link equity?
• Are there any unnecessarily large or slow pages?
• Is the site mobile-first indexed?

What does Topdog do in a log file analysis?

At its core, search engine optimization is a process of continuous improvement that calls for an agile approach, since search engines aren’t always entirely predictable. That means actions need to be adjusted as you go, and the technical side is no exception. When Topdog carries out a log file analysis, here’s what we do:

1. Identifying errors and areas for improvement

We start by working through all the data and answering the questions above. The goal is to pin down exactly which errors need fixing and where the opportunities for improvement lie.

2. Optimising the website’s crawl budget

Search engine crawlers can only work through a certain amount of data from your site at any given moment, so from an SEO perspective, as much of that crawl budget as possible should go to the site’s most important pages. We work out exactly what needs to happen to make that a reality.

3. Improving crawl frequency

Search engines need to see your website as a valuable resource before they’ll crawl it regularly. Here, we look at what it takes to drive more traffic to the sections that Google’s crawlers don’t visit very often.

Who can benefit from a log file analysis?

Sites with the most to gain from a log file analysis include:

• Online retailers
• Media sites
• Marketplaces
• Large websites with lots of content

Where do you find the log file?

To reach your website’s log file, you’ll need access to its web server. If your site sits with a hosting provider, logging into their platform will usually get you there. From that point, download a copy of the log file. The easiest way to do this is with an FTP client (File Transfer Protocol): navigate to it through the client and save a copy locally.