How Analytics works

After signing up for the free Google Analytics site, you are given a piece of code to add to your website pages.  This piece of code (anonymously) collects some information from the user, like what browser and so forth, and sends it back to Google.  It also keeps track of usage statistics, like the time on a single page and more.  The key component, however, is adding the code to each web page you want to track.  I did not add the code to all of the internal tools pages, as that data wouldn’t make any sense trying to lump it in with general website traffic.

The code that Google provides will catch if you go from a web page to a downloadable file on the site, such as all of the PDF files and supplements we have on our site.  However, it is impossible for Analytics to catch traffic that goes directly to a downloaded file.  For instance, if someone were to google ‘dmso in pcr’, and the Google result links directly to one of our protocols PDF that has DMSO in a reaction, Analytics will not catch that.

However, it is possible to catch that type of traffic using the logs for the web server.  Web servers will create a text file that contains a line for each access for each element on the web site.  Since it’s obvious that these files can get very large quickly, it’s standard to do a rotation of these log files.  The previous server system that I was using, however, rotated out log files greater than 7 days old by deleting them entirely.  I replaced that system a little over a month ago, but data previously to that is lost.

Digging into the heart

Initially this started as a project that after about 2 pm on the 15th, I noticed that server traffic jumped up by a very large amount.  Since this was directly after my presentation I figured this was the case, but I was still curious as to where all of the traffic was going.  After looking at the Analytics site and seeing results that did not jive with the server traffic, I made the connection I alluded to above on how it is impossible for Analytics to determine traffic in certain conditions.

To try and see how different the results may be from the Analytics that I used previously, I and analyzed the access log files that I have from the web server.  By themselves the files are mostly useless, but in the hands of a powerful analysis program that can disassemble and group data from access log files to generate reports from it.  Using the quite neat Deep Log Analyzer I took a look at just the week previous.

Since the log file contains all site accesses, and I did not filter any results, there are more elements than in Analytics.  For instance, all of the Tools pages are shown.  The first element is a small program that handles returning the results from the SDS tool, which gets accessed quite a bit, so it is the first element.  Ignoring that and all of the other Tools the trends still follow from the Analytics results (n.b. I’m comparing a week of data to 2 years of Analytics data so…)

Something that I can track with analyzing the logs that I cannot with Analytics are direct downloads that do not come from a link on the site.  This shows the total number of hits to all of the downloaded files in the past week

Definitely our published Protocols generate the most downloaded traffic, something that was not caught at all by Analytics because these files were accessed directly from Google and not from a link on our page.  This also shows much more traffic on our publication PDFs, another thing that wasn’t caught in Analytics.

Looking at the collected Google search terms…

Seems quite different than the terms I found from Analytics.  There is a higher rate of scientific terms.  Unfortunately, the free version of the utility does not allow for the exporting of reports, so I can’t post the dataset.

The page access numbers directly from the log files still correlate with what I found from Analytics, although just from a cursory glance the ratio of Google search terms would likely be evened out some, certainlly different from the 9:1 ratio that I got from Analytics previously.  It would take some more analysis of log files over a longer period of time to really see the discrepancies.  I still stand by my original premise however that a very large portion of the traffic comes from name-searching, but it is probably not as drastic as I originally seen.

When we start using Wordpress for managing content however, it is organized in such a fashion that when we place downloadable content on the site, like my slides from the presentation, Google should recognize that the PDF is a part of some other content so people searching for the terms will end up with the page with the content instead of the downloadable file itself.  Thereby allowing Analytics to pick up on it again.  It will be really interesting to me to convert all of the content to Wordpress in a way that Analytics will pick up on and then re-analyze the data from Analytics in a year.