Web log analyzer for multiple domains

shedlord · Feb 17, 2007

We have a remote server hosting quite a lot of websites (over a thousand). Each site has its own logs in a different folder on the server (eg. /logs/site1.com, /logs/site2.com). We are looking for a web log analyzer which allows us to set up each site individually for separate analysis, and which will automatically download new logs from each log folder on the server on a schedule we set. The idea is we could leave it downloading overnight. Ideally it would also include the ability to summarize the stats across all domains (eg. which sites are most popular? how many visitors were there across all sites? etc)

Anyone know if such a product exists?

thanks

skippy258 · Feb 17, 2007

Hi there just a small question to help me help you. Are your servers based on a Linux or Windows OS and what version of OS is it I ask this just to help me pinpoint the kind of software you might need. I have not run a server for about 2 years but still might be able to help you out in the search for the right program.

Thanks

shedlord · Feb 17, 2007

The servers are windows. We won't be able to host a log analyzer directly on the server (the guys who run it for us say the server won't cope with this). Hence the FTP download question.

shedlord · Feb 18, 2007

Bit more background on my problem.

We have 1000 websites. Each has its own log folder on the server. Occasionally my boss asks me to tell him which of these sites have high traffic. The only way I have of doing this currently is to first decide on a figure for the filesize of a log file which suggests there are a decent number of visitors, then to manually open each log folder and note down which have logfiles that average over this figure.

1000 times!

Is there an easier way to do this?

We have FTP access only. We can't install a log analyzer on the server because it is creaking already. We can ask the 3rd party company who host it for us to send us the logs on DVD however. Even with the logs on a disk locally, not sure how to get around having to manually assess each folder.

shedlord · Feb 19, 2007

Speaking to the company who look after our server, their opinion was that there aren't specific log analyzers for cross-site analysis, and most log analyzers are fairly basic and aimed at single sites with low traffic.

An interesting suggestion they made was Sequioa View,
http://www.win.tue.nl/sequoiaview/
...which I've never seen before. Looks like it would only work with local access (?)

Anyway, I'm close to a semi-decent solution to this problem. Using the ASP below I was able to list the folder structure on our server with the sizes of each folder indicated. Typically, the only folder which isn't yet playing ball is the Logs folder, because I assume it is in reality in a different location to the rest (it just looks like it is in the root when we view the server via FTP).

This script is taken from here...
http://www.brainjar.com/asp/dirlist/
...with just a long script timeout added and the target folder changed.

<%@ LANGUAGE="VBScript" %>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

<% '***************************************************************************
'* ASP Directory Listing *
'* *
'* Do not remove this notice. *
'* *
'* Copyright 1999, 2000 by Mike Hall. *
'* Please see http://www.brainjar.com for documentation and terms of use. *
'***************************************************************************
%>

<% Server.ScriptTimeout=8000
%>
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title>Directory Listing</title>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<link rel="stylesheet" type="text/css" href="/common/default.css" />
</head>
<body>

<div id="demoBox">

<h3>ASP Directory Listing</h3>

<p>All folders and subfolders listed below</p>

</div>

<ul>
<% ListFolderContents("PHYSICAL_LOG_FOLDER_PATH_HERE") %>
</ul>

</body>
</html>

<% sub ListFolderContents(path)

dim fs, folder, file, item, url

set fs = CreateObject("Scripting.FileSystemObject")
set folder = fs.GetFolder(path)

'Display the target folder and info.

Response.Write("<li><b>" & folder.Name & "</b> - " _
& folder.Files.Count & " files, ")
if folder.SubFolders.Count > 0 then
Response.Write(folder.SubFolders.Count & " directories, ")
end if
Response.Write(Round(folder.Size / 1024) & " KB total." _
& vbCrLf)

Response.Write("<ul>" & vbCrLf)

'Display a list of sub folders.

for each item in folder.SubFolders
ListFolderContents(item.Path)
next

'Display a list of files.

' for each item in folder.Files
' url = MapURL(item.path)
' Response.Write("<li><a href=""" & url & """>" & item.Name & "</a> - " _
' & item.Size & " bytes, " _
' & "last modified on " & item.DateLastModified & "." _
' & "</li>" & vbCrLf)
' next

Response.Write("</ul>" & vbCrLf)

Response.Write("</li>" & vbCrLf)

end sub

function MapURL(path)

dim rootPath, url

'Convert a physical file path to a URL for hypertext links.

rootPath = Server.MapPath("/")
url = Right(path, Len(path) - Len(rootPath))
MapURL = Replace(url, "\", "/")

end function %>

Log in or Sign up

Web log analyzer for multiple domains

shedlord Private E-2

skippy258 Private E-2

shedlord Private E-2

shedlord Private E-2

shedlord Private E-2