Efficient reporting in a web application

I know that everyone who has a webpage has Google Analytics as well, and monitors the visitors, clicks and many-many things. Google Analytics is free and cool and really great. (BTW, if you would like to become a real expert in web analitycs check out this blog.) However I have missed one thing very much: I can not track with it how users actually move through my site. I mean that I would like to see that a user arrives to my site, first she views page A, from page A she goes to page B and then to page C and then she leaves. If you want to understand your users I think it is nice to be able to see how they exactly move from page to page. Because of this, I have decided that in addition to using Analytics I will as well do a very simple reporting thing that will basically log every request with a user id and like this I will be able to check user movements. :)

My biggest concern was performance though. I did not want to make a database write for every request, as I thought it would make the read requests slower. So here is what I have done:

I have created a UserActivity table with columns like: IPAddress, Uid, UserName, CretOn, Url. If a user is logged in then I put the username in the UserName column, it is empty otherwise. If user is not logged in only Uid is filled out with a guid I generate for everyone who visits the site for the first time. Than I store the Uid in a persistant cookie, so that I can track the movement of unauthenticated users as well.

Ok so, the interesting part is how do I insert the data in this UserActivity table?

• Reporter class: Reporter class has an Activities property that is a list of UserActivity. The class has an Add method that puts an activity in the Activities list, and it has a Report method, that inserts everything into the database from the Activities list and empties the Activities.

          public void Add(UserActivity activity) {
              lock (this)
              {
                  Activities.Add(activity);
                  if (Activities.Count >= ReportingTreshhold)
                  {
                      Monitor.PulseAll(this);
                  }
              }
          }
  
          public void Report() {
              List copyOfActivities;
              while (true)
              {
                  try
                  {
                      lock (this)
                      {
                          Monitor.Wait(this);
                          copyOfActivities = Activities;
                          Activities = new List();
                      }
                      // Report the Activity to the database
                      ReportingEntities reportingdb = new ReportingEntities();
                      foreach (var item in copyOfActivities)
               {
                          reportingdb.UserActivities.AddObject(item);
                      }
                      reportingdb.SaveChanges();
                  }
                  catch (Exception ex)
                  {
                      ExceptionHelper.RaiseErrorSignal(ex);
                  }
              }
          }
  

• Start a background thread in Global.asax: For the reporting thing to run I start a background thread when the application starts in the Global.asax.

     Thread reporterThread = new Thread(new ThreadStart(Reporter.Report));
     reporterThread.Start();
  

• As I have an ASP.NET MVC application I have created a ReprotingFilter that runs on every request and adds a UserActivity to the Activities list.

          public void OnActionExecuted(ActionExecutedContext filterContext)
          {
              UserActivity activity = new UserActivity();
              HttpRequest currentRequest = HttpContext.Current.Request;
              // We filter out search engines
              if(!currentRequest.Browser.Crawler){
  
                  // Create the UserActivity Object here...
    
                  Application.Reporter.Add(activity);
              }
  
          }
  

Actually I am absolutely satisfied so far with this reporting thing. My site is still quick, user requests are now logged. There was only one big mistake I have made for the fist time: Do not forget to filter out search bot requests, otherwise you will see logged tons of requests from search engines!