There are several ways we measure and report performance in the Sitemorse reports and by looking through some of the client feedback in our recent survey it may be useful to clarify what we do, how we do it and how we report what we find.

The main thing to remember is that we are taking a snapshot of a sites performance.  i.e. if we provide you with a monthly report then the performance figures only relate to the time we were running the report.  We can't definitively say it is representative of your sites performance.  BUT we don't select particular times to run your reports, the times will vary as will the day of the week.  So it should be reasonably representative.

We produce two sets of figures the average figures over the whole test and the figures for the Home Page.

Let's look at the Home Page results first.

We look at the total response time and the speed of the download.  The overall time is obvious how we gather.  Download speed is captured by starting a timer each time we start receiving the first block of data in response to our GET requests and stopping it when the last byte is received.  We know the size of what came down and how long it took so dividing one by the other tells us how many kilobytes per second were received.

Point to remember - line speeds are always quoted in kilobits per second and web page sizes are always quoted in kilobytes.  So you need to divide the kilobits figure by eight to get a rough idea of how the download speed relates to your page sizes.

We then match those against what we consider reasonable times for people connecting via a modem, ADSL or a corporate LAN.  We set these thresholds as 14 seconds, 6 seconds and 4 seconds respectively.  From a modem point of view the limiting factor is the modem.  It can receive data a maximum of about 5 kilobytes per second.  So if we're aiming at 14 seconds you will ALWAYS fail this test if your Home Page is larger than 14 x 5 = 70 kilobytes.  Most Home Pages these days are larger than 70KB so they fail our test.  That doesn't make the test useless, it just means that you have the information about how your Home Page performs for the small number of people that use a modem to browse your site.  You can make the decision to take account of this point and reduce the size of your Home Page, or ignore it.

A 512 kilobit ADSL connection will download at approx. 64 KB per second and a one megabit corporate connection will run at about 128 KB per second.  So if your connection is faster than these you have to look at page sizes of less than 384KB for ADSL and 512KB for corporate to ensure that you pass these two.

We have considered changing these thresholds to reflect the changing landscape of broadband usage and connection speeds but there's so much controversy about what speeds people are actually getting AND quite a lot of sites fail the current tests as they are, so updating them would increase the failure rate.  We'll look at changing them when a high percentage of sites are passing the current tests.

The average figures are captured by starting a timer when we issue a GET request and stopping it when we see the first byte of data.  This gives us the approximate time your infrastructure took to respond. We then start another timer which we stop when the last byte is received.  We do the division and come up the the download speed.  These two figures are then averaged over every GET request done throughout the entire report.  The objective is to attempt to give you an idea of where any performance issues lie.  If you have an average RESPONSE time of 45ms you do NOT need to look at your infrastructure.  Likewise if you have a DOWNLOAD SPEED of 560 kilobytes then you can smile broadly.

If we report RESPONSE times averaging 1350ms the only further information we can give you is the slowest TEN items.  Because we are looking at this from out on the internet we can't gather more detailed info on whether it's your Webserver, Application server or Database server etc that is causing the bottleneck.

More information on how we score sites for the various tests is available here secure.sitemorse.com/benchmark.html