Replaying Web Traffic

Patrick Killelea

Capture Traffic Via A Proxy

Sprocket is a web proxy server which dumps a record of the traffic flowing through it.
A record of web traffic is useful for:
- creating web load tests
- creating monitors for web sites
- figuring out exactly what a browser is doing
Being written in perl and not requiring any libraries, sprocket runs without change on Unix, Macintosh, or Cygwin (on Windows).

Using Sprocket To Record

Start sprocket.

% sprocket > out.http
sprocket listening at 8080. Clear your cache before surfing.

Set your browser to use sprocket as its proxy.
Surf. A transcript of your http traffic will be saved.
sprocket continuously outputs status and diagnostic messages to stderr. If you don't want to see those, you can start sprocket with stderr redirected to /dev/null like this instead:
```
% sprocket > out.http 2>/dev/null
```

Output Format

Sprocket delimits requests and responses with lines of the form:
2837 >>>>>>>> 1199832276.5563
2837 <<<<<<<< 1199832277.0243
First number is the local browser port, corresponding to a thread of execution in the browser.
Second number is a unix timestamp.

Example Transcript

2853 >>>>>>>> 1199832944.7535
GET http://patrick.net/ HTTP/1.1^M
Accept-Encoding: gzip, deflate^M
Host: patrick.net^M
User-Agent: Mozilla/5.0 (Macintosh; U; en-US; rv:1.8.1.10) Gecko/20071115 Firefox/2.0.0.10^M
Accept: text/xml,application/xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5^M
Accept-Language: en-us^M
---------------: ------------^M
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7^M
Keep-Alive: 300^M
Proxy-Connection: keep-alive^M
^M

2853 <<<<<<<< 1199832945.1593
HTTP/1.1 301 Moved Permanently^M
Transfer-Encoding: chunked^M
Date: Tue, 08 Jan 2008 22:55:38 GMT^M
Server: Apache/2.0.54 (Unix) mod_ssl/2.0.54 OpenSSL/0.9.7l PHP/4.4.0 mod_gzip/2.0.26.1a^M
X-Powered-By: PHP/4.4.0^M
Location: http://patrick.net/housing/crash.html^M
--------------: ---^M
Content-Type: text/html^M
^M
6F^M

Please use http://patrick.net/housing/crash.html
instead
^M
0^M
^M

Options

To use a different proxy port, start with the -p option:
```
% sprocket -p 8081 > out.http
```
To bind to a specific IP address on a machine with multiple interfaces, use the -i option:
```
% sprocket -i 10.32.0.106 > out.http
```

Sort HTTP

It's possible that responses will come back in a different order than requests were made.

To prevent confusion, we need to sort so that each response comes immediately after its request.

% sort_http out.http > sorted.http

Replay

Reply plays back http traffic.
Replay validates every response line.
Most response lines are not relevant, or are variable.
Solution: edit the transcript to include only relevant response lines.

Trim

Trim will remove most of the response:

% sprocket > out.http
% sort_http out.http > sorted.http
% trim sorted.http > trimmed.http

HTTP response code and the HTML <title>.*</title> will be left.
These are usually enough to validate we got the page we expected.
You can manually edit the http recording in any text editor, but be sure to leave the ^M's at the end of each request line. Web servers expect them.

Replay Output

For each request, replay outputs one line.

% replay trimmed.http
yyyy mm dd hh mm ss.sss latency bytes url
2008 01 08 15 08 20.876 0.227     486 http://patrick.net/

Replay Options

-d Print debug output to STDERR.
-r Keep redoing a request until the expected proof is obtained.
-s Be completely silent, output nothing.
-t Continue even if proof lines not found in server output.
-v Print complete transcript of the http traffic.

Load Testing

Any language can be used to call multiple copies of replay.
Script can be altered between each interation.

#!/bin/bash -x

((user = 0))
while ((user < 10))
do
    replay trimmed.http > out$user.txt &
    ((user = user + 1))
    perl -pi -e "s/user=user\d+/user=user$user/" trimmed.http
done

Load Testing Example

Load test of real web-based chat site. Site could not be tested with conventional tools, but was testable with sprocket.

Monitoring

Using cron, we can replay traffic at regular intervals.
Using gnuplot, we can plot traffic to make a dashboard.

Cron Example

# MIN   HOUR   DOM    MOY    DOW   Commands
#(0-59) (0-23) (1-31) (1-12) (0-6) (Note: 0=Sun)
*/10    *      *      *      *     replay home.http > home.out
*/10    *      *      *      *     gnuplot home.gp

Gnuplot Configuration File

set term png
set xdata time
set timefmt "%Y %m %d %H %M"
set ylabel "seconds"
set title "homepage latency"
set output "home.png"
plot 'home.out' using 1:7 notitle with lines

Example Monitor

Anything can be monitored this way, including Bay Area housing prices.

Pros and Cons

Commercial Tools	Sprocket
easy ramp up	needs command line knowledge
reporting included	reporting done manually
expensive	free
rigid, limited	flexible
proprietary test language	perl
proprietary data format	text
obscure	clear what is going on