https://medium.com/@francis.john/local-webpagetest-using-docker-90441d7c2513

Local WebPagetest Using Docker

Francis JohnFollowAug 24, 2017 · 5 min read

(Update 8/25/2017: A day after this was posted, traffic shaping on Mac OS was officially supported, however if you are using Docker it will not work as Docker runs in a linux headless agent.)

Recently at NerdWallet we’ve started using SpeedCurve to have a historical record of our WebPagetest results and shiny graphs to help visualize. It’s been a really great tool in terms of measuring performance for a large production website.

However sometimes it can be useful to have an even tighter feedback loop when you are iterating on performance, to get a sense of what works and by how much.

WebPagetest is awesome but I found the documentation a bit lacking so I thought I’d share into my experience in setting up a WebPageTest private instance for local development.

Docker Setup

WebPageTest has two components that need to be in sync with each other, the server and the agent. The server handles the frontend app for WebPagetest and the agent actually runs the tests. The agent polls the server for tests to run, and posts to an endpoint on the server. To spin these up, the WebPageTest folks have created Docker images for both:

https://hub.docker.com/r/webpagetest/agent/https://hub.docker.com/r/webpagetest/server/

If you aren’t super familiar with Docker (as I wasn’t), I’d recommend going through the tutorials to familiarize yourself. After you’ve gotten setup you’ll want to pull down the WebPageTest server and agent images.

$ docker pull webpagetest/server
$ docker pull webpagetest/agent

After pulling down the images you can create the containers.

Run the server

# -d runs the container in "detached" mode in the background
# -p specifies the port to expose.
# --rm will remove the container upon exiting
$ docker run -d -p 4000:80 --rm webpagetest/server

Just after running the server, you should be able to browse to http://localhost:4000 and see the webpage test private instance page to confirm it’s up.

NOTE: Feel free to specify ports other than the one(s) I’m using.

Run the agent

# --network="host" allows the agent to communicate with the server container
# -e specifies an environment variable. WPT requires `SERVER_URL` and `LOCATION` be defined.
$ docker run -d -p 4001:80 \
    --network="host" \
    -e "SERVER_URL=http://localhost:4000/work/" \
    -e "LOCATION=Test" \
    webpagetest/agent

The SERVER_URL should be set to <your server>/work/ — it denotes the URL for the agent to poll. The LOCATION should match one of the locations defined in settings/locations.ini in the WebPagetest server. These define the parameters for that agent you are running (e.g. browsers supported, label, and other goodies). These environment variables are used in the startup script of the wptagent python app.

NOTE: The agent will throw errors if it can’t communicate with the server. Try running the above command without detached mode (remove the -d) and it will spit out errors if there are any.

Start it up

Head over to http://localhost:4000 . You should see the frontend for WebPagetest and on the surface things look good. Right? Let’s check. There’s a built in configuration check if you hit http://localhost:4000/install

If your agent is able to communicate with your server, you should see an agent connected for the Test Location. Easy enough.

However if you’re on OS X like I am, there’s still a bit more work needed.

Traffic Shaping MacOS Workaround (until supported)

Let’s try to run a test. Select the Test Location from the list of locations on your WPT server:

If you attempt to run a test at this point you’re going to see an error Error configuring traffic-shaping:

This is because traffic shaping has not yet been implemented on MacOS, so
we’ll need to implement a couple workarounds.

You can remove traffic shaping by doing two things — setting a dummy value for connectivity in your settings/locations.ini file on your server, and setting --shaper none when the agent is starting up.

I’d recommend creating a new Docker image based on the original WPT agent/server images but with modifications to the configuration.

Server

Setup a new folder that has two files, a Dockerfile and a locations.ini

Dockerfile

FROM webpagetest/server
ADD locations.ini /var/www/html/settings/

locations.ini

[locations]
1=Test_loc[Test_loc]
1=Test
label=Test Location
group=Desktop[Test]
browser=Chrome,Firefox
label="Test Location"
connectivity=LAN

This defines a new docker image that pulls from webpagetest/server , replacing the locations.ini file with one that will work for OS X, particularly setting connectivity to remove traffic shaping.

# -t defines the name we are giving to this image
$ docker build -t local-wptserver .

Agent

Setup a new folder that has two files, a Dockerfile and a script.sh

Dockerfile

FROM webpagetest/agent
ADD script.sh /
ENTRYPOINT /script.sh

script.sh

#!/bin/bash
set -eif [ -z "$SERVER_URL" ]; then
  echo >&2 'SERVER_URL not set'
  exit 1
fiif [ -z "$LOCATION" ]; then
  echo >&2 'LOCATION not set'
  exit 1
fiEXTRA_ARGS=""if [ -n "$NAME" ]; then
  EXTRA_ARGS="$EXTRA_ARGS --name $NAME"
fipython /wptagent/wptagent.py --server $SERVER_URL --location $LOCATION $EXTRA_ARGS --xvfb --dockerized -vvvvv --shaper none

This defines a new docker image with a copy of the existing WPT entrypoint script that tacks on the --shaper none flag.

Make sure script.sh is executable:

chmod u+x script.sh

And now we can build our agent image:

# -t defines the name we are giving to this image
$ docker build -t local-wptagent .

Start it up (for real)

Alright now we should be able to start up our local copies. You may need to first stop the existing docker containers.

Stop the parent containers

$ docker ps
CONTAINER ID        IMAGE
5e2374829136        webpagetest/agent
1cf57d494fc8        webpagetest/server
$ docker stop 5e2374829136
$ docker stop 1cf57d494fc8

Start up your local WPT containers

$ docker run -d -p 4000:80 local-wptserver
$ docker run -d -p 4001:80 \
    --network="host" \
    -e "SERVER_URL=http://localhost:4000/work/" \
    -e "LOCATION=Test" \
    local-wptagent

And voila you should now be able hit http://localhost:4000 and run webpage tests against your local environments:

Happy WebPagetesting!

WRITTEN BY

Francis John

Software Engineer