The Lazy Dev’s Guide to Testing Your Web API
Ryan Kelly
Love/Hate Testing?
Quality assurance = diminishing returns
Early testing efforts rewarded with great improvements
Tests get harder, improvements get more obscure
Process improvement!
Better testing tools to do more with less effort
Invest in laziness!
Be pro-actively lazy! (And I clearly need to re-read Moving Pictures!)
What does Ryan know?
Mozilla web services (Firefox Sync Server)
Tools (WebTest, WSGIProxy, FunkLoad)
enthusiast, not yet expert
WSGI
webtest = WSGI for humans
drive a WSGI application in a temporary in-process environment
Demo
Using webtest to ensure the API operates as expected
Many helpers to perform useful function tests
WSGIProxy
In-memory WSGI “app” that maps WSGI requests to HTTP requests
Can easily run webtest based functional tests against an out-of-process web server
Repurpose functional tests as integration tests and deployment tests
Can pick up anything that goes wrong during deployment
Can use as a basis for simple load testing
FunkLoad
Wants to be a monolithic “do everything” test framework
Standard mode of use requires that tests be implemented specifcally for FunkLoad (this annoys Ryan, just demoing it to illustrate the point)
Can gather benchmarking data
Use benchmarking tools, as they deal with problems you may not have thought of yet (e.g. only recording data while under full load, ignoring ramp up and ramp down periods)
Nice report generation tools
FunkLoad + WebTest
Write an adapter between webtest API and FunkLoad API
Can then reuse any of your webtest based tests as a FunkLoad test
Can use to get differential results to see how your response times are going
Distributed Mode
Spin up clients on multiple hosts
Aggregate benchmarks from multiple clients
Other Approaches
Write FunkLoad tests, use WSGI-Intercept to redirect to in-process WSGI application
Q & A
Just some specific questions about FunkLoad and making in-process WSGI calls on a live application
My Thoughts
Not a lot to add! Some of the ideas here seem similar to those in django-sanetesting, but it in a way that doesn’t drive the inheritance model of the test suite. I may look at migrating the PulpDist tests over to webtest (depending on how complex it is to get the in-process Django environment up and running)