a test suite for ice cream (1/?)

April 4th, 2008 by pam | Posted in programming | 1 Comment »

So, most days, I think I am pretty good at my job, which is [nominally] to write automated functional tests for web applications using twill and nose (and sometimes selenium RC, when I have exhausted all other possible options). That’s what I do, and I don’t suck at it, but often I feel kind of trapped in this silo of narrow knowledge where I can write kickass test generators but then I go and ruin my carefully crafted illusion of competency by saying things like, “tuples can only have two things in them, right? That is why they are called tuples? TWO-pules? Right? … guys?”

Anyway, yesterday I was having problems with variable scope, and today I managed to write a test suite for a new app we’re building, and I am pretty pleased with myself. (It still has some problems, but more on those later.) I don’t know how many people use twill + nose to do the things we do — I never find much about it on the interwebs — so here’s the skinny.

We have a lot of applications that are hooked up to many domains, so while we have many sites (hundreds of them, in fact), we really only have a handful of applications. So the problem has always been something like this:

Let’s say you have come up with a way to sell ice cream over the internet. You set up your site, www.icecreamrules.com, and everything is peachy. Business booms! You decide that you should also sell gelato, so you make www.gelatorulesmore.com, and you build it as basically a skin of the ice cream site. One app, two sites, same functionality.

How do you test it? You could write every test twice, but that’s dumb. It quickly becomes unmaintainable when you decide to also sell frozen custard and milkshakes and sno-cones and Hawaiian shaved ice and those strangely tasty low-fat fudgesicles.

From there, it gets even more complicated because the different types of frozen treats require different site behavior. Most sites have three pages (pick your flavor, where should we send it, thanks for playing), but the milkshake site has an extra page that wants to know if you’d like your milkshake really thick, or thin and drippy. The low-fat fudgesicle site skips the flavor question entirely (it’s fudge-flavored, yo) and just sends you straight to the ordering page.

And THEN maybe you decide to change it up a little and add chocolateicecream.com, which does the same thing except instead of asking for flavor, it asks you for type (gelato, custard, pick your poison). Your app is getting more and more complex, and the tests have to keep up.

So is it still crazy to write a separate test suite for each site? … yeah, still pretty nuts. Okay!

So the best idea we’ve been able to come up with is that there has to be some way to pass parameters to nose via the command line: nosetests –with-site=icecreamrules.com, or nosetests –with-flavor=chocolate. Each site has a configuration file with the relevant details, and the tests rip through them, generate the dataset, go the sites, and make sure everything is copacetic. Right?

Assuming that yes, we’re right, step one is the nose plugin. The code itself is pretty simple, actually, but it requires a lot of thinking about how you’d like to build your suite and what sort of control you’d like to have over the tests. Maybe for our ice cream app we want to run according to site, flavor and type? So the plugin will look something like this:

from nose.plugins import Plugin

class IceCreamParams(object):
    ice_cream_flavor = None

class IceCreamPlugin(Plugin):
    name = 'IceCreamPlugin'

    def options(self, parser, env):
        parser.add_option("--with-flavor", action="store", dest="flavor",
                                 default=None, help="Run against a given flavor.")

        Plugin.options(self, parser, env)

    def configure(self, options, conf):
        Plugin.configure(self, options, conf)

        IceCreamParams.ice_cream_flavor = options.flavor

Obviously that is only for flavor, but it should be pretty clear how to add more options. Once that’s done, write a setup.py file and you’re good to go. IceCreamPlugin will eat up the value you pass in the command line, and that will all live in your IceCreamParams container object. From there, you can write modules to consume that object, by importing it and then calling whatever values you’ve set.

I was going to write up the whole thing, but then I realized that documenting the process is going to take me way longer than it did to actually write the code and I wanted to post something while it was fresh. I will continue showing off my ice cream app tests at a later date, probably by discussing the site configs. Eventually we can hook everything together, and it will be 16 kinds of cool.

But before I do that, I should point out that the shiny test suite I wrote this week was built largely on the bones of the one built by Terry Peppers (with help from Kumar McMillan and Jason Pellerin and Luke Opperman and Chris McAvoy and probably a bunch of other people), and I have basically just cannibalized it for my own whimsical purposes. Terry may also be writing about this whole testing with twill + nose thing, but he has less free time than I do, so I figured I might as well start.

Thoughts?

gimme that returned thing.

April 3rd, 2008 by pam | Posted in programming | 5 Comments »

Wow, I am kind of an idiot sometimes. In my defense, I have been programming for maybe a year and have zero formal training. I’m pretty lucky to work with the people I work with, though, and I think my one saving grace is that I can usually tell when I’m going down the wrong path and I’m not particularly shy about asking for help before I find myself totally lost in the rabbit hole.

I’m sure everyone else knows this already, but hey, maybe someone is interested in the little shit that trips up Python newbies. So, here’s today’s adventure.

def do_something():
    foo = []
    foo.append("bar")
    return foo

if __name__ == "__main__":
    print foo

Obviously, that doesn’t work. I get a NameError: name ‘foo’ is not defined. But if I declare ‘foo’ outside the function, there’s no problem. Works just fine. I knew that was wrong, though, because no one else has to do that. I tried and tried to figure it out, because obviously I was having scope problems and it seemed like something that would be, well, obvious. But finally I broke down and asked someone, and he said, “wait, you’re not assigning the return value of that function?”

Um. In fact, no. So now I have this:

def do_something():
    foo = []
    foo.append("bar")
    return foo

if __name__ == "__main__":
    baz = do_something()
    print baz

And it works! Pretty lame as adventures go, I know, but it is kind of a major revelation to me.

late pycon wrapup

March 27th, 2008 by pam | Posted in programming | No Comments »

I meant to do this a little closer to the event itself, but first I decided I needed to have a site that wasn’t entirely embarrassing. I thought about trying to build myself something, but I eventually threw in the towel and just installed WordPress. It’s easy, it does what I want, I build enough websites, I’d rather spend my time on other things.

ANYWAY.

I’m not really sure what to say about PyCon, actually. It was my first one, and while I gather there was some discontent over the level of commercialization, I had a really great time. I think the best part for me was meeting people: I met a TON of awesome people, some of whom I’ve been secretly fangirling for ages (well, okay, lurking in their blogs) and some of whom I’d never heard of. I went to dinners and parties and happy hours and it was all way more fun than I was expecting it to be. (Although: I’m sorry, no, I don’t know why there are not more women in open source. I’m afraid that I am not plugged into any sort of female programmer zeitgeist.)

Outside of the socialization aspect, I thought it was… I don’t know, okay? I went to Titus & Grig’s testing tutorial on Thursday and acquired some hints I’d like to try out. I went to a few talks, but I don’t know if I just wasn’t going to the right ones or what. They were either too low-level or too high-level for me, and while I tried to go to intermediate talks, I apparently failed spectacularly.

The talks I think I got the most out of were given by my illustrious coworkers: Kumar’s unicode talk (I write a lot of scripts in the futile attempt to bend del.icio.us to my will, and unicode is my arch-nemesis in this regard) and JP’s nose talk. I also quite enjoyed Ian Bicking’s talk on consuming html, despite the heckling, and Titus’ talk about OLPC testing. The py.test talk was something of an eye-opener, actually, which I realized was kind of dumb once I stopped to think about it. See, the way I have learned about python is that I got this job doing testing and was told, “okay, we write tests in python using twill and nose!” And that was kind of it; it didn’t really occur to me that nose is a test framework and that obviously there are others. Like I said, dumb once I thought about it; I just hadn’t really thought about it. I’m too invested in nose to switch at this point, plus the team I work on has standardized on twill+nose, but py.test does have some nice features that I’d love to see somehow turned into nose plugins.

The one bit of coding I got done at the conference was a nose plugin of the very smallest sort; it allows me to do this:

>>> nosetests --with-env=qa

And just pass in an environment variable. I had been either setting it in my .bash_profile or putting it into a settings file, but I’d much prefer to pass it through the command line. That is what I did during the nose mini-sprint, while JP did actual nose things. My python abilities are very narrowly focused, but I’m working on it. Watch this space!

And, if I met you at Pycon, drop me a line and say hi. I almost certainly thought you were awesome and would like to talk again.

so delicious

March 25th, 2008 by pam | Posted in personal | 1 Comment »

So, I’ve been pretty obsessed with del.icio.us lately, and last week my boss wanted to know why. Here we go.

• del.icio.us lets me access my bookmarks from anywhere. Sure, there are crazy ways to sync bookmarks across multiple machines, but I like being able to pull up some link from my friends’ machines, too. Often it’s much easier to find something saved in del than it is to google for it.
• del.icio.us has made actual browser bookmarks useful again. I used to have this insane bookmarks folder and I was always organizing it and moving shit around and I could never FIND anything and often it just ended up being a list of a few thousand links I was never going to do anything with again. Now, though, I have maybe a dozen sites bookmarked. I never ever open my bookmarks folder or look at my list — everything has a keyword assigned or an icon next to the URL box (for instance, I just type “g” in the location bar to look at gmail, and “t” to look at my tickets at work). Bookmarks are actually useful to me now, instead of being just a junk drawer.
• del.icio.us has increased my productivity. The background here is: I get a little antsy reading or hearing about GTD because many of its adherents are so annoyingly self-righteous and preachy about it, but there are a few of its tenets I buy into. The main one is that, when stuff flies into my head, I need to do something with it right away — put it into a system of some sort — and get it out of my head, allowing me to focus on whatever the hell I’m doing. I need to trust my system, and it needs to run mostly in the background. (For example, my mail checks itself every 15 minutes and then my computer notifies me that I have new mail. If I didn’t trust that to work, the system would be useless, because I would always be checking my e-mail manually and I would never get anything done.) One of the ways I use del.icio.us for this is my .checkthisout tag. Any time I find myself looking at a page that I know I am going to need later, but it’s not really relevant to what I am doing right now, I throw it in del and close the goddamn tab. This could be documentation or a blog post on some new python module or a piece of software that looks interesting that I want to test at some point. I subscribe to that tag via RSS, and the stuff there ends up in my ‘inbox.’ (Not my e-mail inbox; the ‘inbox’ is the GTD information-collector, the funnel for the data stream that comes out of my head, the big list of Everything I Need To Deal With In Some Manner, whether that manner is to put it on a do-this-next-week list or to read a blog post or to download some software or to send an e-mail). I periodically go through the inbox and deal with everything that’s in it, and the advantage to this is that I deal with everything I need to deal with at a set time, rather than just sort of as it comes up, which is hugely distracting.
• del.icio.us hits every OCD button I’ve got. (I have many.) I can use it as an exhaustive archive of everything I have ever thought was cool. I can spend all kinds of time thinking about tagging and organizing and bundling and systems. I can think about how to bend it to my will (see above bullet point). Etc.
• del.icio.us also hits a lot of my more academic interests in shit like organically grown folksonomies and organizational structures and data categorization andandand. FASCINATING.
• del.icio.us is for sharing! I have some friends without del accounts (crazy!) or who don’t check them very often, so I have a tag that is their name and they subscribe to that tag via RSS. For my friends with del accounts, I am often sending them things, and I get an awesomely warm-and-fuzzy feeling whenever I see they have saved a link I’ve sent them. It’s like, hey, I sent you something and you think it is awesome enough to keep! Hooray! That is a pretty cool feeling.
• del.icio.us is technologically interesting and accessible. It has a decently well-documented API that I can hit and do cool shit with. I have only been programming for six months or so, and I would consider only the last three months of that to have been “actual” programming (whatever that means). But I am able to write little scripts using del’s functionality to make it better and more useful for myself and for other people. I’ve written a thing to tag broken links as broken, and I’m working on a mass uploader script, and I’m working on an RSS feed freshener, and… those are pretty cool projects, but they’re things that are simple and small enough that I can get my not-very-programmer-like arms around them. And yet, if I can make them awesome, they will be INCREDIBLY useful to a lot of people who are like me.

So, there you have it. That is why I am obsessed with del.icio.us. And why I think everyone else should be, too!

twill + rails

March 25th, 2008 by pam | Posted in programming | No Comments »

My company has an increasingly large stable of web applications. The biggest one is written in PHP (currently being rewritten in python), but in the last year or so we’ve come to rely more heavily on frameworks, mostly Django and Rails.

Testing Rails apps with twill can be tricky, particularly if they’re RESTful. I haven’t quite figured out why that is, but once my dev refactored his code to make it more RESTful, my tests started failing. Said tests are all written in python, so let’s open up a shell and see what that gets us.

>>> from twill import commands as tw
>>> tw.debug('http', 1)
DEBUG: setting http debugging to level 1
>> tw.go(http://qa.xxxxx.com/xxxxx)
connect: (qa.xxxxx.com, 80)
send: 'GET /xxxxx HTTP/1.1\r\nAccept-Encoding:identity\r\nHost:
qa.xxxxx.com\r\nConnection: close\r\nAccept:
text/html; */*\r\n\r\n'
reply: 'HTTP/1.1 500 Internal Server Error\r\n'
header: Date: Tue, 20 Jan 2008 22:10:13 GMT
header: Server: Mongrel 1.0.1
header: Status: 500 Internal Error
header: Cache-Control: no-cache
header: Content-Type: text/html; charset=utf-8
header: Content-Length: 9742
header: Set-Cookie:
_session_id=23e1a01fb82c5356cda6cb34fadfabea; path=/
header: Connection: close
==> at http://qa.xxxxx.com/xxxxx
'http://qa.xxxxx.com/xxxxx'

Rails routes traffic based on headers, so let’s hit this random header testing site.

In twill, you get this:
Accept: text/html; */*
Accept-Encoding: identity

But hitting it in my browser gets me:
Accept: text/xml, application/xml, application/xhtml+xml, text/html; q=0.9, text/plain; q=0.8, image/png, */*; q=0.5
Accept-Encoding: gzip, deflate

The thing to pay attention to, really, is that q= stuff in the browser’s accept headers.

So! After much digging around, there are two ways to fix it (well, three, where the third is ’submit a patch,’ which I haven’t yet done). To fix this issue in twill itself (which is what I’ve done, even though it means my tests are not particularly portable), open up browser.py and take a look around line 63. It reads:

# Ask for MIME type 'text/html' by preference.
self._browser.addheaders = [("Accept", "text/html; */*")]

What that means is that twill’s default accept header is:
Accept: text/html; */*

Which, according to the http spec, is incorrect. What that header does is give equal preference to two different MIME types, which causes Rails to freak out. It doesn’t know what the browser wants, so it doesn’t send anything at all. (That may be an issue in Rails as well — if it can’t figure it out, why doesn’t it pick something rather than send nothing?)

Anyway, the twill-centric fix is to change that line to:

self._browser.addheaders = [("Accept", "text/html; q=0.9 */*")]

Adding the “q=0.9″ sets the priority for */* at .9, leaving the priority for text/html at 1. Now that the two different MIME types have two different priorities, Rails knows what the browser wants and doesn’t freak out.

If you want to fix it without breaking into twill and also keeping your tests portable, you can add this to your tests:

from twill import commands as tw
tw.clear_extra_headers()

That will clear out the bad default accept headers that twill is sending, and will allow your tests to work. Fancy!

generators != lists (?)

March 25th, 2008 by pam | Posted in programming | 4 Comments »

My never-ending quest to write automated tests for web applications mostly involves twill tests, run with nose. Because I am muddling through this more or less on my own, I rely very heavily on documentation and examples, most of which were written by developers for other developers. I’m not a developer, so I spend a lot of time being really confused and asking my devs annoying questions.

Here is today’s issue, probably trivial to anyone who does this regularly, but I’m sure to forget about it and gnash my teeth later.

The twill docs tell me that, “When called from Python, this function [showlinks()] returns a list of the link objects.” Awesome. What I am trying to do is follow link[0], which has no name (so I can’t use follow()). I figured I could get the URL out of the list, and go there. Easy lemon squeezy, right?

But here’s what happens:

>>> showlinks()
Links:
0. ==> http://www.rfc-editor.org/rfc/rfc2606.txt
< generator object at 0x89828 >

Okay, so now I’ve got a generator object to play with. It can make a list, but it doesn’t just return one the way I thought it would. But whatever, let’s make it a list:

>>> links = list(showlinks())
Links:
0. ==> http://www.rfc-editor.org/rfc/rfc2606.txt
>>> links[0]
Link(base_url='http://www.example.com', url='http://www.rfc-editor.org/rfc/rfc2606.txt', text='RFC 2606', tag='a', attrs=[('href', 'http://www.rfc-editor.org/rfc/rfc2606.txt')])

Aaaaand now I have a… list of tuples? I’m not sure what all I was expecting would be in a “link object,” but it wasn’t quite so much information. But I figure I’m almost there. Sadly, not so much.I think my main problem came from that “Link” in the beginning. What is that? I still don’t know. I tried a lot of things that made sense to me at the time:

>>> links[0][1]
Traceback (most recent call last):
File "", line 1, in ?
AttributeError: Link instance has no attribute '__getitem__'
>>> dir(links)
['__add__', '__class__', '__contains__', '__delattr__', '__delitem__', '__delslice__', '__doc__', '__eq__', '__ge__', '__getattribute__', '__getitem__', '__getslice__', '__gt__', '__hash__', '__iadd__', '__imul__', '__init__', '__iter__', '__le__', '__len__', '__lt__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__rmul__', '__setattr__', '__setitem__', '__setslice__', '__str__', 'append', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort']
>>> links[0].url

So… “Link” is the instance of the link object? Which I thought was a list, and type() tells me it’s a list, but it has no __getitem__? Clearly there is something here with instances and objects and types that I’m just not getting. The solution turned out to be fairly simple, and even elegant, for that matter:

>>> links[0].url
'http://www.rfc-editor.org/rfc/rfc2606.txt'

But I confess I’m still at a bit of a loss as to why that’s the answer instead of something else.