Testing is complicated.

A coworker of mine is getting more into the code side of making software work.  I was talking to him recently about increasing the test coverage on the project he’s coming onto and he asked me, that eternal question:

So, what do I test?

The answer, of course, is “everything.”

Less glibly, software testing is a pretty deep topic.  There are a lot of kinds of testing you can do, some very time consuming, others very quick, and with a similar range in effectiveness in finding bugs.  Of course, there are other reasons to do tests (maybe you don’t care about all bugs and just want to be able to make some performance guarantees, for example.)  But most of the time when people talk about software testing they want to get an idea of how buggy the software is; how close its behavior is to the expected/desired.  In this blog post, I’ll talk about a few broad categories of testing.  But first, a quick testing glossary:

• verification: Examining a product to make sure that it does what it was designed to do (what do the blueprints say?)
• validation: Examining a product to make sure that it does what it is supposed to do (what did the customer want?)
• reliability: How likely is it that a product will function when it is needed?
• assertion: Generally, automated tests perform some number of operations with the system, and then inspect some property of the system afterward.  That property is called an assertion, and this term is used pretty universally across testing frameworks.  It is common to talk about a test or an assertion being “strong” or “weak” based on how specific the requirements are.  (x>2 is usually a weaker assertion than x == 6).
• mocking: The practice of creating an object that *acts* like some component of the system, but without being a full implementation.  This is useful in testing because you can guarantee the return values from methods on a mock object will be appropriate for your test.  It’s also nice to not actually make that bank withdrawal when testing your bill-paying application.  You can mock up objects (ie, instances of a class in some programming language) but it is also possible to mock up an external service, or a user account, or whatever environment is needed for testing.

Smoke Testing

Smoke tests just exist to verify that, upon running your software, the magic smoke does not escape from your computer.  They generally make very weak assertions about the system, like, for example, that a homepage in a web application has a title.  But, even in this example, a passing test shows a lot of good things about the system: the templates are configured properly, the system successfully binds to a port, etc.

Unit Testing

Unit tests intend to test a single component of a system.  While the precise definition of a unit is up to individual teams, it is pretty common to discuss unit tests as ones that operate on a single class (in Java code, for example.)  Ideally, in a unit test, all objects except for the one under test should be mocked.

Integration Testing

Integration testing verifies that components in a system interact correctly.  This is the type of testing that you are doing when you set up that server and just check that the homepage comes up.  Again, the line between unit and integration testing is kind of fuzzy.

Black Box Testing

Black box testing uses only the parts of a system that are intended for use by external components.  For example, if you were black-box testing a Queue class, then you would not make any assertions about the internal storage, only based on the standard Queue operations.  You also wouldn’t test or modify any helper classes used in the implementation of the Queue, and code inspection would not be an option.  Black box testing encompasses techniques like:

• fuzz testing, using randomly generated garbage data as input
• equivalence classes, inspecting a component’s specification to identify ranges of input that should be functionally equivalent (thereby allowing the tester to avoid redundant effort
• user testing, simply handing software to a user and asking him to identify places where it works unexpectedly or incorrectly

White Box Testing

White box testing assumes access to the code and internals of a system.  It allows techniques that modify and inspect the code, things like:

• static analysis, where a tool like lint or findbugs inspects code to heuristically detect errors
• code coverage analysis, where a tool like cobertura instruments the system during a run to identify unused sections of code.
• mutation testing, randomly modifying code to gather statistics about how much has to break before automated tests start to fail
• code review (by real live programmers!)

I guess I still haven’t answered the question; what should we test?  It’s going to vary a lot between projects.  A lot of shell scripts I write don’t get tested at all (well, except for the one time I run them), but I’d like to know that the guys working on the software for the next plane I’m on have a pretty solid testing system.  And there are a lot of points in the space between those extrema. In general, I would say probably getting basic tests in place to begin with is good, and of course as a project goes on problem points can be identified.  Automated tests are good for ensuring old behavior doesn’t change unintentionally, but there is a balance to strike between setting up that safety line and tying the whole project down.  Because (especially in web programming) things do actually change from time to time.

You Don’t Need Java to use the JVM

Not too long ago I was looking for a decent web application framework for an application we’re working on at OpenGeo. It’s based on GeoServer and maybe some other Java servers, but the team expressed some concerns about being able to quickly turn around and maintain Java code. So I checked out some alternative JVM languages during my search. Here are my thoughts on the ones I looked at, Jython, Rhino, Groovy, and Scala.

Jython

Jython is an implementation of Python that runs on the JVM. It lets you call Java constructors and methods, as well as extend Java classes and interfaces, and it even maps bean properties to keyword arguments on constructors.  However, the interoperability is not entirely seamless in either direction.  When using overloaded Java methods from Python code, the method is selected on the runtime type of the arguments (nulls are especially troublesome here.)  The recommended way to work around this seems to be just to perform the appropriate conversions manually (not particularly problematic.)  In the other direction, things are a bit more of a hassle: since the Python code is not compiled, classes defined in Python do not exist until the script has been run (and disappear once the JVM exits.)  So, there’s a fair bit of boilerplate required to get a hold of an instance of a Python class, and integration with frameworks that rely on reflection (such as Spring) will require even more (a wrapper class to create the Python object and then delegate to its methods).  Additionally, the base types (String, Integer, etc.) used in Python scripts are not the standard Java classes, so occasionally there is an impedance mismatch due to that.  (The interpreter automatically wraps and unwraps the objects so most of the time it is not a problem.)

Rhino

Rhino is an implementation of JavaScript that runs on the JVM. It also allows you to call Java constructors and methods and extend Java classes and interfaces.  There is also some nice sugar:

• bean properties are mapped to simple properties in JavaScript code (obj.foo = “new value”; instead of obj.setFoo(“new value”); )
• The method overloading problem is handled by providing access through longer keys which include the type signature, as well as the short name.
• If a method expects an interface argument and the interface only declares one method, you can pass in a function object instead of explicitly implementing the interface. (var button = new javax.swing.JButton(); button.addClickListener(function(){…}); )
• Rhino implements the E4X extension for embedded XML literals, so manipulating XML documents is pretty painless.

Rhino has also been around for the longest of any of the languages I looked at and is a fairly robust implementation.  There is a compiler so you can generate real, reflection-friendly classes with it, and, although the JavaScript standard library leaves a bit to be desired, there are projects like Narwhal and JSAN to improve that situation.

Groovy

Groovy is a language designed specifically for use on the JVM, with an eye to interoperability with existing Java libraries, but also providing a more dynamic language.  It provides facilities like optional typing and the “elvis operator” (which works like || in other scripting languages for providing default values in expressions that would otherwise return null):

javascript: var foo = baz.mightBeNull || "defaultFoo";
groovy:     def foo = baz.mightBeNull ?: "defaultFoo";

On the other hand, valid Java code is a mostly valid Groovy syntax, so transitioning an existing codebase is (supposedly) easy.  There are some gotcha’s though.  For example, if you use type checking, it is only implemented at runtime, so you can have the headache of a compile phase followed by a type error.  Again, compiled code is fully accessible to the Java runtime, including reflection and direct instantiation.

Scala

Scala is another language designed with JVM interoperability in mind, although it is much more of a deviation from Java than Groovy.  We’re not using it for the project that inspired this research, but I’ve been working with it on a little side project for some time now.  It is fully interoperable in the sense that Scala classes are Java classes too, but it resorts to some clever compile-time tricks to implement some of its niftier features, like operator overloading and traits (similar to Java’s interfaces, but with the ability to provide default implementations for methods.)  However, it is fully type-checked, using type inference to avoid type declarations all over the code.  It also has a number of features that don’t translate well to Java code:

• Extractors, used for pattern matching; implemented as functions which return instances of a class from the Scala library.  These are usable from Java, but without Scala’s syntactic sugar for using them, they are much less useful.
• Classes may have companion objects (which serve as the holder for any “global” methods; the stuff that you would declare static in Java.)  The way that the Scala compiler implements these means that static methods show up in Java code as part of a second class with a $ at the end of its name (ie, Foo would have a Foo$ class with all the static helpers.)
• Functions as values.  In scala code, calling a function stored in a variable is just like calling a function normally, and there is specialized syntax for function variable types: “var foo: (Int, Int) => Int = (a, b) => a + b”, “foo(1, 2) == 3″.  In Java, such functions show up as instances of scala.Function2<scala.Int, scala.Int, scala.Int> and must be called through their apply() method.

The upshot is that Scala works quite well for working with existing Java code.  But if you are designing a library for use by general Java developers and not just Scala developers, then writing in Scala means that you need to avoid certain language features to avoid making the API cumbersome.

There are, of course, plenty of other languages available for use on the JVM.  JRuby, Clojure, and Jaskell are a few I’ve heard of, but not looked into yet.  However, of the languages I looked at, Rhino and Jython seem better suited as user-facing scripting interfaces, while Scala is a nice alternative to Java for core implementation.  If you are designing a library for consumption by Java developers though, Java itself gives you the best control over the Java classes and method signatures.

For the record, we ended up choosing Django on CPython for the current round of development, with plans to investigate moving to Jython if and when consolidating the code onto the JVM becomes more necessary.

Meme in Scala

Sebastian over at Digifesto recently alerted me to a budding meme being pushed by Eric Florenzano.  The original proposal (from here) goes like this:

Rules:

1. Implement a program that takes in a user’s name and their age, and prints hello to them once for every year that they have been alive.
2. Post these rules, the source code for your solution, and the following list (with you included) on your blog.
3. Bonus points if you implement it in a language not yet seen on the following list!

The List:

1. [Python] http://www.eflorenzano.com/blog/post/trying-start-programming-meme
2. [Bash] http://aartemenko.com/texts/bash-meme/
3. [C] http://dakrauth.com/media/site/text/hello.c
4. [Java] http://adoleo.com/blog/2008/nov/25/programming-meme/
5. [Python 3] http://mikewatkins.ca/2008/11/25/hello-meme/
6. [Ruby] http://stroky.l.googlepages.com/gem
7. [Ruby] http://im.camronflanders.com/archive/meme/
8. [Lisp] http://justinlilly.com/blog/2008/nov/25/back-on-the-horse/
9. [Lua] http://aartemenko.com/texts/lua-hello-meme/
10. [Functional Python] http://aartemenko.com/texts/python-functional-hello-meme/
11. [Erlang] http://surfacedepth.blogspot.com/2008/11/erics-programming-meme-in-erlang.html
12. [Haskell] http://jasonwalsh.us/meme.html
13. [PHP] http://fitzgeraldsteele.wordpress.com/2008/11/25/memeing-in-php-2/
14. [Javascript] http://www.taylanpince.com/blog/posts/responding-to-a-programming-meme/
15. [Single-File Django] http://www.pocketuniverse.ca/archive/2008/november/27/florenzano-factor/

For my entry, I put together a script in Scala.

import scala.Console._

var name = readLine("What is your name? ")
var age  = readLine("How many years old are you? ").toInt
(1 to age).foreach { age => println("%2d) Hello, %s".format(age, name)) }

To run, copy the code into a text file named hello.scala and then run: scala hello.scala You can get the scala interpreter from the Scala website, or your distribution’s package manager.

speaking in tongues

I recently had a chat with a friend of mine about how dissatisfied I feel that I only speak one language (one and a half, counting the few semesters of Spanish I completed in high school).  The compassionate fellow he is, my friend tried to console me, pointing out that I’m at least lucky that the one language I do speak is English (agreed) and that, considering my wealth of programming experience, I am fluent in several languages.

This isn’t the first time I’ve heard someone talk about being fluent in a programming language.  I have never met a developer in person who would use that phrasing, though.  In most programming ‘languages’ it is perfectly possible to write code in any natural language you want.  More importantly, the difference between different programming languages tends to be more along the lines of how processes are structured (if I define a variable here will I be able to see it there? change its type? delete it later? what tools are provided to help me structure data and operations on it? how many lines does it take to write 99 bottles of beer?) than the enormous differences in vocabulary that you see going from, say, English to Italian.  The ‘vocabulary’ of code, the API’s available, are something that don’t need to be memorized for a developer to be proficient; in fact I would say that hardly any developers memorize even built-in APIs (those included with the language as standard libraries) to the extent that they would learn vocabulary in a second language.

2 + 2 means the same thing in Java, JavaScript, PHP, Python, Perl, and Haskell.

I have thought about this for a while and I don’t know of a better word for a programming language than “language.”  But sometimes I really wish there was one.

Outliers in Five Minutes

Recently I was given a copy of Outliers at work to look over as inspiration for how to be effective in the new team I’ll be working with.  Or rather, a copy was purchased for my office, which I promptly took home and didn’t return for a few weeks (it’s a rather short book, when I got around to reading it, I started after dinner and was finished before lunch the next day).  That in mind, I thought the neighborly thing to do would be to summarize it for the other members of my team so they could get up to speed.  I’m putting it online so they can link to it if they so desire.

The email (slightly modified for tone and typos):

Hey guys,

Since I took ‘Outliers’ by Malcolm Gladwell home the day it arrived at the office and prevented either of you from having a chance to look at it, I thought it might be useful to provide the 5-minute version in an email.  I got the impression that Whit wanted to use this book as a framework for jteam discussions, so my notes are in the form of a table with the examples likely to be pulled from the book and the concepts for which they’d be metaphors.

Point: Sometimes we don’t know everything about a subject.
Example: In the 1950’s, medical researchers found that the populace of Roseto, Pennsylvania was much more resistant to disease than the average and after ruling out diet, exercise, and genetic disposition concluded that this must be due to the friendly, relaxed atmosphere of town life.  This was a breakthrough in medical understanding.

Point: Hidden influences can affect success as much or more than aptitude.
Example: Hockey is big in Canada.  Since the junior league tryouts have eligibility requirements including age, athletes born in January have an 11-month head start on their competitors born in December.  This is reinforced by special attention given to star athletes, resulting in the Canadian pro leagues having a preponderance of athletes born in the first quarter of the year.

Point: Excellence, without exception, requires lots of practice.
Example: Studies show that a musician needs at least 10000 hours of cumulative practice to achieve mastery.  8000 hours are required to play professionally, and 4000 is only enough to make it as a public school music teacher.  The 10000 hour requirement for mastery appears to be consistent across disciplines.

Point: It doesn’t make much difference whether you are really smart or just smart enough for the job.
Example: “Termites” (a group of children selected for high IQ by a researcher named Terman) showed little deviation from the national average for success (as measured by adult income, academic awards, etc.).

Point: People skills are important.
Example: Christopher Langan has a freakishly high IQ, but couldn’t convince schools to adjust his class schedule so that he could make the 15-mile journey to campus daily despite not having a car.  Robert Oppenheimer (also quite intelligent) tried to kill his chemistry tutor in grad school, but convinced the review board to just put him on probation.  The difference? People skills.

Point: Experience in a field that’s seeing a surge in demand is advantageous.
Example: Near the turn of the 20th century, Louis Borgenicht had the bright idea to sell aprons in the streets of Brooklyn.  Because he was a Jew and had experience manufacturing and selling clothes, he was able to work his ass off and do quite well for himself.

Point: Sometimes established giants in a field hurt themselves by turning down work that is “beneath them”.
Example: Big-name law firms did not do corporate takeovers or litigation before the 1970’s.  Joe Flom made a killing because he had been forced to work for a smaller firm (that couldn’t afford to turn away that kind of work) and when corporate takeovers became more common he was in a good position to take all that work.

Point: Some people are at a distinct advantage just because of their circumstances growing up.
Example: Children of a smaller-than-average generation do better for themselves because there is more educational infrastructure in place per-capita than preceding generations, and more jobs available per capita when they get out of school.  Maurice Janklow did poorly exiting law school just before the Great Depression; his son Mort Janklow did much better.

Example: Bill Gates had access to a real-time computing system in 8th grade and continued to get access to them pretty much constantly up to the point that he started Microsoft.  This was a pretty unique situation at the time.

Example: Bill Joy was at the University of Michigan at just the right time in order to also get access to real-time computing systems when such systems were quite rare.

Point: People’s behavior is influenced by the culture in which they grow up.  Culture is influenced by the ethnic origins of an area’s population.
Example: Psychological studies show young men from the south get more pissed off than young men from the north when similar stimuli are applied.  This is attributed to the south having ethnic origins in the Scottish highlands while the culture of the north was established by Puritan immigrants.

Point: It’s bad when authority becomes a barrier to communication.
Example: Airplane crashes are often caused by situations where the copilot is too timid to confront the captain about dangerous conditions or poor decisions.  Korean culture’s strict social structure is highlighted as encouraging this effect.

Point: Cultural predispositions affect how diligent people are and how they approach problems.
Example: Rice paddies take a lot of work to produce good yields, so Asians are very hard working people.

Example: English speakers are bad at math because the language is awkward/irregular at expressing numbers.

Point: Cultural predispositions can be overcome.
Example: KIPP (Knowledge is Power Program) schools take poor kids (usually low achievers) and make them do well in school by having 12-hour school days and 4 hours of homework nightly.  (KIPP is a middle-school level program.)

There’s not really a moral to this story; I just wanted to get that on the web.

No Crystal Ball

The OpenGeo team recently created a new, more formal group for JavaScript developers (aka the ‘jteam’) Starting this week, I was supposed to be dividing my time 3:2 between GeoServer work and jteam tasks.

The manager is dealing with some personal obligations and that first week on the new schedule was pushed back a week.

Over the winter break a neat styling tool for GeoServer was announced that made use of a GeoServer extension I’ve been working on on and off for the past 9 months or so. Since then it’s been getting a fair bit of attention from the community since then, I figured I’d be putting a lot of work into polishing it up so we could make it an official extension (basically, put a link to it on the downloads page.)

I ended up fixing random bugs against GeoServer while another developer reviewed the module.

Of those bugs, this one sounded like it would be pretty straightforward to fix. Another sounded pretty tough.

The first took me two days to fix, the second one I resolved in an afternoon.

I am beginning to think that I am not very good at predicting the future.

Happy 2009

Hey, looks like another new year is upon us (I know I missed it by a few days, but give me a break as I’ve been on vacation for a couple of weeks and my brain is still kicking back into gear.) I don’t usually put too much stock in coming up with resolutions for the new year, but this time around I think I’ll make an exception.  My resolution: complain more, but only complain to the right people.

Recently at work I’ve noticed I’m developing a bad habit of, when I have a problem with the way things are being done, complaining to everyone except the person responsible, whether because I think it’s too minor an issue to debate or the culprit is not online/around when I run into trouble or I feel like decisions have been made over my head or whatever.  While out of the office the past couple of weeks I’ve been thinking that over, and I see two big problems with that approach:

• complaining about things to others fosters a predisposition for them to find flaws with their own work, and establishes a precedent that makes following suit seem more acceptable
• not complaining to those responsible means that things won’t get fixed.  Note here that ‘fixed’ might not mean changing what’s done, it could just be giving me that extra bit of perspective that helps me understand why things are being done that way.

These two things feel like a pretty lame combo for a team, so hopefully phasing them out will be a big win.

As long as I’m doing the resolution thing, I think I will also try to post more regularly on this blog.

the right audience

I just woke up from a… wow, 2 hour-long nap.  I wasn’t planning to take a nap; I was trying to read a book about JavaScript, a programming language which interests me only due to the fact that I might eventually have some use for it.  (Not that that’s a bad reason to learn a language; there’s only one language out there that I’ve learned for a reason other than that I thought I might need to get stuff done in it; and I even take a certain glee in the raw mindless-effort-reduction capacity of some languages, a la bash:

15:56 < bmmpxf> dwins: iwilling and I are still having trouble getting
 the data in the postgis.  please stand by.
15:57 < dwins> bmmpxf: what sort of trouble? should I lend a hand?
15:59 < bmmpxf> dwins: Just trying to avoid typing in the password lots of times
16:00 < dwins> bmmpxf: parens to the rescue
16:00 < dwins> (for file in *.shp; do shp2pgsql $file; done) | psql
16:01 < dwins> man I love punctuation

Still, JavaScript the language doesn’t interest me half so much as JavaScript the platform for web application development.)

So what happened?  Was I in the wrong frame of mind (is there some sort of reading flow I should get into?)  Is my brain too feeble to handle more than 37 pages of JavaScript’s splendor without needing to recharge?  Is a book a bad way for a programmer to become acquainted with a language?  Did I miss my daily dose of wake-up pills this morning?

I think it’s just that I picked a book that was written for non-developers (from the preface: “We’re geeky, so you don’t have to be!”) and so dumbed things down a bit much.  I didn’t immediately put it down because, hey, I want to be able to explain things to non-developers too!  But after spending forty pages with half-a-dozen sidebar notes saying ’sorry we included Hello World as an example in a programming text’ and ‘html is a thing you can write in any editor, but don’t use Word!’ I was a little overwhelmed.  Why not just skim, you ask?  This particular book has so few words per page that I found skimming pretty frustrating, it just didn’t work for me.

So, moral of the story: if you read something that’s written for a target audience that clearly doesn’t include you, you’re probably going to feel like you’re going against the flow.  Similarly, if you’re writing a thing (as the OpenGeo team is right now with a serious reworking of the GeoServer documentation) you’re throwing away a lot of your effort if you don’t have the right audience in mind.

Java(Script)?

Recently my manager at OpenGeo, Chris Holmes, asked me about working on some JavaScript projects.  So far at OpenGeo I’ve been using Java pretty exclusively (I’ve spent a bit of time patching up a couple of things in Python), so JavaScript would be a pretty serious change from my normal routine.  Java is a compiled, statically typed, strongly typed language with one runtime that dominates the market (or at least, where we can specify that one particular runtime is supported by the project); JavaScript is interpreted, dynamically typed, weakly typed, and basically has as many interpreters as there are browsers out there, all of which have their own deviations from the standard.

Initially, I told Chris I’d be up for a switch of language, but when I asked some of the guys who are already doing JavaScript they told me it’d be smart to stay away if I could, because it’s just such a pain to deal with cross-browser development in JavaScript.  I’m not really that concerned about it though; I mean, don’t all languages have their weak points?  I can’t just avoid everything that doesn’t inspire outright fanboyism from its users. (Though, to be fair, there’s plenty of kool-aid drinking in the Java world as well; I just don’t have any handy examples in the form of webcomics.)

Anyway, I don’t have any JavaScript projects lined up just yet (and I do have a fair bit of stuff to do in Java), so I guess I’ll just see how things go.  Thoughts from random folks on the interwebs welcome.

(Aside for non-techies: ‘typing’ is the model by which a programming language structures data.  A strongly typed language forces you to use each value in your code as a single type (ie, is this a number or a word or a record representing a country, etc.), while a weakly typed one will try to guess the types based on context (so you can, for example, add the characters “23″ to the number 8 and get 31.  A statically typed system does all this checking before the code ever runs (and usually refuses to run if any of the checks fail) while a dynamically typed system waits until a line is run to check (and will run just fine with code that makes no sense as long as you don’t actually get to that line of code).  Wikipedia probably explains it better.  Anyway, typing is a pretty big factor in a language’s ease-of-use, since it affects how much work the compiler/interpreter can do for you in terms of validating code or figuring things out from context.)

My Very Own Branch

I’m trying something new this week: working on a slew of new features for GeoServer in a branch of my own.  It’s kind of nice to have my own sandbox, although when I finish with the new stuff I’ll be making a pretty substantial patch to the main GeoServer branch, meaning it will probably require a vote at the weekly GeoServer meeting.  Still, I’ll be able to keep my stuff separated and versioned while waiting on the 1.7.0 release, and I think the new features will be pretty exciting at the end.

What am I working on? A few things are on my plate right now:

• Make human-attended configuration for regionating a performance tweak rather than a necessity for regionating to work at all
• Experiment with some alternative ways of expressing the tree (basically, fake the inclusion of all features by drawing the first few in vector form, and including a raster background with the rest)
• Allow users to set custom templates for the KML popups on a higher level than individual layers.

<plug type=”shameless”>

If you want to help me out, you can check out the work in progress and let me know if you see anything broken in Google Earth (aesthetic opinions welcome as well!)  Just visit http://publicus.opengeo.org/dwins_kml/mapPreview.do and click on any of the KML links, then browse around.  (You’ll need Google Earth, of course; you can grab the installer from http://earth.google.com/) There will probably be another update here with more info about checking the different alternatives once I get the different visualization modes working, so stay tuned.

</plug>