No Crystal Ball

The OpenGeo team recently created a new, more formal group for JavaScript developers (aka the ‘jteam’) Starting this week, I was supposed to be dividing my time 3:2 between GeoServer work and jteam tasks.

The manager is dealing with some personal obligations and that first week on the new schedule was pushed back a week.

Over the winter break a neat styling tool for GeoServer was announced that made use of a GeoServer extension I’ve been working on on and off for the past 9 months or so. Since then it’s been getting a fair bit of attention from the community since then, I figured I’d be putting a lot of work into polishing it up so we could make it an official extension (basically, put a link to it on the downloads page.)

I ended up fixing random bugs against GeoServer while another developer reviewed the module.

Of those bugs, this one sounded like it would be pretty straightforward to fix. Another sounded pretty tough.

The first took me two days to fix, the second one I resolved in an afternoon.

I am beginning to think that I am not very good at predicting the future.

Happy 2009

Hey, looks like another new year is upon us (I know I missed it by a few days, but give me a break as I’ve been on vacation for a couple of weeks and my brain is still kicking back into gear.) I don’t usually put too much stock in coming up with resolutions for the new year, but this time around I think I’ll make an exception.  My resolution: complain more, but only complain to the right people.

Recently at work I’ve noticed I’m developing a bad habit of, when I have a problem with the way things are being done, complaining to everyone except the person responsible, whether because I think it’s too minor an issue to debate or the culprit is not online/around when I run into trouble or I feel like decisions have been made over my head or whatever.  While out of the office the past couple of weeks I’ve been thinking that over, and I see two big problems with that approach:

• complaining about things to others fosters a predisposition for them to find flaws with their own work, and establishes a precedent that makes following suit seem more acceptable
• not complaining to those responsible means that things won’t get fixed.  Note here that ‘fixed’ might not mean changing what’s done, it could just be giving me that extra bit of perspective that helps me understand why things are being done that way.

These two things feel like a pretty lame combo for a team, so hopefully phasing them out will be a big win.

As long as I’m doing the resolution thing, I think I will also try to post more regularly on this blog.

the right audience

I just woke up from a… wow, 2 hour-long nap.  I wasn’t planning to take a nap; I was trying to read a book about JavaScript, a programming language which interests me only due to the fact that I might eventually have some use for it.  (Not that that’s a bad reason to learn a language; there’s only one language out there that I’ve learned for a reason other than that I thought I might need to get stuff done in it; and I even take a certain glee in the raw mindless-effort-reduction capacity of some languages, a la bash:

15:56 < bmmpxf> dwins: iwilling and I are still having trouble getting
 the data in the postgis.  please stand by.
15:57 < dwins> bmmpxf: what sort of trouble? should I lend a hand?
15:59 < bmmpxf> dwins: Just trying to avoid typing in the password lots of times
16:00 < dwins> bmmpxf: parens to the rescue
16:00 < dwins> (for file in *.shp; do shp2pgsql $file; done) | psql
16:01 < dwins> man I love punctuation

Still, JavaScript the language doesn’t interest me half so much as JavaScript the platform for web application development.)

So what happened?  Was I in the wrong frame of mind (is there some sort of reading flow I should get into?)  Is my brain too feeble to handle more than 37 pages of JavaScript’s splendor without needing to recharge?  Is a book a bad way for a programmer to become acquainted with a language?  Did I miss my daily dose of wake-up pills this morning?

I think it’s just that I picked a book that was written for non-developers (from the preface: “We’re geeky, so you don’t have to be!”) and so dumbed things down a bit much.  I didn’t immediately put it down because, hey, I want to be able to explain things to non-developers too!  But after spending forty pages with half-a-dozen sidebar notes saying ’sorry we included Hello World as an example in a programming text’ and ‘html is a thing you can write in any editor, but don’t use Word!’ I was a little overwhelmed.  Why not just skim, you ask?  This particular book has so few words per page that I found skimming pretty frustrating, it just didn’t work for me.

So, moral of the story: if you read something that’s written for a target audience that clearly doesn’t include you, you’re probably going to feel like you’re going against the flow.  Similarly, if you’re writing a thing (as the OpenGeo team is right now with a serious reworking of the GeoServer documentation) you’re throwing away a lot of your effort if you don’t have the right audience in mind.

Java(Script)?

Recently my manager at OpenGeo, Chris Holmes, asked me about working on some JavaScript projects.  So far at OpenGeo I’ve been using Java pretty exclusively (I’ve spent a bit of time patching up a couple of things in Python), so JavaScript would be a pretty serious change from my normal routine.  Java is a compiled, statically typed, strongly typed language with one runtime that dominates the market (or at least, where we can specify that one particular runtime is supported by the project); JavaScript is interpreted, dynamically typed, weakly typed, and basically has as many interpreters as there are browsers out there, all of which have their own deviations from the standard.

Initially, I told Chris I’d be up for a switch of language, but when I asked some of the guys who are already doing JavaScript they told me it’d be smart to stay away if I could, because it’s just such a pain to deal with cross-browser development in JavaScript.  I’m not really that concerned about it though; I mean, don’t all languages have their weak points?  I can’t just avoid everything that doesn’t inspire outright fanboyism from its users. (Though, to be fair, there’s plenty of kool-aid drinking in the Java world as well; I just don’t have any handy examples in the form of webcomics.)

Anyway, I don’t have any JavaScript projects lined up just yet (and I do have a fair bit of stuff to do in Java), so I guess I’ll just see how things go.  Thoughts from random folks on the interwebs welcome.

(Aside for non-techies: ‘typing’ is the model by which a programming language structures data.  A strongly typed language forces you to use each value in your code as a single type (ie, is this a number or a word or a record representing a country, etc.), while a weakly typed one will try to guess the types based on context (so you can, for example, add the characters “23″ to the number 8 and get 31.  A statically typed system does all this checking before the code ever runs (and usually refuses to run if any of the checks fail) while a dynamically typed system waits until a line is run to check (and will run just fine with code that makes no sense as long as you don’t actually get to that line of code).  Wikipedia probably explains it better.  Anyway, typing is a pretty big factor in a language’s ease-of-use, since it affects how much work the compiler/interpreter can do for you in terms of validating code or figuring things out from context.)

My Very Own Branch

I’m trying something new this week: working on a slew of new features for GeoServer in a branch of my own.  It’s kind of nice to have my own sandbox, although when I finish with the new stuff I’ll be making a pretty substantial patch to the main GeoServer branch, meaning it will probably require a vote at the weekly GeoServer meeting.  Still, I’ll be able to keep my stuff separated and versioned while waiting on the 1.7.0 release, and I think the new features will be pretty exciting at the end.

What am I working on? A few things are on my plate right now:

• Make human-attended configuration for regionating a performance tweak rather than a necessity for regionating to work at all
• Experiment with some alternative ways of expressing the tree (basically, fake the inclusion of all features by drawing the first few in vector form, and including a raster background with the rest)
• Allow users to set custom templates for the KML popups on a higher level than individual layers.

<plug type=”shameless”>

If you want to help me out, you can check out the work in progress and let me know if you see anything broken in Google Earth (aesthetic opinions welcome as well!)  Just visit http://publicus.opengeo.org/dwins_kml/mapPreview.do and click on any of the KML links, then browse around.  (You’ll need Google Earth, of course; you can grab the installer from http://earth.google.com/) There will probably be another update here with more info about checking the different alternatives once I get the different visualization modes working, so stay tuned.

</plug>

Code Review Review

Mel Chua writes about code review tools in a recent blog post, pondering whether a software code review tool could benefit OLPC (where she’s now employed, doing QA and, knowing her, a zillion other things).  I was about to just make a comment there, but I realized I have a fair bit to say, so full-fledged blog post from me it is.  The only potential benefit she directly mentions is that

there’s this constant loop of feedback and revision happening with code – imagine something going around and around in a positively improving cycle – and when it’s ripe and ready, someone with privs can pluck it from that cycle and make it a commit (as opposed to a linear “go forward… get stuck here… if it doesn’t work discard it out of the waterfall completely” system).

She also links to a video of a Google Tech Talk by Guido van Rossum (of Python fame) about Google’s code review process, along with a tool he wrote to make it easier. (The open-source version can be found here.)  There, he enumerates several benefits of code review:

1. You can catch bugs before they make it into revision control
2. Senior developers can impart knowledge to n00bs
3. Senior developers can verify that said n00bs can be trusted with commit privileges (this is a big one in the open-source world)
4. Team members can familiarize themselves with each other’s strengths and weaknesses
5. In general, you get the benefits of pair programming without the scheduling constraints.

A while back we installed a trial version of Atlassian’s Crucible code review tool at OpenGeo to use for GeoServer development.  I personally ended up using it for only two reviews: a (vast) speed improvement on the code that builds regionated hierarchies in GeoServer, and to review a patch submitted by Wayne Fang of Refractions Research.  Maybe this means that constant code review is not well suited to use by smallish groups (I interact with about 4 developers who actually work on GeoServer on a regular basis, and most of the time we work pretty independently on separate parts of the project.  We are geographically dispersed, so often Important Stuff happens while I am asleep.)  Anyway, I’d say that of the benefits Guido mentioned, only the first couple really came into play.  Interestingly enough, the KML regionating work ended up being a more-or-less complete rewrite, including a new algorithm, so familiarity with the code wasn’t that important in the end.  Much of the discussion around the patch from Wayne was related to style and design concerns (does that class name really signify what it’s actually doing?) rather than behavior, so may have actually been better served by the mailing list.

A couple of other random thoughts:

• Code review is a process.  Including more process means more training needed by people coming onto a project, though this can be mitigated by restricting the complicated bits to senior members, or, ideally, an automated system.  I don’t think it’s a foregone conclusion that adding code review to a project’s policy will make it a better project; it should be considered against the weight in developer ‘activation energy.’  For something like GeoServer, where development is fairly process-heavy anyway, this is pretty marginal (and I’d be interested in seeing where a more serious trial of code review could get us.)  For something with less dependencies, less API to get acquainted with, a quicker test suite, etc, it might be overly burdensome.
• Guido also mentioned that code review will happen whether you set aside time for it or not since any bugs that make it into the codebase will have to be fixed.  It may be a better strategy to just fix the bugs that are caught in QA, directing developer attention to the most egregious bugs first.  Catching bugs by inspection is not exactly bulletproof.  Of course, if you’re relying on code review as a means of maintaining some minimal level of API design quality, going back after the fact is a bit tougher.  But then, you can review API design without going through a line-by-line audit of the code too.
• As I mentioned earlier, I work on a pretty small team.  Patches to GeoServer from developers who don’t already have commit privileges are fairly rare.  I might appreciate code review tools more if I had more occasion to assess whether or not a particular change was up to par for the project I’m working on.

Types In Stereo

There seems to be a lot of (sometimes conflicting) stereotypes about software developers.  They don’t like to talk to people, instead preferring to hide away someplace with no windows, lit only by computer displays.  They don’t care about their appearance and seldom shower.  When they do talk, computers are the only thing they know anything about.  They have atrocious senses of humor.  They like to create listings of how things are defined, especially themselves and the things they work with.

Personally, I try to shy away from this sort of thinking.  I mean, the logical extreme is to become that guy posting on Slashdot about having Asperger syndrome in a bragging tone, which is, well, not that attractive prospect.  (Note to the bored: I didn’t dig too hard trying to find some example comments, but I’m sure they’re not too hard to find.  http://ask.slashdot.org/article.pl?sid=03/06/20/1237229&mode=nested&tid=134 should be a good start.)

However, there is a grain of truth in a lot of these sentiments about the techies. I mean, I’m not too antisocial, and I hated working in a windowless room a couple of years ago when that was what I was doing, but I do kind of hate thinking about my appearance (I shower regularly though!!) Especially though, it’s pretty understandable that software guys talk a lot about software.  It’s not only what we work on, it’s what we work in, since for most software projects software is used to

• manage source code
• generate source code
• communicate about source code
• compile the source code to the finished product
• test the source code

and so on.

The thing is, a lot of the stuff that we work with requires a good deal of familiarity with a wide range of technical concepts.  It does take someone who really loves to work on code to be good at working on code; and it does take a special kind of person to love working on code.  Because honestly, tweaking a few lines of a text file, then hitting [Run] and hoping everything works so you can tweak some more lines that will probably break everything is a pretty painful experience.  It’s one of those things that I love to have done and don’t especially love doing.  So I (and I presume most software guys) end up spending a lot of time thinking about techniques, practices, tools, policies, and other ways of improving that experience.  It doesn’t always leave time for other stuff.

Disclaimer: The Open Planning Project is full of developers who write novels and raise children and cook experimental dishes in their spare time.  I’m not saying it can’t happen, just that it’s no big surprise when it doesn’t.

flow?

Recently I’ve been hearing a lot about the concept of ‘flow’ when working on software.  That is, the idea of there being some special mental state that a developer needs to be in to be at his/her most effective.  I’ve seen it referenced in software team management books, heard it mentioned around the office, read about it on the interwebs.  It’s supposed to be this delicate balance of lots of details in a developer’s head at once, where disrupting him in the middle of things sets him back to square one, wasting the dozens of minutes that it will take for him to resume ‘flow.’

I can’t really speak about anyone who’s not me, but I don’t feel like this is the way I work on things.  In general I take a lot longer than 20 minutes to get an idea of how things fit together (for example, I’ve been working on GeoServer for about 10 months now and I’m still picking things up) and a lot longer than 1 second to lose track of it all.  In fact, one way that I approach tough problems (ie, those that my initial ‘bang my head against it for a couple of hours’ approach fails to resolve) is to just go work on something else and come back a (subway ride, weekend, couple of hours later) and find that I’ve come up with a decent solution subconsciously in the mean time.

I think this concept of ‘flow’ feels like a good way to promote antisocialism in development teams, deemphasizing communication between developers to promote a team of cowboys working on code that requires a lot of headspace to work on.  It feels to me like it would be a lot better to keep communication frequent (keep the bus factor high, that kind of thing) and if it makes simple code a necessity in order to get stuff done, all the better.

So.  What can be done to avoid needing flow?  I find that it’s easier to slip in and out of development when I have a specific behavior I’m looking for, basically a test-driven development model.  Another thing that makes it easier to keep my place is a graphical debugger, which lowers the amount of headspace I have to devote to modelling the software behavior.  Finally, decent documentation (even if it has to be in the form of bugging another developer in an IRC channel) reduces that same headspace requirement by helping me to figure out expected or intended behavior more directly.

Maybe I’d think differently if I worked in an interpreted language; I have little built-in interruptions all day long when I restart GeoServer to try out changes I’ve made.  But I am definitely wary of anything that discourages communication between developers.

I learned stuff today!

So, software is funny. It’s this huge complex thing where tons of abstractions are required to get anything useful done in a reasonable amount of time. (Think you’re dealing with an image? It’s really a grid of colors. Which is really a one-dimensional list of colors with a known width. Which are really just triples of numbers interpreted as red green and blue brightnesses. Which are really just sequences of binary values…) Abstractions are nice though since they keep you from making assumptions and keep things flexible. For example, GeoTools uses an abstraction called a DataStore to keep client code from caring about exactly how geospatial data is stored. Maybe it’s in a database like postgis. Maybe it’s a shapefile. Maybe you’re grabbing the data directly from the web using a WMS server someplace. It doesn’t matter, you just set up a DataStore and make a query and you get some geographic features.

This is really neat, because if someone wants to use a new type of storage, they can just write a new DataStore and all of a sudden everyone using GeoTools can use it. On the other hand, a lot of operations that you might want to run against a dataset still need to be written for each DataStore. So, each thing that a GeoTools Query knows about, a DataStore needs to know about as well. (If you’re wondering why we can’t just have these operations implemented “above” the DataStore in a way that uses that very same abstraction to avoid all this repeated code, see [1].) Today, one of the GeoServer developers wrote up a proposal to add a new feature to the Query object. To my dismay, I found a note to the effect that, since it might not be straightforward to implement this new feature, we should also add a “Capabilities” object to the DataStore, a means for DataStores to advertise which Query features they fail to implement.

The reason such a “cop-out” feature bugs me is that generally, if someone wants to query the data in a certain way, they will need the data that way regardless of whether the DataStore knows how to do it or not. So, you end up with either some code like this:

FeatureSource source = someDataStore.getFeatureSource();
FeatureCollection collection = source.getFeatures(Query.ALL);
Collections.sort(collection, new CustomSorter(mySortCriteria));

(ie, barely even using GeoTools and using a generic sort function instead of the possibly better-optimized features of the underlying database). Or, you might do something like this:

FeatureCollection collection;
FeatureSource source = someDataStore.getFeatureSource();
if (source.getQueryCapabilities().isSortingSupported()){
Query query = new Query(Query.ALL);
query.setSortBy(mySortCriteria);
    collection = source.getFeatures(query);
} else {
    collection = source.getFeatures(Query.ALL);
    Collections.sort(collection, new CustomSorter(mySortCriteria);
}

(that is, check whether GeoTools can sort, and do it yourself if you have to.)  I may be demonstrating a creative failing on my part here, but I can’t think of any situation where a developer would say “I need this sorted… but only if you can do it for me.”  So why provide the lame sorting in datastores directly?  GeoTools already provides an AbstractDatastore which allows general-purpose code to be shared among the DataStores.  So then it would look like this:

FeatureSource source = someDataStore.getFeatureSource();
Query query = new Query(Query.all);
query.setSortBy(mySortCriteria);
collection = source.getFeatures(query);

Sure, it’s an extra line, but now it’s fast when it needs to be.  Plus any bug fixes to the fallback sorting code get pushed out to all the other users of GeoTools since that’s where they live.  So, if this works out so well, why isn’t it an option?

As it turns out, GeoTools has not, thus far, been using this approach for the Query operations already supported.  (For example, there is only one DataStore implementation that actually supports sorting.)  So when we went to add a feature that could build off of other operations (in this case, the proposed paging operation is much more useful if we are imposing an order on the data by sorting it) we couldn’t simply provide a default implementation of the new operation simply because it doesn’t have the option of building on the older ones.  So, doing things ‘right’ in this case would involve updating all of the considerable number of DataStores in GeoTools to use a default sorting function, before even thinking about the paging feature.  Just because our software is Free doesn’t mean our time is worthless, so this will have to wait until we have some time to work on code cleanup as opposed to paying work.

So, the thing I learned today is that sometimes lax design today leads to bad design tomorrow.  That you can’t get rid of.  Because everyone’s dealing with the old way.

background

I work on open source software.  This means that I work with lots of other people who work on open source software.  A neat thing about open-source software is that it basically requires clean code, or at least attention to that sort of thing.  After all, labelling your software as open-source when nobody but you can untangle the spaghetti code enough to get anything done is roughly analogous to giving away free soda while charging for the cups.  So, people that are ‘good’ at the whole open-source thing tend to care about design.

In particular, I work on GeoServer, a web mapping server written in Java.  GeoServer relies on GeoTools to provide database abstractions and other great stuff to help with the geospatial operations, so GeoServer can focus on enforcing security restrictions and providing a decent configuration system and generally bridging the gap between the Web and GeoTools’s Java API.  This works out great, since GeoTools can then provide similar functionality to other geospatial applications such as uDig.  In the typical open-source way of things, this means that all three projects benefit since the more users an open-source project has, the more developers it will have. (generally speaking).

So, inasmuch as I am defined by what I do, that’s who I am.