No Crystal Ball

The OpenGeo team recently created a new, more formal group for JavaScript developers (aka the ‘jteam’) Starting this week, I was supposed to be dividing my time 3:2 between GeoServer work and jteam tasks.

The manager is dealing with some personal obligations and that first week on the new schedule was pushed back a week.

Over the winter break a neat styling tool for GeoServer was announced that made use of a GeoServer extension I’ve been working on on and off for the past 9 months or so. Since then it’s been getting a fair bit of attention from the community since then, I figured I’d be putting a lot of work into polishing it up so we could make it an official extension (basically, put a link to it on the downloads page.)

I ended up fixing random bugs against GeoServer while another developer reviewed the module.

Of those bugs, this one sounded like it would be pretty straightforward to fix. Another sounded pretty tough.

The first took me two days to fix, the second one I resolved in an afternoon.

I am beginning to think that I am not very good at predicting the future.

Happy 2009

Hey, looks like another new year is upon us (I know I missed it by a few days, but give me a break as I’ve been on vacation for a couple of weeks and my brain is still kicking back into gear.) I don’t usually put too much stock in coming up with resolutions for the new year, but this time around I think I’ll make an exception.  My resolution: complain more, but only complain to the right people.

Recently at work I’ve noticed I’m developing a bad habit of, when I have a problem with the way things are being done, complaining to everyone except the person responsible, whether because I think it’s too minor an issue to debate or the culprit is not online/around when I run into trouble or I feel like decisions have been made over my head or whatever.  While out of the office the past couple of weeks I’ve been thinking that over, and I see two big problems with that approach:

• complaining about things to others fosters a predisposition for them to find flaws with their own work, and establishes a precedent that makes following suit seem more acceptable
• not complaining to those responsible means that things won’t get fixed.  Note here that ‘fixed’ might not mean changing what’s done, it could just be giving me that extra bit of perspective that helps me understand why things are being done that way.

These two things feel like a pretty lame combo for a team, so hopefully phasing them out will be a big win.

As long as I’m doing the resolution thing, I think I will also try to post more regularly on this blog.

the right audience

I just woke up from a… wow, 2 hour-long nap.  I wasn’t planning to take a nap; I was trying to read a book about JavaScript, a programming language which interests me only due to the fact that I might eventually have some use for it.  (Not that that’s a bad reason to learn a language; there’s only one language out there that I’ve learned for a reason other than that I thought I might need to get stuff done in it; and I even take a certain glee in the raw mindless-effort-reduction capacity of some languages, a la bash:

15:56 < bmmpxf> dwins: iwilling and I are still having trouble getting
 the data in the postgis.  please stand by.
15:57 < dwins> bmmpxf: what sort of trouble? should I lend a hand?
15:59 < bmmpxf> dwins: Just trying to avoid typing in the password lots of times
16:00 < dwins> bmmpxf: parens to the rescue
16:00 < dwins> (for file in *.shp; do shp2pgsql $file; done) | psql
16:01 < dwins> man I love punctuation

Still, JavaScript the language doesn’t interest me half so much as JavaScript the platform for web application development.)

So what happened?  Was I in the wrong frame of mind (is there some sort of reading flow I should get into?)  Is my brain too feeble to handle more than 37 pages of JavaScript’s splendor without needing to recharge?  Is a book a bad way for a programmer to become acquainted with a language?  Did I miss my daily dose of wake-up pills this morning?

I think it’s just that I picked a book that was written for non-developers (from the preface: “We’re geeky, so you don’t have to be!”) and so dumbed things down a bit much.  I didn’t immediately put it down because, hey, I want to be able to explain things to non-developers too!  But after spending forty pages with half-a-dozen sidebar notes saying ’sorry we included Hello World as an example in a programming text’ and ‘html is a thing you can write in any editor, but don’t use Word!’ I was a little overwhelmed.  Why not just skim, you ask?  This particular book has so few words per page that I found skimming pretty frustrating, it just didn’t work for me.

So, moral of the story: if you read something that’s written for a target audience that clearly doesn’t include you, you’re probably going to feel like you’re going against the flow.  Similarly, if you’re writing a thing (as the OpenGeo team is right now with a serious reworking of the GeoServer documentation) you’re throwing away a lot of your effort if you don’t have the right audience in mind.

Java(Script)?

Recently my manager at OpenGeo, Chris Holmes, asked me about working on some JavaScript projects.  So far at OpenGeo I’ve been using Java pretty exclusively (I’ve spent a bit of time patching up a couple of things in Python), so JavaScript would be a pretty serious change from my normal routine.  Java is a compiled, statically typed, strongly typed language with one runtime that dominates the market (or at least, where we can specify that one particular runtime is supported by the project); JavaScript is interpreted, dynamically typed, weakly typed, and basically has as many interpreters as there are browsers out there, all of which have their own deviations from the standard.

Initially, I told Chris I’d be up for a switch of language, but when I asked some of the guys who are already doing JavaScript they told me it’d be smart to stay away if I could, because it’s just such a pain to deal with cross-browser development in JavaScript.  I’m not really that concerned about it though; I mean, don’t all languages have their weak points?  I can’t just avoid everything that doesn’t inspire outright fanboyism from its users. (Though, to be fair, there’s plenty of kool-aid drinking in the Java world as well; I just don’t have any handy examples in the form of webcomics.)

Anyway, I don’t have any JavaScript projects lined up just yet (and I do have a fair bit of stuff to do in Java), so I guess I’ll just see how things go.  Thoughts from random folks on the interwebs welcome.

(Aside for non-techies: ‘typing’ is the model by which a programming language structures data.  A strongly typed language forces you to use each value in your code as a single type (ie, is this a number or a word or a record representing a country, etc.), while a weakly typed one will try to guess the types based on context (so you can, for example, add the characters “23″ to the number 8 and get 31.  A statically typed system does all this checking before the code ever runs (and usually refuses to run if any of the checks fail) while a dynamically typed system waits until a line is run to check (and will run just fine with code that makes no sense as long as you don’t actually get to that line of code).  Wikipedia probably explains it better.  Anyway, typing is a pretty big factor in a language’s ease-of-use, since it affects how much work the compiler/interpreter can do for you in terms of validating code or figuring things out from context.)

My Very Own Branch

I’m trying something new this week: working on a slew of new features for GeoServer in a branch of my own.  It’s kind of nice to have my own sandbox, although when I finish with the new stuff I’ll be making a pretty substantial patch to the main GeoServer branch, meaning it will probably require a vote at the weekly GeoServer meeting.  Still, I’ll be able to keep my stuff separated and versioned while waiting on the 1.7.0 release, and I think the new features will be pretty exciting at the end.

What am I working on? A few things are on my plate right now:

• Make human-attended configuration for regionating a performance tweak rather than a necessity for regionating to work at all
• Experiment with some alternative ways of expressing the tree (basically, fake the inclusion of all features by drawing the first few in vector form, and including a raster background with the rest)
• Allow users to set custom templates for the KML popups on a higher level than individual layers.

<plug type=”shameless”>

If you want to help me out, you can check out the work in progress and let me know if you see anything broken in Google Earth (aesthetic opinions welcome as well!)  Just visit http://publicus.opengeo.org/dwins_kml/mapPreview.do and click on any of the KML links, then browse around.  (You’ll need Google Earth, of course; you can grab the installer from http://earth.google.com/) There will probably be another update here with more info about checking the different alternatives once I get the different visualization modes working, so stay tuned.

</plug>

Code Review Review

Mel Chua writes about code review tools in a recent blog post, pondering whether a software code review tool could benefit OLPC (where she’s now employed, doing QA and, knowing her, a zillion other things).  I was about to just make a comment there, but I realized I have a fair bit to say, so full-fledged blog post from me it is.  The only potential benefit she directly mentions is that

there’s this constant loop of feedback and revision happening with code – imagine something going around and around in a positively improving cycle – and when it’s ripe and ready, someone with privs can pluck it from that cycle and make it a commit (as opposed to a linear “go forward… get stuck here… if it doesn’t work discard it out of the waterfall completely” system).

She also links to a video of a Google Tech Talk by Guido van Rossum (of Python fame) about Google’s code review process, along with a tool he wrote to make it easier. (The open-source version can be found here.)  There, he enumerates several benefits of code review:

1. You can catch bugs before they make it into revision control
2. Senior developers can impart knowledge to n00bs
3. Senior developers can verify that said n00bs can be trusted with commit privileges (this is a big one in the open-source world)
4. Team members can familiarize themselves with each other’s strengths and weaknesses
5. In general, you get the benefits of pair programming without the scheduling constraints.

A while back we installed a trial version of Atlassian’s Crucible code review tool at OpenGeo to use for GeoServer development.  I personally ended up using it for only two reviews: a (vast) speed improvement on the code that builds regionated hierarchies in GeoServer, and to review a patch submitted by Wayne Fang of Refractions Research.  Maybe this means that constant code review is not well suited to use by smallish groups (I interact with about 4 developers who actually work on GeoServer on a regular basis, and most of the time we work pretty independently on separate parts of the project.  We are geographically dispersed, so often Important Stuff happens while I am asleep.)  Anyway, I’d say that of the benefits Guido mentioned, only the first couple really came into play.  Interestingly enough, the KML regionating work ended up being a more-or-less complete rewrite, including a new algorithm, so familiarity with the code wasn’t that important in the end.  Much of the discussion around the patch from Wayne was related to style and design concerns (does that class name really signify what it’s actually doing?) rather than behavior, so may have actually been better served by the mailing list.

A couple of other random thoughts:

• Code review is a process.  Including more process means more training needed by people coming onto a project, though this can be mitigated by restricting the complicated bits to senior members, or, ideally, an automated system.  I don’t think it’s a foregone conclusion that adding code review to a project’s policy will make it a better project; it should be considered against the weight in developer ‘activation energy.’  For something like GeoServer, where development is fairly process-heavy anyway, this is pretty marginal (and I’d be interested in seeing where a more serious trial of code review could get us.)  For something with less dependencies, less API to get acquainted with, a quicker test suite, etc, it might be overly burdensome.
• Guido also mentioned that code review will happen whether you set aside time for it or not since any bugs that make it into the codebase will have to be fixed.  It may be a better strategy to just fix the bugs that are caught in QA, directing developer attention to the most egregious bugs first.  Catching bugs by inspection is not exactly bulletproof.  Of course, if you’re relying on code review as a means of maintaining some minimal level of API design quality, going back after the fact is a bit tougher.  But then, you can review API design without going through a line-by-line audit of the code too.
• As I mentioned earlier, I work on a pretty small team.  Patches to GeoServer from developers who don’t already have commit privileges are fairly rare.  I might appreciate code review tools more if I had more occasion to assess whether or not a particular change was up to par for the project I’m working on.

What’s up with OpenCore?

So I got into work a bit earlier than usual today and had a chat with a couple of other TOPP developers about stuff. It was kind of rambling, as many casual conversations are; we moved from a discussion of how all interns should give TOPP Talks before they go to the way that we often don’t know what interns are working on to how a recent intern didn’t get to actually work on her supposed project for much of the time she was at TOPP. And then, since the project she was working was a sort of project management tool for TOPP software projects. (At the time of that project’s inception I wrote a pretty high-level blog post about what might be involved in adapting Opencore to serve that purpose.) Nothing has come of that project at this point, and I never hear anything about it so I can only assume that the project’s dead. Anyway. From there we went to a discussion of the state of Opencore and it turned out to be pretty pithy, hence the blog post.

What is Opencore?

Opencore is the software that powers openplans.org. It pulls in different software components and unifies them into projects, which people can create and administer through the web. The intended user base is small non-profits and other organizations that need some basic web presence and organizational tools, but may have better things to do with their funds than hire a tech guru just to have him run a simple website. Right now the components that make up a project are

• a wiki
• a blog
• as many mailing lists as you want
• a task tracker
• and some admin tools to secure and control the various components.

All this sounds pretty decent. But oddly enough many of the projects on openplans.org are software projects, and quite a few (I can name eight or nine without looking) out of the 800 or so projects on the site are run by TOPP employees for TOPP projects. One would expect that, if we’re targeting organizations that would otherwise need to scrounge up technical talent, actual techies would prefer some place that doesn’t dumb things down quite so much. So…

What’s Wrong with Opencore?

It’s kind of weird to look at an OpenPlans project from the point of view of a project admin (which I do from time to time).

The Wiki

Opencore projects have this kind of anemic wiki; it basically is just a bunch of pages that happen to provide a convention for linking to other pages in the same blob. You don’t get a lot of the organization and management tools that a lot of other wikis provide, like change notification, categories (with auto-compiled content listings), special page types such as slideshows, page templates or really anything aside from a wysiwyg editor that is a little annoying to work with. (Maybe this has a lot to do with why so many of our projects deal with software projects; people who are already technically minded don’t mind switching to the source view, but everyone else has only the option of ditching openplans in favor of something that’s more pleasant to use.) A lot of these features are pretty important for managing more than trivial amounts of wiki pages (a good editor is important for managing any wiki pages at all).

The Mailing Lists

The mailing lists in OpenPlans are pretty workable, in that you can interact with them pretty exclusively through email and most of the time that there is something for you to do, you get an email that says ‘reply to this email and hit send to <do something>.’ You get essential capacities like restricting who can send mail to the list and being able to set multiple administrators to share the load of moderation. There’s even a web archive so you can browse the list even when you don’t have your mail client handy. There are a couple of pretty important things missing here though; there’s no facility for digest messages, and when you browse the archives of a list where even a few of the users include message bodies in their replies duplicates of messages dominate.

The Task Tracker

I don’t use the OpenPlans task tracker for any of the projects I’m involved with. It seems that this was a few of the developers’ intro project when they started working at TOPP and hasn’t seen a lot of polish since making it into Opencore. The UI for it suggests that it supports subtasks, priority listings, filtering by priority, assigning tasks to specific project members, deadlines, etc. seem to be in place. I don’t know whether notifications are in place, but they’re the only thing I don’t see mention of that would probably be generally useful for a shared task management tool. So why don’t any of the projects I’m on use this? Probably the mailing list is sufficient for all the non-technical projects, and the technical ones are more demanding of a task management tool so they use something like Trac.

The Admin Tools and Sitewide Stuff

The project management tools are decent, ish. The main problem I have with them is that you can’t message an openplans.org user for any purpose, ever. If you get an join request to your project and you want to find out more information about who the person is before accepting them, you had better hope their profile page has the information you’re looking for. You can’t message them, not even to explain why you are declining their join request. Even after someone joins your project, you can only message them through the project’s mailing list. While there are good reasons (spamming, privacy) behind this decision, I think there are really compelling reasons behind letting people message users and there should be some reasonable compromise to allow messaging in the important scenarios.  Another problem is that you can’t search all of openplans.  There is a global search that can be restricted to ‘people’ or ‘projects’ where it searches profiles or project wikis respectively, but mailing lists and blog posts apparently aren’t worth searching for.  (It looks like they aren’t exposed to Google either, so no easy workaround.)

The Blog

For blogs, Opencore uses WordPress. Since starting this blog at WordPress.com I have been consistently impressed with the quality of the tools provided and the amount of ‘connectedness’ that it provides with things like trackbacks and blogrolls. WordPress is great, and Opencore uses it pretty much uncustomized aside from switching out the rich text editor for the one used in the wiki. There are a few weirdnesses with the integration of WordPress into the site; WordPress provides a second search bar that searches just within the posts in a single blog and there are a few styling problems. In general, it’s a pretty nice experience.

What’s so Bad About That?

I mean, don’t all software projects have some bugs that need to be fixed? The short version is that the WordPress integration demonstrates that Opencore is quite capable of pulling in third-party party tools and making them part of the ‘openplans experience.’ But, it is the one and only piece of third-party software that is used this way. The wiki, the mailing lists, and the task tracker are all projects for which TOPP is a major contributor. We have 20 or so developers who’ve been working on this for two years or so, and rather than having a nice solution based on best-of-breed, open-source tools like WordPress, we have a bunch of things that we are stuck maintaining and which have a lot of ‘catching up’ to do before they are even with tools that are available to us at no cost. It would be different if we were working on something that worked substantially differently from any existing tool, but the fact that we don’t need any conjunctions to describe the components we have demonstrates that’s not the case (“wiki”, “mailing list”, “task tracker”, and “blog” are all common enough classes of tools to have generic names).

So… What Needs to Happen?

I don’t know.  Maybe nothing.  There is a bit more than just the technical factors at work here, and I can’t claim to be familiar with the social/political issues at play, and I’m sure that just throwing out all of the work that’s already been done to have a different, albeit more functional, piece of organization management software would cause a lot of pains both in retraining users (which we can’t do because we don’t have any control over users’ time) and migrating data (since our wiki stores pages in plain HTML and most others use a customized markup.)  Still, the current situtation leaves a lot to be desired.

That is one semantic wiki!

I talked a while back about this talk program I’ve been working on at TOPP (program as in scheduled events, not program as in software.)  So far I’ve been managing it using an OpenPlans project to organize things (one talk == one wiki page, plus a main page that lists all the talks ever given, grouped by whether they’ve been given yet or not (upcoming talks first, click the link above to see what I’m talking about).  So when someone wants to give a talk, there’s a pretty straightforward process:

1. The speaker tells me (s)he wants to give a talk.
2. I say “Great, put a page on the wiki so I don’t forget you told me that.”
3. The speaker puts a page on the wiki.
4. Later I talk to some other people who run things at TOPP and find a good time for a talk, so I go back and rearrange the main page on the wiki.
5. A couple of days before the talk, I send an email to the office saying there’s going to be one.  (The actual timing varies as I am kind of forgetful, and sometimes I spend too much time trying to badger people into giving me a summary to include in the email.)

This left me thinking ‘man, I wish the wiki knew more about the data we’re keeping there so it could automate some of this stuff.’  So when I saw Semantic Mediawiki I was pretty excited, since basically all it is is a tool for making a wiki more aware of what’s in it.  Today I finally got around to installing it and went from a basic LAMP setup to talk management application (with HTML, RSS, and iCal versions of the upcoming talks schedule) in about 2 hours.  It’s pretty cool (by which I mean really awesome in potential, but relies a bit on having lots of people following some not amazingly well-advertised rules.  Semantic Forms looks like it will help, but I haven’t played with that yet.)  As long as all talks have their own page with a [[scheduled for::some date]] tag on it someplace, they get pulled in automagically to the various feeds.  I was even able to embed a nice autogenerated table of upcoming talks in the wiki main page with a search like

{{#ask: [[scheduled for::>now]] | ?speaker | ?audience | ?scheduled for}}

where the stuff in brackets is the query and everything that starts with a ? is some attribute I’d like to include in the table.  Pretty sweet right?  Aside from smart page listings in wiki pages, SMW allows browsing by the different attributes you add.  So in the table when it says ‘audience: Non-techies’ I can follow the link to a page about non-techies, which links to a listing of all pages that are tagged as having that audience in mind, which seems to me like it’s way better than hand-maintaining ‘interesting’ lists of pages.

Anyway, it looks like there’s a lot of potential here, but a little bit of polish should go into it before it’s ready to replace the current ‘David polices the wiki’ approach.

Types In Stereo

There seems to be a lot of (sometimes conflicting) stereotypes about software developers.  They don’t like to talk to people, instead preferring to hide away someplace with no windows, lit only by computer displays.  They don’t care about their appearance and seldom shower.  When they do talk, computers are the only thing they know anything about.  They have atrocious senses of humor.  They like to create listings of how things are defined, especially themselves and the things they work with.

Personally, I try to shy away from this sort of thinking.  I mean, the logical extreme is to become that guy posting on Slashdot about having Asperger syndrome in a bragging tone, which is, well, not that attractive prospect.  (Note to the bored: I didn’t dig too hard trying to find some example comments, but I’m sure they’re not too hard to find.  http://ask.slashdot.org/article.pl?sid=03/06/20/1237229&mode=nested&tid=134 should be a good start.)

However, there is a grain of truth in a lot of these sentiments about the techies. I mean, I’m not too antisocial, and I hated working in a windowless room a couple of years ago when that was what I was doing, but I do kind of hate thinking about my appearance (I shower regularly though!!) Especially though, it’s pretty understandable that software guys talk a lot about software.  It’s not only what we work on, it’s what we work in, since for most software projects software is used to

• manage source code
• generate source code
• communicate about source code
• compile the source code to the finished product
• test the source code

and so on.

The thing is, a lot of the stuff that we work with requires a good deal of familiarity with a wide range of technical concepts.  It does take someone who really loves to work on code to be good at working on code; and it does take a special kind of person to love working on code.  Because honestly, tweaking a few lines of a text file, then hitting [Run] and hoping everything works so you can tweak some more lines that will probably break everything is a pretty painful experience.  It’s one of those things that I love to have done and don’t especially love doing.  So I (and I presume most software guys) end up spending a lot of time thinking about techniques, practices, tools, policies, and other ways of improving that experience.  It doesn’t always leave time for other stuff.

Disclaimer: The Open Planning Project is full of developers who write novels and raise children and cook experimental dishes in their spare time.  I’m not saying it can’t happen, just that it’s no big surprise when it doesn’t.

flow?

Recently I’ve been hearing a lot about the concept of ‘flow’ when working on software.  That is, the idea of there being some special mental state that a developer needs to be in to be at his/her most effective.  I’ve seen it referenced in software team management books, heard it mentioned around the office, read about it on the interwebs.  It’s supposed to be this delicate balance of lots of details in a developer’s head at once, where disrupting him in the middle of things sets him back to square one, wasting the dozens of minutes that it will take for him to resume ‘flow.’

I can’t really speak about anyone who’s not me, but I don’t feel like this is the way I work on things.  In general I take a lot longer than 20 minutes to get an idea of how things fit together (for example, I’ve been working on GeoServer for about 10 months now and I’m still picking things up) and a lot longer than 1 second to lose track of it all.  In fact, one way that I approach tough problems (ie, those that my initial ‘bang my head against it for a couple of hours’ approach fails to resolve) is to just go work on something else and come back a (subway ride, weekend, couple of hours later) and find that I’ve come up with a decent solution subconsciously in the mean time.

I think this concept of ‘flow’ feels like a good way to promote antisocialism in development teams, deemphasizing communication between developers to promote a team of cowboys working on code that requires a lot of headspace to work on.  It feels to me like it would be a lot better to keep communication frequent (keep the bus factor high, that kind of thing) and if it makes simple code a necessity in order to get stuff done, all the better.

So.  What can be done to avoid needing flow?  I find that it’s easier to slip in and out of development when I have a specific behavior I’m looking for, basically a test-driven development model.  Another thing that makes it easier to keep my place is a graphical debugger, which lowers the amount of headspace I have to devote to modelling the software behavior.  Finally, decent documentation (even if it has to be in the form of bugging another developer in an IRC channel) reduces that same headspace requirement by helping me to figure out expected or intended behavior more directly.

Maybe I’d think differently if I worked in an interpreted language; I have little built-in interruptions all day long when I restart GeoServer to try out changes I’ve made.  But I am definitely wary of anything that discourages communication between developers.