Dwins’s Weblog

Smelter – A tool for JavaScript framework development

Posted in Ideas,Open Source Software by dwins on February 28, 2010
Tags: , ,

For much of the JavaScript work we do at OpenGeo, we use jsbuild (from jstools) to concatenate and minify JavaScript sources in the fairly sizable JavaScript libraries we use.  A useful feature that jstools provides is inspecting dependency annotations in the JavaScript to make sure all the scripts are concatenated in the correct order.  This is nice and automatic and everything, but minified JavaScript is not much fun to look at in Firebug.  So, we end up writing “loader” scripts to allow us to use the “raw” sources directly while testing.  The loader scripts are pretty simple; they just add some <script> tags while the page is loading to reference the non-minified sources directly.  A problem with this approach, however, is that the loader script then must enumerate, in order, the raw sources.  Much of the benefit of jsbuild’s dependency analysis is lost if we must also manually maintain a one-dimensional listing of the scripts! For example, look at GeoExt’s loader script in comparison with its build configuration.

In order to address this, I’ve been working on a JS build tool of my own, called Smelter.  Smelter reads jstools build files and dependency information and concatenates and minifies similarly, but it also provides an embedded web server that creates loader scripts based on that same dependencie information.  It also allows switching between minified and loader scripts without modifying the HTML pages that reference them.  Aside from saving developers a bit of duplicated work, this also ensures that the non-minified scripts load in the same order they are included in the minified build, for fewer surprises when switching over to the minified build.

I’ll definitely be dog-fooding this for the JavaScript used in GeoNode, and I’d love feedback from anyone else using it.


Happy 2009

Posted in Development,Ideas by dwins on January 5, 2009

Hey, looks like another new year is upon us (I know I missed it by a few days, but give me a break as I’ve been on vacation for a couple of weeks and my brain is still kicking back into gear.) I don’t usually put too much stock in coming up with resolutions for the new year, but this time around I think I’ll make an exception.  My resolution: complain more, but only complain to the right people.

Recently at work I’ve noticed I’m developing a bad habit of, when I have a problem with the way things are being done, complaining to everyone except the person responsible, whether because I think it’s too minor an issue to debate or the culprit is not online/around when I run into trouble or I feel like decisions have been made over my head or whatever.  While out of the office the past couple of weeks I’ve been thinking that over, and I see two big problems with that approach:

  • complaining about things to others fosters a predisposition for them to find flaws with their own work, and establishes a precedent that makes following suit seem more acceptable
  • not complaining to those responsible means that things won’t get fixed.  Note here that ‘fixed’ might not mean changing what’s done, it could just be giving me that extra bit of perspective that helps me understand why things are being done that way.

These two things feel like a pretty lame combo for a team, so hopefully phasing them out will be a big win.

As long as I’m doing the resolution thing, I think I will also try to post more regularly on this blog.

the right audience

Posted in Development,Ideas by dwins on November 13, 2008

I just woke up from a… wow, 2 hour-long nap.  I wasn’t planning to take a nap; I was trying to read a book about JavaScript, a programming language which interests me only due to the fact that I might eventually have some use for it.  (Not that that’s a bad reason to learn a language; there’s only one language out there that I’ve learned for a reason other than that I thought I might need to get stuff done in it; and I even take a certain glee in the raw mindless-effort-reduction capacity of some languages, a la bash:

15:56 < bmmpxf> dwins: iwilling and I are still having trouble getting
 the data in the postgis.  please stand by.
15:57 < dwins> bmmpxf: what sort of trouble? should I lend a hand?
15:59 < bmmpxf> dwins: Just trying to avoid typing in the password lots of times
16:00 < dwins> bmmpxf: parens to the rescue
16:00 < dwins> (for file in *.shp; do shp2pgsql $file; done) | psql
16:01 < dwins> man I love punctuation

Still, JavaScript the language doesn’t interest me half so much as JavaScript the platform for web application development.)

So what happened?  Was I in the wrong frame of mind (is there some sort of reading flow I should get into?)  Is my brain too feeble to handle more than 37 pages of JavaScript’s splendor without needing to recharge?  Is a book a bad way for a programmer to become acquainted with a language?  Did I miss my daily dose of wake-up pills this morning?

I think it’s just that I picked a book that was written for non-developers (from the preface: “We’re geeky, so you don’t have to be!”) and so dumbed things down a bit much.  I didn’t immediately put it down because, hey, I want to be able to explain things to non-developers too!  But after spending forty pages with half-a-dozen sidebar notes saying ‘sorry we included Hello World as an example in a programming text’ and ‘html is a thing you can write in any editor, but don’t use Word!’ I was a little overwhelmed.  Why not just skim, you ask?  This particular book has so few words per page that I found skimming pretty frustrating, it just didn’t work for me.

So, moral of the story: if you read something that’s written for a target audience that clearly doesn’t include you, you’re probably going to feel like you’re going against the flow.  Similarly, if you’re writing a thing (as the OpenGeo team is right now with a serious reworking of the GeoServer documentation) you’re throwing away a lot of your effort if you don’t have the right audience in mind.

Code Review Review

Posted in Development,Ideas by dwins on October 16, 2008

Mel Chua writes about code review tools in a recent blog post, pondering whether a software code review tool could benefit OLPC (where she’s now employed, doing QA and, knowing her, a zillion other things).  I was about to just make a comment there, but I realized I have a fair bit to say, so full-fledged blog post from me it is.  The only potential benefit she directly mentions is that

there’s this constant loop of feedback and revision happening with code – imagine something going around and around in a positively improving cycle – and when it’s ripe and ready, someone with privs can pluck it from that cycle and make it a commit (as opposed to a linear “go forward… get stuck here… if it doesn’t work discard it out of the waterfall completely” system).

She also links to a video of a Google Tech Talk by Guido van Rossum (of Python fame) about Google’s code review process, along with a tool he wrote to make it easier. (The open-source version can be found here.)  There, he enumerates several benefits of code review:

  1. You can catch bugs before they make it into revision control
  2. Senior developers can impart knowledge to n00bs
  3. Senior developers can verify that said n00bs can be trusted with commit privileges (this is a big one in the open-source world)
  4. Team members can familiarize themselves with each other’s strengths and weaknesses
  5. In general, you get the benefits of pair programming without the scheduling constraints.

A while back we installed a trial version of Atlassian’s Crucible code review tool at OpenGeo to use for GeoServer development.  I personally ended up using it for only two reviews: a (vast) speed improvement on the code that builds regionated hierarchies in GeoServer, and to review a patch submitted by Wayne Fang of Refractions Research.  Maybe this means that constant code review is not well suited to use by smallish groups (I interact with about 4 developers who actually work on GeoServer on a regular basis, and most of the time we work pretty independently on separate parts of the project.  We are geographically dispersed, so often Important Stuff happens while I am asleep.)  Anyway, I’d say that of the benefits Guido mentioned, only the first couple really came into play.  Interestingly enough, the KML regionating work ended up being a more-or-less complete rewrite, including a new algorithm, so familiarity with the code wasn’t that important in the end.  Much of the discussion around the patch from Wayne was related to style and design concerns (does that class name really signify what it’s actually doing?) rather than behavior, so may have actually been better served by the mailing list.

A couple of other random thoughts:

  • Code review is a process.  Including more process means more training needed by people coming onto a project, though this can be mitigated by restricting the complicated bits to senior members, or, ideally, an automated system.  I don’t think it’s a foregone conclusion that adding code review to a project’s policy will make it a better project; it should be considered against the weight in developer ‘activation energy.’  For something like GeoServer, where development is fairly process-heavy anyway, this is pretty marginal (and I’d be interested in seeing where a more serious trial of code review could get us.)  For something with less dependencies, less API to get acquainted with, a quicker test suite, etc, it might be overly burdensome.
  • Guido also mentioned that code review will happen whether you set aside time for it or not since any bugs that make it into the codebase will have to be fixed.  It may be a better strategy to just fix the bugs that are caught in QA, directing developer attention to the most egregious bugs first.  Catching bugs by inspection is not exactly bulletproof.  Of course, if you’re relying on code review as a means of maintaining some minimal level of API design quality, going back after the fact is a bit tougher.  But then, you can review API design without going through a line-by-line audit of the code too.
  • As I mentioned earlier, I work on a pretty small team.  Patches to GeoServer from developers who don’t already have commit privileges are fairly rare.  I might appreciate code review tools more if I had more occasion to assess whether or not a particular change was up to par for the project I’m working on.

What’s up with OpenCore?

Posted in Ideas,Open Source Software by dwins on August 6, 2008

So I got into work a bit earlier than usual today and had a chat with a couple of other TOPP developers about stuff. It was kind of rambling, as many casual conversations are; we moved from a discussion of how all interns should give TOPP Talks before they go to the way that we often don’t know what interns are working on to how a recent intern didn’t get to actually work on her supposed project for much of the time she was at TOPP. And then, since the project she was working was a sort of project management tool for TOPP software projects. (At the time of that project’s inception I wrote a pretty high-level blog post about what might be involved in adapting Opencore to serve that purpose.) Nothing has come of that project at this point, and I never hear anything about it so I can only assume that the project’s dead. Anyway. From there we went to a discussion of the state of Opencore and it turned out to be pretty pithy, hence the blog post.

What is Opencore?

Opencore is the software that powers openplans.org. It pulls in different software components and unifies them into projects, which people can create and administer through the web. The intended user base is small non-profits and other organizations that need some basic web presence and organizational tools, but may have better things to do with their funds than hire a tech guru just to have him run a simple website. Right now the components that make up a project are

  • a wiki
  • a blog
  • as many mailing lists as you want
  • a task tracker
  • and some admin tools to secure and control the various components.

All this sounds pretty decent. But oddly enough many of the projects on openplans.org are software projects, and quite a few (I can name eight or nine without looking) out of the 800 or so projects on the site are run by TOPP employees for TOPP projects. One would expect that, if we’re targeting organizations that would otherwise need to scrounge up technical talent, actual techies would prefer some place that doesn’t dumb things down quite so much. So…

What’s Wrong with Opencore?

It’s kind of weird to look at an OpenPlans project from the point of view of a project admin (which I do from time to time).

The Wiki

Opencore projects have this kind of anemic wiki; it basically is just a bunch of pages that happen to provide a convention for linking to other pages in the same blob. You don’t get a lot of the organization and management tools that a lot of other wikis provide, like change notification, categories (with auto-compiled content listings), special page types such as slideshows, page templates or really anything aside from a wysiwyg editor that is a little annoying to work with. (Maybe this has a lot to do with why so many of our projects deal with software projects; people who are already technically minded don’t mind switching to the source view, but everyone else has only the option of ditching openplans in favor of something that’s more pleasant to use.) A lot of these features are pretty important for managing more than trivial amounts of wiki pages (a good editor is important for managing any wiki pages at all).

The Mailing Lists

The mailing lists in OpenPlans are pretty workable, in that you can interact with them pretty exclusively through email and most of the time that there is something for you to do, you get an email that says ‘reply to this email and hit send to <do something>.’ You get essential capacities like restricting who can send mail to the list and being able to set multiple administrators to share the load of moderation. There’s even a web archive so you can browse the list even when you don’t have your mail client handy. There are a couple of pretty important things missing here though; there’s no facility for digest messages, and when you browse the archives of a list where even a few of the users include message bodies in their replies duplicates of messages dominate.

The Task Tracker

I don’t use the OpenPlans task tracker for any of the projects I’m involved with. It seems that this was a few of the developers’ intro project when they started working at TOPP and hasn’t seen a lot of polish since making it into Opencore. The UI for it suggests that it supports subtasks, priority listings, filtering by priority, assigning tasks to specific project members, deadlines, etc. seem to be in place. I don’t know whether notifications are in place, but they’re the only thing I don’t see mention of that would probably be generally useful for a shared task management tool. So why don’t any of the projects I’m on use this? Probably the mailing list is sufficient for all the non-technical projects, and the technical ones are more demanding of a task management tool so they use something like Trac.

The Admin Tools and Sitewide Stuff

The project management tools are decent, ish. The main problem I have with them is that you can’t message an openplans.org user for any purpose, ever. If you get an join request to your project and you want to find out more information about who the person is before accepting them, you had better hope their profile page has the information you’re looking for. You can’t message them, not even to explain why you are declining their join request. Even after someone joins your project, you can only message them through the project’s mailing list. While there are good reasons (spamming, privacy) behind this decision, I think there are really compelling reasons behind letting people message users and there should be some reasonable compromise to allow messaging in the important scenarios.  Another problem is that you can’t search all of openplans.  There is a global search that can be restricted to ‘people’ or ‘projects’ where it searches profiles or project wikis respectively, but mailing lists and blog posts apparently aren’t worth searching for.  (It looks like they aren’t exposed to Google either, so no easy workaround.)

The Blog

For blogs, Opencore uses WordPress. Since starting this blog at WordPress.com I have been consistently impressed with the quality of the tools provided and the amount of ‘connectedness’ that it provides with things like trackbacks and blogrolls. WordPress is great, and Opencore uses it pretty much uncustomized aside from switching out the rich text editor for the one used in the wiki. There are a few weirdnesses with the integration of WordPress into the site; WordPress provides a second search bar that searches just within the posts in a single blog and there are a few styling problems. In general, it’s a pretty nice experience.

What’s so Bad About That?

I mean, don’t all software projects have some bugs that need to be fixed? The short version is that the WordPress integration demonstrates that Opencore is quite capable of pulling in third-party party tools and making them part of the ‘openplans experience.’ But, it is the one and only piece of third-party software that is used this way. The wiki, the mailing lists, and the task tracker are all projects for which TOPP is a major contributor. We have 20 or so developers who’ve been working on this for two years or so, and rather than having a nice solution based on best-of-breed, open-source tools like WordPress, we have a bunch of things that we are stuck maintaining and which have a lot of ‘catching up’ to do before they are even with tools that are available to us at no cost. It would be different if we were working on something that worked substantially differently from any existing tool, but the fact that we don’t need any conjunctions to describe the components we have demonstrates that’s not the case (“wiki”, “mailing list”, “task tracker”, and “blog” are all common enough classes of tools to have generic names).

So… What Needs to Happen?

I don’t know.  Maybe nothing.  There is a bit more than just the technical factors at work here, and I can’t claim to be familiar with the social/political issues at play, and I’m sure that just throwing out all of the work that’s already been done to have a different, albeit more functional, piece of organization management software would cause a lot of pains both in retraining users (which we can’t do because we don’t have any control over users’ time) and migrating data (since our wiki stores pages in plain HTML and most others use a customized markup.)  Still, the current situtation leaves a lot to be desired.

Types In Stereo

Posted in Development,Ideas by dwins on July 27, 2008

There seems to be a lot of (sometimes conflicting) stereotypes about software developers.  They don’t like to talk to people, instead preferring to hide away someplace with no windows, lit only by computer displays.  They don’t care about their appearance and seldom shower.  When they do talk, computers are the only thing they know anything about.  They have atrocious senses of humor.  They like to create listings of how things are defined, especially themselves and the things they work with.

Personally, I try to shy away from this sort of thinking.  I mean, the logical extreme is to become that guy posting on Slashdot about having Asperger syndrome in a bragging tone, which is, well, not that attractive prospect.  (Note to the bored: I didn’t dig too hard trying to find some example comments, but I’m sure they’re not too hard to find.  http://ask.slashdot.org/article.pl?sid=03/06/20/1237229&mode=nested&tid=134 should be a good start.)

However, there is a grain of truth in a lot of these sentiments about the techies. I mean, I’m not too antisocial, and I hated working in a windowless room a couple of years ago when that was what I was doing, but I do kind of hate thinking about my appearance (I shower regularly though!!) Especially though, it’s pretty understandable that software guys talk a lot about software.  It’s not only what we work on, it’s what we work in, since for most software projects software is used to

  • manage source code
  • generate source code
  • communicate about source code
  • compile the source code to the finished product
  • test the source code

and so on.

The thing is, a lot of the stuff that we work with requires a good deal of familiarity with a wide range of technical concepts.  It does take someone who really loves to work on code to be good at working on code; and it does take a special kind of person to love working on code.  Because honestly, tweaking a few lines of a text file, then hitting [Run] and hoping everything works so you can tweak some more lines that will probably break everything is a pretty painful experience.  It’s one of those things that I love to have done and don’t especially love doing.  So I (and I presume most software guys) end up spending a lot of time thinking about techniques, practices, tools, policies, and other ways of improving that experience.  It doesn’t always leave time for other stuff.

Disclaimer: The Open Planning Project is full of developers who write novels and raise children and cook experimental dishes in their spare time.  I’m not saying it can’t happen, just that it’s no big surprise when it doesn’t.


Posted in Development,Ideas by dwins on July 22, 2008

Recently I’ve been hearing a lot about the concept of ‘flow’ when working on software.  That is, the idea of there being some special mental state that a developer needs to be in to be at his/her most effective.  I’ve seen it referenced in software team management books, heard it mentioned around the office, read about it on the interwebs.  It’s supposed to be this delicate balance of lots of details in a developer’s head at once, where disrupting him in the middle of things sets him back to square one, wasting the dozens of minutes that it will take for him to resume ‘flow.’

I can’t really speak about anyone who’s not me, but I don’t feel like this is the way I work on things.  In general I take a lot longer than 20 minutes to get an idea of how things fit together (for example, I’ve been working on GeoServer for about 10 months now and I’m still picking things up) and a lot longer than 1 second to lose track of it all.  In fact, one way that I approach tough problems (ie, those that my initial ‘bang my head against it for a couple of hours’ approach fails to resolve) is to just go work on something else and come back a (subway ride, weekend, couple of hours later) and find that I’ve come up with a decent solution subconsciously in the mean time.

I think this concept of ‘flow’ feels like a good way to promote antisocialism in development teams, deemphasizing communication between developers to promote a team of cowboys working on code that requires a lot of headspace to work on.  It feels to me like it would be a lot better to keep communication frequent (keep the bus factor high, that kind of thing) and if it makes simple code a necessity in order to get stuff done, all the better.

So.  What can be done to avoid needing flow?  I find that it’s easier to slip in and out of development when I have a specific behavior I’m looking for, basically a test-driven development model.  Another thing that makes it easier to keep my place is a graphical debugger, which lowers the amount of headspace I have to devote to modelling the software behavior.  Finally, decent documentation (even if it has to be in the form of bugging another developer in an IRC channel) reduces that same headspace requirement by helping me to figure out expected or intended behavior more directly.

Maybe I’d think differently if I worked in an interpreted language; I have little built-in interruptions all day long when I restart GeoServer to try out changes I’ve made.  But I am definitely wary of anything that discourages communication between developers.

Show and Tell

Posted in Bio,Ideas,Open Source Software by dwins on May 20, 2008

So, I’ve sort of started this thing at TOPP.  It wasn’t incredibly intentional, it really just started with me getting some non-techies together and giving them the 30-minute tour of What It Is Those Wacky Guys Upstairs Are Doing.  And everyone said “Hey, it would be great if people could do this sort of mass mind meld whenever they wanted.”  So I thought we’d be seeing more of it since, you know, they can.  But it didn’t really happen until one day, in #openplans:

[Wed Mar 26 2008] [17:31:23]      * dwinslow wonders if he should be more proactive about
 selecting and scheduling talks from the wiki or just leave it alone
[Wed Mar 26 2008] [17:31:35] <jhammel>    dwinslow: YES!
[Wed Mar 26 2008] [17:31:41]      * mchua thinks "a good example" is the hardest part of making a
[Wed Mar 26 2008] [17:31:43] <mchua>    dwinslow: YES
[Wed Mar 26 2008] [17:32:00] <dwinslow>    ah.

So I did.   Anyway, I say all this to lead up to this blog post I made elsewhere.  The short version? I would like software to help in scheduling volunteer-proposed and volunteer-run events and am having trouble finding anything ready-made out there.  Maybe I’ll make it happen; it wouldn’t be the first piece of code that I wrote for TOPP Talks.