Dwins’s Weblog

FOSS4G Code Sprint

Posted in Uncategorized by dwins on September 16, 2011

I’m at FOSS4G this week.  I’ll be at the code sprint at the end of the week and thought it might be a good idea to write up some thoughts about what I’d be interested in working on there.  I’m breaking the list down by project.


GeoScript seems to be getting some nice attention at FOSS4G this year.  It adapts the GeoTools geospatial library to be more convenient to use from scripting languages like Python and JavaScript.  I am the main developer of the Scala variant of GeoScript, so I’m happy to see all the excited tweets and other buzz about GeoScript.  I didn’t make it to the talks about it (Spatial Capabilities for Scripting Languages and Scripting GeoServer with GeoScript) due to another presentation I was making, but the tutorial on Thursday had a very nice turnout.  Justin Deoliveira and Tim Schaub put a lot of work into a written tutorial – but only for the Python and JavaScript variants of GeoScript.  So one (really useful) thing that could happen at the code sprint is “porting” that documentation to Scala, and maybe reviewing the GeoScript Scala documentation overall.

I’ve also been eyeing One-Jar as a means of producing an executable Jar distribution of the Scala interpreter with GeoScript and  GeoTools preloaded, as a really simple means of trying out GeoScript (no installer needed) for those who don’t want or need to learn SBT.


gsconfig.py is a Python library for controlling GeoServer remotely.  It’s already in heavy use in the GeoNode project but can be useful for smaller tasks as well – for example, here’s a script using it to copy a layergroup from one GeoServer to another.  Again, documentation would be useful – one thing I feel is really missing from the documentation is some info on hacking it 🙂  There are a lot of corners in the GeoServer REST API which are currently not supported in gsconfig, and it would be nice if it was more accessible for those sorts of improvements to be made by developers other than me.  Fortunately, Python is a nice language and gsconfig does a pretty simple task (effectively just slicing, dicing, and duct-taping XML documents together) and the whole thing is less than 1500 lines of code, so an overview for developers should be pretty attainable.

I also received a brisk reminder during my tutorial on GeoServer Scripting with Python that there is no documentation online about how to actually install gsconfig for use in Python scripts 🙂


mapnik2geotools is a tool (in Scala, of course ;)) to translate map styles between Mapnik XML and OGC’s SLD standard (complete with support for GeoServer extensions and using GeoServer’s REST API to set up the styles in a live GeoServer instance.) It’s a rough project at this point, but I think an interesting one.  It also doesn’t use or need too many exotic features of the Scala programming language, so if you are interested in getting started with Scala, hacking on mapnik2geotools might be a good introduction to the language.  While we have used it a bit at OpenGeo for bootstrapping our OpenStreetMap styling efforts, I haven’t had a lot of time to refine it, and there is a lot of room for improvement.  I am not even going to list ideas here 🙂 but instead refer readers to the Github issues list.

GeoServer CSS

The GeoServer CSS module is another style converter I work on.  There are some interesting problems left to be solved, and some polish to be added to the GeoServer UI related to it.  It would be interesting this weekend to think about styling raster data with CSS.  Automatic conversion of existing SLD documents into CSS would also be an interesting and useful feature.  I haven’t considered the deeper GeoServer integration too much, but the interactive styling page could use some cleanup (and maybe generalization to work with SLD styles too).  It would also be interesting to allow inline CSS in WMS requests, and possibly have a REST API for getting SLD equivalents to CSS, for use with OpenLayers.



Testing is complicated.

Posted in Uncategorized by dwins on December 16, 2009
Tags: , ,

A coworker of mine is getting more into the code side of making software work.  I was talking to him recently about increasing the test coverage on the project he’s coming onto and he asked me, that eternal question:

So, what do I test?

The answer, of course, is “everything.”

Less glibly, software testing is a pretty deep topic.  There are a lot of kinds of testing you can do, some very time consuming, others very quick, and with a similar range in effectiveness in finding bugs.  Of course, there are other reasons to do tests (maybe you don’t care about all bugs and just want to be able to make some performance guarantees, for example.)  But most of the time when people talk about software testing they want to get an idea of how buggy the software is; how close its behavior is to the expected/desired.  In this blog post, I’ll talk about a few broad categories of testing.  But first, a quick testing glossary:

  • verification: Examining a product to make sure that it does what it was designed to do (what do the blueprints say?)
  • validation: Examining a product to make sure that it does what it is supposed to do (what did the customer want?)
  • reliability: How likely is it that a product will function when it is needed?
  • assertion: Generally, automated tests perform some number of operations with the system, and then inspect some property of the system afterward.  That property is called an assertion, and this term is used pretty universally across testing frameworks.  It is common to talk about a test or an assertion being “strong” or “weak” based on how specific the requirements are.  (x>2 is usually a weaker assertion than x == 6).
  • mocking: The practice of creating an object that *acts* like some component of the system, but without being a full implementation.  This is useful in testing because you can guarantee the return values from methods on a mock object will be appropriate for your test.  It’s also nice to not actually make that bank withdrawal when testing your bill-paying application.  You can mock up objects (ie, instances of a class in some programming language) but it is also possible to mock up an external service, or a user account, or whatever environment is needed for testing.

Smoke Testing

Smoke tests just exist to verify that, upon running your software, the magic smoke does not escape from your computer.  They generally make very weak assertions about the system, like, for example, that a homepage in a web application has a title.  But, even in this example, a passing test shows a lot of good things about the system: the templates are configured properly, the system successfully binds to a port, etc.

Unit Testing

Unit tests intend to test a single component of a system.  While the precise definition of a unit is up to individual teams, it is pretty common to discuss unit tests as ones that operate on a single class (in Java code, for example.)  Ideally, in a unit test, all objects except for the one under test should be mocked.

Integration Testing

Integration testing verifies that components in a system interact correctly.  This is the type of testing that you are doing when you set up that server and just check that the homepage comes up.  Again, the line between unit and integration testing is kind of fuzzy.

Black Box Testing

Black box testing uses only the parts of a system that are intended for use by external components.  For example, if you were black-box testing a Queue class, then you would not make any assertions about the internal storage, only based on the standard Queue operations.  You also wouldn’t test or modify any helper classes used in the implementation of the Queue, and code inspection would not be an option.  Black box testing encompasses techniques like:

  • fuzz testing, using randomly generated garbage data as input
  • equivalence classes, inspecting a component’s specification to identify ranges of input that should be functionally equivalent (thereby allowing the tester to avoid redundant effort
  • user testing, simply handing software to a user and asking him to identify places where it works unexpectedly or incorrectly

White Box Testing

White box testing assumes access to the code and internals of a system.  It allows techniques that modify and inspect the code, things like:

  • static analysis, where a tool like lint or findbugs inspects code to heuristically detect errors
  • code coverage analysis, where a tool like cobertura instruments the system during a run to identify unused sections of code.
  • mutation testing, randomly modifying code to gather statistics about how much has to break before automated tests start to fail
  • code review (by real live programmers!)

I guess I still haven’t answered the question; what should we test?  It’s going to vary a lot between projects.  A lot of shell scripts I write don’t get tested at all (well, except for the one time I run them), but I’d like to know that the guys working on the software for the next plane I’m on have a pretty solid testing system.  And there are a lot of points in the space between those extrema. In general, I would say probably getting basic tests in place to begin with is good, and of course as a project goes on problem points can be identified.  Automated tests are good for ensuring old behavior doesn’t change unintentionally, but there is a balance to strike between setting up that safety line and tying the whole project down.  Because (especially in web programming) things do actually change from time to time.

Meme in Scala

Posted in Uncategorized by dwins on November 18, 2009

Sebastian over at Digifesto recently alerted me to a budding meme being pushed by Eric Florenzano.  The original proposal (from here) goes like this:


  1. Implement a program that takes in a user’s name and their age, and prints hello to them once for every year that they have been alive.
  2. Post these rules, the source code for your solution, and the following list (with you included) on your blog.
  3. Bonus points if you implement it in a language not yet seen on the following list!

The List:

  1. [Python] http://www.eflorenzano.com/blog/post/trying-start-programming-meme
  2. [Bash] http://aartemenko.com/texts/bash-meme/
  3. [C] http://dakrauth.com/media/site/text/hello.c
  4. [Java] http://adoleo.com/blog/2008/nov/25/programming-meme/
  5. [Python 3] http://mikewatkins.ca/2008/11/25/hello-meme/
  6. [Ruby] http://stroky.l.googlepages.com/gem
  7. [Ruby] http://im.camronflanders.com/archive/meme/
  8. [Lisp] http://justinlilly.com/blog/2008/nov/25/back-on-the-horse/
  9. [Lua] http://aartemenko.com/texts/lua-hello-meme/
  10. [Functional Python] http://aartemenko.com/texts/python-functional-hello-meme/
  11. [Erlang] http://surfacedepth.blogspot.com/2008/11/erics-programming-meme-in-erlang.html
  12. [Haskell] http://jasonwalsh.us/meme.html
  13. [PHP] http://fitzgeraldsteele.wordpress.com/2008/11/25/memeing-in-php-2/
  14. [Javascript] http://www.taylanpince.com/blog/posts/responding-to-a-programming-meme/
  15. [Single-File Django] http://www.pocketuniverse.ca/archive/2008/november/27/florenzano-factor/

For my entry, I put together a script in Scala.

import scala.Console._

var name = readLine("What is your name? ")
var age  = readLine("How many years old are you? ").toInt
(1 to age).foreach { age => println("%2d) Hello, %s".format(age, name)) }

To run, copy the code into a text file named hello.scala and then run: scala hello.scala You can get the scala interpreter from the Scala website, or your distribution’s package manager.

speaking in tongues

Posted in Uncategorized by dwins on May 18, 2009

I recently had a chat with a friend of mine about how dissatisfied I feel that I only speak one language (one and a half, counting the few semesters of Spanish I completed in high school).  The compassionate fellow he is, my friend tried to console me, pointing out that I’m at least lucky that the one language I do speak is English (agreed) and that, considering my wealth of programming experience, I am fluent in several languages.

This isn’t the first time I’ve heard someone talk about being fluent in a programming language.  I have never met a developer in person who would use that phrasing, though.  In most programming ‘languages’ it is perfectly possible to write code in any natural language you want.  More importantly, the difference between different programming languages tends to be more along the lines of how processes are structured (if I define a variable here will I be able to see it there? change its type? delete it later? what tools are provided to help me structure data and operations on it? how many lines does it take to write 99 bottles of beer?) than the enormous differences in vocabulary that you see going from, say, English to Italian.  The ‘vocabulary’ of code, the API’s available, are something that don’t need to be memorized for a developer to be proficient; in fact I would say that hardly any developers memorize even built-in APIs (those included with the language as standard libraries) to the extent that they would learn vocabulary in a second language.

2 + 2 means the same thing in Java, JavaScript, PHP, Python, Perl, and Haskell.

I have thought about this for a while and I don’t know of a better word for a programming language than “language.”  But sometimes I really wish there was one.

quicky – unintended API design

Posted in Uncategorized by dwins on October 9, 2008

Poking around WordPress.com today, I noticed this blog post on the WordPress.com blog about an updated theme.  It’s funny how changing something like a page template can get so much pushback from users who have somehow managed to make their blogs entirely dependent on it (take a look at the comments to see what I’m talking about).  Kinda reminds me of something I said a while back.