The Kaptain on … stuff

I don’t know exactly how many CSV files I’ve read/written to date, but I’m willing to bet it’s a lot. These kind of files are a simple and common way to exchange data and are interoperable with spreadsheet programs as well, making them more easily accessible to non-programmer types. There is some excellent support out there for reading and writing CSV files and so here, from simplest to most complex, are a few different ways to load and query the data trapped inside your CSV files. But first off…

CSV is Not XML

Or any other suitably constrained data format for that matter. There are some pretty clear ‘rules’ out there for transporting data in this format, but of course hardly anybody listens to them. I found this and this in the first page of a quick Google search.
However, while looking for available sample material on the web to demonstrate with, I found many great examples of how to screw it up: not quoting empty strings, not escaping embedded line breaks, commas embedded in unquoted text, etc. All of these are things that prevent automation from properly interacting with the data. There are also, of course, the CSV ‘formats’ that insist on embedding comments and descriptions, trying to present more than one matrix of information or any of the other crazy things you can do when writing to a format with no real metadata layer.

< \rant>

Unfortunately, there’s nothing as convenient as an xml schema available for describing what information is encoded in the document and, in general, the only potentially useful metadata in an individual file is the header line. The good news is that all of these deficiencies can be overcome if you know they exist. And there’s some pretty nifty tools you can use to take out much of the guesswork. I’ll suggest a couple of ideas for how to deal with files that don’t adhere to the ‘rules’, but the main example I’ve decided on is a straightforward and well formed set representing the periodic table of elements, provided generously by Akiscode. This file is directly machine readable(has no headers or embedded data), quotes all values and just follows all the ‘rules’. Here’s the first five lines as an example.
[table id=4 /]

Groovy All by Itself

Groovy makes dealing with String objects pretty painless. By adding facilities like easy casting and find()/findAll() for quickly turning raw Strings into real data, you can program some fairly complex questions. In this simple example I’m depending on all values being quoted and I am not protecting against casting problems. I’m running this through Maven which really helps to keep test data organized; all you have to do is drop your test file in /src/test/resources and it’s automagically available on the test classpath.
No specific handling is done for different types here; everything is read in as a String and cast to a more specific type as needed. In a ‘real’ application you’re very likely going to encounter mixed alphanumeric data in any given column of data – ‘UNKNOWN’ instead of a number in a column labeled ‘Quantity’ for instance – and be more careful about casting.

[groovy language=”true”]
//load and split the file
InputStream inputFile = getClass().classLoader.getResourceAsStream(TEST_FILE_NAME)
String[] lines = inputFile.text.split(‘\n’)
List<String[]> rows = lines.collect {it.split(‘,’)}

/**
* A little helper method to get rid of the quotes in the input
* and cast values so they can be compared properly.
*/
private double castToDouble(string)
{
return string.replaceAll(‘"’, ”).toDouble()
}

//OK, it’s parsed – let’s ask some questions
private static final int ATOMIC_MASS = 1;

def elementsOver200Mass = rows.findAll {castToDouble(it[ATOMIC_MASS]) > 200}
def elementsBetween10And20 = rows.findAll { row ->
double mass = castToDouble(row[ATOMIC_MASS])
mass <= 20 && mass >= 10
}
[/groovy]

So this approach works fine for well formed input, but falls apart quickly in other cases. For instance, if any columns in the file are missing values the split() function treats them as nulls, leading to different size arrays being stored in the list. Similarly you need to deal with quoted VS non-quoted content, embedded line breaks and other issues individually. Fortunately, some nice guys have done that for us.

OpenCSV

Glen Smith’s OpenCSV library nicely abstracts away a lot of the minutiae of dealing with CSV. It properly deals with escaped characters, multiline input and a host of other issues for you. Give it pretty much anything that satisfies the Reader interface as an input and it’s good to go. The return type of the CSVReader.readAll() method also satisfies the same contract as the plain Groovy version shown above, so we can interact with the parsed results in exactly the same way as in the previous example.
All values are still typed as Strings, so we need to cast in order to compare numeric values.

[groovy language=”true”]
import au.com.bytecode.opencsv.CSVReader

List<String[]> rows = new CSVReader(
new InputStreamReader(getClass().classLoader.getResourceAsStream(TEST_FILE_NAME)))
.readAll()

//same finders as in the Groovy version
def elementsOver200Mass = rows.findAll {it[ATOMIC_MASS].toDouble() > 200}
def elementsBetween10And20 = rows.findAll { row ->
double mass = castToDouble(row[ATOMIC_MASS])
mass <= 20 && mass >= 10
}
[/groovy]

HsqlDB

HsqlDB allows for using flat files as Text Tables, effectively turning a CSV file into a database table. Data values are cast to the types specified in your table definition, so any potential type cast failures happen right up front when loading the data. The benefit is that once the data is read in no further manual processing is necessary. Note: by default the csv file and the database file must be located in the same directory, as a security precaution (can be overridden through configuration).

In addition, leveraging sql makes querying the data extremely easy. Groovy brings in the Sql class to abstract away a lot of the normal Java boilerplate you encounter when working with a ResultSet. I didn’t do any in-depth testing to prove it out, but this is also the only one of the strategies described here that doesn’t require holding the entire data in memory in order to do arbitrary queries. That can be mitigated in the previously shown methods by processing files line by line rather than in bulk if memory usage is a concern.

[groovy language=”true”]
//create the table definition to insert
String tableName = ‘elements’
String tableDefinition = """CREATE TEXT TABLE $tableName (
atomic_number INTEGER PRIMARY KEY,
atomic_mass NUMERIC,
name VARCHAR(255),
symbol VARCHAR(3)
);"""

//create a new file database and a table corresponding to the csv file
Sql sql = Sql.newInstance("jdbc:hsqldb:file:${testdbDir.absolutePath}/testdb", ‘sa’, ”
,’org.hsqldb.jdbcDriver’)
sql.execute(tableDefinition)

//set the source to the csv file
sql.execute("SET TABLE elements SOURCE ‘${TEST_FILE_NAME};all_quoted=true’".toString())

//querying the database that’s wrapping our CSV file
def elementsOver200Mass = sql.rows("SELECT * FROM $tableName WHERE atomic_mass > ?", [200])
def elementsBetween10And20 = sql.rows(
"SELECT * FROM $tableName WHERE atomic_mass <= ? AND atomic_mass >= ?", [20, 10])

//simple db aggregates
def count = 0
sql.eachRow("SELECT count(1) FROM $tableName WHERE atomic_mass <= ?", [20]){row->
count = row[0]
}
def avg = 0
sql.eachRow("SELECT avg(atomic_mass) FROM $tableName".toString()){row->
avg = row[0]
}
[/groovy]

Which One do I Use???

Pick the one that fits best for your use case is the real answer. I didn’t know about the HsqlDB option until recently, but OpenCSV has been a personal standby for years. Then again, if you’re writing something quick and dirty at the script level, the simplicity of just applying a couple of split() operations is pretty appealing. Here’s how I generally decide.

[table id=5 /]

So I was making an attempt to catch up on my Google Reader‘ing this weekend and I came across this post on StackOverflow regarding the shiny new StackExchange API. That and a fresh 1.3.1 drop of Grails seemed like as good a reason as any to hack together a little app suitable for seeing what you can do with said API.
Turns out it’s not a whole lot, at least not yet. The minor detraction of having read-only access and limited connects without an API key are more than offset by seeing such a minimal, clean and easy to use interface. According to the grails stats script, it took about this much effort to implement calls to each and every one of the API endpoints for all of the supported domains:
[xml]
+———————-+——-+——-+
| Name | Files | LOC |
+———————-+——-+——-+
| Controllers | 1 | 42 |
| Groovy Helpers | 1 | 47 |
+———————-+——-+——-+
| Totals | 2 | 89 |
+———————-+——-+——-+
[/xml]

Please note that this is about as minimal as you could reasonably get away with(and kinda ugly to boot), but nevertheless it does manage to implement a UI and backend for exercising the entire API in a total of 2 gsp files and 1 controller action. What’s noticeably missing are some tailored views for each of the different JSON responses and the implementation of additional query parameters on the calls.

StackExchange API

This is really pretty well documented and “consistent”. Calls in some cases require an {id} in the url and that’s about it. Each of the individual calls has its own help page that describes the options, like the badges page for instance. To make things a little more helpful while exploring the API, each of the implemented endpoints in this app are also hyper-linked to the corresponding manual page on api.stackoverflow.com.

RESTful Access

Couldn’t be a whole lot easier than using HttpBuilder, which provides nice and concise ways to execute the GET request, inspect the response and deal with success/failure of the request. Aside from assigning the root domain and method using variables, this is little different from the canonical example on codehaus.
[groovy]
def http = new HTTPBuilder("http://api.$domain")
http.request(GET, JSON) {
uri.path = "/${VERSION}$method"
headers.’User-Agent’ = ‘Mozilla/5.0 Ubuntu/8.10 Firefox/3.0.4’
response.success = { resp, json ->
answer = json
}
response.failure = { resp ->
answer = "Unexpected error: ${resp.statusLine.statusCode} : ${resp.statusLine.reasonPhrase}"
}
}
[/groovy]
The syntax here could probably get even Groovier using the Grails REST client plugin wrappers, and indeed incorporating that plugin is how the HttpBuilder dependency is being provided in this app, but I hit on a working implementation first time out so we’ll leave that for another day, shall we?

What’s it Look Like?

Did I mention it was ugly? Where the API call requires a parameter, you get a text field. Each domain gets its own submit button, officially representing my first attempted usage of g:actionSubmit in Grails. Doesn’t work quite the way I expected, but it does certainly work. Click a button and you’re shown the raw JSON result, along with the call you made. In an ugly fashion, or did I say that already? Still, for a couple of hours of hacking, I’m not unhappy with the result.

StackApps

This is where the apps in the contest get shown off and, in a wonderful display of dogfooding, it’s guess what – another StackOverflow clone! The page for this application can be found here. And the source code is up on github if you’d like to take a look.

Next Steps?

Toying with the idea of porting this to appengine, which would unfortunately mean replacing HttpBuilder and the underlying Apache HttpClient in favor of something that didn’t use a threaded approach, most likely using URL fetch. That and some slightly less ugly tabular and/or tree views of the JSON responses would be kinda nice to have.
🙂

One of the most compelling things about using Groovy is the fluent and concise syntax, as well as the productivity and readability gains that come out of it. But there’s no reason not to take advantage of some of the same techniques and some library support, in this case google-collections, to make Java code easy to write AND to read.

Creating Collections

Groovy really shines for this one, making the creation of Lists and Maps, empty or populated, an absolute breeze. Java has some help for creating basic Lists but begins to struggle when creating maps. This is an area that google-collections can help in, especially in regards to creating immutable Maps.

[groovy language=”true”]
//Empty Lists
List<String> groovyList = []
List<String> javaList = new ArrayList<String>()
List<String> googleList = Lists.newArrayList() //can omit generics

//Populated Lists
List<String> groovyList = ["1", "2"]
List<String> javaList = Arrays.asList("1", "2")
List<String> googleList = Lists.newArrayList("1", "2")

//Immutable Lists
List<String> groovyList = ["1", "2"].asImmutable()
List<String> javaList = Collections.unmodifiableList(Arrays.asList("1", "2"))
List<String> googleList = ImmutableList.of("1", "2")

//Empty Maps
Map<String, String> groovyMap = [:]
Map<String, String> javaMap = new LinkedHashMap<String,String>()
Map<String, String> googleMap = Maps.newLinkedHashMap()

//Immutable Maps
Map<String, String> groovyMap = ["a":"1", "b":"2"].asImmutable()

Map<String, String> javaMap = new LinkedHashMap<String,String>()
javaMap.put("a", "1")
javaMap.put("b", "2")
javaMap = Collections.unmodifiableMap(javaMap)

//OR(works only in Java, will not compile in Groovy)
Map<String, String> javaMap = new LinkedHashMap<String, String>()
{
{
put("a", "1");
put("b", "2");
}
};

Map<String, String> googleMap = ImmutableMap.of("a", "1", "b", "2") //clunky syntax but it works
[/groovy]

Filtering Collections

Groovy provides the very handy ‘findAll’ method that allows you to filter a Collection by applying a Closure. Google-collections provides similar facilities using the Predicate interface and two filter methods available statically on Collections2 and Iterables. This would also be a lot more readable if the Predicate definition were extracted but I wanted to show that it’s still possible to create them in-line quickly.

[groovy language=”true”]
import static com.google.common.collect.Collections2.*

List<Integer> toFilter = [1, 2, 3, 4, 5]
List<Integer> groovyVersion = toFilter.findAll{ it < 3}
List<Integer> googleVersion = filter(toFilter, new Predicate<Integer>()
{
public boolean apply(Integer input)
{
return input < 3;
}
};)
[/groovy]

Joining Collections into a String Representation

This one has come up pretty often over the years, and it’s not surprising that where the JDK hasn’t provided, enterprising developers have added support through libraries. The problem is: given a Collection of objects, create a String representation of that Collection suitable for view from a consumer of the system. And admit it – the first time you hand-coded the algorithm you left a trailing comma, didn’t you? I know I did.
Groovy has fantastic support for this use-case by simply including the ‘join’ method on Lists. Google-collections utilizes static calls on the Joiner class along with a simple DSL-like syntax to achieve the same effect. Throw in a static import to make it even more concise and it does a fine job. These two examples yield the same result.

[groovy language=”true”]
import static com.google.common.base.Joiner.*
def toJoin = [‘a’, ‘b’, ‘c’]
def separator = ‘, ‘

//groovy version
def groovyJoin = toJoin.join(separator)

//google-collections version
def googleJoin = on(separator).join(toJoin)

[/groovy]

And google-collections also supports join for Maps, something missing from Groovy(although not very hard to implement).

[groovy language=”true”]
import static com.google.common.base.Joiner.*
def toJoin = [1: ‘a’, 2: ‘b’, 3: ‘c’]
def separator = ‘, ‘
def keyValueSeparator = ‘:’

//results in ‘1:a, 2:b, 3:c’ which is essentially what is returned from Groovy map.toMapString()
def googleVersion = on(separator).withKeyValueSeparator(keyValueSeparator).join(map)

//results in ‘1 is a and 2 is b and 3 is c’
googleVersion = on(" and ").withKeyValueSeparator(" is ").join(map)

//doing the same in Groovy is slightly more involved, but really not that bad
def groovyVersion = toJoin.inject([]) {builder, entry ->
builder << "${entry.key} is ${entry.value}"
builder
}.join(‘ and ‘)
[/groovy]

Multimaps

Multimaps are one of the more interesting parts of google-collections, at least to me. Have you ever found yourself writing code to create a Map of keys to Lists? Well the various Multimap implementations in google-collections mean you never have to write that boilerplate kind of code again. Here’s a “first-stab” effort to simulate a fairly generic Multimap with pure Java code.
[groovy]
public class JavaMultimap
{
private Map<Object, List<Object>> multimap = new LinkedHashMap<Object, List<Object>>();

public boolean put(Object key, Object value)
{
List<object> objects = multimap.get(key);
objects = objects != null ? objects : new ArrayList<object>();
objects.add(value);
multimap.put(key, objects);
return true;
}
}
[/groovy]

And the same thing in Groovy, achieving a slightly smaller version.

[groovy]
class GroovyMultimap
{
Map map = [:]

public boolean put(Object key, Object value)
{
List list = map.get(key, [])
list.add(value)
map."$key" = list
}
}
[/groovy]

I did some primitive timings comparing Java, Groovy and google-collections Multimaps implementations and, as you’d pretty much expect, google clearly takes the lead. Where things really start to get interesting though is when you start using the Multimap in Groovy code. Imagine if you will that you want to iterate over a collection of Objects and map some of the properties to a List. Here’s a contrived example of what I’m talking about, but applying this same pattern to domain objects in your application or even a directory full of xml files is pretty much the same. If you look closely you’ll notice that Groovy actually makes this a one liner to extract all values for a property across a List of Objects(used in the assertion), but I imagine that Multimap is probably a better alternative for large data sets.

[groovy]
class GoogleCollectionsMultiMapTest
{
private Random random = new Random()

@Test
public void testMultimap()
{
def list = []
10.times {
list << createObject()
}
List properties = [‘value1’, ‘value2’, ‘value3’]
Multimap multimap = list.inject(LinkedListMultimap.create()) {Multimap map, object ->
properties.each {
map.put(it, object."$it")
}
map
}
properties.each {
assertEquals (multimap.get(it), list."$it")
}
}

Object createObject()
{
Expando expando = new Expando()
expando.value1 = random.nextInt(10) + 1
expando.value2 = random.nextInt(100) + 100
expando.value3 = random.nextInt(50) + 50
return expando
}
}
[/groovy]

So Where Does This Get Us?

Between google-collections and the newer guava-libraries that contain it, there’s lots of help available for simplifying programming problems and making your code more readable. I haven’t even touched on the newly available support for primitives, Files, Streams and more, but if you’re interested in reducing the amount of code you write AND simultaneously making it more readable you should probably take a look. There’s a very nice overview available in part one and part two by Aleksander Stensby. And here’s a closer look at what google-collections can do for you.

Source Code

As per usual, source code is available at github as a maven project. Big thanks to the Spock team for sharing how they configure GMaven to properly utilize Groovy 1.7. Please feel free to take a look and comment here. Thanks!

I was reading this article about hibernate validator today and it inspired me to apply a little Groovy to the problem of validating a bean. More specifically, finding out how hard it would be to apply different validation rules to the same classes at runtime. Turns out it’s really pretty simple.
Hibernate validator, if you didn’t already know, is the reference implementation of JSR-303 and it provides the ability to specify by xml or annotation configuration validation rules for pojos.

Where Hibernate Validator Shines

Annotations on domain classes allow for easily validating object state at the time of persistence. Excellent integration with frameworks like JBoss Seam allow this same ability to be utilized for validating web forms on the client-side with little more than an <s:validateAll/> tag. Seam practically hides the entire interaction with validation components from the developer. Since validation rules are defined directly in the domain class, you can (almost) guarantee that no objects with inconsistent state will ever end up being saved in your database. There are certain validations that aren’t possible to verify without actually looking in the database, unique constraints for example, but generally in my experience hibernate validator is extremely easy to configure and work with. Implementing CRUD functionality is pretty trivial, and UIs can achieve consistency since all validations are applied equally.
Alternatively, if you can’t or don’t want to use annotations for some reason, you can specify your validation rules in an xml file. Usually a singleton validation.xml file is made available on the classpath and picked up automagically when a ValidationFactory is created. A simple xml configuration looks like this:

[xml]
<constraint-mappings xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemalocation="http://jboss.org/xml/ns/javax/validation/mapping validation-mapping-1.0.xsd" xmlns="http://jboss.org/xml/ns/javax/validation/mapping">
<default-package>org.kar.test</default-package>
<bean class="ValidateTestableClass">
<field name="name">
<constraint annotation="javax.validation.constraints.NotNull">
</constraint>
</field>
</bean>
[/xml]

and is meant to be applied to this simple class:

[groovy]
class ValidateTestableClass
{
int id
String name
String description
boolean enabled
}
[/groovy]

Comparing with Grails Validation

Grails automatically provides validation capabilities for domain classes and command objects, and enables adding the same behavior to any pogo through a combination of the @Validateable annotation and a static constraints closure. Adding validation support to arbitrary classes also requires specifying which packages to scan for the annotation.
Plugin support from projects like bean-fields simplifies the handling of client-side validation and rendering error markers in the UI, an ability which the Grails framework provides natively by adding an ‘errors’ field directly onto the domain or command object class instances bound to a web form.

Dynamic Constraints

Both the hibernate validator and the Grails strategies for applying validation described here have the same limitation: both are universally applied to all instances of a class. There’s no easy apparent way to override those constraints at runtime, although I suspect that some fancy MOP’ing or configuration could probably be used to accomplish overrides at runtime.
Hibernate validator also supports creation of ad hoc validators by seeding with one or more xml documents. Or if you’re like me and hate hand editing xml, you can leverage Groovy to take a bit of the pain away. Here’s the same xml snippet from above in a Groovy Closure, generated simply by turning the DomToGroovy class loose on the raw xml:

[groovy]
looseConstraint = {
mkp.declareNamespace(xsi: ‘http://www.w3.org/2001/XMLSchema-instance’)
‘constraint-mappings'(xmlns: ‘http://jboss.org/xml/ns/javax/validation/mapping’,
‘xsi:schemaLocation’: ‘http://jboss.org/xml/ns/javax/validation/mapping validation-mapping-1.0.xsd’) {
‘default-package'(‘org.kar.test.objects’)
bean(‘class’: ‘ValidateTestableClass’) {
field(name: ‘name’) {
constraint(annotation: ‘javax.validation.constraints.NotNull’)
}
}
}
}
[/groovy]

Losing all the angle brackets is a good start, but we really haven’t saved a lot of typing. Until you start taking advantage of the ability to define more complicated structures. Note the use of a list structure here to apply NotNull constraints to multiple fields.

[groovy]
strictConstraint = {
mkp.declareNamespace(xsi: ‘http://www.w3.org/2001/XMLSchema-instance’)
‘constraint-mappings'(xmlns: ‘http://jboss.org/xml/ns/javax/validation/mapping’,
‘xsi:schemaLocation’: ‘http://jboss.org/xml/ns/javax/validation/mapping validation-mapping-1.0.xsd’) {
‘default-package'(‘org.kar.test.objects’)
bean(‘class’: ‘ValidateTestableClass’) {
[‘name’, ‘description’].each {
field(name: it) {
constraint(annotation: ‘javax.validation.constraints.NotNull’)
}
}
field(name: ‘id’) {
constraint(annotation: ‘javax.validation.constraints.DecimalMin’) {
element(name: ‘value’, ‘2’)
}
}
field(name: ‘enabled’) {
constraint(annotation: ‘javax.validation.constraints.AssertTrue’)
}
}
}
}
[/groovy]

Applying Dynamic Constraints

Applying constraints is as simple as converting Closures to xml and mapping them to a Configuration object, which then supplies a Validator to use. StreamingMarkupBuilder is utilized to create the xml behind the scenes.

[groovy]
/**
* Create a configuration object passing closures as validation mapping documents.
* @param closures closures to render into validation mapping documents
* @return config
*/
public Configuration createConfig(Closure… closures)
{
Configuration config = Validation.byDefaultProvider().configure()
closures.each {
config.addMapping(new ByteArrayInputStream(GroovyXmlConversionUtil.convertToXml(it).bytes))
}
config
}
[/groovy]

I haven’t tested the use of multiple mappings extensively, but minimally each class you’re configuring must be confined to a single mapping – you can’t extend the validations by layering configurations on top of one another. You should however be able to map constraints for different classes in separate Closures.

Crying out for a Builder!

Going from Closures to xml is a quick and dirty way to test out this functionality, but what would really be nice is a Builder that could create an appropriate validation environment more directly. At the least it would allow for removing the namespace declarations and explicit package naming that make up the bulk of the content.

So what do you get?

1. 1. Ability to declare validations against any existing Java or Groovy class without changing the source code
2. 2. Programmatic ability to create the configuration of validations
3. 3. A choice of which validations to apply at runtime
4. 4. Consistency with the behavior of domain class validation

And what’d I get? A fun bit of quick coding on a Saturday afternoon. Nice! Source code is available on git-hub if you want to check it out.

Recently I got suckered into buying a Roomba at a local store. It was the very basest model on special at $96CD and I gave in to the indulgence to see how it might fare in my home.

Fair Disclosure

We live with 3 pets. Two dogs and one cat. The dogs are black and tan. The cat is gray and white. The end result is that we have hair of every pretty much every possible color just about everywhere, all the time. If you too are a multi-pet owner you may know of the fur dust-devils that can take over your kitchen floor, the inevitable dirt by the door where the dogs come in or the “track proof” kitty litter that inevitably gets tracked. Over time, in order to properly clean up after the pets we’ve invested in a very good vacuum cleaner and a steam cleaner to go with it. Not to mention a weekly two hour visit from a professional cleaner. The Roomba has taken a lot of the work out of it, and that’s a good thing because if there’s one thing I hate it’s doing dishes by hand. Followed closely by vacuuming.

First Impressions

After fully charging the unit, we set it loose on the main traffic areas of our house – the living room and the kitchen. The first time we ran it the area was left pretty much ‘as is’ and the Roomba needed help a couple of times when it got caught up around chairs, table legs and an edge of a couch. The dirt trap was also stuffed to the max with pet hair after an hour. I blame the fact that the couch it DID fit under had probably gone way too long without us pulling it away from the wall. My Bad. At the end of an hour, the floor was very visibly and noticeably cleaner.

Week One

Let me let you in on a little secret. I’m a tech guy. In practical terms what does that boil down to? Wires. I’ve got em. Where I can, they’re routed, fixed and generally hidden, but in the shady recesses behind desks where Roombas might venture they can be found, exposed and vulnerable to the elements. Turns out it’s not that big a deal. As things the Roomba got stuck on revealed themselves, they slowly got evaluated, assessed, and in some cases removed. Getting caught under the edge of one couch? Solved by some $1.99 plastic furniture lifts. Stuck rotating around the legs of a wooden chair for 10 minutes? Chair ended up in the garage. The dirt trap was still pretty much full after every run, one per day in a different part of the house everyday, but at the end of a week the house was looking much better overall. And as a side benefit there was a whole lot less crap lying around on the floor.
🙂

Week Two

So by this time there’s a really big bonus to note about the Roomba over a conventional vacuum. Not only does it not require a lot of direct interaction to get your floors clean, but it also doesn’t make the pets go mental when you turn it on. One sure way to make the dogs and cat disperse in an immediate fashion is to turn on the vacuum cleaner. The noise drives them out of the room as far as they can get and it takes awhile for them to settle down when you finally shut it off.

The Roomba on the other hand doesn’t appear to phase them in the least. It’s (reasonably) quiet for starters. It also isn’t really all that “aggressive”. Our big dog, Aurora, a 13 year old Black Labrador/Dane cross, has a habit of falling asleep right in the middle of the room. The Roomba happily cleans right around her, so much so that she might turn her head to look at it, but rarely is it enough to make her actually stand up. Keep in mind this is when it runs directly into her. The pressure sensor on this piece of hardware is pretty impressive. Before I let it run into Aurora, I did test it out myself first of course – what do you people think I am, a monster? In this way the robot most definitely appears to obey the first law; when it runs into something it immediately stops and tries a different direction. And it stops on a dime. The cat thinks that we bought him a new toy. He chases and jumps on the Roomba like it was covered in catnip. LMAO.

After two weeks we figured out how to properly clean out the unit. Finally breaking down and RTFM helped to figure out how to clean the filter. The chassis is pretty much hollow so a lot of dirt and hair had accumulated there, something you can fix by pretty much shaking the whole thing over a garbage can, preferably outside. The carpet rollers also needed a little attention by this point, as pet hair and assorted other things(thread, plastic twist tie, etc) had been rooted out from the floors of our home and wrapped around the spindle. At this point our cleaning lady, a wonderful woman who has contributed much to keeping our house livable, not only noticed the difference but took the initiative to suggest tackling some of the less “essential” cleaning aspects and generally taking things up a notch: removing all the sink and shower knobs, cleaning in and around them; cleaning the oven(more than the once/yr average); removing all of the window screens, cleaning both them and the windows they’re covering. Did I mention she also takes the dogs out for us once a week? If you’re around #yyj and you want someone hard-working and dependable to give you a hand I can highly recommend this lady. Thank you Sandy!

The trap is still pretty much always full.

Week Three

Stepping it up a notch, the Roomba is now being run 2 or 3 times a day, as much as possible covering the 3 primary areas of the house: the bedroom/bathroom areas, the dining room, and the living room and kitchen. All told it’s about 1800 square feet, broken down in order to roughly 600, 400 and 800 sq/ft chunks. The dirt trap now is starting to not be full. The Roomba is covering more territory every day, all of the electrical cords are tucked away where they can’t be harmed. The cat is now bored with the Roomba and merely takes one tiny step back when it threatens to run into him. The dogs continue to mostly ignore it. The house actually looks better than when we moved in. Despite the two month lifetime stated for the filter, I retired the first filter because it was, quite frankly, toast. Fortunately along with the “special” price 3 more filters were included in the box with the original purchase.

Week Four

Keeping up with things now is a lot easier. The Roomba runs once or twice a day, usually either right after we get home from work(in an area of the house we’re not in), and at night when we’re going to bed. The noise is pretty minimal so running it downstairs in the high-traffic areas at night while we’re sleeping works out really well. It’s also become kind of automatic to deal with. If we’re walking past the place where we charge the Roomba and the light is “green” we pretty much grab it and take it somewhere to do it’s job. If we’re walking past it in the hallway and the light is “red” then it’s time to empty it out and take it back to the charging station. That and remembering to pick up the limited number of things left hanging around on the floor that it can get caught on is pretty much it.

Is it worth it?

Well unless you’ve been paying really bad attention so far, you already know that I think it is. 😉
So here’s the breakdown as I see it.

The Roomba cleans for between 7 and 14 hours per week. I can’t honestly say we ever averaged that much time pushing a vacuum before. The physical effort involved I would estimate at about 5 minutes or so per run, to clear up the floor, turn it loose and then come back and empty it later. So that’s about 30 to 60 minutes a week. And that is kinda the point of a robot isn’t it? Nothing like a division of labor that is 10% planning and 90% labor.

It cleans for approximately an hour every time, and takes a couple or three hours to charge back up to full. I haven’t been explicitly timing this, but it hasn’t varied a lot from this pattern.

It occasionally gets stuck. Personally I just think it’s kinda cool to watch it edge around the top of a staircase, and usually it’s sensible enough to pull back, but every once and awhile it just goes a teeny tiny bit too far and gets stuck slanted and bleating at the edge of the stair. Third Law in full effect here.

Similarly it seemingly hates the half inch foam mat we have under a painting easel, getting caught half-on/half-off the corners of it. With the more expensive Roomba models these are the kind of problems that seem to be usually solved by virtual “walls” which mark boundaries the Roomba should not cross. For us that bought the cheap version, some pretty simple measures close the gap. For the top of the stairs? We already have a gate(s) in place for the pets and it works equally well to keep the Roomba away from places it should not go. For the for mat? Sometimes a chair, sometimes a pair of shoes – whichever one happens to be closer at hand at the time. Weight really isn’t a big factor, but height is. If a surface is high enough and the Roomba bumps into it, it just back off. If it’s too low, the Roomba tries to climb it. Long story short is if there’s anywhere you don’t want your Roomba to go, you can gate it easily with just about anything that will stand in its way and is more than an inch high. As already stated this includes, but is not limited to, pet gates, furniture, people and pets themselves. This pretty much covers the Second Law.

If it gets stuck, well… so what? So far this thing has been so careful that it’s trained us to better behavior. Less things to get stuck on means less things on the floor means less clutter and more things in a closet or on a shelf. If it gets stuck on what’s left, it beeps to tell you so. Give it a kick and set it back on its merry way. The most you lose is <1hr of cleaning time, and you get the balance back with nothing more than moving it around and hitting the button again. Honestly with one simple color/status changing button you get all the information you need about the Roomba just by happening to walk by it. Simple UI FTW!

For the truly interested out there, I tracked this month or Roomba in a Google spreadsheet. Blame my OCD if you want. And for those of you who might be reading this who aren’t total geeks, the Three Laws of Robotics are classic Asimov science fiction.
[table id=3 /]

That’s a whole lot of Gr8ness in the title, don’t you think? Ok, ok – I apologize for the pun. I whipped these up using the open source Gource visualization software and with HD content free on Youtube now, I just had to try that out too.

Gource

Gource is a software version control visualization program with support for Git, Mercurial, and with a tiny bit of extra effort, CVS and Subversion. It’s available as a macport install, as is the ffmpeg program used to stitch together a series of screenshots into the final video.  All of these visualizations were created with Git or Git-svn. When the Gource program is running it brings up an interactive window so you can interact with it to a degree, affecting the camera, zoom, etc.  The default resolution is 1024×768 but I tried to bump it up to 1920×1080. I’m assuming that stretched the limits of the rendering, as there’s quite a bit of pixellation in the final product; either that or the ffmpeg params need to be tweaked. The three visualizations on this page were all created using this one-liner command line invocation (broken down here over several lines for readability).
[bash]
gource -1920×1080 –highlight-all-users –bloom-intensity .3 -a 1 -s .01
–hide-dirnames –hide-filenames –stop-at-end –output-ppm-stream –
| ffmpeg -y -b 3000K -r 60 -f image2pipe -vcodec ppm -i –
-vcodec libx264 -vpre default gource.mp4
[/bash]

Groovy

From the initial commit on August 28, 2003, Groovy has come a long way. I tried the Gource provided python script to translate an svn log file into the required format the first time, but cloning the repository with git-svn seemed to produce better results. It’s interesting to see that the overall structure of the Groovy project doesn’t look like is has changed very much over the years. I recognized a lot of the names you see on the screen. Be sure to watch the 1080p version if you’re having trouble reading the text and see if you can’t find some recognizable committers.

Grails

Starting on Sunday March 14, 2004 here’s how it all unfolded, at least from the perspective of the public Git repository. Grails looks like it grew a little more spontaneously than Groovy; there are more frequent restructurings and some transitions that seem to cover a whole lot of ground in very little time. In particular watch out for the burst of activity in the summer of 2009.

Griffon

September 17, 2008 marks the first commit for Griffon and this short but sweet video explodes right from the start. Benefiting greatly from the Grails model this project shows a lot of progress in a very short time indeed. Most, if not all, of the names you see here you probably caught in the previous two videos as well.

What’s the Point?

Watching these gives me (and maybe you) some appreciation for how much effort over time it takes to bring a new language/framework/technology to maturity and all of the individual contributions that go into it. Gource provides a lot more detail into the actual structure if you want to dig in. For purposes of keeping these images fairly uncluttered I removed file and directory names and just focused on the committers and the general structure. I think I gained a git of insight into how a codebase can evolve, and properly appreciate that organizing the increasing complexity of an ever-growing system over time is not a simple job. Plus they look cool. Enjoy!