Seven Design Considerations for a new DSL in Clojure

I recently wrote a DSL (domain specific language) and tool called Lemur.  Writing an internal DSL in Clojure is a truly fantastic experience (but that's another story).  My goal in this post is to share some questions and decisions that I faced in the hopes that it will be of value to other developers writing their own DSLs.  Ideally, there would be some discussion leading to some agreed principles.

Acknowledgement.  Two projects from which I drew a lot of inspiration from both their design and implementation: Cascalog and Leiningen.  Thank you to the authors of those tools and to the open source ideals which make it possible for me to examine their code.

Lemur and the Jobdef

A very quick intro to Lemur, for background.  Lemur is a tool for launching hadoop jobs on elastic-mapreduce.  Each job is described by a Clojure file, which I refer to as a Jobdef.  The Jobdef uses a DSL to describe the cluster and the jobs that should run.  For reference, here is a sample jobdef file:
(use-base
  'lemur-test.lemur-base)

(catch-args
  [:num-days "Number of days to download" -1])

(add-validators
  (val-opts :required :numeric :num-days))

(add-hooks
  (when-local-test)
    (diff-test-data ["RESULTS" "results"]))

(defcluster marcs-cluster
  :num-instances 1
  :master-instance-type "m1.large"
  :my-root "/Users/mlimotte/projects/lemur/tmp"
  :upload ["./marcs-input.txt" :to "${data-uri}/input.txt"]
  :test-uri "${my-root}/work"
  :keypair "my-keypair"
  :bucket "lemur.marc"
  :jar-src-path "${my-root}/lemur-test-0.0.1-SNAPSHOT-standalone.jar")
(defstep marcs-step
  :main-class "lemur_test.marcs"
  :args.days #(:num-days %)
  :args.stations nil
  :args.data-uri true)

(fire! marcs-cluster marcs-step)
Lemur is described in more detail in this post and full documentation on the wiki at the project site.   I'm hoping that the discussion below does not require the reader to be fully familiar with that documentation, but I provide the link if you want to look deeper.

Domain Specific Language (DSL) Design

For Lemur, I wanted to create an internal DSL.  That is I wanted a DSL that could take advantage of the complete host language for name-spacing, functions, conditionals, looping, etc.  Some of the decisions I made in designing the Lemur DSL follow.

Inheritance (use-base) 

I wanted a way to "inherit" options and functionality.  The use case here is that there are probably many things that your various jobs have in common (i.e. details about your cluster and environment).  These things may be company wide, and they may also be at the team level.  Jobdefs can import base files, which should be included via a (use-base) statement.  The base files are very similar to jobdefs and can include most of the same features using the same syntax/semantics.

Multiple base files can be included (each one is merged, in order, so that the last one mentioned takes precedence).  I discourage chains of inheritance (i.e. JobdefA inherits Base1, which inherits Base2), instead JobdefA inherits Base1 and Base2.  My thought here is that chaining them separates the jobdef writer further from the inherited behavior, where I wanted them to be only one step away from all the options/functionality that might be included (hoping to limit questions like "Where is X specified?")

'use-base' is really an alias for 'require', which is a generic Clojure mechanism for importing code.  I chose to give it a new name in order to highlight the intention when it is used.

Design Consideration: Inheriting functionality is important from a pragmatic standpoint with regard to the DRY principle, but it can make it more difficult to figure out what's going on.  Try to limit the number of places, and be explicit about where users need to look in order to figure out what's going on.

Define-then-kick-off

Lemur follows a model where everything is defined first before any activity is executed.  In this way, various parts of the flow have access to the entire context before they have to take action.  In addition, I want user written functions (hooks, validations, functions to derive config values, etc.) to have access to any of this context.

Design Consideration:  Defer resolution as late as possible.  This lazy approach gives you the most flexibility. 

Global Context

There was a recent post to the Clojure Google Group by Jeffrey Straszheim about his project, The Kiln, which uses the term ball-of-mud.  Meaning that rather than having a very neatly defined set of functions with a handful of well known inputs and outputs, there are a class of problems that require access to this ball-of-mud, because you're not sure what values they may need access to now or in the future.  This situation is exacerbated for a DSL, where you don't know what the user might write.

So, as each bit of the jobdef (or bases) are read in, they are added to the ball-of-mud (I refer to this as the "context").  I use a Map interface for this context so entries can be added with arbitrary names,  multiple Maps can be merged together, and entries can reference other entries in the context.  All these references and merges are resolved at the end (again "before any activity is executed"). 

The context is stored in an Atom.

Design Consideration:  I use a bit of shared state here in order to avoid burdening the DSL user with passing around the context and then capturing the modified context for the next step.  This is a pragmatic choice and works fine as long as Lemur is run as a one-off.  If I ever want to make Lemur into some sort of service with multiple threads and a persistent JVM, than this will need to change.

One of the last things done before real actions are kicked off, is that a data-structure is created from the context, which represents the final result of processing the jobdef.  Data structures in Clojure have a nice printable form, so I can use this structure for dry-run output (it is also saved to a file for later reference, and can be announced on an IRC channel). 

Design Consideration: Think early about how to provide diagnostic output.

Raw Code vs. Hooks

Since the jobdef is a .clj file, you can include arbitrary Clojure code anywhere in the file.  In fact, this was one of my reasons for creating an internal DSL.  After some use, though, I discovered that I wanted Lemur to have some control over those code blocks, in order to:
  1. Avoid execution and provide diagnostic output during a dry-run
  2. Control which code blocks are executed based on the context (i.e. which "profiles" are active). Profiles are just named Maps which might be active or inactive and can override prior values.
  3. Decide when in the execution path the code blocks should be executed
  4. Control how these bits of functionality are "inherited" (via the use-base mechanism)
  5. Ensure that these code blocks get access to the "final" global context
This led me to a hook mechanism.  Hooks allow arbitrary code to be supplied in functions which are handed over to Lemur via an (add-hooks) call.  Arbitrary code blocks are still allowed, but hooks are the preferred mechanism.  This doesn't contradict the goal of an internal DSL... the host language is used to create the functions.  And since the functions are written in a general programming language on top of the JVM, they can do pretty much anything.

Design Consideration: An extension mechanism is nice, but it can be done without completely relinquishing control.

Upload: Declare actions or Declare config

The jobdefs have a declarative way to specify that certain files should be uploaded (to Amazon S3 or to the local file-system) as part of the job set up.  See ':upload [...]' in the sample jobdef above for an example.  I think of this as declaratively specifying what files to upload as part of the config. 

I wrote that functionality before add-hooks. But I think it is more intuitive to think about 'uploads' as an action, rather than a configuration (and therefore, maybe it should be a hook instead).   It is possible now to write a hook to do your uploads, but the :upload option provides some useful syntactic sugar which has not yet been duplicated in a function that could be used as a hook.

Design Point: Don't try to force everything into the same abstraction.  A non-trivial DSL is going to require several abstractions.  Try to determine what they are as soon as you can, so you can slot your functionality into the right one.

Validation

My first implementation let the user supply a single function for doing validation.  A function is infinitely flexible, so this would let the user validate anything.  But in practice, the validation feature was rarely used because the single function was unnecessarily complected.  And these monolithic functions provided very limited room for reuse.

The current implementation, allows the user to supply an unlimited number of functions via (add-validators).  Each function is executed with the full "context" as an argument.  Validation fails if any of the functions fail.  Now I can provide a toolbox of generic validation functions which cover the common situations (e.g. --foo is a required option), and users can write their own, reusable functions as well.

Design Consideration: This is a well known point, but as a reminder, small functions that do one thing well are easier to write, easier to test and easier to reuse.

Why are the validators a collection of functions instead of declarative key/value pairs like defcluster?  After playing with a few examples of what the declarative style would look like, it seemed that while it might make the generic validation declarations a little easier, it made truly custom behavior a little uglier.  And while there is some need for generic validations, many situations (perhaps the majority) involve custom logic.

My compromise was to always take functions for validators, but provide functions that can express common, generic validations in a declarative way as a mini-language.  For example:
(val-opts :required :numeric :num-days)
Which means the 'num-days' option is required and must be numeric.

Design Consideration: Before you implement a feature, write out some examples of how it is used.  Try a few different styles and try to cover a few different use cases in each style.  For the case above, I found that for the use cases I expect, the more pragmatic decision works better than the purer design.

Consistency vs New User Experience

One of the early goals was to make the basic, new-user experience user-friendly; the new user shouldn't have to learn complex syntax which is only needed for advanced features.  I'm not sure this goal was a wise one.  Turns out that there is a lot more functionality needed in order to make the tool interesting and this functionality requires understanding the concepts and abstractions of Lemur.  That learning curve is more the issue than the syntax.

But this goal led me to provide multiple ways of doing things in some cases: a simplistic way with limited functionality and a more full-featured way.  In particular:
  1. (catch-args ...) is a way to specify new command line arguments.  You can do
      (catch-args :foo :bar ...)
    or
      (catch-args [:foo "some doc about foo" "foo's default"] [:bar "doc" "default"] ...)
  2. The :upload directive, discussed above.  Either:
      :upload ["./some-file.txt" "dir/sub-dir" ...]           ;two source locations, dest is a default
    or
      :upload ["./some-file.txt" :to "my-file.txt"            ;this is a src and dest
              ["dir/sub-dir" "${data-uri}/sub-dir"]]         ;this is also a src and dest
And the different styles can be mixed, resulting in ugliness like this:

  :upload ["./some-file.txt" "dir/sub-dir" :to "${data-uri}/sub-dir" ...]
 (catch-args [:foo "doc" 1] :bar ...)

The meaning of these statements with mixed-style is non-obvious.  Perhaps sticking with the slightly more verbose syntax, but being consistent would have been better?

Design Consideration: Weigh the impact of inconsistency very highly in your calculations for user-friendliness.

Final advice for DSL Design

Here's a simple model for design of a DSL.  Start with a layered API, one that defines simple primitives which can be composed for more complex behavior.  The lowest level of the API will match the underlying implementation.  The higher levels will gradually build up more complex abstractions that are more meaningful to the user.  The DSL, then, is a layer of syntactic sugar on top of that, which allows the user to express their needs in a language that is specific to the problem domain.  The user should be able to state what they want, rather than how it should be done and your code will translate that into the proper API calls.

Permalink

Reading BZip2 Files in Clojure

BZip2 compressed files can easily be read in Clojure thanks to Apache Commons Compress. Sample code inside!

Add an Apache Commons Compress dependency to your Leiningen project.clj file like so:

(defproject bz2reader "0.1.0-SNAPSHOT"
  :dependencies [
                 [org.clojure/clojure "1.3.0"]
                 [org.apache.commons/commons-compress "1.4"] ;; Read/write compressed files (BZip2, etc.)
                 ])

Then, you can make a BZip2 capable reader-generating function as follows:

(ns bz2reader.util
  (:require [ clojure.java.io :as io])
  (:import (org.apache.commons.compress.compressors.bzip2 BZip2CompressorInputStream)
   ))

(defn bz2-reader
  "Returns a streaming Reader for the given compressed BZip2
  file. Use within (with-open)."
  [filename]
  (-> filename io/file io/input-stream BZip2CompressorInputStream. io/reader))

This is based on a line from the fs utilites project, which includes a function to uncompress the file on disk. Rather than send the output of our BZip2 stream to a copy function which writes it to disk, we just return the stream for the user to use in the program. You can use the BZip2-enabled stream with any of the normal Clojure I/O methods, just like any other stream.

Here’s an example of how you can print the contents of a BZip2-compressed file to stdout using the above function:

(defn print-bz2-file [filename]
  (with-open [rdr (bz2-reader filename)]
    (doseq [line (line-seq rdr)]
      (println line))))

It’s easy to create similar readers for GZip files, zip files, and so on using the other Apache Commons Compress classes. You can also write BZip2 compressed files similarly, just by piping your regular OutputStream through the encoder.

Permalink

Clojure: Conditionally Importing

I recently ran into a test that needed (org.joda.time.DateTime.) to always return the same time - so it could easily be asserted against. This situation is fairly common, so it makes sense to add support to expectations. However, I didn't want to force a joda-time dependency on everyone who wanted to use expectations. Luckily, Clojure gives me the ability to conditionally import...

Permalink

Runtime Isolation In Immutant

In this article, we're going to take a look at some of Immutant's internals to see how we achieve runtime isolation.

Each application deployed to Immutant gets its very own Clojure runtime. These runtimes are truly isolated from each other - each thinks it is the only Clojure runtime within the JVM. In addition to preventing collisions between application namespaces, it also allows you to optionally use a different Clojure version for each application.

Normally Clojure doesn't support multiple runtimes within the same JVM - the runtime is implemented as static methods/members on the RT class, and there is (usually) only one copy of this class loaded. We'll cover how Immutant achieves runtime isolation given that limitation, along with some of the implications of doing so.

It's (mostly) class loaders

Since the Clojure runtime is implemented as statics on RT, each application needs its own copy of the RT class. To achieve this, we have to do a bit of class loader trickery. Immutant is built on top of JBoss AS7, so we take advantage of the modular isolation provided by jboss-modules.

Under jboss-modules, each module within Immutant (core, web, daemons, messaging, etc.) is isolated from the others, and must explicitly state the packages and classes it exposes along with the other modules it requires. It achieves this by giving each module its own class loader which tightly controls which classes can be loaded by the module. If you are familiar with OSGi, you can think of it as fulfilling similar functionality with regards to isolation, but in a saner and much less complex way. Inside Immutant, each application is considered a module as well and gets its own modular class loader.

Normally, calling Clojure from Java is pretty straightforward:

...
import clojure.lang.RT;
...
RT.var( "clojure.core", "str" ).invoke( "ham", "biscuit" );
...

But, in order to have multiple runtimes, we can't write code quite that simple - any reference to RT from Java will trigger the class to be loaded by the same class loader as the referencing class. In Immutant, this class loader is the one that loads the 'core' module, which is shared by all applications. We really want RT to be loaded only in the application's own modular class loader, which would give each application its own copy. To work around this, we wrap RT in an interface class (ClojureRuntime) and an implementation class (ClojureRuntimeImpl) , and never refer to the implementation class directly.

ClojureRuntime has a bit of AS7 and Java boilerplate in it, but the important parts are:

public static ClojureRuntime newRuntime(ClassLoader classLoader, String name) {
    ClojureRuntime runtime;
    try {
        runtime = (ClojureRuntime)classLoader.loadClass( "org.immutant.runtime.impl.ClojureRuntimeImpl" ).newInstance();
    } catch (Exception e) {
        throw new RuntimeException( "Failed to load ClojureRuntimeImpl", e );
    }

    runtime.setClassLoader( classLoader );
    runtime.setName( name );

    return runtime;
}

public abstract Object invoke(String namespacedFunction);

public abstract Object invoke(String namespacedFunction, Object arg1);

public abstract Object invoke(String namespacedFunction, Object arg1, Object arg2);

public abstract Object invoke(String namespacedFunction, Object arg1, Object arg2, Object arg3);

// ...and many more!

newRuntime is responsible for loading an instance of ClojureRuntimeImpl in the given class loader, which will be the application's modular class loader. This class loader is handed off to the ClojureRuntime instance (we'll see why in a bit).

invoke is the real interface into Clojure - all calls from Java go through it. Its implementation handles looking up the given function Var and invoking it. It comes in 22 varieties to match the arities of Var#invoke.

Its implementation looks like this (we'll only look at one of the invoke definitions, they are all the same except for arity):

public Object invoke(String namespacedFunction, Object arg1, Object arg2) {
    ClassLoader originalClassLoader = preInvoke();
    try {
        return var( namespacedFunction ).invoke( arg1, arg2 );
    } finally {
        postInvoke( originalClassLoader );
    }
}

It looks almost identical to the RT.var() example above. The interesting bits here are the preInvoke and postInvoke methods. These two methods are responsible for setting the thread context class loader (TCCL) to the application's modular class loader before invoking the function and restoring the original TCCL afterward (Clojure uses the TCCL internally for its class loading). Threads used for deployment and web request handling in Immutant come from pools that are shared across all applications, so we have to set the TCCL for the thread on each invocation since we don't know where it has been.

In addition to restoring the TCCL, postInvoke handles the removal of a couple of Clojure's internal thread locals - we'll explore why in the next section.

It's also thread locals

Clojure uses ThreadLocals in a few places (Agent.nested, LockingTransaction.transaction, Var.dvals) to manage state, and doesn't explicitly remove them from the current thread when they are no longer needed. In two cases (LockingTransaction.transaction, Var.dvals) not removing the ThreadLocal causes a hard reference to a Clojure class to be retained in the thread's ThreadLocalMap, which holds a reference to its class loader, which in turn holds references to every single class that it loaded. This prevents any of those classes from being unloaded, so every application undeploy results in a chunk of memory that cannot be garbage collected. If you undeploy and redeploy enough, you'll eventually exhaust the heap, which isn't good for anyone.

In addition to making your heap very tired, leaving thread locals set in thread from a global that will possibly be next used by another application is a security risk, since it is possible for that other application to access the thread local data.

To prevent these issues, postInvoke calls remove on each of these thread locals after each invocation. Take a look at ClojureRuntimeImpl if you are interested in how this is implemented.

Why doesn't Clojure remove these thread locals itself?

Mainly because it hasn't needed to. It wasn't designed to allow more than one runtime per JVM, so the security aspect of shared threads wasn't an issue. And since the leaks are caused by retained classes, they aren't a concern in a single runtime environment since there will only ever be one copy of the class.

Performance

All this muckery does come at a price - invoking through ClojureRuntime is slightly slower than invoking a Var directly. It adds a ~60 nanosecond overhead to each call based on this simple benchmark:

>>>> RT.var.invoke: Execution time for 1000000 calls: 162.405ms (162ns/call)
>>>> ClojureRuntime.invoke: Execution time for 1000000 calls: 219.0318ms (219ns/call)

For every single immutant component (messaging, caching, daemons, etc) except web, we only invoke through ClojureRuntime at deployment. For the web component, we invoke on every request, which means an extra 60ns per request.

Fare thee well

We hope you've enjoyed this peek at some of the guts of Immutant. If you have any comments or questions, feel free to leave them below, or get in touch via the usual methods.

Permalink

Distributed Actors in Clojure

Here's another post on a topic that have been discussed since the dawn-of-time, is there is nice and idiomatic way to write Erlang/Actor style distributed programs in Clojure? There has certainly been a few attempts, but Rich's post (above) still holds true today.

First some clarification; I am not primarily thinking about number-crunching, map/reduce-y stuff, where Clojure has a pretty good story;

Akka and the Erlang legacy

I am trying to write programs that solve problems in the areas where Erlang typically excels such as;
  • Event-driven, asynchronous, non-blocking programming model
  • Scalability (location transparency etc)
  • Fault tolerance (supervisors, "let it crash")
The closest we've got on the JVM is Akka, which claims to have all features (and more) listed above. Akka is the "killer app" for Clojure's sister-language Scala, and is very feature rich and performant. Levering it's power in a safe and idiomatic way is certainly appealing.

However, interfacing to Akka from Clojure is not nice, and certainly not idiomatic. Some work is clearly needed in order to improve Akka/Clojure interrop. The bigger question is if it's worth pursuing? Even if the interrop is made as pain-free as possible, how badly will it clash with Clojure's underlaying design and philosophy? For instance; Akka comes with a STM, how nasty will that be when used in conjunction with Clojure's own?

Wishful thinking

Ideally, Clojure should support distributed actors in it's core, that looks, behaves and interrops nicely with it's other concurrency primitives. It's pretty easy to create a ideal-world straw-man for how this might look from a code/syntax perspective; Termite is a good place to start. Here is a cleaned-up version of the hello-world examples in the gist above.

Many problem arises, serialisation is a big one. Since Clojure's data structures can contain "anything", like Java objects, some limitations needs to be applied to strike a good usability / performance balance. Limiting the stuff you can distribute amongst actors to mimic Erlangs atoms/lists/tuples are probably a fair trade off (all you need is a hashmap right?), and maybe baking in Google Protobuf for efficiency.

For data transport / socket stuff, I'd vote for using a message queue such as 0MQ or maybe even RabbitMQ, this would simplify and empower matters greatly.

With all that in place, it would be possible to build Clojure equivalents of Erlang's OTP, Mnesia etc, now that's a world I want to live in! :)

More reading

  • Learn you some Erlang for Great Good
    Quickly get into the Erlang frame of mind
  • A vision for Erlang-style actors in clojure-py
    Part1 and Part2
  • Exlir
    ErlangVM language with support for Lisp-style macros
  • Joxa
    Clojure-style Lisp for the ErlangVM
  • Avout
    Distributed STM for Clojure, for synchronously updating of shared state.
  • Jobim
    An attempt to mimic the Erlang programming model in Clojure

Permalink

nyc clojure meetup notes

On Wednesday, I went to my first NYC Clojure meetup, which took place at Google's 8th ave offices. I got there way too early, and ended up standing around in the lobby with Rich Hickey waiting to get in the building. There were three speakers presenting at the meetup; Rich was there to talk about the new clojure reducers library, David Nolen gave a talk on ClojureScript performance, and and Kovas Boguta talked about the future of REPLs on the web.

David Nolen, one of the key developers on clojure script, started off by showing us some JVM vs V8 spectral norm benchmarks, showing that clojure script is quite fast. He talked a bit about protocols in clojure script, and how they use the "satisfies?" method to determine if an object conforms to a protocol. The clojure script developers have implemented satisfies using bitmasking, because bitwise operations in javascript are extremely fast. I hadn't used clojure script at all before this talk, but am definitely inspired to check it out.

Next up was Kovas Boguta, talking about the future of the REPL. Long story short, Kovas has built an extremely cool web based REPL that works with both clojure and clojure script. He demonstrated the mathematica "REPL", an extremely sophisticated environment that's built from a very simple concept: the symbolic representation of an expression doesn't need to be the same as the display, and in fact separating the two can lead to a considerably more natural programming environment. The closest analogy in the traditional coding world I can think to this concept is interface builders in xcode and visual studio, that show you a graphical representation of a UI element, but it's backed by C#/objective-C code all the same. Kovas mentioned how crucial tag literals were to this kind of implementation, something I haven't really grokked yet and need to explore further.

The final talk of the evening was the author of clojure, Rich Hickey, talking about the new reducers library. Rich noted that it used to be that the easiest way to make your code faster was to wait 18 months and buy yourself some new hardware. This is an old joke, but his point was this is no longer true, we are adding cores now, not clock cycles. The inherent sequential nature of programs, even in functional programming, does us no favors. The idea behind reducers is to decouple a lot of the "how" aspects of higher order functional manipulation: map, filter, fold, functions that are fundamentally using reductions in some way. Rich has blogged about reducers extensively, and does a far better job than me of explaining, so I won't rehash what he talked about. He did come up with a very cool analogy to illustrate how order is often unneeded, using a bag of apples. A lot of times, you just want to filter out all the rotten apples (filter), or take all the stickers off the apples (map), you don't care a thing about order in this case. Yet, map and filter give you order, like it or not. If you're baking a pie, you might not even care about the full bag of apples, just the ones you need, but you're going to get the full bag anyways. Anyway, I am going to do a poor job of rehashing Rich's talk if I go any further, I hope it's posted online so we can re-watch and try to understand a bit more. The endgame of his lecture was a paralellized fold, that was very cool to see.

I'll definitely be going to these meetups in the future, the speakers were all very impressive, and it's really interesting to talk to folks about what they are working on.

Permalink

An orange glow

I'm happy to announce that as of yesterday Light Table has joined a host of amazing startups for the Summer 2012 batch of YCombinator. Under the guidance of YC we believe we have the best possible chance to not just make Light Table a reality, but to help craft a new future for the way we create software.

Light Table is a bit more orange

This does, however, mean that circumstances have changed. We rallied together as a community to ensure that this project had a future. While being a part of YC doesn't affect our plans or change our use of Kickstarter as a means of accelerating the release of Light Table, many pledged with the explicit purpose of making sure that it had a chance. With YC that chance is now safe and we wanted to let our supporters know there will be no hard feelings if this changes your desire to pledge. We want to thank you all the same for being a part of the push that has gotten us this far. In the end, the important part is that we were able to come together around this ideal and now we can work to make it a reality.

Thank you to everyone who has helped get us here and I look forward to showing you some of the work we've been doing in the next few days.

Permalink

Starting Clojure

I’ve wanted to put together a long-form introductory Clojure screencast for some time.  I had an opportunity to do this in grand style yesterday in a live O’Reilly webcast, but, for various reasons, I wasn’t fond of how that came together.  So, I cut another live coding screencast that introduces, in various levels of detail:  Preview Text:  ...

Permalink

Syntactic Pipelines

Lately I’ve been thinking about Clojure programs written in this “threaded” or “pipelined” style: 

(defn large-process [input]
  (-> input
      subprocess-one
      subprocess-two
      subprocess-three))

If you saw my talk at Clojure/West (video forthcoming) this should look familiar. The value being “threaded” by the -> macro from one subprocess- function to the next is usually a map, and each subprocess can add, remove, or update keys in the map. A typical subprocess function might look something like this: 

(defn subprocess-two [data]
  (let [{:keys [alpha beta]} data]
    (-> data
        (assoc :epsilon (compute-epsilon alpha))
        (update-in [:gamma] merge (compute-gamma beta)))))

Most subprocess functions, therefore, have a similar structure: they begin by destructuring the input map and end by performing updates to that same map.

This style of programming tends to produce slightly longer code than would be obtained by writing larger functions with let bindings for intermediate values, but it has some advantages. The structure is immediately apparent: someone reading the code can get a high-level overview of what the code does simply by looking at the outer-most function, which, due to the single-pass design of Clojure’s compiler, will always be at the bottom of a file. It’s also easy to insert new functions into the process: as long as they accept and return a map with the same structure, they will not interfere with the existing functions.

The only problem with this code from a readability standpoint is the visual clutter of repeatedly destructuring and updating the same map. (It’s possible to move the destructuring into the function argument vector, but it’s still messy.)

defpipe

What if we could clean up the syntax without changing the behavior? That’s exactly what macros are good for. Here’s a first attempt: 

(defmacro defpipe [name argv & body]
  `(defn ~name [arg#]
     (let [{:keys ~argv} arg#]
       ~@body)))

(macroexpand-1 '(defpipe foo [a b c] ...))
;;=> (clojure.core/defn foo [arg_47_auto]
;;     (clojure.core/let [{:keys [a b c]} arg_47_auto] ...))

That doesn’t quite work: we’ve eliminated the :keys destructuring, but lost the original input map.

return

What if we make a second macro specifically for updating the input map? 

(def ^:private pipe-arg (gensym "pipeline-argument"))

(defmacro defpipe [name argv & body]
  `(defn ~name [~pipe-arg]
     (let [{:keys ~argv} ~pipe-arg]
       ~@body)))

(defn- return-clause [spec]
  (let [[command sym & body] spec]
    (case command
      :update `(update-in [~(keyword (name sym))] ~@body)
      :set    `(assoc ~(keyword (name sym)) ~@body)
      :remove `(dissoc ~(keyword (name sym)) ~@body)
      body)))

(defmacro return [& specs]
  `(-> ~pipe-arg
       ~@(map return-clause specs)))

This requires some more explanation. The return macro works in tandem with defpipe, and provides a mini-language for threading the input map through a series of transformations. So it can be used like this: 

(defpipe foo [a b]
  (return (:update a + 10)
          (:remove b)
          (:set c a)))

;; which expands to:
(defn foo [input]
  (let [{:keys [a b]} input]
    (-> input
        (update-in [:a] + 10)
        (dissoc :b)
        (assoc :c a))))

As a fallback, we can put any old expression inside the return, and it will be just as if we had used it in the -> macro. The rest of the code inside defpipe, before return, is a normal function body. The return can appear anywhere inside defpipe, as long as it is in tail position.

The symbol used for the input argument has to be the same in both defpipe and return, so we define it once and use it again. This is safe because that symbol is not exposed anywhere else, and the gensym ensures that it is unique.

defpipeline

Now that we have the defpipe macro, it’s trivial to add another macro for defining the composition of functions created with defpipe: 

(defmacro defpipeline [name & body]
  `(defn ~name [arg#]
     (-> arg# ~@body)))

This macro does so little that I debated whether or not to include it. The only thing it eliminates is the argument name. But I like the way it expresses intent: a pipeline is purely the composition of defpipe functions.

Further Possibilities

One flaw in the “pipeline” style is that it cannot express conditional logic in the middle of a pipeline. Some might say this is a feature: the whole point of the pipeline is that it defines a single thread of execution. But I’m toying with the idea of adding syntax for predicate dispatch within a pipeline, something like this: 

(defpipeline name
  pipe1
  ;; Map signifies a conditional branch:
  {predicate-a pipe-a
   predicate-b pipe-b
   :else       pipe-c}
  ;; Regular pipeline execution follows:
  pipe2
  pipe3)

The Whole Shebang

The complete implementation follows. I’ve added doc strings, metadata, and some helper functions to parse the arguments to defpipe and defpipeline in the same style as defn. 

(def ^:private pipe-arg (gensym "pipeline-argument"))

(defn- req
  "Required argument"
  [pred spec message]
  (assert (pred (first spec))
          (str message " : " (pr-str (first spec))))
  [(first spec) (rest spec)])

(defn- opt
  "Optional argument"
  [pred spec]
  (if (pred (first spec))
    [(list (first spec)) (rest spec)]
    [nil spec]))

(defmacro defpipeline [name & spec]
  (let [[docstring spec] (opt string? spec)
        [attr-map spec] (opt map? spec)]
    `(defn ~name
       ~@docstring
       ~@attr-map
       [arg#]
       (-> arg# ~@spec))))

(defmacro defpipe
  "Defines a function which takes one argument, a map. The params are
  symbols, which will be bound to values from the map as by :keys
  destructuring. In any tail position of the body, use the 'return'
  macro to update and return the input map."
  [name & spec]
  {:arglists '([name doc-string? attr-map? [params*] & body])}
  (let [[docstring spec] (opt string? spec)
        [attr-map spec] (opt map? spec)
        [argv spec] (req vector? spec "Should be a vector")]
    (assert (every? symbol? argv)
            (str "Should be a vector of symbols : "
                 (pr-str argv)))
    `(defn ~name
       ~@docstring
       ~@attr-map
       [~pipe-arg]
       (let [{:keys ~argv} ~pipe-arg]
         ~@spec))))

(defn- return-clause [spec]
  (let [[command sym & body] spec]
    (case command
      :update `(update-in [~(keyword (name sym))] ~@body)
      :set    `(assoc ~(keyword (name sym)) ~@body)
      :remove `(dissoc ~(keyword (name sym)) ~@body)
      body)))

(defmacro return
  "Within the body of the defpipe macro, returns the input argument of
  the defpipe function. Must be in tail position. The input argument,
  a map, is threaded through exprs as by the -> macro.

  Expressions within the 'return' macro may take one of the following
  forms:

      (:set key value)      ; like (assoc :key value)
      (:remove key)         ; like (dissoc :key)
      (:update key f args*) ; like (update-in [:key] f args*)

  Optionally, any other expression may be used: the input map will be
  inserted as its first argument."
  [& exprs]
  `(-> ~pipe-arg
       ~@(map return-clause exprs)))

And a Made-Up Example

(defpipe setup []
  (return  ; imagine these come from a database
   (:set alpha 4)
   (:set beta 3)))

(defpipe compute-step1 [alpha beta]
  (return (:set delta (+ alpha beta))))

(defpipe compute-step2 [delta]
  (return
   (assoc-in [:x :y] 42)  ; ordinary function expression
   (:update delta * 2)
   (:set gamma (+ delta 100))))  ; uses old value of delta

(defpipe respond [alpha beta gamma delta]
  (println " Alpha is" alpha "\n"
           "Beta is" beta "\n"
           "Delta is" delta "\n"
           "Gamma is" gamma)
  (return)) ; not strictly necessary, but a good idea

(defpipeline compute
  compute-step1
  compute-step2)

(defpipeline process-request
  setup
  compute
  respond)

(process-request {})

;; Alpha is 4 
;; Beta is 3 
;; Delta is 14 
;; Gamma is 107

;;=> {:gamma 107, :delta 14, :beta 3, :alpha 4}

Permalink

On Lisp in Clojure, chapter 10

I am continuing to translate the examples from On Lisp by Paul Graham into Clojure. The examples and links to the rest of the series can be found on github.

Chapter 10 presented some interesting problems. Some of them were the result of rampant mutability, and I have skipped those, but much of the rest applies to Clojure.

Stuart Halloway also has written a post on this chapter on his blog.

Section 10.1 Number of Evaluations

The difference between the correct version of the for loop and the multiple evaluation version is pretty straight forward. By binding ~stop to gstop#, stop only gets evaluated once. 


;; correct version
(defmacro for' [[var start stop] & body]
  `(let [~var (atom ~start) gstop# ~stop]
     (while (< (deref ~var) gstop#)
         ~@body
         (swap! ~var inc))))

;; subject to multiple evaluations
(defmacro for' [[var start stop] & body]
  `(let [~var (atom ~start)]
     (while (< (deref ~var) ~stop)
       ~@body
       (swap! ~var inc))))

The the problem with the last version of the for loop really belongs in the next section.


;; incorrect order of evaluation
(defmacro for' [[var start stop] & body]
    `(let [gstop# ~stop ~var (atom ~start)]
       (while (< (deref ~var) ~stop)
         (swap! ~var inc)
         ~@body)))

Section 10.2 Incorrect Order of Evaluation

When I read the for macro labeled as having the incorrect order of evaluation, I thought Graham meant that the counter was being incremented at the top of the loop, and so I wrote my loop that way, and thought it was kind of a silly example. Then I tried the call to for in this section. First, Graham shows us an example where the order of evaluation gives an interesting result. 


(def x (atom 10))
(+ (reset! x 3) @x)
;; => 6

In the version of the for loop with the incorrect order of evaluation, the stop variable appears first, so that gets evaluated, which sets x to 13. Then, start gets bound to the value of x, which is now 13, so the loop never actually runs. In the correct version of the for loop, in the let expression, the start is bound first, to 1, and then stop is bound to 13. I think Graham is right when he says that a caller has a right to expect start to be evaluated before stop because they appear left to right in the argument list. He is definitely right when he says this is a pathological way to call for'. 


(let [x (atom 1)]
  (for' [i @x (reset! x 13)]
        (println @i)))

Section 10.3 Non-functional Expanders

This section shows a lot of awful things that can happen when you mix macros and mutation. But since we are in Clojure, we already avoid mutation when possible. Moving on...

Section 10.4 Recursion

Graham shows us how write a function with recursion and then shows us how to rewrite the same function in a more imperative manner. He does this because in this section he shows a potential pitfall of recursion in macros, and the imperative loop is an alternative.

The imperative version of our-length is a little extra painful in Clojure. Rather than mutating the list and the counter in a while loop, I am going to stick with the recursive function. We will just be careful when we recurse in macros.



(defn our-length [x]
  (loop [lst x acc 0]
    (if (empty? lst) acc
        (recur (rest lst) (inc acc)))))

Graham's ntha function works just fine. Rewriting it as the macro, nthb causes an infinite loop in the macro expansion.

 
(defn ntha [n lst]
  (if (= n 0)
    (first lst)
    (ntha (- n 1) (rest lst))))

(defmacro nthb [n lst]
  `(if (= ~n 0)
     (first ~lst)
     (nthb (- ~n 1) (rest ~lst))))

(macroexpand-1 '(nthb 2 [1 2 3 4 5]))
Graham shows a couple of ways to rewrite nth as a macro that doesn't lead to an infinite loop. I have just rewritten the version with the recursion in the macro. 

(defmacro nthe [n lst]
  `(loop [n# ~n lst# ~lst]
     (if (= n# 0)
       (first lst#)
       (recur (dec n#) (rest lst#)))))

Graham also presents a pair of examples of writing a an `or` macro that sidestep the pitfalls of a recursive macro. In the first, the macro calls a recursive function. The second macro does its own recursion. This seems to be less difficult to do safely in Clojure, because recursion is done with `recur` rather than a function calling itself by name. 


(defn or-expand [args]
  (if (empty? args)
    nil
    (let [sym (first args)]
       (if sym
         sym
         (or-expand (rest args))))))

(defmacro ora [& args]
  (or-expand args))

(defmacro orb [& args]
  (loop [lst args]
    (if (empty? lst) false
        (let [sym (first lst)]
          (if sym sym (recur (rest lst)))))))

Permalink

in which three programming methods are compared

There are, roughly speaking, three ways to develop large user-facing programs, which we will refer to here as 0) the Unix way, 1) the Emacs way, and 2) the wrong way.

The Unix way has been expounded upon at length many times. It consists of many small programs which communicate by sending text over pipes or using the occasional signal. If you can get away with using this model, the simplicity and universality it offers is very compelling. You hook into a rich ecosystem of text-based processes with a long history of well-understood conventions. Anyone can tie into it with programs written in any language. But it's not well-suited for everything: sometimes the requirement of keeping each part of the system in its own process is too high a price to pay, and sometimes circumstances require a richer communication channel than just a stream of text.

This is where the Emacs way shines. A small core written in a low-level language implements a higher-level language in which most of the rest of the program is implemented. Not only does the higher-level language ease the development of the trickier parts of the program, but it also makes it much easier to implement a good extension system since extensions are placed on even ground with the original program itself. I wrote about this in an earlier post on the live-development model Emacs offers:

If you have to use some tacked-on "plugin mechanism" to customize it, then you’re going to be limited at the very least by the imagination of the author of the plugin mechanism; only the things he thought you would want to do with it are doable. But if you’re using the exact same tools as the original authors were using to write the program in the first place, you can bet they put all their effort into making that a seamless, powerful experience, and you'll be able to access things on an entirely new level.

-in which a subject is attempted to be approached objectively, though such a thing is actually impossible

It's worth noting that this is the model under which Mozilla is developed. The core Mozilla platform is implemented mostly in a gnarly mash of C++, but applications like Firefox and Conkeror are primarily written in JavaScript, as are extensions. Following the Emacs way accounted for Firefox's continuing popularity even back when it was getting trounced by competitors in terms of JavaScript performance. Chrome's extension mechanism is laughably simplistic in comparison.

Finally for completeness sake, the wrong way is simply to write a large monolithic application in a low-level language, usually C++. Often half-hearted attempts at extension mechanisms are bolted on to programs developed this way, (usually in order to check off another box on a features list) but they are invariably frustrating and primitive and don't end up offering extension developers the same access to program internals that the developers of the original program itself have.

The Unix way makes particularly explicit the notion of composing small programs, but the Emacs way shines when a single runtime process plays host to a number of independent programs that can interact with each other gracefully. For instance, the Magit version control interface can run in the same Emacs instance as a SLIME session controlling a lisp project. They coexist in a complimentary way and compose together without interference. So rather than saying there are three ways to write large user-facing programs, it might be more accurate to say that there are zero good ways to write large user-facing programs and two ways to compose a number of small programs into a coherent system.

This is especially interesting to me right now since it has come to my attention that when it was rewritten in the transition from version 2 to version 3, GNOME has switched to the second way via an embedded JavaScript runtime, which means things are about to get very interesting.

Permalink

BrowserChannel and the Rijksmuseum API at The Next Web HackBattle

BrowserChannel and the Rijksmuseum API at The Next Web HackBattle

The last The Next Web conference in Amsterdam on the 23rd to 25th of April 2012 included a two day HackBattle. During this HackBattle I built a website combining the API from the Rijksmuseum with my clj-browserchannel library.

The Rijksmuseum API contains all the works from the museum in digital format, with loads of metadata. The website I build displays a selection from these works and allows people to add 'notes' to the paintings. When a note is placed, everybody looking at that painting will see the note pop-up on the site. The idea behind this was that this could be used in a group tour setting through the museum. Of course the real-time aspect of the website is provided by clj-browserchannel. This group tour aspect is also the motivation behind the simple design of the website, which makes it usable on at least Android smartphones.

Screenshot of a note on a painting:


How big?!

Part of the metadata of a painting is its size. Rather than just including this information in text, the website has a 'How big?!' button. When 'How big?!' is enabled a painting is shown with a common object as an overlay, in the proper scale. This is meant to communicate the real-life size of the work. For smaller paintings a soccer ball is used for the comparison. The large 'Nachtwacht' by Rembrandt uses a Dutch bicycle to show its dimensions.

Van Gogh's portrait with a soccer ball:

This project made me able to complete my goal for the conference: plugging Clojure on stage. This was done during the 1-minute presentations of the HackBattle entries on stage in the Green Room.

The code is here: rijks-hackbattle. Many thanks to the kind people from TNW and the Rijksmuseum for organizing the HackBattle.

Permalink

The Future of Web Dev is (almost) here!

Hoverboard resized 600

Hoverboards probably won't be on the shelves until 2015, or possibly 2014, but alpha versions of amazing, bleeding-edge web development frameworks are available today!

Recently, there have been some exciting developments in this space. Several teams are working on their own solutions with the goal of drastically simplifying the process of developing a web application.

Derby

Meteor

Firebase

SocketStream

Capsule & Thoonk

Space Magic

These frameworks boast a few common features I am very excited about:

Sharing code between client and server

I feel terrible every time I have to duplicate model validation logic in both a server-side language and javascript. These new frameworks are based on javascript, or languages that compile to javascript (like coffeescript), which means everything you write can be run on the server or in the browser.

Notably, clojure, with the invention of clojurescript, also has the capability to compile to javascript, although I am not aware of anyone working to leverage the concept of real-time model syncing and live binding into a framework. I would love to see this.

Automatic real-time model syncing and view rendering

Every web app I've written has had semi-specialized code dedicated to translating UI interactions into changes to a data model, and then communicating those changes to and from the server. These frameworks allow you to define UI-model bindings declaritively and then handle propagation between UI, model and server automatically and in real time. And, if one client makes a change to a model that is also powering another client's UI, the second client receives the updated model data and has their UI re-rendered automatically as well. These are powerful ideas and will simplify or eliminate many of the typical complications that currently hinder web development.

What's the catch?

The catch is that these technologies are still very alpha. They are either missing important features, very buggy, or regularly making breaking changes to their APIs. I've spent some time with Derby in particular, and it's definitely bleeding-edge (because I got cut, bad). But, that hasn't lessened my strong belief that the fantastic ideas implemented in these solutions will be part of mainstream web development in the near future. I'm rooting for all of these guys.

Permalink

Against the Grain: How We Built the Next Generation Online Travel Agency using Amazon, Clojure, and a Comically Small Team

;; Acknowledgements

Flowers and chocolates to the Room Key team members who contributed.

Extra special thanks and props to Lawrence Krubner for excellent constructive criticism and feedback during the writing of this story.


;; Who Gives a Shit?  …You Might?

You should read this story if you want to learn about the choices I made as CTO at a little startup in Charlottesville, Virginia between late 2007 and 2011.

You should read it if you want to hear about real-world mistakes a CTO made.

You should read it if you’re trying to build your own company.

You should read it if you’re a technology professional interested in new technologies and their impact on real problems.

You should read it if you’re Paul Graham.  Or if you want to be like him.

This epic poem captures, in an off-the-cuff way, much of the story of what happened at hotelicopter over many years.  It features numerous omissions, many exaggerations, some half-truths, and a few lies.  It also contains a few nuggets of information that may be of some interest if you’re out there in the trenches, doing your own startup, or working on some tech.  Namaste.


;; The Ballad of hotelicopter

Once upon a time (circa 2006), two bright, shiny, newly-minted graduates from the University of Virginia’s Darden School of Business decided to launch a startup.  They came up with a doozy of an idea: a mashup of the best of Facebook, Tripadvisor and Kayak.  It would be a hotel metasearch engine that would use social recommendations to find you the best hotel.  It was one hell of an ambitious idea. They won business plan competitions.  They high-fived each other over lattes. They started working on a prototype.  Eventually they got the attention of a serial entrepreneur who had taken his own travel company public a few years earlier, and they managed to convince him to be their lead angel investor.

You would think two smart, earnest and hardworking MBAs would figure out that they’d bitten off too much to chew with this ambitious plan, and they did, eventually, but it took until early 2009.

Not long after arriving at the company in late 2007, I had argued with the founders to adopt a B2B approach, but to no avail.  Mea culpa.

In our first pivot, we reoriented the business, away from the social and reviews aspects, stripping it down to just a Kayak-like metasearch site focused solely on hotels.  “We’ll be the next consumer destination for hotel bookers! w00t!”  That was the story, anyway.

The company raised a sizeable chunk of money from a small group of wealthy angel investors, and re-branded itself as hotelicopter (it was previously known as VibeAgent).

Some of you may recall our 2009 April Fool’s stunt, which to this day stands as the singular best piece of guerilla marketing chutzpah I’ve ever seen.  The former CEO of hotelicopter still gets a tip of my hat for that one!

[http://www.telegraph.co.uk/news/newstopics/howaboutthat/5083003/Millions-of-web-users-fall-for-hotelicopter-April-Fool.html]

Despite my arguments that we concentrate on a B2B model, hotelicopter pressed on with an ambitious plan to build traffic using SEO and SEM. But, for two smarty-pants MBAs, one smarty-pants CTO (myself), and an experienced CMO, no one really stepped back far enough from the day to day to do the math.  Every year, the other competitors in the space spend about a BILLION dollars on marketing.  Our budget was, ummm… about $100,000.  Even the wildly successful flying hotel prank couldn’t save us.

The flying hotels came home to roost about a year later, in early 2010, when our CMO quit and we collectively realized that ::cough:: Colin was right about going B2B.  (You really should listen to me. All the time.)

Over time, I was able to convince our CEO to adopt the customer development methodology we needed to match our agile product development approach.  But that’s a story for another day.

The fun now really began.  The founders and our lead investor handed me all the rope I could carry.  I had more than enough to hang myself and the rest of the company along with me.  I knew we had to be nimble, and we had to build a platform, not a web site.  We needed something scalable, something that could grow easily to the much maligned “web scale”.  It had to support a three-sided platform, with publishers, travelers and hotel suppliers.  We needed a solution that could integrate easily on a spectrum of publisher levels: from white-labeled web sites that we hosted, to portable widget-like search solutions, to API-level integrations.  The platform had to accommodate the stone-age interfaces of hotel suppliers, and the twitch-game timing of web marketing that our publishers demanded.  Oh, and I had to do this with a four person team.

At the time, the company’s technology stack hadn’t evolved much from its prototype: a monolithic LAMP application, slathered in the worst kind of PHP you can imagine, with a giant spaghetti hairtarball of relational data behind it.  In a brief consulting gig I took with them before joining as CTO, I had extracted the core metasearch functionality from the big ball of mud, rewritten it in Ruby using the async IO framework EventMachine, and set it stand-alone.  But the rest of the system looked like a total loss.  Incidental and unwarranted complexity overwhelmed the existing architecture.

You may be aghast that this was the case, but in the 15 years or so that I’ve been working with startups, I can tell you that this state of affairs was absolutely normal.  Par for the course.

For example, at that point, the site ran out of one ginormous subdirectory with hundreds of PHP files scattered like chunks of gorgonzola on your salad, sticking to one another with tenacious glee. There was a “lib” directory, which you think would hold much of the supporting library code, but a good fraction of that actually lived in “site”, and some in “server”.  The previous programming staff had felt it good and worthwhile to roll their own half-assed MVC framework, including a barely-baked library for page caching (which broke and took the site down at regular intervals), and components for database abstraction that only worked with - wait for it - MySQL.  Every single goddamn file was littered with SQL, like bacon bits on this demonic salad.  There was a “log” directory, but the search logs weren’t kept there, they were in “server”.  Etc., etc.  It made you want to eat a gun.

The database was even worse.  “Facebook-meets-Kayak-meets-Tripadvisor” sounded so good during the business plan competitions, but no one knew what they were doing when they built it, and the data model… There was no data model, really.  Hundreds of tables with distressingly similar names.  There were columns in tables that contained string concatenations of fields from other columns in other tables, generally glommed together with pipes or semicolons, or some other ick, except when they weren’t.  There were missing indexes, huge, ponderous unused indexes, replication that worked by sheer luck, and every single fucking field on every table was prefixed with “hotel_”.  It was nightmarish.  “Normal form?”  “Sure, it’s normally a clusterfuck.”

We dubbed this hairy mudball salad “PHP Hell”.

I suppose this could be construed as indirectly throwing rocks at PHP. Hmm.  Yep, that’s pretty much true.  Discuss.

Oh, yeah, lest I forget:  There were no tests.  None.

Faced with the oft-recurring question of “Evolve or Big Rewrite (tm),” I resolved to do the latter.  Risky!  But, at the time it seemed justifiable.  And now in retrospect, it was what saved us.  To their eternal credit, the founders and our lead angel investor gave deep, patient and ongoing support to an anxiety-inducing and time consuming process.

Taking a deeeeep breath, I chucked out the old team, and chucked out our entire code base, from a standing start at the start of 2010.  We left the existing site running, on life support, while we wiped the slate clean.

The time had come for a different class of developers, and we needed them to have the latitude to work to the best of their ability.  One by one I fired the old team members or they left, and in their place I hired veteran, self-starting software craftsmen. Folks who can’t help but code, whose intellectual curiosity is matched only by their desire to make lasting and significant contributions to the success of the company they work at.  Folks who read xkcd, issue pull requests, hack robots after dinner at home, play Mario Kart obsessively, and who have long, hard, passionate arguments about SCRUM tools and unit testing. Bad ass muthafuckas.

This redemptive catharsis played out over the course 2010, with the size of the team remaining more-or-less constant as we went.

I hired programmers based on their previous work.  Everyone had to submit a code sample.  I was hiring craftsmen!  If you were hiring a woodworker to hand build chairs for your dining room, you’d want to see the chairs that he’d made previously, right?  Software is no different.  I was looking for folks who were very, very good, and I didn’t really give a damn what their backgrounds were.  I ended up with a guy with a computer engineering degree who had recently been wielding a soldering iron, a Berkley grad who’d majored in Jazz, a messenger biker, and a Brit whose background was English Lit - The Bard, to be precise.

The next big decision I had to make, after “Big Rewrite”, and “New Team”, was “Traditional Infrastructure or Cloud?”  Our experience with our traditional hosting provider had been good.  Well, as good as it can be.  Frankly, I was tired of trying to guess when we’d run out of space in our rack, and whether or not the next rack over had a 4U or 8U slot available, and blah blah fucking blah.

I considered Rackspace, but previous experience with their cloud offering had been underwhelming.  That really only left Amazon Web Services (AWS).  I confess I didn’t make this decision very scientifically.  Maybe my age and the fact that I’ve done this stuff every damn day for twenty years has baked it all into my subconscious, but I figured we’d go for broke.  Hell, I’d fired the team, hired a bunch of new guns, and we had not a single damn line of source code to start with.  Why not, right?  In for a penny, in for a pound.

Turns out, this decision looks prescient in hindsight too.  Embracing the true nature of on-demand computing infrastructure deeply, fundamentally changes the game of architecting big distributed systems.  When obtaining computes, bandwidth, resources, databases, the whole shmear allllll turns into function calls, the world changes. This realization came by degrees, and as it did, we incorporated this learning aggressively into the evolving architecture.

This shuffling all transpired over the course of just a few months, and concurrently with it, I was constantly sketching and re-sketching the outlines of a highly scalable, modular architecture.  The team kicked the pieces around generally, and worked in earnest on the most critical and/or least-likely-to-change bits.

We kicked into high gear in the Spring of 2010.

As we chugged along, Tom Southall, now our Front-End Development Manager, came up with a nutty idea.  He suggested that we rebuild the application as a single-page javascript app.  Today that seems obviously like the right approach, but in early 2010, it didn’t seem so obvious yet.  We couldn’t find any frameworks worth a damn, no best practices, few resources, and no one who had any more clue how to do that correctly than we did.  I held some skepticism at first, but I had deliberately hired folks who would lean way out on the sharp edge. With a miniscule team, we needed to grab any advantage we could, and leverage the crap out of it.  This was just that sort of opportunity. I’ve always given these software craftsmen wide berth, and this was no exception.  I told him to prove that it would work, and it did. Brilliantly!

Today I love saying “Why spend money on computes and bandwidth to render HTML?  We let the browser do that for us.”  Fine work, Mr. Southall.

Similarly, Matt Mitchell, a senior developer at Room Key now, came to me saying, “Colin, there’s this tool called Solr, and I think it will make searching our hotel database much easier.”  Again, I can only shake my head at what happens when you hire smart people, treat them like adults, and actually listen to what they come up with.  Solr quickly went from a piecewise solution to our searching needs to something far more interesting.  In our problem domain, stale reads of certain data are tolerable, and exploiting that was a lever we could pull.  Eventually, we ended up baking an instance of Solr/Lucene directly into our individual application processes, making it possible to achieve true linear horizontal scalability for the application.

As of this writing, we’re on track to having 9 million uniques a month, from zero at the start of 2012.  We did so with absolutely no fuss, no late nights, no hand wringing, and a laughably small amount of additional opex spend.  But I digress.

With the revelation about Solr, we were able to decouple the front end of the stack, which had stringent performance and scalability requirements, from the back end, into which we could now stuff all of our messy relational data and processing intensive activities.  With a giant wall between them.

At this point, we were still focused on using Ruby/Eventmachine as a cornerstone of our technology stack.  But here we hit a snag.

Yep.  I’m gonna beat up Ruby, because it was a mistake.

I started using Ruby back before there was a Rails.  Before the invention of fire.  (Late 2000.)  No one loved Ruby more than I did, both as an individual practitioner, and as a CTO.  Ruby rocks.

Except.  Yeah, except it doesn’t scale.  ::ducks::

OK, that’s not completely fair.  Ruby does scale.  But it doesn’t scale well, or easily, and in benchmarking and stress testing, I was seeing that we were going to have a use a small truck load of resources at AWS, or spend a bunch of preciousssss developer time making it scale.  It was too expensive.  I wanted high user::machine density, and I didn’t want to have my developers do handstands to get it.

Yeah, I could’ve done lots of things.  I didn’t have time to do that shit.  Not with a four person team.  Not with cash running out, and promises to keep, and no time to work for Ruby.  I needed something that would work for me.  This thing had to be FAST.  It had to drive hardware to the limit without driving us crazy.

The bottom line is that it was too much work to make Ruby go fast enough.

I knew I didn’t want to sacrifice the pure programming joy that Ruby delivers.  Ruby makes smart, intense, SEAL-team-dangerous developers happy.  It’s a great big chainsaw kitana of object deliciousness.  It gets out of your way.  I wanted something that made programming fun.

So, Java was out.  Wayyyyy out.

I also wanted something with maturity, libraries, support… something with gravitas.  Python?  Meh.

I tried Scala, and threw up a little in my mouth.

I looked at Go.  I tried to like it.  Then IO.  Erlang.  Haskel.

Finally, I looked at Clojure.

You can read my blog entry about my satori experience with Clojure; I won’t belabor it from an individual practitioner viewpoint, here. Instead, let me tell you that as the CTO at a cash-strapped startup, Clojure was the answer to a prayer.  Just like Paul Graham says about the averages.  [http://www.paulgraham.com/avg.html]

A little background on our application might help.  Our job is to give prospective hotel bookers a view into what their options are.  At the time we were making the decision to migrate from Ruby to Clojure, the system was using so-called “realtime” rates and availability checks with hotel suppliers to get that information.  That meant that when a visitor conducted a search, we would spin up dozens of individual HTTP requests to hotel supplier sites to get rates and availability data at that very moment.  We’d parse the responses, collate them, and present them back to the UI in just a few seconds.

You might ask why we didn’t cache that information, but suffice it to say there were significant business drivers for that decision.

Managing this concurrent (and long-running) IO was a major theme for us, and Ruby did so reasonably well using EventMachine.  However, we had to normalize the returned data into a single unified data model, and none of our partners had simple (or compact) XML representations of the data, so not only did we have IO issues, but CPU-bound processing issues as well.  The combination of the two made the EventMachine implementation suffer from less-then-stellar throughput, and because of Ruby’s green threads implementation and global interpreter lock, we had to run oodles of Ruby processes on each box to achieve reasonable throughput.

Perhaps just as importantly, the reactor pattern’s upside-down flow of control style of programming was (and is) a pain in the ass.  It was hard to read, hard to maintain, and generally obstreperous.

I can already year you Ruby folks protesting “Fibers!” and so on. Heh.  Have fun storming that castle.

Clojure was a whole different story, and addressed these issues admirably, for all the reasons you’ll discover when you look into it further.  (Hint, hint.)

While the team was wrestling with other pieces of the system, I personally prototyped the piece of our stack with the highest scalability and performance demands using Clojure, and benchmarked. It was immediately obvious that it was a game changer.  Thus we began our journey from being a mostly-Ruby shop to a almost-solely-Clojure shop.

Again, to their credit, hotelicopter’s founders didn’t bat an eyelash.  Lisp, Ruby, blah blah blah.  Just get it done, Colin.

I’m OK with that.

Bear in mind that at this point, we already had a running business - the consumer-oriented hotel metasearch engine.  It was running on the awful PHP code, and we were putting just enough time into it so it wouldn’t completely fall over.

Meantime, we had done the leg-work to figure out what it was we could and should build as the first step towards our new B2B platform.  This new platform was what I had resolved to build in Clojure on AWS.

We began picking off pieces and building them, learning Clojure as we went.  It turned out to be a very steep learning curve, but despite that we were doing reasonably well with the new language and environment within a few months.

Furiously coding away, in true MVP (Minimum Viable Product) style, we launched early customers while still filling in the gaps in the platform.  It worked!  It was fast, reliable, simple, and scalable. It looked like we had a winner.

I’m gonna pause this part of the story, and pull another thread.  I’ll weave them back together shortly.


;; The Saga of Hotel Distribution - Or - How to Boil a Frog

Back in the Good Old Days (tm), before the invention of the Intertubes, to book a hotel you went to see a human being.  A travel agent.

This agent of travel was endowed with special powers.  Namely, the power to access an arcane oracular system known as a GDS - a “Global Distribution System”.  This system allowed the travel agent to search for hotel rates and availability, and to book your rooms!  How exciting, and how quaint!

When a travel agent booked a room for you like this, they were paid a 10% commission by the hotel.

Back then, the Internet was a weird, fringy thing.  The hotel chains and hotel owners figured it was a flash in the pan.  Like laserdisks, or Segas.

The earliest Online Travel Agencies (OTAs) connected to the GDSs to get inventory and book rooms.  But then something else started to happen…

When the hotel suppliers were approached by these fledgling OTAs to sell hotel rooms directly - not through the GDSs - they figured, sure. Why not?  We have some “distressed inventory” - some hotel rooms that we can’t sell, chronically, and we’ll give these online travel agents this cruddy inventory.  We’ll sell it to them wholesale, cuz we’re not gonna make a dime from it any other way.  And they can do what they will with this inventory.

So they started giving inventory directly to the OTAs.  And the OTAs started selling.  Before you knew it, the hotel suppliers started giving the OTAs non-distressed inventory, at a discount.  Not much, ya know.  Just a little.

Bit by bit… Like the frog that doesn’t figure out it should jump out of the cold pan of water on the stove until it’s boiling and too late… The water started getting hotter.  And hotter.

This distribution channel for hotels opened a pandora’s box of problems.  For one thing, the customer was at arm’s length from the hotel.  The hotel didn’t get to shape the customer experience of booking.  The hotel was at the mercy of the OTA as far as being compared to other hotels.  The customer wasn’t exposed to messaging about the hotel’s loyalty program, or even basic branding.  The list goes on and on.

But maybe the very worst part was (and is) how much it cost the hotel. Usually about 30% - three times as much as paying a travel agent’s commission in the Good Old Days.

The poor frogs, er, hotels, were waking up to the fact that they were now hooked on OTAs for distribution.  But, they couldn’t quit the crack cocaine of selling inventory through the OTA channel - it was too much volume.  Too big to fail!  Boiling water!

It got worse.

After 9/11, the bottom fell out of the flights market.  Airlines were scrabbling to stay alive, and commissions for selling airline tickets plummeted.  In a competitive frenzy, one by one, the OTAs stopped charging customers fees for booking airlines.

All of this spelt trouble for the OTAs.  Flights had been the lion’s share of their revenue.  They had quarterly numbers to make.  They needed to make up this lost revenue somewhere… but where?

Oh, yeah.  Let’s get it from the hotels!

It was a bad scene.  Finally, the hotels cried “Uncle!”.  Some of the biggest hotels in the world got together and said, “Let’s do something.  Let’s start our own OTA.  We’ll own it!  It will send visitors directly to our own web site to book! And we’ll be able to set the commission.  Let’s make it low, like it was in the Good Old Days! Huzzah!”  So say we all.

So around the beginning of 2010, they formed a joint venture.  That joint venture, Room Key, needed a technology platform.  One that looked suspiciously like hotelicopter’s.


;; Against the Grain

See, these two subplots DO intersect.

The joint venture began looking for a company to acquire or to partner with, to find a web platform that could grow to meet their ambitions. They looked at hotelicopter.  For a variety of serendipitous reasons, we looked pretty good.  Soon enough, the acquisition process was in full swing, and they put hotelicopter under the microscope.  Actually it felt more like a colonoscopy.

Every single one of the technology choices I made as CTO of a scrappy startup were called into question.  They questioned anything that didn’t fit the model of how they do business.  Ie., everything.  The culture clash was epic.

They wondered if Amazon was reliable.  This might seem like a strange thing, but in their world, the world of proprietary infrastructure, which is remarkably unreliable, it made sense.  Literally, they wanted to know how many nines, how many OC-48s, disaster recovery, etc.  They wondered if Amazon scaled.  They wondered if our architecture scaled. They wondered if our code was fast enough.

When I say “wondered”, here, I mean wondered in the way a dentist pulling an abscessed tooth out of your mouth wonders if he’ll need to stand up and put one foot on the chair to get more leverage, or maybe just use the drill some more?

They questioned the choice of Clojure.  They asked for justification why we wouldn’t be using Java.  They wondered where we’d ever be able to find enough programmers to work on such a fringy language.  They couldn’t understand what Solr was, much less why we used it.  Their collective eyes glazed over at discussions at the genetic algorithms we used to optimize Solr weights.  They disputed my assertions about our ability to test the system.  And so on, and so on, ad nauseam.

The choices I’d made never felt more against the grain than during the due diligence carried out on hotelicopter.  I’ll touch on a couple of the juicy ones.

Regarding Amazon, there were a variety of questions, ranging from “What is it?” to “Isn’t operating your own infrastructure cheaper?” to “Does it really scale?” to “Does your application scale?”  The silly questions were easy enough to rebut, but the last two - “Does Amazon scale?” and “Does your app scale on Amazon?” were persistent and difficult to explain.  To address these issues, it seemed far more effective to “Show, don’t tell,” and so we used developer time to build load and stress tests, and we ran them.  No real surprise, we found a couple of bugs, but the point was amply made that yes, Amazon does in fact work as advertised, and more importantly, our architecture would scale to handle the anticipated load.

Explaining AWS was a pain the butt.  Explaining Clojure was a whole different animal.

Here, the questions bordered on incredulous criticism.  A few choice derisive phrases came up, including “toy language”, “your pet language”, and so on.

There were concerns about finding talent that had a kernel of truth in them.  My response was that I was looking for the veteran software craftsmen I described above, and they were damned hard to find no matter what, and that pretty much anyone we hired would have to be trained to use the language.  I’m not sure that went over so well.

Another question was, “How is Clojure suited to large programming teams?”  My response was that I never intended us to have a large programming team.  Programming in the large is a higher-order “code smell” (“architecture smell”?) that means you’re doing it wrong. Decoupled, distributed systems mean you shouldn’t be worrying about this problem any more.  “Just pass messages.”  I don’t think that was quite what they expected to hear, either.

I laid some of the skepticism to rest once I was able to explain that Clojure was a JVM-hosted language, which meant that much of the Java ecosystem could be leveraged, including debugging tools, profiling, etc.  Although no one said so aloud, I think that they took this to mean that if the acquisition was completed, the system could be migrated to Java.  Heh heh.

Finally, there were questions about the performance characteristics of Clojure.  Those were easy enough to address by pointing to the results of the scaling tests we conducted.

Looking back, I think that two things made the “sell” of Clojure and AWS (and all of our other beating-the-averages decisions) possible: 1) the empirical results we could show, and 2) the fact that Clojure, and hence the system, was hosted on the JVM.  Ultimately I think the former is what sealed the deal; I didn’t have to have *arguments* about the characteristics of our system.  Instead, I could simply point to pretty graphs and charts.

The due diligence dragged on and on, but you already know the punchline.  Eventually we got the thumbs up, and so hotelicopter was acquired by the joint venture in the Fall of 2011, and became Room Key.  I like to think that when that happened, the state of the art for Online Travel Agencies (OTAs) just got a little better.

You might wonder why we didn’t refuse the offer, and continue to remain hotelicopter, free and wildly tipping over apple carts in the hospitality industry.  The truth is that we weren’t making enough money fast enough, and our very, very patient investors had waited long enough.  They wanted out, and I can’t blame them.  We exited handsomely, and although it was no home run, it was a respectable double.  I’m not complaining.

;; Things We Flubbed and Things We Did Right

Wrong

  • Not firing fast enough
  • PHP, then Ruby
  • Not enough research into “What’s out there?” that we could leverage (Solr, machine learning, etc.)


Right

  • Supportive management
  • The right culture
  • Big Rewrite (tm)
  • Rewrite mercilessly (Our core search system is now on its 5th generation.)
  • AWS (Beanstalk, SDB, RDS, Cloudwatch, SNS, S3, MapReduce, mostly)
  • “Risky pieces 1st”  RP1.
  • Judiciously adopt cutting-edge solutions that yield leverage
  • Clojure
  • Solr
  • Genetic algorithms
  • Single-page JS App
  • Roll our own JS framework


There are a few things on hit I didn’t cover above, like our use of genetic algorithms, and for that I apologise.  But this is already a bit of a War and Peace, and if you’ve made it this far you deserve a break.  Thank you for your time and attention, and good luck out there.


;; Quotes From the Characters

“There was no bureaucracy or process or politics to restrain team members from taking risks and doing good work. Of course, everyone was accountable, and if you took a risk, you had to justify it with results. But the important thing was that the culture encouraged and inspired good workers to do their best.”

“Clojure takes a similar world-view. Unlike Java, where two-thirds of what you write is for the compiler, Clojure really gets out of your way. When you write Java, you’re constantly thinking about the bureaucracy/process that the compiler demands. The compiler enforces the kind of oversight and restrictive management that big slow-moving corporate cultures love. In a way, it lets managers manage their coders without having to stand over their shoulders.”

“Clojure, on the other hand, trusts the developer entirely and merely asks him to express his intent. This allows good developers to do really good work. Of course, it also allows bad developers [to] really make a mess. So as an organization using Clojure, you have to make a different choice. Instead of hiring potentially mediocre programmers and throwing them into an environment that polices them, you hire really good developers and trust them.”

— Andrew Diamond, Senior Developer formerly with hotelicopter / Room Key

“I don’t think the value of the AWS infrastructure can be overstated. The number of things we don’t worry about and the number of people we don’t employ would be considered black magic by a good portion of the 1990s computer industry.”

— Chris Hapgood, Senior Room Key Developer

“If you have a gut feeling, go with it … but follow through. I remember when I suggested that we chuck mongo and use Solr only. This was risky, and I paid in sweating bullets… but only for one ridiculously stressful afternoon :) —- totally worth it ha.”

“Team is everything. Everyone we have complements [sic] each other. Personality is critical. Passion is a must. Everyone on our team rocks.”

“Oh, leave your ego at home. Trust me it feels good.”

“Don’t be afraid of a huge challenge or change, embrace it. Even if it hurts at first.”

“Stay positive when around your team members. It’s too easy to complain, and it’s contagious.”

— Matt Mitchell, Senior Room Key Developer

“A good while before I joined hotelicopter and certainly before there were any of today’s burgeoning array of frameworks such as Sammy.js, backbone.js and so on, I recognised that Javascriopt could be leveraged for much more than simple form validation and mouse-overs. It was a powerful and underused tool and potentially an entire application could be built using Javascript living on a single web page.”

“Before joining the hotelicopter team, I had already begun to experiment with rudimentary single-page apps using AJAX to pull in JSON data and which manipulated the browser DOM accordingly. I was already very encouraged by the speedy and responsive user experience that resulted. “

“Then I arrived at hotelicopter and found an old-school server-side app with PHP-generated HTML and lots of sluggish round-trips to the server. The product director John Demarchi and I started to formulate ideas for a next-generation hotel search UI. His existing vision (which I immediately subscribed to) was the idea of what he termed “site-as-application” (or in other words what we would nowadays term a web-app) - a site that looked and responded more like a traditional desktop application.”

“The idea is so commonplace now that it hardly seems extraordinary at all, but at the time it was a relatively unknown and undeveloped concept that had only just started to create buzz among designers and front-end developers. And of course John’s ideas for a UI gelled perfectly with my interest in the idea of a single-page Javascript app that consumed JSON formatted data from an API, thereby completely decoupling it from the back-end. “

“So when the dev team sat down with Colin and started to thrash out what a newly architected search engine might look like, it seemed like a natural fit and so, with his blessing, I set about building the first prototype and consulting with the back-end team on an API… except this germ of an idea eventually grew into something even better. “

“One day, Colin suggested, “why not build this thing to be portable?” and so the UI became not just a single-page web application but an SDK comprising data models and mutually aware UI components that could be placed on any web page anywhere and bring fully featured (and monetisable) hotel content and metasearch functionality to anyone who needed it. The potential was (and still is) enormous.”

— Tom Southall, Room KeyFront End Development Manager

 ;; Executive Summary

  • The right culture. (Build the right team.  Listen to them.  Get out of the way.)
  • Didn’t Evolve.  Did the Big Rewrite (tm).
  • Not our own infrastructure. Amazon.
  • Not PHP.  Not Ruby.  Clojure.
  • Solr.
  • Single-page JS App.
  • Kudos to a patient management team.


;; Postscript

Re-reading this saga, and pondering Andrew’s observation about big company bureaucracy / process, I’m struck that the adage, “Your code will end up reflecting the culture of your company,” (roughly paraphrased) is deeply, profoundly true.  The problems of our work are truly sociological, not technological.

It was a really fun ride, and Room Key is going to be even more fun. I’m still looking for kick-ass developers.  Send me some code.

Permalink

Agile Development with Clojure

If you've ever spent any time learning Rails then you probably read one of the editions of Agile Web Development with Rails, and if you're like me (skeptical & pedantic) then you probably asked yourself: what the hell does Rails have to do with Agile development? At the time, I assumed that Dave and David were merely capitalizing on the buzz around Agile; however, even if that's the case, I think they did manage to highlight one of my favorite aspects to building websites with Rails: The ability to make a change, reload the page and see the results makes you a much more agile programmer - where 'agile' is defined as: Characterized by quickness, lightness, and ease of movement; nimble

It turns out, it's not very hard to get that same productivity advantage in Clojure as well. I would go so far as to say that the ability to change the server while it's running is assumed if you're using emacs+slime; however, what's not often mentioned is that it's also possible (and trivial) to reload your server code (while it's running) even if you're using IntelliJ, scripts, or anything else.

The majority of the servers I'm working on these days have some type of web UI; therefore, I tie my server side code reloading to a page load. Specifically, each time a websocket is opened the server reloads all of the namespaces that I haven't chosen to ignore. The code below can be found in pretty much every Clojure application that I work on.
(defonce ignored-namespaces (atom #{}))

(defn reload-all []
  (doseq [n (remove (comp @ignored-namespaces ns-name) (all-ns))]
    (require (ns-name n) :reload )))
Like I said, when I open a new websocket, I call (reload-all); however, the (reload-all) fn can be called on any event. When discussing this idea internally at DRW, Joe Walnes pointed out that you could also watch the file system and auto-reload on any changes. That's true, and the important take-away is that you can easily become more productive simply by finding the appropriate hook for what you're working on, and using the code above.

The ignored-namespaces are important for not reloading namespaces that don't ever need to be reloaded (user); other times you'll have a namespace that doesn't behave properly if it's reloaded (e.g. I've found a record + protocol issue in the past, so I don't dynamically reload defrecords in general).

The change-reload webpage-test loop is nice for making changes and seeing the results very quickly - and I strongly prefer it to having to stop and start servers to see new functionality.

© Jay Fields - www.jayfields.com

Permalink

Germination X: Player characters

After another code sprint on Germination X, I’ve added player characters (avatars) to the game. Based on the falmouth focus group feedback this seemed one of the major things missing that players felt would improve the game.

The character design came from the Mandrake plant, which has been cropping up in groworld projects for some time – as a magical plant known to resemble human forms as it grows.

This change also allowed a much needed clean up and decluttering of the user interface for the game – as navigation now happens by moving your mandrake around by clicking on things. Your location is persistent, so when you log in you go back to the same place. All existing players in the game have been given mandrakes scattered around the world.

As the mandrakes represented a new type of entity in the world, able to move around, and linked to players – this was quite a lot of work, particularly in terms of updating the existing database. I had a lot of trouble doing this manually with MongoDB’s save command in the script interface. This seemed to be creating duplicate records (and creating very hard to track down bugs) that took a long time to find. The better approach seems to be to upgrade the database automatically in the game code, by checking a stored version number – and mapping over entries like this:

;; map over each player in the db
(db-map!
 (fn [player]
   (println "upgrading" (:name player) "to include tile and avatar")
   ;; get a random tile in the world
   (let [tile (first (db-get-random-one :tiles {}))]
     (db-update! ;; update it, adding an avatar
      :tiles tile
      (merge tile {:entities
                   (cons (make-avatar
                          (:id player)
                          (:name player)
                          (make-vec2      ;; random position
                           (rand-int 5)   ;; in the tile
                           (rand-int 5))
                          (:layer player) ;; show the player's score on the avatar
                          (count (:flowered-plants player)))
                         (:entities tile))}))
     ;; add the tile location to the player
     ;; so we can find the avatar again
     (merge player {:tile (:pos tile)})))
 :players)

This also means that the latest code will work with snapshots I’ve taken of the game world regardless of how old they are. This turns out to be really important – I can try some changes and rewind the whole world back to the same starting point, as well as testing code locally on a copy of the current public world version.

Permalink

understanding clojure concurrency, part 1

This is part 1 of a 2 part series I’m writing about Clojure concurrency

where we are today

We demand more from our computer programs than ever before, and with those demands comes the demand to process more data, to do it faster and do it more efficiently. For decades, programmers writing user-land code have been able to mostly ignore developing with concurrency in mind, instead counting on faster clock speeds to make their programs run faster. We’ve had multi-threaded runtimes for a long time, but the added complexity of writing concurrent code often makes it not worth the effort.

I could talk all about Moore’s law, and how ignoring it leads to all sorts of peril, but I’d rather not go that direction. Instead, I like to think of learning concurrent programming as a chance to maximize the resources I’ve been given as a developer. If I was a truck driver, I would be tasked with keeping my truck as full as possible as I make runs across the country to deliver my goods. If I drove with my truck half empty all the time, you’d say I’m being inefficient. If we want to maximize the resources that hardware makers are creating, we’re going to have to learn to get better at multi-threaded, parallel and concurrent programming. 24 core servers are the norm these days, and utilizing them is really important. To do any less would simply be a waste.

When you think about the means of production and value creation, the goal is to take raw materials and turn them into something more valuable than you started with. If you drill oil out of the ground, the goal is to take that oil and refine it, distribute it, and turn it into something more valuable. Programmers use power, CPUs, silicon and hard drives as raw materials. We combine them with our ideas and our programs to convert the raw materials into something more valuable than what we started with. Creating good software is far more expensive than the hardware part, but why not squeeze the most out of the raw materials we’ve already been given?

As a Rubyist learning Clojure, I’ve grown used to doing concurrency the “Ruby way” - lots of forked processes and minimal control. Headache usually ensues. Sure, Ruby has the JRuby and Rubinius runtimes, which have their own strengths and weaknesses, but both are usually demoted to second class citizens in the Ruby community. Perhaps you’ve worked in a runtime that forces a process level parallelism pattern. After spending some time with Clojure, I’ve realized that it doesn’t have to be this way. Writing concurrent code can be simple, and can help you write programs that do more than you thought was possible.

what’s so hard about concurrent programming?

The general wisdom that I hear about concurrent programming is, “stay away from it unless you know what you’re doing”. Writing truly multi-threaded programs has been an art rarely embraced outside from those writing system level code or perhaps someone looking for extreme levels of pain. One of the big problems concurrent programs face is dealing with shared data. Most parallel and concurrent programs deal with shared data in a way that’s hard to reason about and painful to get correct. Locks, semaphores, mutexes are all important and necessary tools, but they often approach the problems of concurrent programming from the completely wrong angle. Concurrent locks were invented at a time where a differently architected machine ruled the world. Gone are the days of single or dual core computers, and gone are the days when locks will help you squeeze performance out of your programs with ease. Locks just don’t scale the way they used to. It’s time to embrace something different.

why clojure for concurrency?

Clojure addresses all these concurrency headaches with several features.

  1. Persistent, immutable data structures for default collection types
  2. A software transactional memory system
  3. Several semantic language level concurrency primitives

We’ll talk about each one of these items shortly.

This list of features comes baked in to Clojure itself, and as such, they are considered part of the core design of Clojure. Can these features be added to other programming languages with libraries? Yes, they can, but often at a cost to completeness and support. Having concurrency constructs at the language level offers power and flexibility that few libraries can rival. For most other languages, putting these primitives at the library level often means rewriting your application using solid thread-safe data structures (which may or may not exist), and rethinking the flow control of your entire program. In Clojure, switching your code to run concurrently can sometimes mean as little as changing a ‘map’ function to a ‘pmap’! All your data is persistent and immutable by default.

persistent data ptructures

Clojure is able to achieve simple concurrent functionality because its core sequence data structures are persistent. That doesn’t mean persistent like database persistent, but rather persistent in its immutability and its structural sharing between data writes. We’ll go into this in more depth shortly.

most data structures

When you think about most of your core collection based data structures: arrays, hash maps and sets it’s common to think of these as ‘cubby holes’ where your data lives. For example, when you want to switch an item in the middle of an array, you find the memory address for that position and then you change the data at that location. When you mutate the value at a specific location and replace it with another, the value that was there previously is gone forever. Rich Hickey, the creator of Clojure refers to this term as ‘place oriented programming’, where data lives in a certain ‘cubby hole’ in the computer now, and might be replaced with another value at an unknown point in the future.

Why do mutable data structures make it hard to program concurrently? Because to perform a safe write or read operation, the programmer has to be diligent about mutex locking. When a program makes heavy use of locks, it then has to be concerned with incidental complexities like lock ordering, deadlock, lock contention, race conditions, and other nasty problems that are sometimes next to impossible to debug. What makes these impossible to debug? In a multithreaded program, one of the hardest things to reason about is the state of your data. You have to ask yourself questions like, “Is it safe to read this data without acquiring a lock?”, “Am I guaranteed this data won’t change while I’m working with it?”, “Am I allowed to pass this data pointer to another thread safetly?”. All of these questions are incidental complexity that comes from working with mutable data structures. Your end-user doesn’t care that you had to spend weeks getting your code thread safe, they only care that it works fast and lets them do what they want. Let’s give them what they want, shall we?

clojure data structures

How are Clojure’s collection data structures different? Like I mentioned earlier, Clojure data structures are persistent, which means they are both immutable and share structure with previous generations when writes occur.

Let’s talk first about immutability. One of the amazing qualities of immutable data is its benefit in multi-threaded programming. Wondering “Am I guaranteed this data won’t change while my thread is reading it or holding on to it?” becomes a non-issue. Why? Because that piece of data you’re holding a reference to is never going to change. It’s immutable, so reading from it and passing it to another thread are incidental complexities that go away. You can hold on to that piece of data as long as you want, and never have to worry about it changing out from under you. In practice, this means that every time you make a change to a Clojure collection, you’re getting back a new immutable collection. Immutability is a boon to the multi-threaded programmer because it allows read operations to scale with significantly less mental overhead and program complexity.

Clojure’s data structures aren’t just immutable, but are also persistent. With immutability, we know that we get an immutable collection every time we make a modification to an existing collection, but this does not mean that Clojure copies the entire collection every time you make a change to it. Instead, under the hood Clojure does structural sharing between one generation of the collection and the next. Because the Clojure collections use tree structures internally to represent data, only a small amount of data needs to be created or updated because of a change. This is called persistence. If the entire collection needed to be copied every time a write operation occured, performance would take a nose-dive for data that changes quickly. Persistence helps the multi-threaded programmer think about their problem domain instead of worrying about collection performance.

I would argue that immutability resolves most difficult problems of concurrency. When you choose immutability, you easily sidestep many of the pitfalls and perils that await you in an uncontrolled, mutating world.

Let’s think a little bit about how the real world works and how that might relate to parallel and concurrent programming. If you’re watching a professional baseball game you’re ‘reading’ the state of the world without having to worry about who else is reading it. You can happily eat your hotdog and drink your beer and watch the game without having to think about what the other fans are doing. Your eyeball is receiving images at a certain point of time based on the state of the world when you saw it. You don’t have to stop everyone else from observing the game, so only you can watch - that would be just absurd. Instead everyone can saftely watch the game as their own independent person.

When you think about scaling up read operations, thinking in terms of immutability is a huge win. If all you’re doing is concurrent reads, immutable and persistent data structures are usually all you need.

In the next post, I’ll cover some of the other concurrency primitives that Clojure gives you - and how to leverage and use them effectively

Permalink

Anatomy of a Reducer

Last time, I blogged about Clojure's new reducers library. This time I'd like to look at the details of what constitutes a reducer, as well as some background about the library.

What's a Reducing Function?

The reducers library is built around transforming reducing functions. A reducing function is simply a binary function, akin to the one you might pass to reduce. While the two arguments might be treated symmetrically by the function, there is an implied semantic that distinguishes the arguments: the first argument is a result or accumulator that is being built up by the reduction, while the second is some new input value from the source being reduced. While reduce works from the 'left', that is neither a property nor promise of the reducing function, but one of reduce itself. So we'll say simply that a reducing fn has the shape:

(f result input) -> new-result

In addition, a reducing fn may be called with no args, and should then return an identity value for its operation.

Transforming Reducing Functions

A function that transforms a reducing fn simply takes one, and returns another one:

(xf reducing-fn) -> reducing-fn

Many of the core collection operations can be expressed in terms of such a transformation. Imagine if we were to define the cores of map, filter and mapcat in this way:

(defn mapping [f]
  (fn [f1]
    (fn [result input]
      (f1 result (f input)))))

(defn filtering [pred]
  (fn [f1]
    (fn [result input]
      (if (pred input)
        (f1 result input)
        result))))

(defn mapcatting [f]
  (fn [f1]
    (fn [result input]
      (reduce f1 result (f input)))))

There are a few things to note:

  • The functions consist only of the core logic of their operations
  • That logic does not include any notion of collection, nor order
  • filtering and kin can 'skip' inputs by simply returning the incoming result
  • mapcatting and kin can produce more than one result per input by simply operating on result more than once

Using these directly is somewhat odd, because we are operating on the reducing operation rather than the collection:

(reduce + 0 (map inc [1 2 3 4]))
;;becomes
(reduce ((mapping inc) +) 0 [1 2 3 4])

Reducers

We expect map/filter etc to take and return logical collections. The premise of the reducers library is that the minimum definition of collection is something that is reducible. reduce ends up using a protocol (CollReduce) to ask the collection to reduce itself, so we can make reducible things by extending that protocol. Thus, given a collection and a reducing function transformer like those above, we can make a reducible with a function like this:

(defn reducer
  ([coll xf]
   (reify
    clojure.core.protocols/CollReduce
    (coll-reduce [_ f1 init]
      (clojure.core.protocols/coll-reduce coll (xf f1) init)))))

Now:

(reduce + 0 (map inc [1 2 3 4]))
;;becomes
(reduce + 0 (reducer [1 2 3 4] (mapping inc)))

That's better. It feels as if we have transformed the collection itself. Note:

  • reducer ultimately asks the source collection to reduce itself
  • reducer will work with any reducing function transformer

Another objective of the library is to support reducer-based code with the same shape as our current seq-based code. Getting there is easy:

(defn rmap [f coll]
  (reducer coll (mapping f)))

(defn rfilter [pred coll]
  (reducer coll (filtering pred)))

(defn rmapcat [f coll]
  (reducer coll (mapcatting f)))

(reduce + 0 (rmap inc [1 2 3 4]))
;=> 14

(reduce + 0 (rfilter even? [1 2 3 4]))
;=> 6

(reduce + 0 (rmapcat range [1 2 3 4 5]))
;=> 20

From Reducible to (Parallel) Foldable

While it is an interesting exercise to find another fundamental way to define the core collection operations, the end result is not much different, just faster, certainly something a state-of-the-art compilation and type system (had we one) might do for us given sequence code. To stop here would be to completely miss the point of the library. These operations have different, fundamentally simpler semantics than their sequence-based counterparts.

How does one define parallel mapping/filtering/mapcatting etc? We already did! As long as the transformation itself doesn't care about order (e.g. as take does), then a reducer is as foldable as its source. As with reduce, fold bottoms out on a protocol (CollFold), and our reducer can extend that:

(defn folder
  ([coll xf]
     (reify
      ;;extend CollReduce as before

      CollFold
      (coll-fold [_ n combinef reducef]
        (coll-fold coll n combinef (xf reducef))))))

Note that:

  • folder has the same requirements as reducer - collection + reducing function transformer
  • when fold is applied to something that can't fold, it devolves to reduce

Thus the real definitions of reducers/map et al use folder (while take uses reducer):

(defn rmap [f coll]
  (folder coll (mapping f)))

(defn rfilter [pred coll]
  (folder coll (filtering pred)))

(defn rmapcat [f coll]
  (folder coll (mapcatting f)))

Thus a wide variety of collection transformations can instead be expressed as reducing function transformations, and applied in both sequential and parallel contexts, across a wide variety of data structures.

The library deals with several other details, such as:

  • the transformers all need a nullary arity that just delegates to the transformed reducing function
  • the transformers support a ternary arity where 2 inputs are supplied per step, as occurs with reduce-kv and map sources
  • all of the reducers are curried

These additions are all mechanical, and are handled by macros. It is my hope that the above will help illuminate the core logic underlying the library.

Background

Much prior work highlights the value of fold as a primary mechanism for collection manipulation, superior to iteration, although most of that work was done in the context of recursively defined functions on lists or sequences - i.e. fold implies foldl/foldr, and the results remain inherently sequential.

The two primary motivators for this library were the Haskell Iteratee library and Guy Steele's ICFP '09 talk.

Haskell Iteratees

The Haskell Enumerator/Iteratee library and its antecedents are an inspiring effort to disentangle the source of data and the operations that might apply to it, and one of the first I think to reify the role of the 'iteratee'. An enumerator makes successive calls to the iteratee to supply it items, decoupling the iteratee from the data source. But the iteratee is still driving in some sense, as it is in charge of signaling Done, and, it returns on each step the next iteratee to use, effectively dictating a single thread of control. One benefit is that even operations like take can be defined functionally, as they can encode their internal state in the 'next' iteratee returned. OTOH, and unlike reducers, the design wraps the result being built up in a new iteratee each step, with potential allocation overhead.

Being an automaton in a state, an iteratee is like a reified left fold, and thus inherently serial. So, while they form quite a nice substrate for the design of, e.g. parsers, iteratees are unsuitable for defining things like map/filter etc if one intends to be able to parallelize them.

Guy Steele's ICFP '09 talk

Organizing Functional Code for Parallel Execution or, foldl and foldr Considered Slightly Harmful

This talk boils down to - stop programming with streams, lists, generators etc if you intend to exploit parallelism, as does the reducers library.

Where reducers diverges from that talk is in the structure of the fork/join parallel computation. Rather than map+reduce, reducers uses reduce+combine. This reflects 2 considerations:

  • It is accepted fork/join practice that at some point you stop splitting in half and handle the leaves 'sequentially'
    • if the best way to do that at the top is reduce, why not at the bottom as well?
  • map forces a result per input

You can see the awkwardness of the latter in the map/reduce-oriented definition of parallel filter in the talk, which must 'listify' items or return empty lists, creating a bunch of concatenation busy-work for the reducing step. Many other collection algorithms suffer similarly in their map/reduce-oriented implementations, having greater internal complexity and wrapping the results in collection representations, with corresponding creation of more garbage and reduction busy-work etc vs the reducing function transformer versions of same.

It is interesting that the accumulator style is not completely absent from the reducers design, in fact it is important to the characteristics just described. What has been abandoned are the single initial value and serial execution promises of foldl/r.

Summary

I hope this makes reducers easier to understand, use and define.

Rich

Permalink

Deliberate Practice

“Education comes from within; you get it by struggle and effort and thought.”
— Napoleon Hill

We all know the feeling of flow while coding. It's that moment when you hit a consistent stride and leave the surrounding world. It becomes effortless to glide over problems in an autopilot state and obstacles shrink from the size of mountains to the size of pebbles. Of course, being able to hit this flow means that you have practiced the problems you're facing to the point where it's of second nature. But, how did you get here? You practiced in two ways. First, you practiced moving forward to a place outside of your boundaries. Second, you also practiced the skills acquired from this action. Typically, practice is associated with the latter, but what is missing from this view of practice is the need to feel uncomfortable again. For me, when I begin to see problems as pebbles, it becomes time to deliberately seek a new mountain to climb. Both the acquisition and the polish of skills are important, but I want to shed some light on the practice of pushing boundaries, better known as deliberate practice.

As programmers there are many ways to do this. Here are a few that I find the most helpful.

Impose Artificial Constraints

In order to challenge myself I like to constrain the way I write solutions to problems. I once attended a code retreat where we were not allowed to use if or case statements in writing Conway's Game Of Life. This constraint forced a movement from a comfortable solution to one that left my pair and I feeling uncomfortable with the proceedings. We were forced to think in new ways that expanded our ideas of what constituted a solution. The solution we eventually came up with was a challenge to write.

When the event was over I thought back on my experience and realized that not only did I challenge myself, but I had a lot of fun doing it! It's interesting to see the creatively that flows from constraints. Now, when I approach problems that would have forced me into a solution involving if statements, I can feel comfortable moving outside of that solution and benefit from the understanding my new perspective has brought.

Learn New Languages & Paradigms

Steve Kim has a great post about learning a new language yearly. I want to echo his sentiment as an important part of deliberate practice. When I begin to feel comfortable with a language I know it's time to pick up a new one. The first language I felt comfortable with was Java. While in school I was forced into taking my first jump to another language when, as a class, we dove head first into C++. It was extremely intimidating at first and I started to feel discouraged (I have to manage memory?!?). Thankfully, I didn't back down and when I got the hang of it I was able to write some fun code for an embedded project. If I had stayed inside of my Java comfort zone I would have surely missed the chance to work on the embedded project.

The next big jump for me was to Clojure. This endeavor was another tough challenge because it introduced new paradigms. After some time and many struggles it started to click. Now, Clojure is one of my favorite languages to work with.

When you're feeling comfortable in a language pick up a new one. If you're a Java programmer, like I was, try out C++. If you're a Ruby programmer try out Python. When you feel comfortable in a certain paradigm switch to a new family of languages. There are many paradigms to explore including:

Switch Domains

The development community has many subcommunities such as web developers, front end developers, embedded developers, game developers, and so forth. One way to deliberately practice my craft is to extend myself into other communities. There are new ideas to be shared in the areas of design patterns, system structure, and much more when developers begin to blur the lines between each of these subcommunities. As I said earlier I took on an embedded project. While it was hard at first, the practice of the embedded design craft is important to the learning process and the movement towards greatness. I encourage everyone to take a look at the opportunities to deliberately practice in new domains.

Challenge Yourself

Practice is a process which will ultimately move your skills into a better state. Part of treating software as a craft is to continually raise the bar and engage the surrounding community to raise the bar. When programming begins to feel a little bit mundane seek out an exciting new mountain to climb. I challenge everyone to take a portion of practice time to deliberately practice.

Permalink

Clojure with Python syntax, part II

A couple months ago, I posted some thoughts I was having about writing a DSL that would make it possible to write Clojure without the s-expressions, the nesting, and back-and-forth indentation style that is so common in Lisp programs. I said the following would be the goals of the project, if I were to do it:

  • Eliminate the Lisp syntax barrier to getting programmers to try Clojure.
  • Make Clojure acceptable to universities that want to avoid Lisp syntax.
  • Provide a stepping stone so that programmers can learn Clojure concepts, then when they want more control over their language, they can start writing Clojure without giving up any of their existing code.
The post was motivated by my attempts to get others interested in Clojure. I'm a true believer in Clojure - I don't think there's another language that offers anything close to the experience. It's just a well thought out language that was implemented by someone with a lot of programming experience. The meaning of the term "pragmatic" has changed in recent years to mean "excuse for hypocrisy", so I won't say Clojure is pragmatic, but it certainly is a practical approach to functional programming. I think it is the best choice for those learning to program as well as those who want to get work done.

That is why I was disappointed at the way Clojure's syntax, which is quite an improvement on the Lisps that came before, is still viewed as clumsy and hard to read. I've always loved the consistency of Lisp syntax, and have always felt that the advantages of consistency outweigh any of the complaints. Too bad that's not true for everyone.

I've come to accept that humans treat their first experience as the default. Once you've reached puberty, you can forget about speaking a foreign language without an accent, because you will always, to some extent, try to put the new language into the form of the old. An accent is nothing more than pronouncing the new language using the "idiomatic" form of your native tongue.

I see no compelling reason that Lisp syntax should be a barrier to using Clojure. If your "default" is that a language should look like Python, then you should be able to write Clojure in that form. You'd lose the power of macros, but how is that a problem for someone who doesn't know Clojure or any other Lisp? When they're ready to take the leap to start writing macros, they can start working with s-expressions.

I did some searching on Google before my previous post. I had hoped this would already have been done, but I couldn't find anything. Richard Gabriel and Guy Steele even have a section in this paper devoted to alternative Lisp syntax. The problem is that all of those alternatives are ugly, much worse than Scheme or Common Lisp, so it's not at all surprising that none of them took off.

Basically every similar project has the same fatal flaw. They're all trying to be Lisp without parenthesis. In my experience, the objection to Lisp has little to do with parenthesis. Even when you end up with something like this: )))))))), the problem is that you've nested eight things in one statement, usually with varying levels of indentation, so it's not really the eight parens that causes the grumbling. Python, Ruby, or C programmers are unlikely to have eight levels of nesting in their programs.

The thing to realize with alternative Lisp syntax is that you can't save macros or prefix notation. Clojure != macros. Clojure != prefix notation. Clojure does so much so well that there is no reason to fear the loss of macros. I know about languages like Dylan. Dylan is not in any way worthy of comparison with Clojure. As a matter of fact, Clojure with Python syntax wouldn't even be a Lisp anymore, and that's okay, because I don't really care about Lisp. I could still write my Clojure with parens, it wouldn't make any difference how others write theirs, since it would still be Clojure. I can sit in my basement on Saturday night as a hobbyist writing macros, but I'd rather use Clojure all day at the office, even if I'm not using Lisp.

I haven't had time to dig into it as deeply as I'd like. Here's what I have concluded on the basis of my research:
  • I don't know anything about writing a DSL, parsing, or code generation.
  • I'm an intermediate Clojure user, so I don't yet understand some of the features of the language, and I don't understand most of the language as well as I should.
  • I don't use Python much these days (I don't think I've written a line of Python code in the last year). I don't understand what is so readable about Python, it looks the same to me as any other language.
  • Nobody would care if I did write it. I could show it to the people I know, but being realistic, few people would use it.
Combining my current time constraints with the huge cost of learning everything I'd have to learn, and comparing with the benefits, I can't justify doing it. I nevertheless think it would be a worthwhile project for someone who has no kids and is looking for a way to kill some time.

Permalink

Lithium: an x86 assembler for Clojure

Ah, the golden days of childhood’s hackage. Don’t you have fond memories of them?

I got my first PC when I was 10. It was a 486DX2/66 with 4 megs of RAM and a 170 meg HDD; it ran DOS and had lots of things installed on it, notably Turbo Pascal 6. I hacked a lot in it. These were pre-internet days when knowledge was hard to come by, especially for someone living in a small town in Poland; my main sources were the software I had (TP’s online help was of excellent quality), a couple of books, and a popular computing magazine that published articles on programming. From the latter, I learned how to program the VGA: how to enter mode 13h, draw pixels on screen, wait for vertical retrace, manipulate the palette and how to combine these things into neat effects. One of the very first thing I discovered was when you plot every pixel using sum of its coordinates modulo 40 as color, you get a nice-looking diagonal stripes effect. Because of the initially incomprehensible inline assembly snippets appearing all over the place, I eventually learned x86 assembly, too.

stripes

Back to 2012: I’ve long been wanting to hack on something just for pure fun, a side pet project. Writing code for the bare metal is fun because it’s just about as close as you can get to wielding the ultimate power. And yet, since Clojure is so much fun too, I wanted the project to have something to do with Clojure.

So here’s Lithium, an x86 16-bit assembler written in pure Clojure and capable of assembling a binary version of the stripes effect.

To try it, clone the git repo to your Linux or OS X machine, install DOSBox, launch a REPL with Leiningen, change to the lithium namespace and say:

(run! "/home/you/lithium/src/stripes.li.clj")

FAQ

(Well, this is not really a FAQ since nobody actually asked me any questions about Lithium yet. This is more in anticipation of questions that may arise.)

What’s the importance of this?

None whatsoever. It’s just for fun.

How complete is it?

Very incomplete. To even call it pre-pre-alpha would be an exaggeration. It’s currently little more than pure minimum required to assemble stripes.li.clj. Output format wise, it only produces bare binaries (similar to DOS .COMs), and that’s unlikely to change anytime soon.

Do you intend to continue developing it?

Absolutely. I will try to make it more complete, add 32- and possibly 64-bit modes, see how to add a macro system (since the input is s-expressions, it should be easy to produce Clojure macros to write assembly), write something nontrivial in it, and see how it can be used as a backend for some higher-level language compiler (I’m not sure yet which language that will turn out to be).

Permalink | Leave a comment  »

Permalink

Clojure Threading Operators

When I first started learning Clojure, wrapping my head around the different thinking was hard enough. Then, I saw the threading macros -> and ->>, and I wondered when those would ever be used outside a textbook. It wasn’t until I started doing something complicated that I saw the need to use them.

In the test-key-exclusion example the rows I’m interested in for my report (the ones that are not in the “other” report) are gathered, but nil values appear with any value that did match, because there is no interest in reporting a non-problem. Those values are filtered out with (filter (complement nil? , and then a vector of vectors is created with the non-matching results. ->> is quite useful as documentation to explain what is really going on.

(defn test-key-exclusion
"This function takes csv-data1 (the includees) and tests to see
if each includee is in csv-data2 (the includeds). This function
uses full rows, so ssn, lnam, and fnam can be reported."

[csv-data1 pkey-idx1 csv-data2 pkey-idx2 lnam-idx fnam-idx]

; This is a case for using the thrush ->> operator in Clojure.
; Insert each result into the last position of the next call.

(->> (map #(ret-non-match-rows csv-data2 pkey-idx2 pkey-idx1 %1) csv-data1)
(filter (complement nil?))
(map (fn [row]
(vector (nth row pkey-idx1 nil)
(nth row lnam-idx nil)
(nth row fnam-idx nil))))))


Permalink

I am adjusting to a mobile digital life: the acceleration of convenience

The first time I used a computer was in 1962 when I was 11 years old. It was not convenient: my Dad had to take me to his office to use one of the early timesharing systems using a local teletype machine with punched tape for saving and reloading stuff. A few years later I took an extension course at a local UC campus, FORTRAN with punched cards with nice keypunch facilities: definitely a large improvement!

In the 1980s life really got good: my own Xerox 1108 Lisp Machine and an Internet connection.

In the decades starting in 1990 and 2000 there were continual improvements but progress was gradual: better Internet connections, the development of the Web, improvements in software tools like code repositories, IDEs, etc.

Skipping ahead to the present time, I am trying to adjust my digital life, including my work and writing flows, to a more mobile lifestyle.

Probably the biggest change for me is that I used to keep just about everything that I worked with and touched in a repository (first cvs, then svn, and now git). I no longer need incremental backups of all of my work because for low priority stuff I can always dig back in time using simple Time Machine backups of my MacBook Air. I do still use a lot of git repositories for high value assets like:

  • Software development projects for customers
  • My open source projects at github
  • My own high value projects such as software and other assets for my web properties and commercial software products
This is a major change for me. I keep other assets that don't really need versioning on DropBox:
  • All assets for my writing projects: books I have written in the past, my current writing project, etc.
  • A million + 1 small code snippets and small bits or program code organized by all languages that I use (Common Lisp, Java, Ruby, Scheme, Haskell, Python, Prolog, etc. (whenever I figure out a new coding technique, how to do something generally useful, etc., I always like to save a little code snippet to refer to later)
  • Favorite pictures taken in my lifetime (old ones are now digitized)
  • Favorite videos taken in my lifetime (lots of digitally converted 8mm videos from my childhood through recent vacation videos: everything!)
  • All of my personal notes organized by travel logs, personal writing, etc.
  • A large fraction of the huge amount of music that I have purchased, organized by artist (and sometimes also by album)
  • Most of the eBooks that I have ever purchased if they are not in the Kindle format.
For Kindle books not purchased through Amazon, I email them to my Kindle/Amazon email address and let Amazon manage keeping everything in-sync on all of my devices. Kindle platform syncing the current reading location across devices is a huge convenience since I routinely read on my Kindle, iPad, and MacBook Air.

I am careful to not keep anything that is highly proprietary on DropBox (i.e., assets for customer projects). I used to also keep well organized directories of useful reading material found on the web (e.g., PDFs on AI, machine learning, NLP, server deployments, etc., etc.). I have very recently made the rather large step in throwing all of this material into Evernote, backing it up, and deleting it.

The really big win in relying on DropBox, Evernote, and the Kindle platforms is being able to switch between computers easily and also have most of my stuff available on my iPad and Droid cellphone. I use several computers and it is a slight nuisance doing a git pull every time I switch computers. I like to use mobile devices for reading and general thinking time and this is a lot easier now. I have been running Apple's beta OS X Mountain Lion that has iCloud integration. There is a chance that if iCloud is very well implemented that I may slowly transition away from DropBox to iCloud, and switch to using an iPhone. However, DropBox is very well implemented so Apple would need to make iCloud's implementation across all Apple devices very, very good for me to make the switch.

Saving the last big win for last: a mobile digital life promotes more of what Clojure creator (and general programming mentor) Rich Hickey calls "hammock time": the time you spend away from the computer thinking. I still use a pad or paper and a pen for away from a computer thinking time, but I find mobile devices augment this activity nicely.

Permalink

On Lisp in Clojure chapter 9

I am continuing to translate the examples from On Lisp by Paul Graham into Clojure. The examples and links to the rest of the series can be found on github.

Chapter 9 is presented for the sake of completeness. Clojure's let bindings solve most of the problems Graham describes in this chapter. You can solve the rest of the problems by putting a # at the end of any variable name you don't want to be subject to capture. e.g. x#.

Stuart Halloway also has written a post on this chapter on his blog. He gives a good description of how these capture issues relate to Clojure.

Section 9.1 Macro Argument Capture

We can write Clojure macros that run into some the same argument capture problems. 


(defmacro for' [[var start stop] & body]
  `(do
     (def ~var (atom ~start))
     (def limit ~stop)
     (while (< (deref ~var) limit)
       ~@body
       (swap! ~var inc))))

;; seems to work
(for' [ x 1 10] (println (str "current value is " @x)))

;; fails with error
(for' [ limit 1 10] (println (str "current value is" @limit)))

The version that is supposed to fail silently actually works just fine. 


(let [limit 5]
  (for' [i 1 10]
        (when (> @i limit)
          (println (str @i) ) )))

Section 9.2 Free Symbol Capture

Translating the examples from this section does not lead to the same errors. The w referred to in simple-ratio is different from the w referred to in gripe. I don't know why the gripe macro doesn't get fooled by the parameter in simple-ratio, but it doesn't. 


(def w (atom []))

(defmacro gripe [warning]
  `(do (swap! w #(conj % (list ~warning)))
       nil))

(defn simple-ratio [v w]
  (let [vn (count v) wn (count w)]
    (if (or (< vn 2) (< wn 2))
      (gripe "sample < 2")
      (/ vn wn))))

Section 9.3 When Capture Occurs

The first couple of code snippets just show let binding to describe the free variables. The first capture example is almost identical in Clojure. 


(defmacro cap1 []
  `(+ x 1))

The next several capture examples don't apply in Clojure. Even if you do have an x defined in your environment, this macro won't compile: 


(defmacro cap2 [var]
  `(let [x 2 ~var 3]
     (+ x ~var)))

When the macro expands, x gets expanded to a namespace qualified symbol, such as user/x, but you can't use let to bind a value to a namespace qualified symbol. In the first example from this chapter, the for loop, I used def to bind my variables, because I wanted to go out of my way to get the error.

The only way to define a cap2 macro in Clojure is like this: 


(defmacro cap2 [var]
  `(let [x# 2 ~var 3]
     (+ x# ~var)))

The # symbol after the variable name causes the macro expansion to do a gensym, which will give a unique name to x#, which no doubt is how Graham will have us solve the problem, after he finishes warning us about it.

Section 9.4 Avoiding Capture with Better Names

This section doesn't give any examples.

Section 9.5 Avoiding Capture by Prior Evaluation

The failing call to before doesn't fail in Clojure. Even with the caller using def from within a do block. Probably this is because we can't bind to a namespace qualified symbol in a let. Oh, don't call my position function on a value that doesn't exist in the sequence. 


(defn position [needle haystack]
  (loop [acc 0 lst haystack]
    (cond (empty? lst) nil
          (= needle (first lst)) acc
          :default (recur (inc acc) (rest lst)))))


(defmacro before [x y seq2]
  `(let [seq# ~seq2]
     (< (position ~x seq#)
        (position ~y seq#))))

(before (do (def seq2 '(b a)) 'a) 'b '(a b))

Here is the Clojure version of the new improved for loop. Graham defined the code to be executed as a lambda, and then looped over the invocation of that function. I am just going to bind my end value as limit# and move on. 


(defmacro for' [[var start stop] & body]
  `(let [~var (atom ~start) limit# ~stop ]
     (while (< (deref ~var) limit#)
       ~@body
       (swap! ~var inc))))

Sections 9.6 and 9.7

We have seen the for loop several times, and we have already mentioned that gensym in CLISP is done with appending a # to the var name. Clojure is broken up into namespaces, not packages. Namespaces make name collisions less common, but not impossible, just as Graham describes for Lisp.

Section 9.8 Capture in Other Name Spaces

I wasn't able to reproduce the error in Clojure. What mac thinks is fun is different than what my let binding does for fun. 


(defn fun [x] (+ x 1))

(defmacro mac [x] `(fun ~x))

(mac 10)

(let [fun  (fn [y] (- y 1))]
  (mac 10))

Section 9.9 Why Bother

Graham's answer to the question is a good one. And Rich Hickey's implementation of Lisp solves a lot of the problems for us.

Permalink

Clojure at cf.Objective()?

I'll be at cf.Objective() this coming week, manning the Railo booth - we're Gold sponsors again! - and, for once, not speaking! However, if you want to learn more about Clojure, especially how we're using it at World Singles integrated into CFML, feel free to track me down and ask me about it. I'll be easy to find: I'll be at the Railo booth nearly all the time (during the day - I'm only planning to attend a handful of sessions; I expect other Railo team members will be attending many of the sessions - Christian Ready, Gert Franz and Mark Drew are all giving talks).

These are the only sessions I'm planning to attend, so you'll know where I am when I'm not at the Railo booth:

  • The TDD List - Bill Shelton (Thursday, 10:15am)
  • Concurrency Zen - Marc Esher (Thursday, 1:45pm)
  • Railo 4Ever - Gert Franz / Mark Drew (Thursday, 4:15pm) - come and hear about all the cool new stuff in Railo 4.0!
  • Getting closure on Closures - Mark Mandel (Friday, 9:00am) - probably
  • CFML Mythbusters - Mark Drew (Friday, 1:45am) - probably
  • A/B Testing with Squabble - Mark Mandel (Saturday, 10:15am)

I'll also probably go to the Birds of a Feather sessions: either CFML Frameworks or Agile (Friday, 7:00pm) and either CFML Code Editors or Ideas for the Future of CFML (Friday, 8:00pm).

Given the sessions I'm planning to attend, it won't surprise you that I'll also be more than happy to talk about those topics in a Clojure context too - clojure.test, the Expectations framework and the autoexpect continuous testing tool, generative testing; concurrency the easy way (in a side-effect free functional language); closures and what a functional language brings to the table!

See y'all in Minneapolis!

Permalink

Clojure/West video schedule

Long overdue, the Clojure/West videos will start appearing on InfoQ next week.

Videos are listed by the week of release. Slides can be found at the GitHub slide repo.

  • 5/14/2012: Pallet, DevOps for the JVM - Antoni Batchelli
  • 5/21/2012: Why is a Monad Like a Writing Desk? - Carin Meier
  • 5/28/2012: Why Prismatic goes faster with Clojure - Bradford Cross
  • 6/4/2012: The Datomic Architecture and Data Model - Rich Hickey
  • 6/4/2012: Evident Code, at Scale - Stuart Halloway
  • 6/11/2012: Composing Statistical Graphics On the Web - Kevin Lynagh
  • 6/11/2012: Programming with Values in Clojure - Alan Dipert
  • 6/18/2012: Clojure-powered Startups: a three-part story - Paul deGrandis
  • 6/18/2012: What sucks about Clojure...and why you'll love it anyway - Chas Emerick
  • 6/25/2012: Load testing with Clojure - Andy Kriger
  • 6/25/2012: clojure @ runa :: dynamic pricing through DSLs - Amit Rathore
  • 7/2/2012: The Taming of the Deftype - Baishampayan Ghose 
  • 7/2/2012: Building tools to help kids fight ADHD - Alan Whitaker
  • 7/9/2012: Knockbox, an Eventual Consistency Toolkit - Reid Draper
  • 7/9/2012: Introducing Immutant - Jim Crossley
  • 7/16/2012: Building User Interfaces with Seesaw - Dave Ray
  • 7/16/2012: Clojure in the Clouds - Micah Martin
  • 7/23/2012: Accessing Real-World APIs from Clojure - Pat Patterson
  • 7/23/2012: Scalable Realtime Computation with Storm's Clojure DSL - Nathan Marz
  • 7/30/2012: Real World Clojure - Doing Boring Stuff With An Exciting Language - Sean Corfield
  • 7/30/2012: Crunching Numbers With Clojure - Daniel Solano Gomez
  • 8/6/2012: Distilling Java Libraries - Zach Tellman
  • 8/6/2012: SOLID Clojure - Colin Jones
  • 8/13/2012: Practical core.logic - Ryan Senior
  • 8/13/2012: Clojure, JRuby on Rails, and You - Allen Rohner
  • 8/20/2012: Swarm Coding - Phil Hagelberg
  • 8/27/2012: Thinking in Data - Stuart Sierra
  • 8/27/2012: Laziness: the Good, the Bad, and the Ugly - Paul Stadig
  • 9/3/2012: The generative generation - Aaron Bedra
  • 9/3/2012: Adopting Continuous Testing in Clojure - Bill Caputo
  • 9/10/2012: Engineering(,) A Path to Science: "I don’t want to die in a language I can’t understand" - Richard Gabriel
  • 9/10/2012: Namespaces, Symbols, and Vars, Oh My! - Craig Andera
  • 9/17/2012: Bootstrapping Clojure at Groupon - Tyler Jennings
  • 9/17/2012: DSLs in Clojure - Jim Duey
  • 9/24/2012: ClojureScript Anatomy - Michael Fogus
  • 9/24/2012: Macros are Hard! - David McNeil
  • 10/1/2012: Beyond Ninjas: DOM manipulation with ClojureScript and Domina - Luke VanderHart
  • 10/1/2012: Building Libraries for ClojureScript - Learning to Love Google Closure - Creighton Kirkendall
  • 10/8/2012: Distributed Apps: The Joys of Testing and Debugging - Chris Houser

Permalink

Prismatic's Global Newsfeed

One of our users' favorite things about Prismatic is that it's highly targeted -- personally, I love surfing topic feeds about robots, San Francisco food, and Clojure -- but they also want to keep up on the most important events happening around the world, across a much wider range of topics.

That's the purpose of our Global feed, which we're making public today on the heels of opening up hundreds of thousands of interest feeds on Wednesday. This post will explain the vibe of our Global feed, how it differs from other "Top News" sites like Reddit or Google News, and how we arrived at its current implementation.

What's Popular?

A first idea to single out important events is to sort recent stories by the total number of times they have been shared on social networks like Twitter. This was the basic idea behind an earlier iteration of our Global feed.

With a quick glimpse at the data, however, the problems with this approach become obvious. Of the twenty most-shared articles on Twitter over the past few days, thirteen are about social media or technology, and seven are from a single publisher (here are three representative examples). While social media, technology, and funny videos (biting social commentary notwithstanding) are undoubtedly important parts of the zeitgeist, when more than a hundred such stories appear before any mention of important global events we have clearly failed to capture the perspective we're aiming for.

Thus, while shares on sites such as Twitter are a powerful source of information about the content people are enjoying and engaging with across the internet, simply picking the most-shared articles has an unacceptable bias towards tweeting cats. To single out stories that nobody wants to miss about the most important events happening around the world, we had to look for other signals.

Who's Writing About It?

Counting shares of individual articles ignores an key fact: when a truly important event happens, there are usually lots of different articles written about it. Perhaps the best-known embodiment of this idea is Google News, which collects, organizes, and displays thousands of articles (primarily from traditional publishers) about each newsworthy event.

Of course, to be able to put this idea to work ourselves we need to know more than just which articles are out there and who is sharing them: we also need to know what each article is about, so that we can cluster together all of the articles about a single event. This is a technically difficult problem; fortunately, we have recently implemented a system that solves it quite well.

After developing this system, the next Global feed experiment we tried was to sort the story clusters, preferring those with the greatest number of articles about the same event. While this was a definite improvement over pure social sorting, it still didn't quite have the right feel. There were still too many tech stories, and some of the important and highly shared but more unique stories were lost under the news wire.

Our 'Global Feed'

After investigating social- and aggregation-based ranking independently, we were pretty sure that the two could be combined to yield a really great Global feed. Now, it was just a matter of figuring out how to best combine the signals to produce the vibe we were looking for.

Getting this right took a lot of experimentation, but we finally settled on a simple scheme that seems to work quite well. First, we only consider events with at least 5 stories about them as eligible for inclusion the Global feed. Then, we rank all of the stories about such events as usual, but give a small bonus to stories with many related articles with lots of shares.

This allows the content to speak for itself, and relies on our stock ranking pipeline to produce a final feed consisting of high-quality stories about important events, chosen from a diverse set of publishers and topics. Moreover, like the rest of Prismatic, it only gets better if you sign in -- you'll see more stories by publishers and about topics you care about, comments by people you know, and more.

Check out the Global feed, and please let us know what you think by filling out this brief survey.

Permalink

Mori: Persistent Data Structures for JavaScript

I've created several functional programming utility libraries in the past for JavaScript and CoffeeScript. Today they have become obsolete with the release of Mori. Mori is a simple bridge from vanilla JavaScript to ClojureScript's persistent data structures and the supporting API. Mori even includes Rich Hickey's new reducer work for allocation free operations on collections.

1
2
3
4

var v1 = mori.vector(1,2,3);
var v2 = mori.conj(v1, 4);
v1.toString(); // => '[1 2 3]'
v2.toString(); // => '[1 2 3 4]'

So maybe your boss won't let you use Lisp - well use Mori - it's just ~20k of gzipped JavaScript.

Permalink

Permalink

Recursion Exists For A Reason

I’ve survived another Clojure recurs or not recurs crisis. I understand why Clojure has sequences, but recursion is built into the language, and why its use represents a non-functional programming style is news to me. I have albeit helpful comments to that effect.

A lot of list like things can be abstracted into sequences,  without having to know what kind of sequence, so I am beginning to get why someone would want to use map and other sequence-oriented functions.

However, sometimes, you may want to take the difference between two similar spreadsheet reports. A value in one column  (a key) of Report A should be found in Report B in another column.

I spent a lot of extra learning time trying to take a list of keys present in Report A and see if they were also present in Report B using only Clojure’s sequence functions. Personally, I wanted to write a function to perform a match, return nil, if the match was found, or the search row if the match was not found. How hard is that?

I was frustrated, because it felt like all my logic was being re-routed, rather than part of it, like learning how to use map, filter, and so on.

The data looks like this:

[["0123456789" "SMITHFIELD" "HAM"]["1123456789" "LITTLE" "CHICKEN"] ...]

Here is the solution, which uses loop .. recur.

(defn ret-non-match-rows
"Expects a sequence of sequences, like what is returned from clojure-csv.
Returns nil if there's a match; else returns failing row."

[s-o-s cmp-col-idx inq-row-idx inq-row]

(let [inq-row inq-row]
  (loop [[row & remain-seq] s-o-s pos 0]
    (let [cmp-val (nth inq-row inq-row-idx nil)]
    (cond
    (not row) inq-row
    (= cmp-val (nth row cmp-col-idx)) nil
    :not-found (recur remain-seq (inc pos)))))))

(defn test-key-exclusion
"This function takes csv-data1 (the includees) and tests to see
if each includee is in csv-data2 (the includeds). This function
uses full rows, so ssn, lnam, and fnam can be reported."

[csv-data1 pkey-idx1 csv-data2 pkey-idx2 lnam-idx fnam-idx]

; This is a case for using the thrush ->> operator in Clojure.
; Insert each result into the last position of the next call.

  (->> (map #(ret-non-match-rows csv-data2 pkey-idx2 pkey-idx1 %1) csv-data1)
       (filter (complement nil?))
       (map (fn [row]
         (vector (nth row pkey-idx1 nil)
         (nth row lnam-idx nil)
         (nth row fnam-idx nil))))))

Used with a map statement, a sequence is returned that contains a lot of nils (hopefully) and a few non-nill entries. It is those entries that go into a missing report.

So, the moral of the story is, sometimes you want to traverse a sequence and stop when a condition, like a key match, has occurred. But do not give up on sequences.


Permalink

Prismatic's Global Newsfeed

One of our users' favorite things about Prismatic is that it's highly targeted -- personally, I love surfing topic feeds about robots, San Francisco food, and Clojure -- but they also want to keep up on the most important events happening around the world, across a much wider range of topics.

That's the purpose of our Global feed, which we're making public today on the heels of opening up hundreds of thousands of interest feeds on Wednesday. This post will explain the vibe of our Global feed, how it differs from other "Top News" sites like Reddit or Google News, and how we arrived at its current implementation.

What's Popular?

A first idea to single out important events is to sort recent stories by the total number of times they have been shared on social networks like Twitter. This was the basic idea behind an earlier iteration of our Global feed.

With a quick glimpse at the data, however, the problems with this approach become obvious. Of the twenty most-shared articles on Twitter over the past few days, thirteen are about social media or technology, and seven are from a single publisher (here are three representative examples). While social media, technology, and funny videos (biting social commentary notwithstanding) are undoubtedly important parts of the zeitgeist, when more than a hundred such stories appear before any mention of important global events we have clearly failed to capture the perspective we're aiming for.

Thus, while shares on sites such as Twitter are a powerful source of information about the content people are enjoying and engaging with across the internet, simply picking the most-shared articles has an unacceptable bias towards tweeting cats. To single out stories that nobody wants to miss about the most important events happening around the world, we had to look for other signals.

Who's Writing About It?

Counting shares of individual articles ignores an key fact: when a truly important event happens, there are usually lots of different articles written about it. Perhaps the best-known embodiment of this idea is Google News, which collects, organizes, and displays thousands of articles (primarily from traditional publishers) about each newsworthy event.

Of course, to be able to put this idea to work ourselves we need to know more than just which articles are out there and who is sharing them: we also need to know what each article is about, so that we can cluster together all of the articles about a single event. This is a technically difficult problem; fortunately, we have recently implemented a system that solves it quite well.

After developing this system, the next Global feed experiment we tried was to sort the story clusters, preferring those with the greatest number of articles about the same event. While this was a definite improvement over pure social sorting, it still didn't quite have the right feel. There were still too many tech stories, and some of the important and highly shared but more unique stories were lost under the news wire.

Our 'Global Feed'

After investigating social- and aggregation-based ranking independently, we were pretty sure that the two could be combined to yield a really great Global feed. Now, it was just a matter of figuring out how to best combine the signals to produce the vibe we were looking for.

Getting this right took a lot of experimentation, but we finally settled on a simple scheme that seems to work quite well. First, we only consider events with at least 5 stories about them as eligible for inclusion the Global feed. Then, we rank all of the stories about such events as usual, but give a small bonus to stories with many related articles with lots of shares.

This allows the content to speak for itself, and relies on our stock ranking pipeline to produce a final feed consisting of high-quality stories about important events, chosen from a diverse set of publishers and topics. Moreover, like the rest of Prismatic, it only gets better if you sign in -- you'll see more stories by publishers and about topics you care about, comments by people you know, and more.

Check out the Global feed, and please let us know what you think by filling out this brief survey.

Permalink

Software developers are not carpenters: on "Why do web sites andsoftware take so long to build? And why is it so hard?"

This evening it happened to me to read this quite interesting article about why software is so hard to write. In the article, if you don't care or do not want to read it, the blogger reports the decision of a friend of his quitting his (probably highly paid) job as software developer because he got tired of it. The motivation he (the one who quit) suggested are that software is complex to build and is the only large-scale human artifact that is mostly built by hand. Hand-made stuff, he argues, is more fragile and error prone that machine-made stuff. He also points out that good software developers do not agree on the right way to do things, etc. Eventually, he arguments that in civil engineering architects (or engineers) design stuff, politicians and administrators regulate it and carpenters build it. That thing is not possible with software.

So I have to say that: I'M FUCKING TIRED OF PEOPLE COMPARING BUILDING SOFTWARE TO BUILDING HOUSES. It is not the same thing. Moreover, I believe that comparisons with other forms of engineering has been detrimental for software development and lots of wrong practices and costly mistakes came from this illusion. I also think that people who really believe that building software is like building houses should be building other stuff. Neither software or houses (not houses I'm going to live in). Am I saying Rational F*cking Unified Process?

Humans have been building houses and bridges for millennia (yeah) and software since 50 years. It is quite natural that techniques have been developed. Software engineering is young. Really. So please, have patience. We will surely agree on the principle. That will happen earlier if people realize that no matter the process, unskilled developers are not going to build a working and maintainable piece of software.

Regarding the objection of software development being too manual, last time I checked, gcc and javac are exactly machine building software, not a bunch of underpaid third world workers. So, we do have machines building software. Compilers, build systems... testing infrastructures. We have lots of machines that help to build software, much in the same way that we use machines to build other things. We do not use machines to design most non trivial stuff.

And we continuously create new kind of machines doing stuff for us. Moreover, in the last 50 years we have been rising the abstraction level: since CPUs basically compute the same stuff, it means that we have more "machines" at work. People do not write opcodes. We write Python (Lisp? Clojure? Ruby?) code that is compiled and/or interpreted by a machine that runs on a machine called operating system that offers uses the services provided by a machine called kernel that uses relatively high-level services provided by the hardware. I don't tell the computer to write a 0 in some position of the hard disk!

Of course, we don't have machines that convert pointed-haired-boss-level babbling and unclear requirements to executable software. In other words, software developers are not skilled in divination. Perhaps haruspicy [0] or augury should be added to CS curricula. The problem is people do not even know exactly what they want. The best we can do is to keep them involved in the development process and use continuos feedback to deliver the product they expect.

"Name one other thing in the world, he said, that is used by so many people and which is created entirely by hand"

Well, many things come to my mind. Books. Laws. Theorems. You don't have machine writing books (well, in fact IA tried to build machines performing some limited "artistic" tasks, with mixed results -- the more constrained the piece of art, the better the chances --). I do not think we have machines writing laws. Human have been creating laws for millennia. We *wrote* them down. And still, there is not consensus on what is right. Laws are bugged and there are people exploiting bugs in them (and that is not even closely illegal: it is the highly paid profession of the lawyer). We don't have nor want machines writing laws for us. Experts do not agree in what is a right law. Laws are continuously changed, modified, patched, deployed (in Italy we have the concept of "decreto attuativo" which I'm not even sure it has a clear translation in English -- the idea is that the law is so vague that you need some "lower level" laws to actually explain how the law should be implemented). Laws are unjust. And nobody fucking complains that the Romans could build bridges lasting millennia (I could have a roman bridge right under my feet right now) and we can't have unambiguous laws. Well, romans were pretty good to write laws as well...

So essentially the point is that people that software should be like houses, because it is a product. I think that software should be like poetry, because it is an idea.

----
[0] The problem is that if you believe write requirements based on sheep guts then maybe you should start the noble profession of PHB.
[1] [0]s/sheep guts/birds flight/

Permalink

OpenJDK 1.8.0-jdk8-b35 seems to break Leiningen 2

Is it only me experiencing an issue with running the most recent build of Leiningen 2 with the recent OpenJDK 8 x64-lambda JDK with Lambda from jdk8 branch (b35) (OpenJDK-OSX-8-x64-lambda-jdk-b35-20120507) on Mac OS X?

With the version of OpenJDK selected in the Java Preferences, lein2 finishes with a stacktrace.

Here’s the full log of the steps to reproduce the issue. Is it Clojure 1.4 or lein2?

jacek:~/oss/leiningen
$ git pull -u
Already up-to-date.

jacek:~
$ java -version
openjdk version "1.8.0-jdk8-b35"
OpenJDK Runtime Environment (build 1.8.0-jdk8-b35-20120507)
OpenJDK 64-Bit Server VM (build 24.0-b07, mixed mode)

jacek:~/oss/leiningen/leiningen-core
$ lein install
Copying 33 files to /Users/jacek/oss/leiningen/leiningen-core/lib
No namespaces to :aot compile listed in project.clj.
Created /Users/jacek/oss/leiningen/leiningen-core/leiningen-core-2.0.0-SNAPSHOT.jar
Wrote pom.xml
[INFO] Installing /Users/jacek/oss/leiningen/leiningen-core/leiningen-core-2.0.0-SNAPSHOT.jar to
/Users/jacek/.m2/repository/leiningen-core/leiningen-core/2.0.0-SNAPSHOT/leiningen-core-2.0.0-SNAPSHOT.jar

jacek:~
$ lein2 version
Exception in thread "main" java.lang.VerifyError: (class: ordered/map/OrderedMap, method: count signature: ()J) Expecting to find long on stack, compiling:(ordered/map.clj:26)
    at clojure.lang.Compiler.analyzeSeq(Compiler.java:6462)
    at clojure.lang.Compiler.analyze(Compiler.java:6262)
    at clojure.lang.Compiler.analyze(Compiler.java:6223)
    at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5618)
    at clojure.lang.Compiler$FnMethod.parse(Compiler.java:5054)
    at clojure.lang.Compiler$FnExpr.parse(Compiler.java:3674)
    at clojure.lang.Compiler.analyzeSeq(Compiler.java:6453)
    at clojure.lang.Compiler.analyze(Compiler.java:6262)
    at clojure.lang.Compiler.analyzeSeq(Compiler.java:6443)
    at clojure.lang.Compiler.analyze(Compiler.java:6262)
    at clojure.lang.Compiler.access$100(Compiler.java:37)
    at clojure.lang.Compiler$DefExpr$Parser.parse(Compiler.java:518)
    at clojure.lang.Compiler.analyzeSeq(Compiler.java:6455)
    at clojure.lang.Compiler.analyze(Compiler.java:6262)
    at clojure.lang.Compiler.analyzeSeq(Compiler.java:6443)
    at clojure.lang.Compiler.analyze(Compiler.java:6262)
    at clojure.lang.Compiler.analyze(Compiler.java:6223)
    at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5618)
    at clojure.lang.Compiler$LetExpr$Parser.parse(Compiler.java:5919)
    at clojure.lang.Compiler.analyzeSeq(Compiler.java:6455)
    at clojure.lang.Compiler.analyze(Compiler.java:6262)
    at clojure.lang.Compiler.analyze(Compiler.java:6223)
    at clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5618)
    at clojure.lang.Compiler$FnMethod.parse(Compiler.java:5054)
    at clojure.lang.Compiler$FnExpr.parse(Compiler.java:3674)
    at clojure.lang.Compiler.analyzeSeq(Compiler.java:6453)
    at clojure.lang.Compiler.analyze(Compiler.java:6262)
    at clojure.lang.Compiler.eval(Compiler.java:6508)
    at clojure.lang.Compiler.load(Compiler.java:6952)
    at clojure.lang.RT.loadResourceScript(RT.java:359)
    at clojure.lang.RT.loadResourceScript(RT.java:350)
    at clojure.lang.RT.load(RT.java:429)
    at clojure.lang.RT.load(RT.java:400)
    at clojure.core$load$fn__4890.invoke(core.clj:5415)
    at clojure.core$load.doInvoke(core.clj:5414)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.core$load_one.invoke(core.clj:5227)
    at clojure.core$load_lib.doInvoke(core.clj:5264)
    at clojure.lang.RestFn.applyTo(RestFn.java:142)
    at clojure.core$apply.invoke(core.clj:603)
    at clojure.core$load_libs.doInvoke(core.clj:5298)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invoke(core.clj:603)
    at clojure.core$require.doInvoke(core.clj:5381)
    at clojure.lang.RestFn.invoke(RestFn.java:551)
    at leiningen.core.project$eval22$loading__4784__auto____23.invoke(project.clj:1)
    at leiningen.core.project$eval22.invoke(project.clj:1)
    at clojure.lang.Compiler.eval(Compiler.java:6511)
    at clojure.lang.Compiler.eval(Compiler.java:6501)
    at clojure.lang.Compiler.load(Compiler.java:6952)
    at clojure.lang.RT.loadResourceScript(RT.java:359)
    at clojure.lang.RT.loadResourceScript(RT.java:350)
    at clojure.lang.RT.load(RT.java:429)
    at clojure.lang.RT.load(RT.java:400)
    at clojure.core$load$fn__4890.invoke(core.clj:5415)
    at clojure.core$load.doInvoke(core.clj:5414)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.core$load_one.invoke(core.clj:5227)
    at clojure.core$load_lib.doInvoke(core.clj:5264)
    at clojure.lang.RestFn.applyTo(RestFn.java:142)
    at clojure.core$apply.invoke(core.clj:603)
    at clojure.core$load_libs.doInvoke(core.clj:5298)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invoke(core.clj:603)
    at clojure.core$require.doInvoke(core.clj:5381)
    at clojure.lang.RestFn.invoke(RestFn.java:482)
    at leiningen.core.main$eval3$loading__4784__auto____4.invoke(main.clj:1)
    at leiningen.core.main$eval3.invoke(main.clj:1)
    at clojure.lang.Compiler.eval(Compiler.java:6511)
    at clojure.lang.Compiler.eval(Compiler.java:6501)
    at clojure.lang.Compiler.load(Compiler.java:6952)
    at clojure.lang.RT.loadResourceScript(RT.java:359)
    at clojure.lang.RT.loadResourceScript(RT.java:350)
    at clojure.lang.RT.load(RT.java:429)
    at clojure.lang.RT.load(RT.java:400)
    at clojure.core$load$fn__4890.invoke(core.clj:5415)
    at clojure.core$load.doInvoke(core.clj:5414)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.core$load_one.invoke(core.clj:5227)
    at clojure.core$load_lib.doInvoke(core.clj:5264)
    at clojure.lang.RestFn.applyTo(RestFn.java:142)
    at clojure.core$apply.invoke(core.clj:603)
    at clojure.core$load_libs.doInvoke(core.clj:5298)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invoke(core.clj:603)
    at clojure.core$require.doInvoke(core.clj:5381)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.main$main_opt.invoke(main.clj:324)
    at clojure.main$main.doInvoke(main.clj:427)
    at clojure.lang.RestFn.invoke(RestFn.java:436)
    at clojure.lang.Var.invoke(Var.java:423)
    at clojure.lang.AFn.applyToHelper(AFn.java:167)
    at clojure.lang.Var.applyTo(Var.java:532)
    at clojure.main.main(main.java:37)
Caused by: java.lang.VerifyError: (class: ordered/map/OrderedMap, method: count signature: ()J) Expecting to find long on stack
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:259)
    at clojure.lang.RT.classForName(RT.java:2039)
    at clojure.lang.Compiler$HostExpr.maybeClass(Compiler.java:957)
    at clojure.lang.Compiler$HostExpr.access$400(Compiler.java:736)
    at clojure.lang.Compiler$NewExpr$Parser.parse(Compiler.java:2473)
    at clojure.lang.Compiler.analyzeSeq(Compiler.java:6455)
    ... 93 more

With the version of OpenJDK changed to 1.7.0-jdk7u4-b21, it works fine.

jacek:~
$ java -version
openjdk version "1.7.0-jdk7u4-b21"
OpenJDK Runtime Environment (build 1.7.0-jdk7u4-b21-20120427)
OpenJDK 64-Bit Server VM (build 23.0-b21, mixed mode)

jacek:~
$ lein2 version
Leiningen 2.0.0-SNAPSHOT on Java 1.7.0-jdk7u4-b21 OpenJDK 64-Bit Server VM

jacek:~
$ lein2 repl
nREPL server started on port 60220
Welcome to REPL-y!
Clojure 1.4.0
    Exit: Control+D or (exit) or (quit)
Commands: (user/help)
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
          (user/sourcery function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
Examples from clojuredocs.org: [clojuredocs or cdoc]
          (user/clojuredocs name-here)
          (user/clojuredocs "ns-here" "name-here")
nil
user=> (clojure-version)
"1.4.0"
user=> (leiningen.core.main/leiningen-version)
"2.0.0-SNAPSHOT"

Permalink

Lemur: Declarative launching of Hadoop jobs on Elastic-Mapreduce

In this post, I'm announcing a new tool for launching Hadoop jobs locally and on AWS's elastic-mapreduce, (EMR) which I call Lemur.  I built this tool for The Climate Corporation (TCC), purveyors of Weather Insurance to the Agriculture Industry.  I want to thank TCC for allowing me to open source this work, as well as a great deal of feedback and the original use cases.

Overview

Lemur is a tool to launch hadoop jobs locally or on EMR, based on a configuration file referred to as a jobdef. The jobdef file describes your EMR cluster, local environment, pre- and post-actions (aka hooks) and zero or more "steps".  A step is Amazon's name for a task or job submitted to the cluster. Lemur reads your jobdef, at the end of your jobdef you execute (fire! ...) to make things happen.  Lemur is implemented in Clojure as an internal DSL. You should be able to use it for many tasks with only a brief understanding of Clojure's syntax.

What Lemur is Not

Lemur is NOT a replacement for the Ruby elastic-mapreduce cli.  There is some overlap, but I did not attempt to reproduce functionality (like --list, --terminate, etc) that works perfectly well in that tool.  Also, Lemur is not a job scheduler (a la Oozie, Quartz or Azkaban).  You might decide to use one of those tools to trigger Lemur which knows how to run your job.

The Problem

A typical hadoop job likely accepts command line arguments to tell it where the input files are, where the output should go, and specify other parameters and options that are specific to the analysis that the job does. In addition, you need to specify all the options for your EMR cluster (cluster size, instance types, spot market, etc). And then maybe you need some custom actions run when the cluster is started, so you need to specify one or more bootstrap-actions. You may also need to upload the JAR that contains your hadoop job to S3, if it's not already deployed there.  And if you run this job frequently, there may be other tasks you need to do before the cluster launches (e.g. download some input files from a FTP site, modify some config file, look at the size of your input data to dynamically determine how big the cluster should be and so on).  The point is that once you move beyond a sample or proof-of-concept job to a real, operationalized job; there are frequently many additional concerns that need to be handled.  A common solution is a Bash (or other) script wrapping calls to elastic-mapreduce.  If you've tried this, you probably know that for non-trivial scenarios these scripts can quickly become complex and riddled with DRY violations (although re-use is possible in Bash, it's not pretty and often not done). 

In one case, TCC had a Ruby Rakefile that was over 1300 lines long-- it contained logic for running 3 or 4 different jobs.  And even with 1300 lines, it didn't do everything.  It made use of a another in-house, Clojure tool that had a lot of the common things in the TCC environment (i.e. which keypair to use, specification of all the bootstrap actions, etc.). It included some nice features though:
  • automatically build the hadoop job jar for each run if the source files have changed
  • launch the job on elastic-mapreduce, or run it locally in Hadoop standalone or pseudo-distributed mode
  • do a test run where the output is automatically verified
Lemur attempts to support all of these use-cases and features and more; through a declarative, but extensible interface.  You write a jobdef file for each job.  The jobdef contains name-value pairs specifying most options.  You can include custom logic in the form of expressions or functions, and you can 'inherit' common functionality from one or more 'base' files.  Most jobdefs at TCC make use of two base files (one is common to all TCC jobs, the other is team specific).

Features

  • Launch EMR cluster and submit step(s); or run against local hadoop (usually hadoop standalone for dev and testing)
  • Basic configuration options include: 
    • Bootstrap actions
    • Hadoop config
    • Uploads (files to transfer to S3, or local)
    • Cluster details (num instances, master instance type, etc)
    • Output paths to use for data, logs, main jar, etc. 
    • Support for spot market instances
  • Profile support provides packages of options and functionality that can be enabled or disabled as one switch.  (e.g. you can have a :test profile or a :live profile)
  • Validation for your command line options and environment before launching EMR
  • Override configured options via command line
  • Hooks for actions that should be triggered before or after job launch.  For example:
    • One hook in use at TCC does a diff on the results of a local run (i.e. an integration test)
    • Another hook posts a detailed message to IRC (hipchat) when a new job is started
  • Optionally wait for an EMR job to complete
  • A dry-run feature, so you can check the final cluster configuration, hooks that will be executed, hadoop job arguments, etc.
  • All the details from dry-run (cluster/step config, etc) are persisted with each job run (to STDOUT and saved to a YAML file alongside your output)
  • All settings can be literal values, interpolated strings (e.g. set the S3 bucket to "com.your-co.${env}.hadoop"), or functions
  • Import ("inherit") common options, functionality and behavior to avoid duplication
  • Pass-through command-line options, allows you to specify extra args on the command line that are meaningful to your hadoop main function, but are unknown to lemur or your jobdef
  • Most of TCC's actual hadoop jobs are written with Cascalog.  But Lemur is agnostic to this detail.  They could be Cascading, Java, Hive, Pig, Scalding, Streaming, etc.

    Examples of Jobdefs and running Lemur 

    These are intended to give you the flavor of Lemur.  Actual documentation is on the project Github site (linked below).

    Here's a trivial example.  It inherits from two base files, and defines a one node cluster and a job with one step.   Things like the keypair, which jar to use, a unique and predictable path to use for the output, how to launch this job either on EMR or hadoop standalone, etc would all be handled by the base files or generic Lemur functionality.
    (use-base 'climate.lemur-base 'climate.another.lemur-base)

    (defcluster simple-cluster
      :num-instances 1
      :master-instance-type "m1.large")

    (defstep simple-step
      :main-class "climate.another.Simple"
      :args.passthrough true
      :args.data-uri true)

    (fire! simple-cluster simple-step)
    A more involved example follows.  This one also has a single step, but includes extra command line options with validation, some custom logic to download some data from ftp before the job runs, and specification of an integration test (diff-test-data).  The integration test specifies 4 output directories that should be verified.  Again, many details are handled in the base files; like the location of the test input data, for example, which in the TCC case is a common git repo with a sub-directory for this app-name.  The app-name in this case is "esat", implied by the name given on the (defcluster) line.
    (use-base  'climate.lemur-base 'climate.another.lemur-base)

    (require '[climate.another.esat :as esat])

    (catch-args
      [:download? "Download and save esat data from the Esat ftp server." true]
      [:num-days "Number of days to download." 1]
      [:days-file "A file of days to process yyyy-mm-dd, one per line." nil])

    (add-validators
      (val-opts :file :days-file)
      (val-opts :required :numeric :num-days))

    (defn- ftp-download [eopts]
      (when (:download? eopts)
          (if (dry-run?)
            (println (format "Download %s days from %s" (:num-days eopts) (:repo eopts)))
            (esat/ftp (:repo eopts) (:num-days eopts))))

    (add-hooks
      ftp-download
      (when-local-test)
        (diff-test-data ["FILENAMES" "filenames"]
                               ["OBSERVATIONS" "observations/values"]
                               ["HISTORY ERRORS" "histories/errors"]
                               ["METRICS" "save-observations/metrics"]))

    (defcluster esat-cluster
      :upload [:days-file :to :data-uri]
      :num-instances 1
      :live {:num-days 12})

    (defstep esat-step
      :main-class "climate.another.esat"
      :args.repo "${repo}"
      :args.store nil
      :args.days "${days-file}"
      :args.stations nil
      :args.passthrough true
      :args.data-uri true)

    (fire! esat-cluster esat-step)

    The Project

    This project is open-sourced with an Apache 2.0 license and is available on github.  Help and more examples can be found on Github.

    Permalink

    in which korean hardware pleases

    I just got a new laptop, and while I wouldn't post about it normally I've had a number of people ask me what I thought of it. For the last five years or so I've been a Thinkpad user exclusively, but the latest models of their lightweight X series have offered disappointing screens with 1366x786 being the only resolution option. Since my dignity prevents me from buying a laptop with fewer vertical pixels than my phone, I broadened my search this time around and came across the 13-inch Samsung Series 9, aka NP900X3B.

    zuse

    At first the most striking thing about it is its thin profile, but once you've started using it you realize it's the screen that really sets it apart. Samsung is the largest manufacturer of LCDs in the world, so it makes sense that the screen would be the distinguishing factor on their high-end models. The resolution is 1600x900, which I couldn't find in anything else smaller than 14 inches, but it's the 400 cd/m2 of brightness that really sets it apart. With summer right around the corner, this perfect for working outdoors.

    While I'm pretty thrilled with the new machine, the thin profile forces some difficult compromises with the keyboard. It's of the standard chicklet design, which is a drag coming from the Thinkpad. The X200s Thinkpad I was using previously had a very comfortable response and depth to it that you don't find with chicklet keybords. The Samsung also lacks a trackpoint, drainage tray, and dedicated volume buttons. The X200s isn't appreciably heavier than the Samsung; (only ½ a pound more) but it is over twice as thick. Personally if it's light enough I don't find thickness to be a problem; I'd much rather have a comfortable keyboard than a 0.4-inch profile. But it does turn heads.

    The speakers on ultraportable laptops like the X200s typically come across as a token effort to fill a feature checkbox, but the Samsung's are fairly respectable. I was pleased to see the lack of the hardware wifi kill switch that plagued me on my Thinkpad. It's also got a bigger selection of ports than you'd expect with the slim profile: USB 2, USB 3, micro-HDMI, combined mic/headphones, and Ethernet, though the latter must be used with the provided adapter.

    I'm running Debian Wheezy on it because I read on a blog post that a newer kernel was needed for the wifi drivers, but apparently this was only the case with an earlier model of the Series 9 that used the Broadcom chipset; The NP900X3B uses an Intel chipset that has been well-supported for some time. Everything worked perfectly out of the box except the keys for adjusting the keyboard backlight. The camera, external monitor port, and multitouch trackpad work as you'd expect. The suspend functionality, which has traditionally lagged behind Mac OS X, resumes in a couple short seconds.

    While I would love to see this same screen on a laptop with the luxurious keyboard, carbon fiber body, and replaceable battery of a Thinkpad, the Series 9 is quite slick and should keep me happy for a number of years.

    Permalink

    datomic – demonic transforms – to do or not to do?

    While working with the new Datomic database, we’re having to get used to using ns-qualified keywords in our maps. To make day-1 go faster, I wrote these versions of the demonic insert and load functions:

    and also:

    The idea was to use them as follows (for instance):

    where datomic-key->regular-key is something like:

    The idea behind all this was that a map like:

    Could become something like:

    This initially seemed to be more usable and familiar. However, we’ve since moved away from it, seeing some value in having keys that are indeed namespaced to the entity they belong to (for example user entity has keys called :user/first-name and :user/last-name).

    Just thought we’d share.


    Tagged: clojure, code, database, datomic, demonic, tech

    Permalink

    Copyright © 2009, Planet Clojure. No rights reserved.
    Planet Clojure is maintained by Baishamapayan Ghose.
    Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
    Theme by Brajeshwar.