Thomas Lee

The other day I was discussing Rails’ form processing behavior with Ben, when the topic of editing multiple associations in a single form came up. Effectively, he needed to be able to manipulate a collection of records associated with a parent via a has_many association. For each item in this collection, he wanted something like two drop down lists — nothing unreasonable. He said it was similar to recipe 13 from Advanced Rails Recipes — “Handle Multiple Models in One Form” — except that he had to use two fields instead of the one textfield, and it was proving to be surprisingly tough.

I know what you’re thinking: surely this can be extended to more than one form field per item without any problem? Up until I read the code myself a few months ago, I wouldn’t have thought it would be so difficult either — but it’s true. The parameter parsing code is quite quirky and, without reading the detail of the source, borderline unpredictable. I’ve had a few ideas for dealing with more complicated forms in Rails ever since I first ran into this issue. Only recently have I actually gotten around to doing something about it. Here’s an abridged version of an email I just sent to the rails-core mailing list:

Hi,

…

A colleague of mine recently ran into a related problem when dealing with collections of records in Rails forms, which prompted me to finally do something about the parameter parsing behavior. Now, while I’m sure we’re not the only ones hitting this wall, I’m aware that proposals to change to the parameter parsing semantics in Rails is likely to be met with a little caution, hesitation — possibly even terror. With that in mind, I’ve put up a plugin so everybody can give this a go without having to apply any nasty patches:

http://www.vector-seven.com/git/rails/plugins/form_collections.git

Please see the README (attached) for the full details.

…

Cheers,
Tom

This is a plugin which greatly simplifies the existing Rails parameter parsing code, and makes it a whole lot easier to deal with collections in your forms. I encourage all you Rails developers out there to take a look when you get the opportunity, I’d love to hear your opinions on this one. You can get a copy of the git repository using the following command:

$ git clone http://www.vector-seven.com/git/rails/plugins/form_collections.git

UPDATE: Here’s the contents of the README file:

> FormCollections
> ===============
>
> This plugin effectively rewrites UrlEncodedPairParser and provides the
> following features/changes:
>
> 1. The parameter parsing algorithm is much simpler
>
> There's all sorts of wacky stuff happening in the current implementation of
> UrlEncodedPairParser#parse. This plugin does away with that. A stack is no
> longer used during the parse. Parsing an Array vs. parsing a Hash is no longer
> all that different.  See lib/patch_url_encoded_pair_parser.rb.
>
> 2. Parsing "a[b][0][c]=6" yields {"a" => {"b" => [{"c" => "6"}]}}
>
> Previously this would be parsed to: {"a" => {"b" => {"0" => {"c" => "6"}}}}
>
> In other words, parameters that were formerly treated as hashes with a numeric
> index are now actually arrays.
>
> This is important, because the order in which form fields are present may be
> important to the back-end processing code. Here, the array in "b" will
> preserve the ordering specified -- generating an Array rather than a Hash for
> numeric indices. Most other scenarios should essentially work as before, with
> the exception of a few error cases (e.g. parsing "a/b@[c][d[e][]=f" yields
> {"a/b@" => {"c" => {"d[e" => ["f"]}}} instead of {"a/b@" => {"c" => {}}}).
>
> The main problem here is that if an array is instantiated with a value at
> index 1000000, then we'll have 999999 nil elements. I figure this can easily
> be worked around with an Array-like structure that consumes only as much memory
> as it needs but otherwise acts as an Array, or a hard limit set via
> configuration variable.
>
> 3. fields_for now treats Arrays properly ...
>
> Example:
>
>  <% fields_for @post.comments do |comment_form, comment| %>
>    <%= comment.new_record? ? "" : comment_form.hidden_field(:id) %>
>    <%= comment_form.hidden_field :post_id %>
>    <%= comment_form.text_field :content %>
>    <hr />
>  <% end %>
>
> This will generate the following HTML for an array with two elements (one a new_record?, the other existing):
>
>  <input id="comments_0_id" name="comments[0][id]" type="hidden" value="1" />
>  <input id="comments_0_post_id" name="comments[0][post_id]" type="hidden" value="1" />
>  <input id="comments_0_content" name="comments[0][content]" size="30" type="text" value="a test comment" />
>  <hr />
>  <input id="comments_1_id" name="comments[1][post_id]" type="hidden" value="1" />
>  <input id="comments_1_content" name="comments[1][content]" size="30" type="text" value="" />
>  <hr />
>
>
> Example
> =======
>
> Just install the plugin, and the new behavior should already be in effect.
>
> Copyright (c) 2008 Thomas Lee ..., released under the MIT license
>

The second part of my Scala compiler construction tutorial has been a long time coming. This post is, unfortunately, not the second part of the article — although that is coming soon. Honest.

Since part 1 was published, Scala 2.7 has been released which — among other things — introduced changes to the parser combinator library. Changes that meant the source code from part 1 will not compile in Scala 2.7. Sorry about that. Updated, working code can be found at the end of this post.

So what exactly has changed? Well, let’s see …

keyword() no longer discards its result

A bit of a refresher first:

In both Scala 2.6 and 2.7, the keyword implicit is called whenever you use a string in your “grammar rules”. For example:

def sum = expr ~ "+" ~ expr

Is the equivalent of:

def sum = expr.~(keyword("+").expr)

In Scala 2.6’s parser combinator library, the result of the keyword() call was actually the UnitParser — that is, a parser that would discard its result. At the time, that meant we could use the tilde operator (”~”) to create a sequenced parser and everything was fine:

def sum = expr ~ "+" ~ expr ^^ ((left : Expr, right : Expr) => Sum(left, right))

In 2.7, the keyword() implicit returns a Parser[String] rather than a UnitParser. This means we have to either deal with the newly introduced tokens as follows:

def sum = expr ~ "+" ~ expr ^^ ((left : Expr, op : String, right : Expr) => Sum(left, right))

Or alternatively …

Use <~ and ~> to indicate the important parts of a parse rule

Let’s start with something simple based on the 2.6 combinators:

def bracketExpr = "(" ~ expr ~ ")" ^^ ((e : Expr) => e)

Again, in 2.7 we know that keyword() is no longer a UnitParser, so we have to deal with it like so:

def bracketExpr = "(" ~ expr ~ ")" ^^ ((l : String, e : Expr, r : String) => e)

Alright, this compiles and does what we expect. But why should we keep those strings around if we don’t need them? They sure do clutter up the code a whole bunch.

<~ and ~> can be used to include or discard the result of a given parser. <~ builds a parser that takes the result of two parsers (the parsers to its left and right) and builds a parser that takes the result of both and discards the result of the one on the right. Conversely, ~> yields a parser that takes the result of both parsers and discards the result on the left.

So, using these two operators we can rewrite bracketExpr as follows:

def bracketExpr = "(" ~> expr <~ ")" ^^ ((e : Expr) => e)

Ah. Much better.

The ^^^ operator

I had to go digging in the Scala source code to work out what exactly this one does.

First, let’s take a look at some 2.6 code:

def simpleExpr = term * (
    "+" ^^ ((x : Expr, y : Expr) => Add(x, y)) |
    "-" ^^ ((x : Expr, y : Expr) => Sub(x, y))
)

In 2.6, this can parse zero or more repetitions of “term” interleaved by “+” and “-” (check out part 1 if you need a refresher on how the * combinator works). In 2.7 it’s a compile-time error because of the fact keyword() is no longer a UnitParser (are you seeing a pattern here? :)), and ^^ is trying to pass the resulting String on to the anonymous method in each case.

If we use the ^^^ operator here, we can effectively discard the result of the keyword parse, and build a parser that uses a simple anonymous method to parse the current pair of terms:

def simpleExpr = term * (
    "+" ^^^ ((x : Expr, y : Expr) => Add(x, y)) |
    "-" ^^^ ((x : Expr, y : Expr) => Sub(x, y))
)

Exactly what we’re after. This compiles and behaves as expected.

What else?

There may be more changes to the parser combinator library which I haven’t covered here, but I’m not going to go looking for any more changes since the updated code seems to work just fine. This should be enough to at least understand the updated code for the compiler described in part 1 without needing to deal with any cryptic compiler errors.

Finally, the new code!

Special thanks to Harshad for sending through working code for 2.7 ages ago which I never got around to posting here. This code, along with the Scala API docs, was used to figure out just what had changed since 2.6. The code below is derived from some code he sent to me a few months back.

I’m really sorry this has been so long in the making. I’ll try to get around to writing the “real” part two of this article. In the meantime, here’s the updated code. Thanks! Please post or email any comments or questions.

Well, that wasn’t as awful as I was expecting it to be. Everything should be back to normal now.

I’m in the middle of a major upgrade. Things may be a little weird until the upgrade is complete!

I just finished reading Tony Morris’s blog post on Scala implicits and saw the following comment:

To me, implicits look a lot like global variables, which is why I don’t like them.

Or maybe I missed something?

Uh, yep — you missed something I was going to reply in the comments to Tony’s post but as per usual it became far too long, so I’m posting it here instead.

Let’s try something a little less abstract than what was covered in Tony’s article. I haven’t actually tested any of this code to check that it compiles (lazy, lazy, lazy!), but it should be pretty straightforward & clarify what implicits can be used for:

class Person(name : String) {}
class Conversation(person1 : Person, person2 : Person) {
    def greet() = {
       println(person1.name + " says: Hello, " + person2.name)
       println(person2.name + " says: Oh, hello " + person1.name)
    }
}
implicit def stringToPerson(name : String) : Person = new Person(name)

With this implicit in effect, we can create a Conversation without even mentioning the Person class:

val conv = new Conversation("John", "Mary")
conv.greet()

See how we passed two strings to the Conversation constructor, even though it’s supposed to take two Persons?

Global variables don’t really come into it, it’s more about implicit type conversion.

Why would you do this? Well for one, it lets you create faux multimethods:

class FrameWrapper(frame : JFrame) {
    def title_=(s : String) = frame.setTitle(s)
    def visible_=(b : Boolean) = frame.setVisible(b)
}
implicit def wrapFrame(frame : JFrame) : FrameWrapper = new FrameWrapper(frame)

This means we can call FrameWrapper methods on a JFrame and the Scala compiler will figure out what we mean:

val frame = new JFrame
frame.title = "My Application"
frame.visible = true

Normally you’d have to call setTitle/setVisible at the second and third lines of the code above because JFrame doesn’t have any public attributes called “title” and “visible”. However, with the implicits in effect the Scala compiler will wrap your JFrame in a FrameWrapper instance to call that method. A weak example to be sure, but it really is a nice way to “scala-ize” any existing Java code. All this is statically checked too, so it won’t try to set your children on fire when you’re not looking.

Nifty eh?

In many open source software projects, full backwards compatibility is ensured between minor point releases (for example, 1.2.3 to 1.2.4). Generally speaking, these releases are made mostly to get important defect and/or security fixes to the public in a relatively timely manner. PHP is a notable exception to the rule: I haven’t been following development all that closely of late, but in the past it was not uncommon to break backwards compatibility between point releases.

Another major exception to the rule that hits a little closer to home is Ruby. In the past, backwards-incompatible changes have crept into point release changes — I’m uncertain as to whether or not this was intentional. However, the Ruby project also provides what’s known as a “patchlevel” release for each given point release. The patchlevel counts the number of patches applied to any given point release. Generally speaking, these seem to be bug and security fixes.

If you’re interested, the patchlevel release of your Ruby installation can be seen like so:

[ tom ] ~
$ ruby --version
ruby 1.8.x (YYYY-MM-DD patchlevel XXX) [i486-linux]

Anyway, over the past few days I’ve been watching the discussion leading up to patchlevel releases of Ruby 1.8.6 and 1.8.7. It’s been an interesting experience. You can follow the discussion from message #17499 here.

First there is an announcement that new versions of Ruby will drop in three days’ time. Shortly after this, the announcement that both versions are failing numerous rubyspecs (57 failing for 1.8.7!). Then a report that memory leaks may be present in both versions, followed by a report that one of the tests hang in Win32. Eventually a request to delay the release came through, along with a recommendation by Charles Nutter (the guy who brought us JRuby) to implement continuous integration.

I’m really surprised at a few things:

• The original decision to make a release was based upon “I think current 1.8.6/1.8.7 is [more] stable than p230/p22″. Please, please tell me that the decision that the trunk is more stable than a previous release wasn’t based on a gut feeling. Please.
• No continuous integration. It’s very important for software projects to maintain and continuously run a set of regression test. Otherwise, sooner or later they will break their users’ code.

Ruby is a great language and I applaud the work of the developers — maintaining a beast like the Ruby core for naught but love must be hard work. However, I have to say that I think their fumbling to get a stable release out the door on short notice — especially in the face of their recent security problems — is a concern for anybody relying on the Ruby code base. All that said, it sounds like the project is already taking steps in the right direction based on an earlier email to the list from Matz.

One day in the near future I’m sure that cutting a Ruby release won’t be quite so painful, but the lesson here is clear: Exercise your tests frequently — and ideally, automatically — or getting a stable release out the door might be nigh on impossible.