Ruby Releases Are Scary (Or: How CI Can Save Your Ass)
July 4th, 2008
In many open source software projects, full backwards compatibility is ensured between minor point releases (for example, 1.2.3 to 1.2.4). Generally speaking, these releases are made mostly to get important defect and/or security fixes to the public in a relatively timely manner. PHP is a notable exception to the rule: I haven’t been following development all that closely of late, but in the past it was not uncommon to break backwards compatibility between point releases.
Another major exception to the rule that hits a little closer to home is Ruby. In the past, backwards-incompatible changes have crept into point release changes — I’m uncertain as to whether or not this was intentional. However, the Ruby project also provides what’s known as a “patchlevel” release for each given point release. The patchlevel counts the number of patches applied to any given point release. Generally speaking, these seem to be bug and security fixes.
If you’re interested, the patchlevel release of your Ruby installation can be seen like so:
[ tom ] ~
$ ruby --version
ruby 1.8.x (YYYY-MM-DD patchlevel XXX) [i486-linux]
Anyway, over the past few days I’ve been watching the discussion leading up to patchlevel releases of Ruby 1.8.6 and 1.8.7. It’s been an interesting experience. You can follow the discussion from message #17499 here.
First there is an announcement that new versions of Ruby will drop in three days’ time. Shortly after this, the announcement that both versions are failing numerous rubyspecs (57 failing for 1.8.7!). Then a report that memory leaks may be present in both versions, followed by a report that one of the tests hang in Win32. Eventually a request to delay the release came through, along with a recommendation by Charles Nutter (the guy who brought us JRuby) to implement continuous integration.
I’m really surprised at a few things:
- The original decision to make a release was based upon “I think current 1.8.6/1.8.7 is [more] stable than p230/p22″. Please, please tell me that the decision that the trunk is more stable than a previous release wasn’t based on a gut feeling. Please.
- No continuous integration. It’s very important for software projects to maintain and continuously run a set of regression test. Otherwise, sooner or later they will break their users’ code.
Ruby is a great language and I applaud the work of the developers — maintaining a beast like the Ruby core for naught but love must be hard work. However, I have to say that I think their fumbling to get a stable release out the door on short notice — especially in the face of their recent security problems — is a concern for anybody relying on the Ruby code base. All that said, it sounds like the project is already taking steps in the right direction based on an earlier email to the list from Matz.
One day in the near future I’m sure that cutting a Ruby release won’t be quite so painful, but the lesson here is clear: Exercise your tests frequently — and ideally, automatically — or getting a stable release out the door might be nigh on impossible.
Categories: Compilers, Ruby, Software Development |
The dark ages of not using CI for stuff like this should be well and truly behind us. Unfortunately, it’s not the case. There are still a surprising amount of projects, small and large, unknown and well-known, that are built without some form of CI. In my view that’s just nuts.
10 years ago, getting a CI setup together was a pain in the ass, and I can understand why people would consider not “wasting time” on it. These days you can have a basic CI build set up in under an hour. There really isn’t an excuse for not doing it.
I guess we should at least be grateful that they use source control
You know, I spent a shameful amount of time trying to get 1.8.7 to compile on Solaris, before deciding that maybe it wasn’t my fault and reverting back to 1.8.6, which worked like a charm.
Just sayin’.
CI is great, but in my opinion, it really should be a last line of defense. While not as bad as human QA finding bugs (or, even worse, end users), the feedback that CI provides is still far too delayed. I’d argue that any project should strive to make sure every developer can run the full suite on their desktop (and make it fast enough that they can do it often).
@OJ: haha, very true
@Ben: It’s not so much about finding broken tests as it is about communication. CI ensures that your tests will — at some point — be exercised rather than trusting developers to run their test suite prior to every commit. And when they break, *everyone* gets to know about it. As I’m sure you’re aware, even the most diligent programmers sometimes get lazy and/or make honest mistakes.
This is Ruby, which been subject to far more developer scrutiny than your average internal software project — and still 57 specs were broken in the trunk, seemingly without anyone being aware of it.
@Ben: I don’t agree. It should be the first line of defense upon check-in. If a build is broken (either by a failed compilation or a unit test failure) that can be throw up in the face of the developer in a heartbeat - well before anyone else in the testing cycle gets their hands on it.
Yes, devs should “get latest”, build and run the test suite before checking in, but the CI server should still be there to make sure everything is honky-dory.
As Tom said, you can’t trust every developer to do the right thing. But on top of that, there’s always a risk of different devs checking in different source files at the same time resulting in a broken build … even if they’ve followed the correct process.
@tom & OJ
You bring up a good point about communication - CI does add value by letting every developer know that the tests will be run (and that everyone will be alerted if there are problems). And, as you said, it’s nice to have it in place to catch those mistakes we all end up making sooner or later. For those reasons and others, I think CI is valuable and should be used.
That said, I still think that CI should really be the safety net or the last line of defense for automated tests (OJ - I agree it should be the first line of defense upon check-in, but teams should have ‘defense’ way before check-in). I think sometimes teams come to rely on CI too much because their test suite has gotten too large or too slow to run on individual developer machines. I’d like to see teams address the problem of getting fast pre-checkin feedback via tests first and foremost and use CI as a safety net. Ideally, CI would catch very few problems, because most would be caught before.
@Ben: That’s true I have seen that in the past as well. It’s becoming more common to find teams that just rely too heavily on the CI build. The purpose of CI builds should be made clear to the dev team (if it’s not already!) in that it is not a catch-all. Just because a CI build is in place it doesn’t allow them to be cowboys who think that everything is fine unless the CI build breaks.