Comments on: Ruby Releases Are Scary (Or: How CI Can Save Your Ass)

By: OJ

OJ — Mon, 07 Jul 2008 09:26:17 +0000

@Ben: That’s true I have seen that in the past as well. It’s becoming more common to find teams that just rely too heavily on the CI build. The purpose of CI builds should be made clear to the dev team (if it’s not already!) in that it is not a catch-all. Just because a CI build is in place it doesn’t allow them to be cowboys who think that everything is fine unless the CI build breaks.

By: Ben

Ben — Sat, 05 Jul 2008 18:59:01 +0000

@tom & OJ

You bring up a good point about communication - CI does add value by letting every developer know that the tests will be run (and that everyone will be alerted if there are problems). And, as you said, it’s nice to have it in place to catch those mistakes we all end up making sooner or later. For those reasons and others, I think CI is valuable and should be used.

That said, I still think that CI should really be the safety net or the last line of defense for automated tests (OJ - I agree it should be the first line of defense upon check-in, but teams should have ‘defense’ way before check-in). I think sometimes teams come to rely on CI too much because their test suite has gotten too large or too slow to run on individual developer machines. I’d like to see teams address the problem of getting fast pre-checkin feedback via tests first and foremost and use CI as a safety net. Ideally, CI would catch very few problems, because most would be caught before.

By: OJ

OJ — Sat, 05 Jul 2008 10:43:47 +0000

@Ben: I don’t agree. It should be the first line of defense upon check-in. If a build is broken (either by a failed compilation or a unit test failure) that can be throw up in the face of the developer in a heartbeat - well before anyone else in the testing cycle gets their hands on it.

Yes, devs should “get latest”, build and run the test suite before checking in, but the CI server should still be there to make sure everything is honky-dory.

As Tom said, you can’t trust every developer to do the right thing. But on top of that, there’s always a risk of different devs checking in different source files at the same time resulting in a broken build … even if they’ve followed the correct process.

By: tom

tom — Sat, 05 Jul 2008 03:21:49 +0000

@OJ: haha, very true

@Ben: It’s not so much about finding broken tests as it is about communication. CI ensures that your tests will — at some point — be exercised rather than trusting developers to run their test suite prior to every commit. And when they break, *everyone* gets to know about it. As I’m sure you’re aware, even the most diligent programmers sometimes get lazy and/or make honest mistakes.

This is Ruby, which been subject to far more developer scrutiny than your average internal software project — and still 57 specs were broken in the trunk, seemingly without anyone being aware of it.

By: Ben

Ben — Fri, 04 Jul 2008 13:40:13 +0000

CI is great, but in my opinion, it really should be a last line of defense. While not as bad as human QA finding bugs (or, even worse, end users), the feedback that CI provides is still far too delayed. I’d argue that any project should strive to make sure every developer can run the full suite on their desktop (and make it fast enough that they can do it often).

By: Phill

Phill — Fri, 04 Jul 2008 01:11:42 +0000

You know, I spent a shameful amount of time trying to get 1.8.7 to compile on Solaris, before deciding that maybe it wasn’t my fault and reverting back to 1.8.6, which worked like a charm.

Just sayin’.

By: OJ

OJ — Thu, 03 Jul 2008 22:35:32 +0000

The dark ages of not using CI for stuff like this should be well and truly behind us. Unfortunately, it’s not the case. There are still a surprising amount of projects, small and large, unknown and well-known, that are built without some form of CI. In my view that’s just nuts.

10 years ago, getting a CI setup together was a pain in the ass, and I can understand why people would consider not “wasting time” on it. These days you can have a basic CI build set up in under an hour. There really isn’t an excuse for not doing it.

I guess we should at least be grateful that they use source control