Apr

11

Evaluating Open Source Software Projects and Companies

Filed Under Technology | 1 Comment

Undertaking evaluation of open source software for the ? time (too many times), I decided to look for new insights into evaluating software.

As a little background, I always try to understand first what I absolutely need for my personal project. Then, as I compile a list of software that most closely solves the needs of my personal project, I’ve always reviewed open source software projects for a business model to drive growth and hints of momentum. I comb in the search engine indices and the projects’ forums (one great gauge is that many of the projects with the best momentum are inside my CPanel, and others are on simplescripts via Bluehost). But I’ve never compiled a list of criteria to systematize my evaluation and this morning I decided to tackle the problem for a couple hours.

I hoped that by 2009 some one had put together a reliable, systematic way of evaluating open source projects. I got excited when I discovered OpenBRR, backed by the likes of O’Reilly and Intel.  Then I realized my excitement was premature as the last news on the initiative was in 2006 and the home page hadn’t been updated since May 2007. Finally, the phpBB forum powering the community section of the website wasn’t even working.

phpBB : Critical Error

Error creating new session

DEBUG MODE

SQL Error : 1016 Can’t open file: ‘phpbb_sessions.MYI’ (errno: 145)

INSERT INTO phpbb_sessions (session_id, session_user_id, session_start, session_time, session_ip, session_page, session_logged_in, session_admin) VALUES (‘7900ec6a20082801c3ed5e2cc176727e’, -1, 1239455790, 1239455790, ‘d839a34e’, 0, 0, 0)

Line : 187
File : sessions.php

It’s a little ironic that an initiative established to help companies know which open source projects have momentum and are trustworthy has neither momentum nor reliable, up-to-date information. Oh well.

I was much more encouraged by the direction and momentum of Coverity’s Architecture Library. It provides quick insights into a project’s feature set, but not exactly what I’m after. I’m looking more at how to choose from my list of potential solutions.

Hands-down, David Wheeler has compiled the most comprehensive document for evaluation.  It’s also the most up-to-date (revised March 14, 2009). It’s a little overwhelming for the scale I’m working on, but has a well-thought out methodology for a high-stakes decision with the budget to do this level of research.

On a pratical level, my favorite insights came Matt Johnston, a JAVA engineer. He suggests:

  1. studying the documentation
  2. checking the community for activity outside the sponsoring company
  3. reviewing the project lifetime and release cycle
  4. and looking for signs of momentum (this is much more important for younger projects)

For example, with last release date of 2005, osCommerce, by his criteria, is dead and does not have momentum. Magento Commerce, on the other hand, is backed by a business model and has had two major version since it’s 2007 launch. It is also being talked about a lot in the community.

I found some great insights from the comments on a CIO article, Evaluating an Open-Source Project’s Business Model. In addition to Matt’s ideas, these insights included:

  1. checking for formal support
  2. a roadmap that can be influenced by the community
  3. ISV endorsements
  4. partners
  5. and local user groups.

Interestingly, the comment authors’ companies (Pentaho, Alfresco, etc.) I’ve seen pop-up in articles in many places online, signs of active, vibrant open source companies.
Finally, Keith Harrison-Broninski recommends:

  1. looking for a standard license (especially important for corporate adoption)
  2. number of committers who post regular updates
  3. and whether the project is backed by major VCs or companies.

I’m now going to conquer my current problems with these insights. Enjoy.