Using Ruby and SQLite to create Linux clusters that take advantage of idle nodes and bypass expensive software solutions.

My friend Dave Clements always is game for a brainstorming session, especially if I’m buying the coffee. Recently, we met at the usual place and I explained my problem to him over the first cup. My office had a bunch of Linux nodes sitting idle and a stack of work lined up for them, but we had no way to distribute the work to them. Plus, the deadline for project completion loomed over us.

Over the second cup of coffee, I related how I had evaluated several packages, such as openMosix and Sun’s Grid Engine, but ultimately had decided against them. It all came down to this: I wanted something leaner than everything I’d seen, something fast and easy, not a giant software system that would require weeks of work to install and configure.

After the third cup of coffee, we had it: Why not simply create an NFS-mounted priority queue and let nodes pull jobs from it as fast as they could? No scheduler, no process migration, no central controller, no kernel mods—simply a collection of compute nodes working as fast as possible to complete a list of tasks. But there was one big question: was accessing an NFS-mounted queue concurrently from many nodes possible to do safely? Armed with my favorite development tools—a brilliant IDE named Vim and the Ruby programming language—I aimed to find out. link