Nice article in the Linuxjournal
Using Ruby and SQLite to create Linux clusters that take advantage of idle
nodes and bypass expensive software solutions.
My friend Dave Clements always is game for a brainstorming session,
especially if I’m buying the coffee. Recently, we met at the usual
place and I explained my problem to him over the first cup. My office had a
bunch of Linux nodes sitting idle and a stack of work lined up for them,
but we had no way to distribute the work to them. Plus, the deadline for
project completion loomed over us.
Over the second cup of coffee, I related how I had evaluated several
packages, such as openMosix and Sun’s Grid Engine, but ultimately had
decided against them. It all came down to this: I wanted something leaner
than everything I’d seen, something fast and easy, not a giant
software system that would require weeks of work to install and configure.
After the third cup of coffee, we had it: Why not simply create an
NFS-mounted priority queue and let nodes pull jobs from it as fast as they
could? No scheduler, no process migration, no central controller, no kernel
mods—simply a collection of compute nodes working as fast as possible
to complete a list of tasks. But there was one big question: was accessing
an NFS-mounted queue concurrently from many nodes possible to do safely?
Armed with my favorite development tools—a brilliant IDE named Vim
and the Ruby programming language—I aimed to find out. link