From Python to Ruby on Rails to Erlang…

We started working on SlideAware Workgroups about a year ago. After collecting feedback from Alpha users in November 06, we decided to build the simpler SlideAware Personal which allows users to publish presentation to a URL and conduct live slideshows. SlideAware Personal has been live since March 07. SlideAware Workgroups will be released in the next few months. Both applications share the same architecture.

There are really two front-ends:

• The PowerPoint toolbar which provides buttons to allow users to very easily send slides to SlideAware or get feedback from co-workers. It is written in Visual Basic Application and leverages external DLLs for image and flash generation, file compression, etc.

• The web application leverages Dojo and Scriptaculous and interacts with the server using XmlHttpRequest. The user interface is entirely done in JavaScript in the browser. The JavaScript code manipulates the business objects (slides, presentations, tags, users,  etc.) and generates the user interface using Dojo widgets. The server is invoked to persist and query objects. It serves data as JSON instead of sending back HTML snippets.

The back-end has gone through several iterations.

We wrote the first version (0.1) of SlideAware Workgroups using Python and TurboGears. TurboGears combines a template engine (Kid), an object-relational mapping manager (SQLObject). Data (slides, presentations, tags, users) was stored in a SQLite database, while text indexes for the slides’ content were stored using Lucene. We leveraged xml-rpc to communicate between the python server and the java server (Lucene). We had a Jython wrapper on the java side.

We decided to switch to Ruby on Rails for the second version (0.2) of SlideAware Workgroups. Rails has a much larger following and finding solutions to issues is therefore a lot easier. Because the presentation layer is entirely done in the browser in JavaScript, our back-end code is fairly lean and was ported to Ruby on Rails in a matter of weeks. We kept SQLite, xml-rpc and Lucene. We added Lighttpd in the mix as our web server.

At that point, we started wondering about how our heterogeneous back-end stack could be made available, reliable and scalable. We had been hearing for a while about Erlang, the concurrent programming language initially developed at Ericson,  and Mnesia the real-time distributed database written in Erlang that is part of Erlang’s Open Telecom Platform,  and Yaws the web server written in Erlang.

We liked the idea of replacing our multiple language stack (Lighttpd + RoR + SQLlite + XMLRpc + Jython + Lucene) with an Erlang-only solution (Yaws + Mnesia + Erlang).

We were particularly intrigued by the promises of:

• High concurrency: Yaws leverages Erlang’s  light-weight threading system and supposedly performs very well under high concurrency. Mnesia’s tables can be replicated across nodes to allow more reads to happen concurrently.
• High performance: Mnesia resides in memory and is built to allow fast real-time lookups.
• High reliability: Erlang flagship project, the AXD301 which has over 2 million lines of Erlang has achieved a NINE nines reliability (yes, you read that right, 99.9999999%).

So we made the switch for version 0.5 of SlideAware Workgroups.

In my next post, I’ll tell you whether Erlang lived up to our expectations…

Comments

[Brett]

"Premature optimization is the root of all evil."

How many majillion business critical powerpoint slides are you guys really serving?

Sounds like you're having a lot of fun anyway.

[bmm6o]

your NINE nines link is broken, should be http://www.sics.se/~joe/thesis/armstrong_thesis_2003.pdf

[Vijay Chakravarthy]

@Brett: I'd say "Quotes taken out of context is the root of all evil", but you knew that anyway :).

We are definitely having a lot of fun. We are not trying to optimize the product in advance, but rather making sure that the cost of such optimizations, should we need to make them, is low.

We believe Erlang enables this in our problem domain..
Regards,
Vijay

[John]

Just curious: How do you do the fulltext search (Lucene) in Erlang?

[Vijay Chakravarthy]

Hi John,
We wrote our own search engine in Erlang. It didnt need to have all the features of lucene, since in our problem space, we only needed to do simple boolean search.

Writing a inverted index based search engine in Erlang is quite straightforward, mainly due to two reasons:
1. Mnesia and dets can easily handle "the last n additions" problem. This allows you to structure a search engine as a combination of merging responses from two queries -- the last n additions to the index which havent been merged into the file based index, and the results from the file based inverted index.

2. For file based inverted indexes (like the segments in lucene), the bit syntax allows you to write very efficient (Golomb coded) structures, while the standard library has some really nice modules like file_sorter which are useful..

For us, the problem was even simpler, since typically PowerPoint documents do not contain large amounts of text, so a simple mnesia based inverted index with some clever optimizations suffices. It also solves the problem of making the search engine highly available.

Regards,
Vijay

[Dennis Peterson]

As some C++ guy said, "Belated pessimization is the leaf of no good." :)

I wouldn't expect you to share anything too specific to your business, but sometime if you wanted to write up a tutorial about writing search engines in Erlang, I'd really be interested...

[Vijay Chakravarthy]

Hi Dennis,
We will certainly consider doing that. We have benefited a lot from Erlang, and want to give back to the community as much as possible.
Regards,
Vijay

[Vijay Chakravarthy]

Joe Armstrong has an inverted index based search engine in
his forthcoming book on erlang, FYI..

http://www.pragmaticprogrammer.com/titles/jaerlang/

Regards,
Vijay

[Mike McNally]

"Because the presentation layer is entirely done in the browser in JavaScript"

Looks like I can't even get into my account without having Flash installed (which is not an option for me as my platform is unsupported). So is "in Flash" what you mean by "in Javascript"? Or is the Flash requirement (which is unmentioned in the FAQ, incidentally) only there as a barrier to non-Flash users from gaining access to the application?

[Didier Prophete]

Hi Mike,

We use Flash to render PowerPoint slides into the browser. This allows us to get scalable renderings without having to worry about browser incompatibilities.

Thanks for pointing out that the Flash requirement is not in the FAQ. We will add this shortly.

Out of curiosity: which platform do you use ?

-Didier

[mike]

you moved from TG to rails because rails has a bigger community to help with problems, fine. but then you moved.....to erlang ? what are there, 5 websites on the whole planet using erlang right now ? the rationales for those two platform switches are incongruent.

[Vijay Chakravarthy]

Well Mike,
I think that if you dig a little deeper, you will find that there is a substantial amount of work that has been done for erlang. In fact, I invite you to take a look at erlang.org, specifically at the OTP documentation.

Erlang and OTP have existed for a lot longer than Ruby and Rails. And to counter your point, there are a lot more people being served by Erlang technology than by Ruby or Rails. The fact that they are not being served by a web site interface is largely irrelevant.

[Jenkers]

Wolfgang, where's the followup? We're dyin here! :)

[Mike McNally]

Re: what platform -> I'm on 64-bit Linux, and Adobe has been very remiss in supporting the platform.

[Seth W. Klein]

On platforms not able to install flash: I'm on PPC (iBook) Linux, and flash has never been available (unless you can get the qemu hackery working, and even then it's dog slow, I'm told).

[Florian]

have you tried using gnash (if available for your platform, which is probable) ?