This article describes the motiviation behind, and the techniques used to create, Delancey. Delancey is an online bookmark manager that enhances the del.icio.us social bookmarking service. Delancey presents an alternate view of a user‰Ûªs bookmarks by keeping track of which ones are used most frequently.
Preface:
As I was writing a draft of this article, I learned that del.icio.us would be acquired by Yahoo!. So there it is. Just one week after Delancey was released, del.icio.us was snapped up. Coincidence?
While I write about a connection in jest, there is actually an element of truth there. Not about Delancey having anything to do with the del.icio.us acquisition itself -- Delancey is just one tiny del.icio.us enhancment among hundreds -- but the same spirit that made Delancey possible may also explain the Yahoo! deal.
Why is del.icio.us a good fit for Yahoo? It is probably not just the users -- there are maybe 300,000 del.icio.us users in total, and demonstrably far less than that in active daily users. To Yahoo!, a network with 100 million unique visitors each month, the incremental gain of adding del.icio.us users is statistically negligiable. (Not to mention, most of them were probably Yahoo! users anyway.) Nor would it be del.icio.us' technology itself. Yahoo already effectively already has all del.icio.us has and more with it's My Web products.
No, my guess is that Yahoo! bought del.icio.us because of a) Joshua Schachter, and b) what del.icio.us represents. Yahoo! has already demonstrated that they get Web 2.0, and are moving full-speed into that world. Joshua not only gets Web 2.0, he is helping invent it. And that is precisely what del.icio.us represents. Not the bookmarks, not the tagging, but the open-ended nature of the service. Technology whose true value increases exponentially as it expands. At the heart of this is an open API, a foundation on which services like Delancey (and those 100's of other applications) can be built. Similar to the motivation behind acquiring Flickr, if you can tap into this collective spirit, into the power of "other people's work," then there are no limits on what can be done.
(And be sure to read a great writeup on social tagging by my good friend, Sean.)
Motivation:
Joshua has said that he created del.icio.us because he wanted a better way to organize his own bookmarks. In other words, he built something that he knew would be useful, if for no other reason than he would be using it. Along those lines, I built Delancey because I wrote something like it years ago, and wanted to try it again.
And that seems to be a successful methodology: I set the homepage of all of my web browsers to Delancey last week, and found that it really works amazingly well as a simple starting point for browser sessions.
Praise for Delancey:
A handful of the nice things other people have said about Delancey:
"This is the best add-on to de.licio.us I've seen. It's useful. It's visually appealing. It actually improves instead of complicates."
"Delancey is for the first time convincing me to use delicious and tags in the way I think they've always been intended."
"You're brilliant for making it work with delicious, rather than reinventing the wheel."Statistics:
Delancey took off in the first 48 hours -- around 5,000 people visited Delancey in the first 2 days. Over 800 people clicked on at least one of their bookmarks via Delancey. Over 200 claimed their del.icio.us username. All told, there were just shy of 4000 bookmark clicks in total during that initial burst.
The top five bookmarks in terms of total clicks across all users (not including the demo user) are:
Technology:
Delancey is divided into two parts. The first part is a backend service that listens for incoming REST requests and manages the bookmark state. This is the part that communicates with del.icio.us and with the local Delancey database. The core of this backend service is a Perl module called DelanceyService.pm that runs inside an Apache httpd process via mod_perl. The Delancey code is written on top of Essex, the component toolkit that I wrote earlier this year. Essex forms the foundation for almost everything in the new project. Delancey uses an old version of MySQL (3.23, which comes installed on RedHat Fedora Core 2), but up-to-date version of everything else.
The second part of Delancey is a rather involved bit of JavaScript called delancey-0.1.js. This JavaScript (which relies on parts of a pre-release of prototype.js and an alpha build of md5.js) is responsible for making AJAX requests to the Delancey backend server and presenting an rich user interface.
The JavaScript portion of Delancey consists of over 2,500 lines of code. The backend code is only half as long. And in terms of the time invested in each, I'd estimate that the JavaScript took almost 80% of the total hours. (Averaging 1 hour a day for 5 weeks. Yikes! Not my most efficient coding ever.)
A major factor in the inefficient development process was the learning curve in JavaScript. Over the years I've taught myself C, C++, Java, Perl, and enough Python, PHP, Lisp, Tcl, etc., to write programs of comparable complexity to Delancey. But I'll admit it -- this was the hardest language to write a real program in that I've ever learned.
It's not the language itself, which is generally rather reasonable and predictable. (Though the hoops that one must jump through to even approximate object-oriented programming are appalling.) It is the libraries and toolkits, particularly the DOM and DHTML extensions, that are required for browser-based programming that really hold JavaScript back. The documentation, or lack thereof, with JavaScript code is a major limitiation. The differences between browsers is maddening. (Delancey currently works in Firefox, Safari, and Opera -- and getting it to do so took perhaps 30% of my time.) I've blocked a bit of it out of my head so I'm calm right now, but there were moments that I simply had to get up and walk away from the keyboard in frustration.
Another way to explain the frustration is that I rewrote delancey.js from scratch 3 times. I hardly went back and changed a single line of the backend code once it was written. This is partly because I was learning as I went along, but partly because patterns that made sense and felt natural in server-side coding didn't have an intuitive analogy on the client. I'm still wrapping my head around cross-platofrm event-driven programming, especially when the event models differ so idiosyncratically from client to client.
Challenges:
There were several challenges in building Delancey.
The first was on how to get the bookmark data to the client. I wanted to reuse the bookmarks from del.icio.us, rather than ask users to invest their time in yet another bookmark server. (Though as you can see from the code, it would trivial to modify Delancey to store bookmarks in Delancey if it made sense to so.) I wanted to use AJAX-techniques to keep the page-refresh times down and provide a more desktop-like experience. But there was a problem -- the XMLHttpRequest object can't make calls to two different servers. Unlike del.icio.us director, I couldn't use the approach of adorning del.icio.us, as the client then wouldn't be able to call out to unto.net. So I had to compromise and have the client connect to the Delancey server, and have the Delancey server in turn connect to del.icio.us.
Which led to the second challenge -- how could Delancey make proxy requests to del.icio.us without asking the user for their del.icio.us password? Fortunately, this is where the power of the open del.icio.us network was fully realized. Although the official del.icio.us APIs are too limited to be useful here (more on this in a later post), the JSON-based feed URLs provide just enough public information to power Delancey. The Delancey server is thus able to make HTTP requests to del.icio.us (caching the results to reduce load) and merge the source bookmark data with the local additions, such as display names and click counts.
Also related was a self-imposed challenge: I wanted to keep usernames, passwords, and even URLs out of the Delancey database. If for some reason the Delancey database were to fall into the hands of some nefarious organization, they wouldn't immediately know anything about your surfing habits. (Nor would I, for that matter.) Nothing short of complicated public/private key systems would keep things truly secure, but by aggressively using one-way hashes calculated on the client, I could at least keep usernames and other identifying data out of the database. While the code hasn't been externally audited, I am at least confident in the effort I made.
And that challenge -- keeping usernames out of the Delancey database -- was further amplified when I decided that I wanted to allow users some way to "claim" their username. I didn't want to add full-fledged Delancey accounts (that will have to wait for the Orchard core to come online), but I did want some way for users to lock down their Delancey bookmarks so that only they could access them. Doing so proved to be rather tricky. The first several ideas I had all involved sending a secret token to the user's del.icio.us account, and then using that token (which only they would know) as a means of gaining access to Delancey. But this would have required some way of sending a private bookmark on del.icio.us -- an frequently-requested feature, but one not available at this time. The solution was to share a secret between the client and Delancey (in the form of a password), and use a public bookmark to verify the user on del.icio.us. In the end I am happy with the elegance of the solution, especially given the limitations of available APIs. (If anyone wants me to I can write up a full description of the Delancey/del.icio.us claim system.)
The only other real challenge was, as I said, the DHTML coding itself. Cross-browser inconsistencies are frustrating. High quality (i.e., accurate and comprehensive) online crosss-browser documentation is non-existent. Decent JavaScript libraries are few and far between. (Something to consider for library designers -- if you design widgets, consider being back-end neutral, and consider breaking up your scripts to be more modular and lightweight.) One bright spot out there is QuirksMode.org, a site without which I wouldn't have had the patience to finish debugging Delancey. Two JavaScript library ideas for others: 1) a "crossplatform.js" that either patches the inconsistencies across browsers, or provides a workaround, and 2) a cross-browser implementation of either the GTK+ or the WinForms 2.0 widget libraries.
Open Source:
All of the code that I wrote to build Delancey is licensed under a Creative Commons Attribution-ShareAlike Common Deed. (I.e., you can use it, even commercially, but you need to give credit if you do.) You can view the source code for version 0.1 of Delancey (the version currently online) in the Subversion repository for Delancey 0.1. I haven't invested any time at all in making it easy for someone else to install, but at the very least it might prove inspirational (or alternately, mind-numbingly confusing) to another developer.
The Delancey APIs are also public, though the documentation is a bit sparse. But the nice thing is that the API docs are generated on-the-fly from the configuration files, so they are guaranteed to be up-to-date.
Future enhancements:
A list of things that I would like to see in Delancey:
- A way of letting people use Delancey even if they've never heard of del.icio.us. This will probably have to wait until the Orchard core is up and running.
- Support for Furl, or other non-del.icio.us bookmark backends. This shouldn't be too hard, as all of the del.icio.us interaction is abstracted by a agnostic web-services API.
- An optional Greasemonkey script that downloads a list of your bookmarks and pings the Delancey servers whenever you visit a page you've previously bookmarked. Thus you can increment and keep track of bookmarks counts even if you don't get to the page directly via Delancey.
- An optional Greasemonkey script that dynamically modifies the del.icio.us website to display and track click counts.
- A preference panel for different display options and sort orders. The code is already there, but there is no interface for the user to change their preferences.
- A dynamic "all" tag that shows what you've clicked on across all your tags.
- A "zeitgeist" page that displays everyone's most clicked on URLs. (Anonymously and in aggregate, of course.)
To be honest, now that Joshua has all of the support he needs at Yahoo!, I'll probably put further Delancey enhancements on the back burner. That's the downside of the acquisition, but one that will probably work out best for del.icio.us users in the long run.
Thanks:
I want to thank Robert, Greg, Stephen, Jessica, Chris, and everyone that has written about Delancey and made this so much fun.