Archive for the 'Ruby on Rails' Category
Catalog Choice registers half a million users! Monday, January 28th, 2008 by Matt
Catalog Choice on the Today Show.
On January 24, Catalog Choice saw its biggest day yet, when it was covered in a fantastic piece on NBC’s “The Today Show”:
Over the course of the day, the catalogchoice.org website saw over two million page views, and registered 60,000 new user accounts, bringing the total number of registered users, three days later, to over 500,000!
In addition, “Catalog Choice” was the number one search term for the day on Google:
Coping with the traffic.
Coping with a sudden increase in traffic, orders of magnitude more than typical, was a challenge. The front-end web application servers quickly became overloaded, and later the back-end DB server became overloaded (we were servicing over 2,000 DB queries per second!) Since it’s still not possible (with our hosting providers, at least) to bring on additional servers on-demand, we quickly made several modifications to the application:
We made a number of layout modifications in the application that would allow us to cache content to a far greater extent.
These same modifications also targeted the reduction of DB queries.
With these modifications, we were able to cope well with the secondary traffic surge.
Lessons learned.
It’s quite possible that Catalog Choice is now one of the largest Ruby on Rails applications running on the internet, in terms of number of users. Over the past few months of operation, we’ve learned some lessons:
Although not related to Rails, we’ve learned that it’s a good idea, especially for a site with this broad of a user base, to be conservative on the use of client-side technology. When originally launched, we had implemented elegant page transitions, catalog finder live type-ahead, and other similar UI features — all done with JavaScript (AJAX) in a way that gave the site a desktop-application feel. We considered this acceptable practice, as we were designing for IE 6/7, Safari 2/3 and Firefox 2/3.
However, when you have 500,000 users, even 1% on older browsers represents quite a large crowd! So we’ve since modified the site to work in a far more traditional manner, relying very little on client-side JavaScript, and where necessary, degrading very gracefully.
For hosting, our infrastructure, like many these days, is based on virtual machines. We have N number of front-end web application servers, each practically maxed out in terms of CPU and memory. Based on the experience with the Today Show traffic, we’re thinking now that it might be better to have 2N front-end servers, each with half the CPU and memory, since it’s a lot easier to quickly add CPU and memory to an existing server (to meet demand), than it is to bring on additional VMs. (This is, assuming 2N front-end servers with half the memory are roughly comparable in cost to N servers with double the CPU and memory, which might not be the case.)
It has been a very exciting experience to watch the site grow, analyze the usage patterns, and adjust the application and its user interface to not only improve the usability and user experience, but to adapt to the changing user profiles (i.e. now that over 500,000 of our visitors are no longer first-timers, and that we have over 1,000 merchants in the system.)
How the site is doing.
When the site first launched, the consumer response was (and continues to be) nothing short of amazing. It is clear that this site is meeting a very big need in the United States; that is the reduction of unwanted paper catalogs. The industry’s response was, expectedly, lukewarm, especially after the Direct Marketers Association (the DMA) issued an email to all its members to “Just say no!” to Catalog Choice.
However, with half a million vocal consumers behind it, Catalog Choice has become an influential heavyweight. A website feature we launched last week alerts users to which specific merchants have refused to honor their opt-out requests, and provides the merchants customer support telephone number, just in case the consumer would like to give them a call. Within 24 hours, after being inundated with phone calls from angry customers, we had merchants changing their minds :-)
A misconception in the industry (promoted by the DMA) is that Catalog Choice seeks to do away with catalogs altogether. That couldn’t be further from the truth. Catalog Choice is about doing away with just those catalogs that are unsolicited and unwanted.
All in all, Catalog Choice has been a fantastic project for MakaluMedia. We’re fortunate to be one of very few companies having the opportunity to build and operate such a large-scale Rails site, and a site that serves such a meaningful social purpose!
Phenomenal response to Catalog Choice Tuesday, November 6th, 2007 by Matt
HARDLY A MONTH HAS PASSED since we announced the launch of Catalog Choice, a website that MakaluMedia was chosen to design and develop by an amazing team of funding and supporting organizations, and the response to the site has been extraordinary.
In less than 25 days, more than 100,000 users have registered, opting out of more than 800,000 catalogs. And for us, as user experience designers and product developers, one of the most exciting statistics is the near 50% conversion ratio, meaning that nearly one out of every two visitors to the site has registered.
Update November 21, 2007: We are about at the one month point now, and have over 180,000 users, opting out of nearly two million catalogs!.
Catalog Choice has also been featured on the national television networks CNN, NBC and ABC (see videos below), and has been well received in the press and blogosphere:
- The New York Times
- The Boston Herald
- The Chicago Tribune
- Guy Kawasaki
- Joel on Software
- Adam Engst at TidBITS
The success of the site has served to validate much of our beliefs regarding user experience design. Some of principles that stand out include:
- Simple is better. Less is more.
- A successful user experience derives from near obsessive attention to detail.
- Don’t outsource your user support. On the contrary, put your best people on it. People love getting a personal response from somebody that knows what they’re talking about. You’ll reap the benefits in trust, patience and tremendous value in terms of feedback to the process of continually improving the user experience.
We’ve also learned a wealth about effectively operating a high-volume Ruby on Rails application. (On one particular day, the ABC “Good Morning America” spot was the very top news article on Yahoo News, and we saw 20,000 new user registrations in a period of four hours.) Having a great hosting partner helps a lot!
Finally, this project is proof positive that talented and motivated people, supported by an effective agile process and paying a lot of attention to clear communications, can build a successful project, even if separated great distances geographically. Our Catalog Choice team includes MakaluMedia staff located in the United States, Germany, Spain and Ireland!
And with that, we bring this article to a close. Our next project milestones include one million catalog opt-outs, and hopefully down the road one millions users! And remember, if you’re located in the US, and ready to simplify your life while helping the environment, head on over to Catalog Choice and sign up for a free account:
Be sure to check out the following video clips.
- Catalog Choice on NBC
- Catalog Choice on CNN
- Catalog Choice on ABC Good Morning America
Stop those catalogs with Catalog Choice! Wednesday, October 10th, 2007 by Matt
The idea.
Growing up in the US, I remember how annoying it was to receive commercial sales calls on the telephone late in the evening, when the family was sitting around trying to enjoy some quality time together. This problem was later solved (mostly) through the introduction of the “Do Not Call,” registry — a legislated service which marketers must consult before they can call you. If your name is on the “Do Not Call” list, it’s illegal for them to call you.
A similar problem exists today with paper catalogs. American consumers are simply inundated with unsolicited, undesired paper catalogs from a multitude of merchants. While receiving these catalogs can be just as annoying as receiving the marketing phone calls, there’s a deeper, more concerning problem. The production of the 19 billion paper catalogs sent each year in the US takes a huge toll on the environment.
19 BILLION. We throw around terms daily like million, billion, trillion, without really having a feel for the magnitude of those numbers. Consider this, it takes 11 days for a million seconds to pass. It takes 32 years for a billion seconds to pass! So, 19 billion catalogs — that’s a lot of wasted paper!
Earlier this year, my company MakaluMedia was offered an opportunity to participate in a project backed by some of the US’ largest environmental and non-profit organizations to help address this project. Today represents the culmination of several months of hard work on this project, and I’m proud to announce the launch of Catalog Choice:
Catalog Choice is a free service, that allows people to find and opt-out of the undesired catalogs they receive. The concept is very simple; you sign up, search for a catalog you receive, and opt-out. We take care the rest, and in about six weeks’ time, you should no longer receive the catalog.
The application.
Given that the service targets all American consumers, including a large segment who may not be computer saavy, a major objective was making Catalog Choice simple, inviting and easy-to-use. (Well, in fact, those should be objectives of any product really.)
Making something simple is anything but simple :) It involves everything from the visual aesthetic, to the interaction workflows, to the choice of the wording on the site. (For example, we opted against using the term “Sign up”, as some people find that idea intimidating, and rather went for “Get started now”.) We hope we’ve succeeded, but we’ll be monitoring the visitor’s experiences closely to tune and adjust the application as needed.
Technically, the site was built using the Ruby on Rails application framework, which is simply fantastic.
The launch.
The big launch happened yesterday, and shortly thereafter we were mentioned on a couple of very popular websites:
That certainly got the traffic flowing, allowing the powerful Engine Yard servers to finally break a sweat. :)
And the day ended with the good news that next week we’re going to get some major national media coverage!
The team.
I almost feel guilty to be paid to do work that’s so enjoyable, and the biggest part of that satisfaction comes from being part of the tremendous team we have in MakaluMedia. In this Catalog Choice project, Mike Rohde, an “icon” in the logo design world (couldn’t help saying that ;), designed the logo, Alex Bendiken applied his wizardry to design the application’s user interface and user experience (Alex, by the way, is the guy that won the Slashdot redesign contest!), the Rails application development included Christian Kumpf (one of our crack engineers in Germany, who happened to build an ultra-stable satellite telemetry processing system last year!), Alex (yes, same one; he does Rails application development too!), Brian Cooke (maker of famous Mac OS X applications) and our latest additions, Alvaro Silva and Luke Brown, systems work by Irish legend Niall O Broin (who will never have a weblog, so don’t get your hopes up) and, finally, project management by Fausto “I dream in SQL” Roveda, and myself.
All in all the project has been a ton of fun, and hopefully it’ll be a big success.
We need your help.
This project has been especially satisfying as it represents something that could have a major positive impact on the environment. But for that to happen, it needs to be used on a very large scale. And in that respect, we could really use your help.
As a free service, and given the high costs of marketing, the team is hoping in a big way that the service will spread via word of mouth. If you’d be willing to link to the site from your own website or blog, we’ve got some hopefully nice looking badges available here:
http://www.catalogchoice.org/#about
And if you use the service, please consider using the “Invite a friend” feature. Thanks so much.
Miscellaneous updates.
- To handle all the customer service incoming emails, Catalog Choice are using a service that absolutely rocks, MailTank. It is fantastic.
Zero to Launch in Two Weeks (Rock-am-Ring Photo Site) Thursday, June 14th, 2007 by info
Last week, around Monday or Tuesday, we got an urgent request from our longtime German customer, Marek Lieberberg Konzertagentur:
We need a Rock-am-Ring photo upload and sharing site, ASAP!
So we hunkered down, conceived a rough concept, and Alex got to work from scratch designing and coding. About two weeks later, we just launched:
Rock-am-Ring Photo Sharing Site
This site is a good example of the productivity gains that can be realized (by great developers, of course!) with the Ruby on Rails web application framework.
Our new site features include:
User accounts, created from temporary accounts (i.e. every visitor instantly becomes a temporary user, which is migrated when they create their permanent account) with profiles. (This is an innovative technique that we’ll blog about soon.)
User uploading and management of images.
Probably the most advanced file upload interface on the web. You can select multiple files for upload (and we locally pre-filter by image size and type!), and they are sequentially uploaded and processed, with full interactive feedback to the user (You can literally upload hundreds of files at once.) We plan to publish this solution as Open Source.
Photo voting with AJAX (Digg-style, thumbs up/down). Notice we don’t display the count before voting, to avoid bias.
Photo commenting, with email notification, and 30 minute post-editing.
Community-based moderation. If three people mark an image as inappropriate, it’s no longer displayed, and we’re notified to consider suspending the user’s account.
Advanced slideshow viewer, with link emailing.
Photo tagging
AJAX implementation of carousel slider
Automatic creation of multiple sizes of uploaded files
Photo EXIF information display.
Lots of nice lightbox UI effects (signup, login, etc.)
Bilingual user interface (German/English)
Amazon S3 image storage and serving
Multi-tier server architecture
And lots and lots more…
There are still lots of features to add, but the main focus now is on database query optimization, to improve the page load times, and to support the large amount of traffic. (In the first 36 hours, we’ve got 900 images, and serviced 4 million database queries!)
All in all, this was a pretty amazing accomplishment, both in terms of what was produced and the time in which it was produced.
It was also a great (and fun!) team effort. Thanks to:
Alex, for working like a madman to get this designed, and coded in a week and a half.
Niall and Arto, for figuring out how to keep the servers handling the requests.
Christian, for being late to his movie, while providing the German translations.
Alvaro, Martin and Christian, for testing.
Nespresso, for creating the caffeine that kept us going.
Spam Filters, Alien Technology and Ruby on Rails Wednesday, July 5th, 2006 by Arto
When Paul Graham’s A Plan for Spam made its dramatic entrance into the anti-spam battle four years ago, it heralded the beginning of the end for spam — as we knew spam back then, anyway. Applying a simple statistical approach, based on word frequency analysis with a naive Bayesian classifier, Graham described how to create a spam filter accurate enough (99+%) that false positives effectively ceased to be an issue.
The central idea of the Graham Algorithm was quickly adopted en masse by spam filters, and as a result, the spam arms race has in the past few years tipped in favor of the good guys. “Successful� spam has devolved into exactly what Graham predicted it would: “some completely neutral text followed by a URL.� For me personally, the combination of good server- and client-side filters has made spam yesterday’s problem. (Well, that, and using Gmail as a front-end for lower-priority e-mail; spam all you want, it’s Google’s problem and they’re up to the task.)
Recently at MakaluMedia, we’ve succeeded in applying similar text classification principles to another unrelated problem area, with the intent of forcing the computer to do the tedious job it was invented for, allowing us super-apes, in turn, to spend more time under a palm tree on the nearby beach, sipping tinto de verano and working out answers to deep existential questions, or whatever else it is that one does on the beach (note to self: need more practice).
The exact details of this covert project will have to await its escape into the wild, should it ever evolve the capability for that. For the time being, some of the technically more interesting tidbits will have to do as fodder for my ramblings.
First, as with most of our internal development, and an ever-increasing percentage of our client projects, this system was developed using the high-productivity Ruby on Rails framework, and reached the magical 0.1.0 mark (i.e., pre-alpha, but usable enough to solve many of the developers’ own needs) in less than a man-week of intensive coding (not to forget the skimming of a good number of research papers related to the subject).
However, to ensure a permanent gap on the competition (after all, it sometimes seems like half the world has already jumped onto Rails), we also pulled out the big guns: the top-secret alien technology known as Lisp.
(Don’t be fooled by the devious code name, intended to confound us earthlings with spurious ideas relating to speech defects — this is seriously powerful stuff: exposure is guaranteed to subtly but permanently alter your brain structure in ways not yet fully understood. In fact, the aliens have theorized that the Universe may actually be one giant Quantum Lisp machine, explaining how it is possible that lots of irritating, seemingly superfluous parentheses can act as magic incantations conveying an apparently inexhaustible power as per the principle of Clarke’s Third Law. But that’s neither here nor there for our present purposes.)
In our case, we simply integrated into the Rails system an interpreter for a subset of Scheme, a Lisp derivative; thus no doubt confirming Greenspun’s Tenth Rule once again. (Well, to be fair, the Lisp interpreter in question is only some few hundred lines of fairly elegant Ruby code.)
The system’s top-level classification and scoring algorithms are implemented in this Lisp subset, allowing us to easily fine-tune and try out new tweaks at runtime, and perhaps in the future letting us semi-automatically pit various competing implementations against each other in a manner not dissimilar to a genetic algorithm.
Due to the combined RAD-factors of Rails and Lisp, we quickly proceeded through a number of intermediate prototypes along the way, starting from a short-lived Ferret-based implementation, evolving to a hand-rolled tokenizer and SQL-backed corpora storage, and eventually ending up with the current version that delegates the content classification to a Dr. Strangelove-inspired piece of excellent software called the CRM114 Controllable Regex Mutilator.
(Speaking of CRM114, I was surprised to not find any existing Ruby bindings for it, and thus took the time to transcribe a previous Python wrapper into a Ruby version, to be released shortly.)
Anyway, the system appears to work more or less according to spec, but definitely still needs some more tweaking before embarking on world domination. For one thing, all this number crunching is, well, rather heavy (let’s just say it’s a good thing we have A/C in the server room where the development box is located). Although CRM114 itself is pretty light on its feet, we’re dealing with an exponentially growing data set, and the next challenge will be to put some checks on resource consumption.
So, for the time being, we’re not going to let the development box interact with our space systems department, to prevent any non-regulated growth or inadvertent contact with the aliens. More updates to follow as they happen.
Note to self: too many tinto veranos can make you forget there’s a fine line between tongue-in-cheek surrealism and plain-bad geek humor.
Update 2006/11/06: I’ve released the Ruby interface to CRM114 on RubyForge.