Building an API and mobile platform for 10 million users
Four days before the turkey was served up at Christmas last year, we threw the switch on the biggest project we’ve ever done at globaldev. We launched our new mobile counterpart to our desktop application for nearly 200 of our biggest sites.
At our partner conference in October, we had a prototype ready to show which had the majority of the key features of the standard web-app implemented: search and viewing profiles, plus a few smaller features. Once that had been given the green light both from internal stakeholders and external partners, we had just under eight weeks to take our prototype and build it into a production-ready system capable of handling up to 10% (and growing) of our daily traffic.
Not Just a mobile version
Although the visible output of the project was the mobile application that members use, we wanted to do more. For some time now we’ve been discussing the future direction of our dating platform: how to make it easier to work with, more flexible and easier to look after operationally. A large amount of the business logic within the platform is held within the core ColdFusion codebase that powers the primary web application, but (for a number of reasons) we’ve been building less and less in ColdFusion, and more in separate Ruby tasks and applications.
So as part of the mobile project, we started the process of transitioning away from one single ColdFusion logic-base, and created a number of distinct network services that control specific domains within the platform: search, sites, members (users) and communication. On top of this we built a simple API router which exposes the internal services externally, while providing security measures such as API key handling, session tokens and so on.
The benefits of a more SOA-based approach are many, not least making it much easier for our growing number of developers to start developing and testing individual components without requiring a complete understanding of the whole system to be productive. We’re certainly not saying we got it right first time, but what we have is successfully running the mobile application, and we can continue working on these services under the hood without impacting the front-end system, as long as the API doesn’t change too much.
The mobile app and most of the services are Rails apps, and all of them serve up JSON over HTTP. We used Rails and ActiveRecord for the services due to the ease and speed of development, but going forward we may replace this with something more custom, as we know there are a number of points where we’ve had to make compromises in the short term. The search service is less complicated in terms of routing and database access –– it effectively exposes just a single endpoint (
search) and talks to Sphinx – so we’ve tailored that slightly more and built it as a Sinatra app, using Sequel where necessary. Everything’s running on Ruby 1.9.3.
The router is slightly different. Its sole job is to take external requests and, after authenticating and authorising the request, to pass that request on to the relevant internal service (and then pass back the result.) Rather than build this around a web framework not built for this purpose, and knowing that this part had to be very efficient (every API request would go through this system) one of our engineers built a new Rack-based framework, open-sourced as Octarine.
Another piece of the puzzle was the additions we made to our existing internal ColdFusion API. Due to the complexities of even apparently simple tasks (for example, creating a new member also has to send them a verification email, add them to various email CRM schedules, store tracking information, apply a few different rules based on which site the member’s signing up on, etc.) and our desire to ship iteratively, we decided to effectively route all write traffic back through our existing ColdFusion logic.
We’ve used our ColdFusion API (the framework for which we’ve extracted into RESTfulCF) for a while for other internal tools, so it was a case of expanding the API to support the additional features and pieces of functionality we required for the mobile application.
The requests themselves still go through the appropriate service, but those writes then get proxied back to the ColdFusion API. This meant we could cut a huge chunk of time off the delivery, but still leaves us with the ability to migrate those writes back in to the services themselves without changing the service-layer API.
Where to put everything?
This project was the first we’ve completed where almost all new code is running on cloud instances rather than our own physical hardware. We’re using Rackspace’s RackConnect infrastructure to provide a bridge between this new cloud instance based system and our fixed infrastructure (database servers etc) meaning that we have can effectively treat the cloud instances as part of our own infrastructure, and we don’t pay for any data transfer.
We’re currently using Puppet to both build and manage cloud instances, as well as deploy code for the various services. To keep things simple for launch, a single request stays on one box: each instance contains the mobile app, API router and all four services. They’re each run through Thin processes, proxied by nginx.
This gave us the ability to quickly scale the application should demand require it: spinning up a new box, building the required software (Ruby, nginx etc), getting all the code checked out and configuring it, and adding the box to the load-balanced pool takes around 45 minutes. It’s not lightning quick, but it’s more than good enough for our needs at the moment.
The end result
Launching the mobile app in the way we did, with the timescales we gave ourselves, was a hugely ambitious task and, to be honest, one we nearly failed at. It was only through the momentous dedication of the team at globaldev that we pulled through, and we’ve had nothing but high praise from all who’ve seen the result.
We’ve certainly learned an enormous amount, both about the massive capabilities of our team, but also about how not to do things in the future. But most of all, we’ve learned that, given a challenge, we can trust our team of developers and engineers can pick it up and run with it.
If you want to the see the fruits of our labour, visit Dating Republic (you might have seen our advertising on London buses or the Tube) on your iPhone, Android or BlackBerry.
We’re really happy with the mobile application: it looks great (thanks to our user experience team) and it actually exceeded our expectations from an end-user perspective. In the process of delivering that, we had to compromise on some of our original goals for the API stack (the router and services), so we’re now working out how to take those forward. Some of the thoughts we’re investigating at at the moment include:
- Does threaded JRuby give us better resource utilisation across the services?
- The API router is a prime candidate for running in an evented manner using either EventMachine or Ruby 1.9’s fibers, due to the fact that every request spends 95% of its time waiting for other internal services.
- How can we improve caching between the layers to skip the need for hitting the apps at all? We’re already running a custom build of nginx to which one of our operations team added some custom memcached hashing logic to, but we want to go further.
- Starting to consider the additional security and availability implications of opening up the API to particular partners to allow them to build their own applications on top of our platform.
- Starting to package up our existing business logic into the service layer, with the eventual aim of having all our own apps running through the single API.
This post was originally published on the globaldev site.