Tim Blair

Programmer, app architect, tinkerer, husband, father, cat owner, runner, adventurer, eater of cakes, pedant, enjoyer of proper beer, frisbee thrower, guitar player, curator of unfinished and ever-growing lists…

Page 2

Building an API and mobile platform for 10 million users

Four days before the turkey was served up at Christmas last year, we threw the switch on the biggest project we’ve ever done at globaldev. We launched our new mobile counterpart to our desktop application for nearly 200 of our biggest sites.

At our partner conference in October, we had a prototype ready to show which had the majority of the key features of the standard web-app implemented: search and viewing profiles, plus a few smaller features. Once that had been given the green light both from internal stakeholders and external partners, we had just under eight weeks to take our prototype and build it into a production-ready system capable of handling up to 10% (and growing) of our daily traffic.

 Not Just a mobile version

Although the visible output of the project was the mobile application that members use, we wanted to do more. For some time now we’ve been discussing the future

Continue reading →

Eliminate the impossible to find the improbable

Yesterday we released a new feature on the platform called “New Members Today” — those members that are interested will receive a daily email containing profiles of new members that may be of interest to them. We’ve been running this across a sub-section of the member-base, but expanded this to all members.


As with anything, we spent time testing our changes as fully as we could in both development and staging environments. We needed this to be right, as we’d soon be sending emails out to millions of members. Unit tests passed. Integration testing went well. Full-stack tests sent the correct content to the right members. We were good to go: the button was pressed and everything was deployed to production.

At midday, the scheduler kicked things off, and we were in business: items started appearing on the queues, metrics started increasing as messages were spooled, and emails

Continue reading →

Tales of the unexpected

Sometimes things go wrong. The results of the “something” going wrong are usually fairly obvious — searches not returning, systems not starting, payments not processing — but working out why that “something” stopped working (or started working differently) is the challenge.


Earlier this week we noticed that for the previous week or so, we’d been making a huge number more lookups to an external system for retrieving the latitude and longitude for members’ postcodes (which we use for distance calculations between members,) but we couldn’t place our finger on what had happened: no one had changed the lookup code in months. We permanently cache all data locally to avoid excessive remote calls, and that data also remained intact.

Looking at what we had changed, we had a hunch that the symptom was caused by our new member profile pages: it was the only significant release during the

Continue reading →

Live schema changes on high volume tables

We’ve just finished some fairly serious data shuffling in the WLD database.

We’ve been working on changes to reduce the amount of storage space private messages take up, aiming to decrease the storage requirements by around 80%, and also to dramatically lower the database hit each time a new message is sent (there’s a write-query reduction of around 75%).

However, one of the changes required a database table schema change to the main messages table (the table that stores the references to messages for each member). The problem we had: there were nearly a quarter of a billion records in that table.

We estimated that running an ALTER TABLE on that, waiting for the changes to replicate over all the slaves, and then for the slaves to catch up again, could take up to two days, which would mean two days of effective downtime. Obviously, this wasn’t an option, so we had to come up with an

Continue reading →

memcached invalidation for sets of keys

There are only two hard things in Computer Science: cache invalidation and naming things.
— Phil Karlton

We use memcached extensively throughout the White Label Dating platform, primarily for write-through data caching. For example, composite member and site objects are serialised to JSON, stored with simple keys such as member:1234567 or site:1234, and deserialised and the values injected into object constructs when required.

Although this data-caching is what takes up most of our caching space, we also store fully rendered HTML pages in a read-through fashion. We do this rather than using a caching reverse proxy such as Squid or Varnish to give us the ability to control expiration and invalidation of the cached items directly.

Due to the heavy customisation of each logged-in member’s pages, we can only perform full-page caching on the visitor and search results pages; with people

Continue reading →