The main duty of the employee would be to work directly with the customers’ operations teams to detect, analyze and fix performance problems and plan application scalability in accordance with customers’ needs/requests.

The employee would work in a small team of highly professional consultants helping the company to handle growing demand for web applications performance consulting services.

Position is in Toronto. Telecommuting is not possible.

Being a consulting-related position, it requires an extensive experience with highly-loaded web-applications scalability and, based on our customers’ needs, an expert knowledge of the following technologies:

• Programming Languages: Ruby, PHP, C/C++, C#, Google Go
• Databases: MySQL, Microsoft SQL Server
• Operating Systems: MacOS X, Linux, Windows

This position requires a Bachelor or Master’s degree in computer sciences. Fluent oral and written Russian language and an intermediate written English language skills are required.

For more details please contact hr (аt) scalexis.com.

It is a complex problem that usually consists of two smaller ones: individual cache elements invalidation (you need to keep an eye on your data changes and invalidate cached pages when related data changes) and full cache purges (sometimes your site layout or page templates change and you need to purge all the cached pages to make sure users will get new visual elements of layout changes). In this post I’d like to look at a few techniques we use at Scribd to solve cache invalidation problems.

Since we use haproxy to balance our traffic in the cluster it is hard to predict where should we send a purge request. So we fan those out to all cache servers.

To make sure cache purging won’t slow the site down, especially considering we need to do more that just a simple cache purge (submit documents to search indexes, etc, etc), we just spool a “document changed” request to a queue and then have a set of asynchronous processes that do all the work in background.

Next, The Hard Problem – handling full cache purges w/o killing our backend servers with 5x-10x traffic (our normal hit ratio is ~90-95%).

We’ve spent a lot of time thinking about this problem and the first idea we came up with was to have a loop process somewhere that would iterate all documents we have cached and purge them one by one… but that does not seem to be a practical solution when you have tens of millions documents (and few page versions per document) and obviously the solution would not scale with constantly growing documents corpus.

So we kept brainstorming and finally got one idea that works just perfectly for us: what if we’d be able to take our traffic and define a function f(t) that would return a percentage of the traffic that should be purged at any moment in time. So we did it – we’ve implemented a nginx module that would version our cache by assigning every cached page a revision (using a custom HTTP-headers + Vary-caching) and would be able to slowly migrate the cache from one revision to another over a pre-defined period of time.

Here is an overview of the requests/responses flow in our web/cache/application cluster:

X-Cache-Revision value selection algorithm is the following:

Having this kind of logic in place allows us to do so called “slow” cache purges that could take any time from a few minutes (that still helps to reduce the load spike generated by the hottest content) up to many hours (this is what we normally use) or days (never used this option, but it is definitely possible).

Here is an example 100% cache purge over an 8 hour interval:

1. Daily hit ratio graph:
   
2. Weekly hit ratio graph:
   

As you can see, during those slow purges our cached pages would be slowly updated without putting too much pressure on the backend. Cache hit ratio would slowly degrade and then slowly get back to its normal levels, but with our normal (6-8 hours) purges hit ratio never gets lower that 65-70% which makes it possible for us to save huge amounts of money on not having 90% spare capacity just for the cache purge load surges (we used to have lots of spare application cluster capacity before introducing this approach).

Update (June 5th, 2010): Request/response flow and cache revision selection algorithm added.

Today, six months after the first release of the gem and we’ve moved it to gemcutter (which is now the official gems hosting) and we’re already at version 1.6.11. The gem was downloaded more than 2000 times. There are (at least) 10+ large users that rely on this gem to scale their products out. And (this is the most exciting) we’ve added tons of new features to the product.

Here are the main features added since the first release:

• Much better multi-database migrations support including default migrations connection changing.
• We’ve added ActiveRecord associations preload support that makes it possible to move eager loading queries to the same connection where your finder queries go to.
• We’ve improved ActiveRecord’s query logging feature and now you can see what connections your queries executed on (and yes, all those improvements are colorized ).
• We’ve added an ability to temporary remap any ActiveRecord connections to any other connections for a block of code (really useful when you need to make sure all your queries would go to some non-default slave and you do not want to mess with all your models).
• The most interesting change: we’ve implemented some basic sharding functionality in ActiveRecord which currently is being used in production in our application.

As you can see now DbCharmer helps you to do three major scalability tasks in your Rails projects:

1. Master-Slave clusters to scale out your Rails models reads.
2. Vertical sharding by moving some of your models to a separate (maybe even dedicated) servers and still keep using AR associations
3. Horizontal sharding by slicing your models data to pieces and placing those pieces into different databases and/or servers.

So, If you didn’t check DbCharmer out yet and you’re working on some large rails project that is (or going to be) facing scalability problems, go read the docs, download/install the gem and prove them that Rails CAN scale!

DB Charmer – ActiveRecord Connection Magic Plugin

DbCharmer is a simple yet powerful plugin for ActiveRecord that does a few things:

1. Allows you to easily manage AR models’ connections (switch_connection_to method)
2. Allows you to switch AR models’ default connections to a separate servers/databases
3. Allows you to easily choose where your query should go (on_* methods family)
4. Allows you to automatically send read queries to your slaves while masters would handle all the updates.
5. Adds multiple databases migrations to ActiveRecord

Installation

There are two options when approaching db-charmer installation:

• using the gem (recommended)
• install as a Rails plugin

To install as a gem, add this to your environment.rb:

And then run the command:

To install db-charmer as a Rails plugin use this:

Easy ActiveRecord Connection Management

As a part of this plugin we’ve added switch_connection_to method that accepts many different kinds of db connections specifications and uses them on a model. We support:

1. Strings and symbols as the names of connection configuration blocks in database.yml.
2. ActiveRecord models (we’d use connection currently set up on a model).
3. Database connections (Model.connection)
4. Nil values to reset model to default connection.

Sample code:

The switch_connection_to method has an optional second parameter should_exist which is true by default. This parameter is used when the method is called with a string or a symbol connection name and there is no such connection configuration in the database.yml file. If this parameter is true, an exception would be raised, otherwise, the error would be ignored and no connection change would happen.

This is really useful when in development mode or in a tests you do not want to create many different databases on your local machine and just want to put all your tables in a single database.

Warning: All the connection switching calls would switch connection only for those classes the method called on. You can’t call the switch_connection_to method and switch connection for a base class in some hierarchy (for example, you can’t switch AR::Base connection and see all your models switched to the new connection, use the classic establish_connection instead).

Multiple DB Migrations

In every application that works with many databases, there is need in a convenient schema migrations mechanism.

All Rails users already have this mechanism – rails migrations. So in DbCharmer, we’ve made it possible to seamlessly use multiple databases in Rails migrations.

There are two methods available in migrations to operate on more than one database:

1. Global connection change method – used to switch whole migration to a non-default database.
2. Block-level connection change method – could be used to do only a part of a migration on a non-default db.

Migration class example (global connection rewrite):

Migration class example (block-level connection rewrite):

By default in development and test environments you could skip this :second_db connection from your database.yml files and rails would create the tables in your single database, but in production you’d specify it and get the table created on a separate server and/or in a separate database.

This behaviour is controlled by the DbCharmer.migration_connections_should_exist configuration attribute which could be set from a rails initializer.

Using Models in Master-Slave Environments

Master-slave replication is the most popular scale-out technique in a medium-sized and large database-centric applications today. There are some rails plugins out there that help developers to use slave servers in their models but none of them were flexible enough for us to start using them in a huge application we work on.

So, we’ve been using ActsAsReadonlyable plugin for a long time and have made tons of changes in its code over the time. But since that plugin has been abandoned by its authors, we’ve decided to collect all of our master-slave code in one plugin and release it for rails 2.2+. DbCharmer adds the following features to Rails models:

Auto-Switching all Reads to the Slave(s)

When you create a model, you could use db_magic :slave => :blah or db_magic :slaves => [ :foo, :bar ] commands in your model to set up reads redirection mode when all your find/count/exist/etc methods will be reading data from your slave (or a bunch of slaves in a round-robin manner). Here is an example:

Default Connection Switching

If you have more than one master-slave cluster (or simply more than one database) in your database environment, then you might want to change the default database connection of some of your models. You could do that by using db_magic :connection => :foo call from your models. Example:

Sample model on a separate master-slave cluster (so, separate main connection + a slave connection):

Per-Query Connection Management

Sometimes you have select queries that you know you want to run on the master. This could happen for example when you have just added some data and need to read it back and not sure if it made it all the way to the slave yet or no. For this situation and a few others there is a set of methods we’ve added to ActiveRecord models:

1) +on_master+ – this method could be used in two forms: block form and proxy form. In the block form you could force connection switch for a block of code:

In the proxy form this method could be used to force one query to be performed on the master database server:

2) on_slave – this method is used to force a query to be run on a slave even in situations when it’s been previously forced to use the master. If there is more than one slave, one would be selected randomly. This method has two forms as well: block and proxy.

3) on_db(connection) – this method is what makes two previous methods possible. It is used to switch a model’s connection to some db for a short block of code or even for one statement (two forms). It accepts the same range of values as the switch_connection_to method does. Example:

Associations Connection Management

ActiveRecord models can have an associations with each other and since every model has its own database connections, it becomes pretty hard to manage connections in a chained calls like User.posts.count. With a class-only connection switching methods this call would look like the following if we’d want to count posts on a separate database:

Apparently this is not the best way to write the code and we’ve implemented an on_* methods on associations as well so you could do things like this:

Notice: Since ActiveRecord associations implemented as proxies for resulting objects/collections, it is possible to use our connection switching methods even without chained methods:

Starting with DbCharmer release 1.4 it is possible to use prefix notation for has_many and HABTM associations connection switching:

Named Scopes Support

To make it easier for DbCharmer users to use connections switching methods with named scopes, we’ve added on_* methods support on the scopes as well. All the following scope chains would do exactly the same way (the query would be executed on the :foo database connection):

And now, add this feature to our associations support and here is what we could do:

Documentation

For more information on the plugin internals, please check out the source code. All the plugin’s code is ~100% covered with a tests that were placed in a separate staging rails project located at github. The project has unit tests for all or at least the most of the parts of plugin’s code.

What Ruby and Rails implementations does it work for?

We have a continuous integration setups for this plugin on MRI 1.8.6 with Rails 2.2 and 2.3. We use the plugin in production on Scribd.com with MRI (rubyee) 1.8.6 and Rails 2.2.

Who are the authors?

This plugin has been created in Scribd.com for our internal use and then the sources were opened for other people to use. All the code in this package has been developed by Alexey Kovyrin for Scribd.com and is released under the MIT license. For more details, see the LICENSE file.

If you have any comments on this project, feel free to contact me here in comments or by email. And, of course, patches are welcome (only when covered with tests).

First of all, let’s take a look at a typical Scribd’s document URL: http://www.scribd.com/doc/1/Improved-Statistical-Test.

As we can see, it consists of a document-specific part (/doc/1) and a non-unique human-readable slug part (/Improved-Statistical-Test). When a user comes to the site with a wrong slug in the document URL, we need to make sure we send the user to the correct URL with a permanent HTTP 301 redirect. So, obviously we can’t simply send our requests to the squid because it’d cause few problems:

• When we change document’s title, we’d create a new cached item and would not be able to redirect users from the old URL to the new one
• When we change a title, we’d pollute cache with additional document page copies.

One more problem that makes the situation even worse – we have 3 different kinds of users on the site:

1. Logged in users – active web site users that are logged in and should see their name at the top of the page, should see all kinds of customized parts of the page, etc (especially when a page is their own document).
2. Anonymous users – all users that are not logged in and visit the site with a flash-enabled browser
3. Bots – all kinds of crawlers that can’t read flash content and need to see a plain text document version

All three kinds of users should see their own document page versions whether the page is cached or not.

So, how do we solve these two problems? Here is how.

First of all, to fix the URLs problem we’ve decided to rewrite the URL before it goes to a squid server. We change URLs to look like this: http://www.scribd.com/doc/1?__enable_docview_caching=1. This makes the document URL dependent only on a unique document ID that never change and sends an additional parameter to the backend to signal that the page could potentially be cached. The slug is sent to backend using an HTTP-header (X-Scribd-Slug) so that backend could check the slug and return a redirect if needed.

To make sure we won’t respond with a cached page to a request with an invalid URL (invalid slug basically), we use Vary: X-Scribd-Slug HTTP header which is implemented in Squid (only late 2.6 and 2.7) and makes it check specified headers in a request before responding with a cached content. If the header of the cached content is different then the header in the request, squid proxies the resuest to backend where we could handle the cache miss as we want.

Next, to resolve the users problem we’ve created a small nginx module that looking at a request headers could tell you whether the user is a bot or not and whether he’s logged in or an anonymous visitor. This module basically exposes a $scribd_user_id variable to our configs and we could use the variable to do separate configuration for different kinds of users.

At this point we do not cache document pages for logged in users so we basically have two copies of each page in the cache: flash-enabled document page and an inline document page. We do this separation by changing our cache URLs one more time: we add a scribd_user_id=$scribd_user_id variable (anonymous = 0, bot = -1) to the cache URL: http://www.scribd.com/doc/1?__enable_docview_caching=1&scribd_user_id=0.

And last, not not least, we use two really awesome Squid features called stale-while-revalidate and stale-if-error (AFAIR, they were invented in Yahoo! and then described by their squid admin).

Option stale-while-revalidate allow us to quickly serve content from the cache while doing background re-validation requests to the Rails backend. Option stale-if-error basically allows us to serve content from the squid cache when Rails backend is down/dead/slow.

All these changes allowed us to handle more traffic with less hardware and what is even more important, they helped us improve user experience with the site: response times dropped 2-3 fold and much less people see our Ouch! pages (HTTP 50x errors when backend is dead or overloaded). Here is an example of one of our servers’ hit ratio and traffic savings daily graphs:

This it with the logged in users and complex URLs handling in Scribd caching architecture. Next post in this series will explain how we do cache invalidation in Scribd. Stay tuned.

And 3 years ago in March 2006 I’ve decided to start my own blog. I took a standard template and started the blog on a separate domain while the domain was on its own domain name… This spring my wife made me a great birthday present – she’s created me a custom blog design that has all the stuff I wanted from my own web site for a long time. My friend Dima Shteflyuk has helped me with creating a wordpress template from Tanya’s mockups and here we are – now I’ve decided to merge my blog and my web site into a single web entity called http://kovyrin.net/. Welcome to my new blog/site/whatever!

Originally loops plugin was created to make our (Scribd.com) own loops code more organized. We used to have tens of different modules with methods that were called with script/runner and then used with nohup and other not so convenient backgrounding techniques. When you have such a number of loops/workers to run in background it becomes a nightmare to manage them on a regular basis (restarts, code upgrades, status/health checking, etc).

After a short time of writing our loops in more organized ways we were able to generalize most of the loops code so now our loops look like a classes with a single mandatory public method called run. Everything else (spawning many workers, managing them, logging, backgrounding, pid-files management, etc) is handled by the plugin itself.

The major idea behind this small project was to create a deadly simple and yet robust framework to be able to run some tasks in background and do not think about spawning many workers, restarting them when they die, etc. So, if you need to be able to run either one or many copies of your worker or you do not want to think about re-spawning dead workers and do not want to spend megabytes of RAM on separate copies of Ruby interpreter (when you run each copy of your loop as a separate process controlled by monit/god/etc), then I’d recommend you to try this framework — you’ll like it.

For more information, visit the project site and, of course, read the sources

All this time (few months now) I was using starling noticed really bad thing in how it works: if workers die (really die, or lock on something for a long time, or just start lagging) and queue start growing, the thing could kill your server and you won’t be able to do something about it – it just eats all your memory and this is it. Since then I’ve started looking for a better solution for our queuing, the technology was too cool to give up. I’ve tried 5 or 6 different popular solutions and all of them sucked… They ALL had the same problem – if your queue grows, this is your problem and not queue broker’s :-/ The last solution I’ve tested was ActiveMQ and either I wasn’t able to push it to its limits or it is really so cool, but looks like it does not have this memory problem. So, we’ve started using it recently.

In this small post I’d like to describe a few things that took me pretty long to figure out in ruby Stomp client: how to make queues persistent (really!) and how to process elements one by one with clients’ acknowledgments.

First, we need to connect to the queue server and this code is the same for clients and servers:

Second form looks better for me because it allows you to specify if you want client to be reliable (lock on errors and try to reconnect, etc) or just want client to raise an error in case of any problems.

When you’ve connected to the server, you can push your data to a queue:

And this is where you have an ability to specify if you want your data to be persistent (survive server crashes, etc): headers is a hash that could have an element :persistent => true, which would do the thing. So, your code would looks like this (for example):

Now, when you have your data submitted to the queue, you need to be able to read it and process with some script. This is as simple as the following code:

Again, we wanted to receive messages one by one and acknowledge successful processing in our code. This is simple too. You need:

• Pass :ack => :client as an element in your headers hash
• Call client.acknowledge(msg) in the loop if you’re sure that an element could be removed from the queue

This is basically it with the stuff I wanted to explain today. If you’ve never tried to work with any queue servers, try today and maybe tomorrow you won’t be able to imagine your systems architecture without such a component

Few times I was creating a git branches and hacking this action trying to implement some sort of page-level caching to make things faster. But all the time results weren’t as good as I’d like them to be. So, branches were sitting there and waiting for a better idea.

Few months ago my good friend has joined Scribd and we’ve started thinking on this problem together. As the result of our brainstorming we’ve managed to figure out what were the problems preventing us from doing efficient caching:

• First of all, a lots of code in the action is changing the page view if our visitor is a bot (no, not a cloaking, just some minor adjustments of the view).
• Second problem was a set of differences in the view for anonymous and logged in users.
• And finally, third problem was the fact that the page has a few blocks that change pretty dynamically: document stats pane and comments lists.

All these problems when combined were creating a lots of pain when I was trying to cache a whole page. When we’ve figured them out, we’ve started thinking on how could we generalize possible combinations of those factors and possible approaches to caching.

There is a well known idea in web applications development: the fastest web app action is an action that does not require any code to be executed on your application server. So, first idea we’ve tried to think about was some approach that would definitely reduce the number of hits on our app servers. This idea was based on HTTP protocol features related to Last-Modified and E-Tag headers. But there was a problem – not so many users go to the same page twice so even if we’d make the page cacheable, it wouldn’t help too much. But the idea of full page caching outside of the application was really good and we’ve started playing with it to figure out how to use it in production.

Long time ago, when Internet was slow and expensive many ISPs and large companies were trying to reduce their traffic w/o hurting users’ experience. Then caching proxy servers were born. The idea of those servers was to handle all web requests going from a network (ISP or a company office) and try to cache as much content as possible so when the same or some other user would request a cached page, proxy server would return it really fast. If we’d implement support for those Last-Modified headers, all proxy servers would be happy to cache our pages. But there was a problem – no one uses caching proxies in 2008 So, we’ve got an idea – why can’t we place such a server in front of our application and make it cache content for all users in the world? (Yes, we knew about a caching reverse proxies before – I’m just trying to explain the flow of our thoughts and words when we were brainstorming the problem).

The only problem with this approach would be to differentiate logged in users, anonymous users and bots. Considering the fact that our proxy server could be placed between the app and our web servers (nginx), we’ve decided to create a nginx module that would translate the same document page URLs to a set of URLs, which would be different for all those 3 kinds of users.

When all those problems with different kinds of users were solved, we’ve decided to solve the last one – non-cacheable dynamic stats pane. The solution was pretty simple – we’ve added a small ajax call to the page which would update stats on the cached version of our page for all real users while bots will see the same page, but with a bit stale stats pane.

Long story short, the results is really great. Application servers load reduced by 50-70%, database servers load is reduced by 30-60%, response times dropped down to 150-200 msec from 500-750 msec. As an additional positive effect of the caching we’ve managed to remove all fragments caches from the application and free more of memcached resources for data caches. Here are a few cacti graphs of our servers load/traffic (the caching was introduced on Oct 9th at night):

Main MySQL command counters:

Uploaded with plasq‘s Skitch!

One of our Application Servers CPU Usage:

Uploaded with plasq‘s Skitch!

One of our Application Servers Load Average:

Uploaded with plasq‘s Skitch!

Unfortunately there are a lot of things to share related to this caching experience, so I’ve decided to make a series of posts that would explain all the problems we had and solutions we’ve found for each of the following parts of the caching system:

• Logged in Users and Complex URLs Handling
• Rails code to support Last-Modified headers and how we purge caches
• Squid Server setup (configs and hardware)
• Nginx module development

So, if you’re interested in details, subscribe to this blog’s RSS feed and in a few days you’ll see the first article from this series.

Bounces-handler package is a simple set of scripts to automatically process email bounces and ISP‘s feedback loops emails, maintain your mailing blacklists and a Rails plugin to use those blacklists in your RoR applications.

This piece of software has been developed as a part of more global work on mailing quality improvement in Scribd.com, but it was one of the most critical steps after setting up reverse DNS records, DKIM and SPF.

The package itself consists of two parts:

• Perl scripts to process incoming email:
  • bounces processor — could be assigned to process all your bounce emails
  • feedback loops messages processor — more specific for Scribd, but still – could be modified for your needs (will be released soon).
• Rails plugin to work with mailing blacklists

For more information, please check our README file. If you have any questions, comments or suggestions, please leave them here as a comments and I’ll try to reply as soon as possible.