DB Charmer – ActiveRecord Connection Magic Plugin

DbCharmer is a simple yet powerful plugin for ActiveRecord that does a few things:

1. Allows you to easily manage AR models’ connections (switch_connection_to method)
2. Allows you to switch AR models’ default connections to a separate servers/databases
3. Allows you to easily choose where your query should go (on_* methods family)
4. Allows you to automatically send read queries to your slaves while masters would handle all the updates.
5. Adds multiple databases migrations to ActiveRecord

Installation

There are two options when approaching db-charmer installation:

• using the gem (recommended)
• install as a Rails plugin

To install as a gem, add this to your environment.rb:

And then run the command:

To install db-charmer as a Rails plugin use this:

Easy ActiveRecord Connection Management

As a part of this plugin we’ve added switch_connection_to method that accepts many different kinds of db connections specifications and uses them on a model. We support:

1. Strings and symbols as the names of connection configuration blocks in database.yml.
2. ActiveRecord models (we’d use connection currently set up on a model).
3. Database connections (Model.connection)
4. Nil values to reset model to default connection.

Sample code:

The switch_connection_to method has an optional second parameter should_exist which is true by default. This parameter is used when the method is called with a string or a symbol connection name and there is no such connection configuration in the database.yml file. If this parameter is true, an exception would be raised, otherwise, the error would be ignored and no connection change would happen.

This is really useful when in development mode or in a tests you do not want to create many different databases on your local machine and just want to put all your tables in a single database.

Warning: All the connection switching calls would switch connection only for those classes the method called on. You can’t call the switch_connection_to method and switch connection for a base class in some hierarchy (for example, you can’t switch AR::Base connection and see all your models switched to the new connection, use the classic establish_connection instead).

Multiple DB Migrations

In every application that works with many databases, there is need in a convenient schema migrations mechanism.

All Rails users already have this mechanism – rails migrations. So in DbCharmer, we’ve made it possible to seamlessly use multiple databases in Rails migrations.

There are two methods available in migrations to operate on more than one database:

1. Global connection change method – used to switch whole migration to a non-default database.
2. Block-level connection change method – could be used to do only a part of a migration on a non-default db.

Migration class example (global connection rewrite):

Migration class example (block-level connection rewrite):

By default in development and test environments you could skip this :second_db connection from your database.yml files and rails would create the tables in your single database, but in production you’d specify it and get the table created on a separate server and/or in a separate database.

This behaviour is controlled by the DbCharmer.migration_connections_should_exist configuration attribute which could be set from a rails initializer.

Using Models in Master-Slave Environments

Master-slave replication is the most popular scale-out technique in a medium-sized and large database-centric applications today. There are some rails plugins out there that help developers to use slave servers in their models but none of them were flexible enough for us to start using them in a huge application we work on.

So, we’ve been using ActsAsReadonlyable plugin for a long time and have made tons of changes in its code over the time. But since that plugin has been abandoned by its authors, we’ve decided to collect all of our master-slave code in one plugin and release it for rails 2.2+. DbCharmer adds the following features to Rails models:

Auto-Switching all Reads to the Slave(s)

When you create a model, you could use db_magic :slave => :blah or db_magic :slaves => [ :foo, :bar ] commands in your model to set up reads redirection mode when all your find/count/exist/etc methods will be reading data from your slave (or a bunch of slaves in a round-robin manner). Here is an example:

Default Connection Switching

If you have more than one master-slave cluster (or simply more than one database) in your database environment, then you might want to change the default database connection of some of your models. You could do that by using db_magic :connection => :foo call from your models. Example:

Sample model on a separate master-slave cluster (so, separate main connection + a slave connection):

Per-Query Connection Management

Sometimes you have select queries that you know you want to run on the master. This could happen for example when you have just added some data and need to read it back and not sure if it made it all the way to the slave yet or no. For this situation and a few others there is a set of methods we’ve added to ActiveRecord models:

1) +on_master+ – this method could be used in two forms: block form and proxy form. In the block form you could force connection switch for a block of code:

In the proxy form this method could be used to force one query to be performed on the master database server:

2) on_slave – this method is used to force a query to be run on a slave even in situations when it’s been previously forced to use the master. If there is more than one slave, one would be selected randomly. This method has two forms as well: block and proxy.

3) on_db(connection) – this method is what makes two previous methods possible. It is used to switch a model’s connection to some db for a short block of code or even for one statement (two forms). It accepts the same range of values as the switch_connection_to method does. Example:

Associations Connection Management

ActiveRecord models can have an associations with each other and since every model has its own database connections, it becomes pretty hard to manage connections in a chained calls like User.posts.count. With a class-only connection switching methods this call would look like the following if we’d want to count posts on a separate database:

Apparently this is not the best way to write the code and we’ve implemented an on_* methods on associations as well so you could do things like this:

Notice: Since ActiveRecord associations implemented as proxies for resulting objects/collections, it is possible to use our connection switching methods even without chained methods:

Starting with DbCharmer release 1.4 it is possible to use prefix notation for has_many and HABTM associations connection switching:

Named Scopes Support

To make it easier for DbCharmer users to use connections switching methods with named scopes, we’ve added on_* methods support on the scopes as well. All the following scope chains would do exactly the same way (the query would be executed on the :foo database connection):

And now, add this feature to our associations support and here is what we could do:

Documentation

For more information on the plugin internals, please check out the source code. All the plugin’s code is ~100% covered with a tests that were placed in a separate staging rails project located at github. The project has unit tests for all or at least the most of the parts of plugin’s code.

What Ruby and Rails implementations does it work for?

We have a continuous integration setups for this plugin on MRI 1.8.6 with Rails 2.2 and 2.3. We use the plugin in production on Scribd.com with MRI (rubyee) 1.8.6 and Rails 2.2.

Who are the authors?

This plugin has been created in Scribd.com for our internal use and then the sources were opened for other people to use. All the code in this package has been developed by Alexey Kovyrin for Scribd.com and is released under the MIT license. For more details, see the LICENSE file.

If you have any comments on this project, feel free to contact me here in comments or by email. And, of course, patches are welcome (only when covered with tests).

First of all, let’s take a look at a typical Scribd’s document URL: http://www.scribd.com/doc/1/Improved-Statistical-Test.

As we can see, it consists of a document-specific part (/doc/1) and a non-unique human-readable slug part (/Improved-Statistical-Test). When a user comes to the site with a wrong slug in the document URL, we need to make sure we send the user to the correct URL with a permanent HTTP 301 redirect. So, obviously we can’t simply send our requests to the squid because it’d cause few problems:

• When we change document’s title, we’d create a new cached item and would not be able to redirect users from the old URL to the new one
• When we change a title, we’d pollute cache with additional document page copies.

One more problem that makes the situation even worse – we have 3 different kinds of users on the site:

1. Logged in users – active web site users that are logged in and should see their name at the top of the page, should see all kinds of customized parts of the page, etc (especially when a page is their own document).
2. Anonymous users – all users that are not logged in and visit the site with a flash-enabled browser
3. Bots – all kinds of crawlers that can’t read flash content and need to see a plain text document version

All three kinds of users should see their own document page versions whether the page is cached or not.

So, how do we solve these two problems? Here is how.

First of all, to fix the URLs problem we’ve decided to rewrite the URL before it goes to a squid server. We change URLs to look like this: http://www.scribd.com/doc/1?__enable_docview_caching=1. This makes the document URL dependent only on a unique document ID that never change and sends an additional parameter to the backend to signal that the page could potentially be cached. The slug is sent to backend using an HTTP-header (X-Scribd-Slug) so that backend could check the slug and return a redirect if needed.

To make sure we won’t respond with a cached page to a request with an invalid URL (invalid slug basically), we use Vary: X-Scribd-Slug HTTP header which is implemented in Squid (only late 2.6 and 2.7) and makes it check specified headers in a request before responding with a cached content. If the header of the cached content is different then the header in the request, squid proxies the resuest to backend where we could handle the cache miss as we want.

Next, to resolve the users problem we’ve created a small nginx module that looking at a request headers could tell you whether the user is a bot or not and whether he’s logged in or an anonymous visitor. This module basically exposes a $scribd_user_id variable to our configs and we could use the variable to do separate configuration for different kinds of users.

At this point we do not cache document pages for logged in users so we basically have two copies of each page in the cache: flash-enabled document page and an inline document page. We do this separation by changing our cache URLs one more time: we add a scribd_user_id=$scribd_user_id variable (anonymous = 0, bot = -1) to the cache URL: http://www.scribd.com/doc/1?__enable_docview_caching=1&scribd_user_id=0.

And last, not not least, we use two really awesome Squid features called stale-while-revalidate and stale-if-error (AFAIR, they were invented in Yahoo! and then described by their squid admin).

Option stale-while-revalidate allow us to quickly serve content from the cache while doing background re-validation requests to the Rails backend. Option stale-if-error basically allows us to serve content from the squid cache when Rails backend is down/dead/slow.

All these changes allowed us to handle more traffic with less hardware and what is even more important, they helped us improve user experience with the site: response times dropped 2-3 fold and much less people see our Ouch! pages (HTTP 50x errors when backend is dead or overloaded). Here is an example of one of our servers’ hit ratio and traffic savings daily graphs:

This it with the logged in users and complex URLs handling in Scribd caching architecture. Next post in this series will explain how we do cache invalidation in Scribd. Stay tuned.

And 3 years ago in March 2006 I’ve decided to start my own blog. I took a standard template and started the blog on a separate domain while the domain was on its own domain name… This spring my wife made me a great birthday present – she’s created me a custom blog design that has all the stuff I wanted from my own web site for a long time. My friend Dima Shteflyuk has helped me with creating a wordpress template from Tanya’s mockups and here we are – now I’ve decided to merge my blog and my web site into a single web entity called http://kovyrin.net/. Welcome to my new blog/site/whatever!

Originally loops plugin was created to make our (Scribd.com) own loops code more organized. We used to have tens of different modules with methods that were called with script/runner and then used with nohup and other not so convenient backgrounding techniques. When you have such a number of loops/workers to run in background it becomes a nightmare to manage them on a regular basis (restarts, code upgrades, status/health checking, etc).

After a short time of writing our loops in more organized ways we were able to generalize most of the loops code so now our loops look like a classes with a single mandatory public method called run. Everything else (spawning many workers, managing them, logging, backgrounding, pid-files management, etc) is handled by the plugin itself.

The major idea behind this small project was to create a deadly simple and yet robust framework to be able to run some tasks in background and do not think about spawning many workers, restarting them when they die, etc. So, if you need to be able to run either one or many copies of your worker or you do not want to think about re-spawning dead workers and do not want to spend megabytes of RAM on separate copies of Ruby interpreter (when you run each copy of your loop as a separate process controlled by monit/god/etc), then I’d recommend you to try this framework — you’ll like it.

For more information, visit the project site and, of course, read the sources

All this time (few months now) I was using starling noticed really bad thing in how it works: if workers die (really die, or lock on something for a long time, or just start lagging) and queue start growing, the thing could kill your server and you won’t be able to do something about it – it just eats all your memory and this is it. Since then I’ve started looking for a better solution for our queuing, the technology was too cool to give up. I’ve tried 5 or 6 different popular solutions and all of them sucked… They ALL had the same problem – if your queue grows, this is your problem and not queue broker’s :-/ The last solution I’ve tested was ActiveMQ and either I wasn’t able to push it to its limits or it is really so cool, but looks like it does not have this memory problem. So, we’ve started using it recently.

In this small post I’d like to describe a few things that took me pretty long to figure out in ruby Stomp client: how to make queues persistent (really!) and how to process elements one by one with clients’ acknowledgments.

First, we need to connect to the queue server and this code is the same for clients and servers:

Second form looks better for me because it allows you to specify if you want client to be reliable (lock on errors and try to reconnect, etc) or just want client to raise an error in case of any problems.

When you’ve connected to the server, you can push your data to a queue:

And this is where you have an ability to specify if you want your data to be persistent (survive server crashes, etc): headers is a hash that could have an element :persistent => true, which would do the thing. So, your code would looks like this (for example):

Now, when you have your data submitted to the queue, you need to be able to read it and process with some script. This is as simple as the following code:

Again, we wanted to receive messages one by one and acknowledge successful processing in our code. This is simple too. You need:

• Pass :ack => :client as an element in your headers hash
• Call client.acknowledge(msg) in the loop if you’re sure that an element could be removed from the queue

This is basically it with the stuff I wanted to explain today. If you’ve never tried to work with any queue servers, try today and maybe tomorrow you won’t be able to imagine your systems architecture without such a component

Few times I was creating a git branches and hacking this action trying to implement some sort of page-level caching to make things faster. But all the time results weren’t as good as I’d like them to be. So, branches were sitting there and waiting for a better idea.

Few months ago my good friend has joined Scribd and we’ve started thinking on this problem together. As the result of our brainstorming we’ve managed to figure out what were the problems preventing us from doing efficient caching:

• First of all, a lots of code in the action is changing the page view if our visitor is a bot (no, not a cloaking, just some minor adjustments of the view).
• Second problem was a set of differences in the view for anonymous and logged in users.
• And finally, third problem was the fact that the page has a few blocks that change pretty dynamically: document stats pane and comments lists.

All these problems when combined were creating a lots of pain when I was trying to cache a whole page. When we’ve figured them out, we’ve started thinking on how could we generalize possible combinations of those factors and possible approaches to caching.

There is a well known idea in web applications development: the fastest web app action is an action that does not require any code to be executed on your application server. So, first idea we’ve tried to think about was some approach that would definitely reduce the number of hits on our app servers. This idea was based on HTTP protocol features related to Last-Modified and E-Tag headers. But there was a problem – not so many users go to the same page twice so even if we’d make the page cacheable, it wouldn’t help too much. But the idea of full page caching outside of the application was really good and we’ve started playing with it to figure out how to use it in production.

Long time ago, when Internet was slow and expensive many ISPs and large companies were trying to reduce their traffic w/o hurting users’ experience. Then caching proxy servers were born. The idea of those servers was to handle all web requests going from a network (ISP or a company office) and try to cache as much content as possible so when the same or some other user would request a cached page, proxy server would return it really fast. If we’d implement support for those Last-Modified headers, all proxy servers would be happy to cache our pages. But there was a problem – no one uses caching proxies in 2008 So, we’ve got an idea – why can’t we place such a server in front of our application and make it cache content for all users in the world? (Yes, we knew about a caching reverse proxies before – I’m just trying to explain the flow of our thoughts and words when we were brainstorming the problem).

The only problem with this approach would be to differentiate logged in users, anonymous users and bots. Considering the fact that our proxy server could be placed between the app and our web servers (nginx), we’ve decided to create a nginx module that would translate the same document page URLs to a set of URLs, which would be different for all those 3 kinds of users.

When all those problems with different kinds of users were solved, we’ve decided to solve the last one – non-cacheable dynamic stats pane. The solution was pretty simple – we’ve added a small ajax call to the page which would update stats on the cached version of our page for all real users while bots will see the same page, but with a bit stale stats pane.

Long story short, the results is really great. Application servers load reduced by 50-70%, database servers load is reduced by 30-60%, response times dropped down to 150-200 msec from 500-750 msec. As an additional positive effect of the caching we’ve managed to remove all fragments caches from the application and free more of memcached resources for data caches. Here are a few cacti graphs of our servers load/traffic (the caching was introduced on Oct 9th at night):

Main MySQL command counters:

Uploaded with plasq’s Skitch!

One of our Application Servers CPU Usage:

Uploaded with plasq’s Skitch!

One of our Application Servers Load Average:

Uploaded with plasq’s Skitch!

Unfortunately there are a lot of things to share related to this caching experience, so I’ve decided to make a series of posts that would explain all the problems we had and solutions we’ve found for each of the following parts of the caching system:

• Logged in Users and Complex URLs Handling
• Rails code to support Last-Modified headers and how we purge caches
• Squid Server setup (configs and hardware)
• Nginx module development

So, if you’re interested in details, subscribe to this blog’s RSS feed and in a few days you’ll see the first article from this series.

Bounces-handler package is a simple set of scripts to automatically process email bounces and ISP‘s feedback loops emails, maintain your mailing blacklists and a Rails plugin to use those blacklists in your RoR applications.

This piece of software has been developed as a part of more global work on mailing quality improvement in Scribd.com, but it was one of the most critical steps after setting up reverse DNS records, DKIM and SPF.

The package itself consists of two parts:

• Perl scripts to process incoming email:
  • bounces processor — could be assigned to process all your bounce emails
  • feedback loops messages processor — more specific for Scribd, but still – could be modified for your needs (will be released soon).
• Rails plugin to work with mailing blacklists

For more information, please check our README file. If you have any questions, comments or suggestions, please leave them here as a comments and I’ll try to reply as soon as possible.

The problem with browsing was pretty simple to describe and as hard to fix – we have large data set which consists of a few tables with many fields with really bad selectivity (flag fields like is_deleted, is_private, etc; file_type, language_id , category_id and others). As the result of this situation it becomes really hard (if possible at all) to display documents lists like “most popular 1-10 pages PDF documents in Italian language from the category “Business” (of course, non-deleted, non-private, etc). If you’ll try to create appropriate indexes for each possible filters combination, you’ll end up having tens or hundreds of indexes and every INSERT query in your tables will take ages.

So, we were doing many weird things to solve this problem, like having really huge covering indexes for most popular browsing situations, having powerful (and expensive) dedicated servers for browsing queries, etc. Nothing helped and we ended up reducing filters number to 1-2 per query. But with continuously growing documents base we’d really love to help people find interesting content and filters is what we’d really want to add to the project.

Long story short, we’ve decided to try to use Sphinx search engine for really unusual thing – we decided to move all SELECT queries for our documents to a small (proof of concept) sphinx installation on one of our DB servers and use Sphinx’ full-scan feature. After a few hours of development I’ve created a simple to use module which mimics ActiveRecord’s find method interface and allows us to perform queries like the following (sometimes even more powerful than AR):

After a short testing we’ve got our performance benchmarks results and here are some facts:

• documents count – 3000000+
• re-indexing time – 20-30 seconds for an entire data set (4 large joined tables)
• longest query (surprisingly – a query without any filters) takes about 0.5 sec on one core
• generic query with 1-2 filters takes 0.01-0.05 sec on one core
• queries with many filters take 0.01 sec and less on one core
• queries time could be reduced almost linearly when adding more cores (splitting index on many pieces).
• sphinx scales really easily on more than one box if we’ll need it to.

As you can see here, it is really useful sometimes to think “out of the box” when you work with some tools – we were using sphinx for a quite a long time now, but this solution for browsing was like a blessing and really helped us to keep up with service popularity growth.

• More MySQL data types added: DECIMAL (both old and new), DATE, TIME
• CHAR data type handling improved in table definitions generator
• Indexes filtering added to page_parser
• 64-bit stat() support added to all tools
• Linux has no isnumber() function so we define our own implementation (pretty simple)
• Lots of fixes in create_defs.pl script – now it generates definitions which could recover your data in 80% cases w/o any changes.
• Min/max record size calculation fixed in constraints-based parser.
• Nullable fixed-size columns support is fixed.
• Debug logging is much cleaner now.

As always, if you need any help with your recovery, we would love to help.

Looking at this sessions-related situation we have created really simple (and, at the same time, really useful for large Rails projects) plugin, which replaces ActiveRecord-based session store and makes sessions much more effective. Below you can find some information about implementation details and decisions we’ve made in this plugin, but if you just want to try it, then check out our project site.

FastSessions is a sessions class for ActiveRecord sessions store created to work fast (really fast). It uses some techniques which are not so widely known in developers’ community and only when they cause huge problems, performance consultants are trying to help with them.

The Problem

Original ActiveRecord sessions store is slow. It is fine for some low traffic blogs, but it is too slow to use it on some big/large/huge sites. First of all, it is slow because ActiveRecord is slow. It is powerful ORM framework, but it is overkill for such simple task as a sessions management.

That is why people created SqlSession store. It works with mysql directly with database APIs and works much faster than original AR session store. But it is still slow because:

• it creates/updates session on each hit – even dumb bots crawling your sites create thousands of thousands of useless records in your sessions table, 99% of hits do not require any session updates!
• it uses 32-char string as a key for sessions records – all databases work with string keys MUCH slower that with integers keys, so it would be much better to use integers, but we have so long session ids and all session stores use these session ids as a key.
• it uses auto_increment primary key, which causes table-level locks in InnoDB for all MySQL versions prior to 5.1.21. These table-level locks with unnecessary inserts cause really weird problems for large sites.

The Solution

FastSessions plugin was born as a hack created for Scribd.com (large RoR-based web project), which was suffering from InnoDB auto-increment table-level locks on sessions table.

So, first of all, we removed id field from the table. Next step was to make lookups faster and we’ve used a following technique: instead of using (session_id) as a lookup key, we started using (CRC32(session_id), session_id) – two-columns key which really helps MySQL to find sessions faster because almost all lookups use crc32 field only to find needed record.

And last, but most powerful change we’ve tried to make was to not create database records for empty sessions and to not save sessions data back to database if this data has not been changed during current request processing.

All of these changes were implemented and you can use them automatically after a simple plugin installation.

Controversial Decisions

Many plugin users would never think about one problem we’ve introduced when removed that auto-increment primary key, so I’d like to describe it here. The problem is following.

InnoDB groups all data in tables by primary key. This means that when we create auto-increment primary key and insert records to a table, our sessions records are grouped together and saved sequentially on the disk. But if we’ll make pretty random value (like crc32 of a random session id) a primary key, then every session record will be inserted in its own place and it will generate some random I/O which is not so good for I/O bound servers.

So, we decided to let the user choose what primary key to use in his deployment of our plugin, so if you’re going to use this module with MySQL 5.1.22+, then you’d like to set

because it will provide you with consecutive data inserts in InnoDB. Another cases when you’d like to use it is when your MySQL server is I/O bound now and you do not want to add random I/O because of randomized primary key.

Working With Old AR Sessions Table

If you do not like to loose old sessions created with default AR sessions plugin, you could set

and then all session reads will fall back to old sessions table if some session_id was not found in default fast sessions table. Old sessions table name could be set using

variable.

Installation

This plugin installation is pretty simple and described in a few steps below:

1. Install this plugin sources in your vendor/plugins directory (it could be ./script/plugin install, or piston import command – it is up to you) from our SVN reposipory. For example:

   1	$ piston import http://rails-fast-sessions.googlecode.com/svn/trunk/ vendor/plugins/fast_sessions

2. Enable ActiveRecord session store in your config/environment.rb file:

   1 2 3 4 5

   Rails::Initializer.run do |config|   ......   config.action_controller.session_store = :active_record_store   ...... end

3. Create migration for your new sessions table:

   1	$  ./script/generate fast_session_migration AddFastSessions

4. Open your newly created migration and change table_name and use_auto_increment parameters of the plugin (if you want to).
5. Run your migration:

   1	$  rake db:migrate

6. Start your application and try to perform some actions which would definitely save some data to your session. Then check your fast_sessions table (if you did not renamed it) for a records.

Downloading

Most recent version if this plugin could be found on the project’s site or in SVN repository

Author

This plugin has been created by Alexey Kovyrin. Development is sponsored by Scribd.com.