How often do you think about the reasons why your favorite RDBMS sucks? 
Last few months I was doing this quite often and yes, my favorite RDBMS is MySQL. The reason why I was thinking so because one of my recent tasks at Scribd was fixing scalability problems in documents browsing.
The problem with browsing was pretty simple to describe and as hard to fix – we have large data set which consists of a few tables with many fields with really bad selectivity (flag fields like is_deleted, is_private, etc; file_type, language_id , category_id and others). As the result of this situation it becomes really hard (if possible at all) to display documents lists like “most popular 1-10 pages PDF documents in Italian language from the category “Business” (of course, non-deleted, non-private, etc). If you’ll try to create appropriate indexes for each possible filters combination, you’ll end up having tens or hundreds of indexes and every INSERT query in your tables will take ages.
Read the rest of this entry »
How often do we think about our http sessions implementation? I mean, do you know, how your currently used sessions-related code will behave when sessions number in your database will grow up to millions (or, even, hundreds of millions) of records? This is one of the things we do not think about. But if you’ll think about it, you’ll notice, that 99% of your session-related operations are read-only and 99% of your sessions writes are not needed. Almost all your sessions table records have the same information: session_id and serialized empty session in the data field.
Looking at this sessions-related situation we have created really simple (and, at the same time, really useful for large Rails projects) plugin, which replaces ActiveRecord-based session store and makes sessions much more effective. Below you can find some information about implementation details and decisions we’ve made in this plugin, but if you just want to try it, then check out our project site.
Read the rest of this entry »
I’m returned from my 1-week vacation today and want to say – I’ve never been so productive as I was there 
Blue ocean, hot sun and white sand really helped me to finish my work on the first release of one really awesome project.
Today I’m proud to announce our first public release of the Data Recovery Toolkit for InnoDB – set of tools for checking InnoDB tablespaces and recovering data from damaged tablespaces or from dropped/truncated InnoDB tables.
Read the rest of this entry »
It’s been a long time since we’ve started this project and it is time to make a checkpoint. So, I’ve decided to release final 1.0 version and make 1.X branch stable while all serious development with deep architectural changes will be done 2.X branch (trunk at this moment).
Changes from previous release:
- Perl semaphores implementation caused huge memory leaks (mmmd_mod).
- Now we do not send any commands to hard offline hosts with dead TCP/IP stack to prevent mointoring problems for other hosts.
- Removed legacy StartSlave method from agent code which caused problems on some Perl versions
- Added a few fixes to prevent non-exclusive roles from moving. This caused internal status structures to be corrupted.
- Made all mysql checks properly report errors occurring (previously they were resulting in an OK response). Thanks to Phillip Pearson.
- Some memory leaks found in mysql checkers and as a quick fix we’ve added an idea of “Maximum Checks Before Restart” to all checkers. If you want some checker to restart after N checks, simply add “restart after N” to your checker declaration.
- Added some more docs to the project site.
New version can be obtained here or from the project’s SVN repository.
One of MMM users reported that they’re experiencing really weird memory leaks in checker processes used by MMM. After a deep investigation I’ve found out that Perl part of the checker and checker modules does not leak (at least I didn’t found these leaks), so I think it could be caused by some problems in MySQL DBD module (client uses Ubuntu server).
So, I’d like to ask all users to check if their checker processes use more memory than expected and if yes, what OS, MySQL libraries versions and Perl version used on their servers.
Thanks in advance for any help.
This week aside from tons of different tasks I was working on one of MMM users complaint regarding some issues with MMM on Solaris 10. I knew that this OS has not so user (admin) friendly environment (especially for people with strong GNU-related background), but had no other options and decided to install Solaris 10 in VMWare Fusion on my desktop.
Installation was a bit strange comparing to Debian/RHEL/Ubuntu and FreeBSD where I have a strong experience, but I’ve managed to install it successfully. The major problem after my first boot was a lack of knowledge about how things could be done in Solaris… Below I’ll describe what generic Linux admin could do with Solaris to make it easier to use and more friendly for GNU-addicted mind
Notice: If you’re Solaris administrator, please, don’t read further because I’m pretty sure that all these suggestions would look dumb for you (I knew some old solaris admins before and they hated GNU environments).
Read the rest of this entry »
Yes! We did it! Just before I went to bed at 3:40AM (heh ), I noticed, that someone submitted new video to BTV. I’ve approved it instantly and then noticed a following information in the top part of the site: “We have 1000 videos / 569+ hours of high-quality content!” So, we broke trough 1k videos barrier! And we’ll try to do 2k videos before the end of this year, we have some new features in closed beta testing which would greatly increase number of submitted videos each day.
Thanks to all our users for their contributions and support! Stay with us and we’ll try to give you more features really soon.
Today I’ve started moving (actually copying) all information I have about MMM to the project’s wiki.
If someone is interested in adding some information there or modifying my descriptions (yes, I know about by awful English and I hope someone will fix my mistakes), ping me (contacts are here) and I’ll add you to the project members list.
Thanks all for participation and support.
So, we did it! New Best Tech Videos site version has been released today. Of course, it could have some issues, and we are really looking forward for your feedback on our support forums.
Let’s review main improvements we’ve made here:
- First of all, I want to mention our first step to socialization of the BTV – all our users could signup now and get their own lists of favorited, commented and voted videos and much more – they could have their own personal RSS feeds.
- Next cool thing we’ve prepared for you is user-generated content! You can find some great videos on the Net and share them with fellow BTV readers. At this moment our posting system works in pre-moderated mode to keep really high level of videos we post here, but later we want to delegate moderation to our most active members.
- As I mentioned before – we have our own video rating system. All our users could vote for videos they like and then we’ll let users to decide, what videos they would like to watch based on these votes.
So, these are just small part (but most valuable) of our ideas. If you want to have a look at all of them, just sign up and start using the service – you gonna like it, we’re sure!
Thanks to all our readers, without your help this release would not happen!