Last few days one of our customers (one of the largest Ruby on Rails sites on the Net) was struggling to solve some really strange problem – once upon a time they were getting an error from ActiveRecord on their site:
1
| (ActiveRecord::StatementInvalid) "Mysql::Error: Lock wait timeout exceeded; try restarting transaction: UPDATE some_table..... |
They have innodb_lock_wait_timeout set to 20 seconds. After a few hours of looking for strange transactions we were decided to create s script to dump SHOW INNODB STATUS and SHOW FULL PROCESSLIST commands output to a file every 10 seconds to catch one of those moments when this error occurred.
Today we’ve got next error and started digging in our logs…
Read the rest of this entry →
I’m returned from my 1-week vacation today and want to say – I’ve never been so productive as I was there 😉 Blue ocean, hot sun and white sand really helped me to finish my work on the first release of one really awesome project.
Today I’m proud to announce our first public release of the Data Recovery Toolkit for InnoDB – set of tools for checking InnoDB tablespaces and recovering data from damaged tablespaces or from dropped/truncated InnoDB tables.
Read the rest of this entry →
Today I was developing one small merb application for one of our projects and needed to see ActiveRecord logging on console like I do in Rails. After a short research I’ve found out that merb_active_record plugin passes its MERB_LOGGER to AR by default so I decided to try to change merb log level and here they are – my pretty colored AR logs!
So, if you want to see ActiveRecord logs in your application in development mode, then you need to add one line to your conf/environments/development.rb file:
1
2
| puts "Loaded DEVELOPMENT Environment..."
MERB_LOGGER.level = Merb::Logger::DEBUG |
That’s it for now. Long live merb! 🙂
Today I was working on one small consulting task and our client asked for an upgrade from MySQL 5.0 to 5.1. It was pretty easy and task was successfully finished and reported to the customer… But few hours after my report I’ve got an email from customer with something like “WTF? Where is my 5.1?!”. I was shocked when I saw happily running 5.0 on their server w/o anything related to my 5.1 installation…
After some short investigation I’ve found out, that it was cpanel (dumb software for dumb system administrators) – it noticed, that installed mysql version (5.1) is not the same as it thought it should be (5.0), so without any warnings or notices it removed all 5.1 rpms and installed “brand new” 5.0.
Here I’d like to say GREAT THANKS to mysql team for such a great software which did not screwed up user’s data in such situation. But what idiots in cpanel development team decided, that is it appropriate and acceptable to perform such operations?! As an administrator and as a software developer I do not understand them – I just can’t understand such approach….
So, enough complaining – here is a piece of useful information for my readers: If you’re so unlucky to have cpanel installed on your server and you’d like to upgrade your mysql manually, then you can perform following operations:
1
2
3
| # touch /etc/mysqlupdisable
# chattr +i /etc/mysqlupdisable
# service cpanel restart |
After these small changes your cpanel will forget about mysql upgrades and you’ll be able to do what you want and not what some dumb developers decided you should do.
It’s been a long time since we’ve started this project and it is time to make a checkpoint. So, I’ve decided to release final 1.0 version and make 1.X branch stable while all serious development with deep architectural changes will be done 2.X branch (trunk at this moment).
Changes from previous release:
- Perl semaphores implementation caused huge memory leaks (mmmd_mod).
- Now we do not send any commands to hard offline hosts with dead TCP/IP stack to prevent mointoring problems for other hosts.
- Removed legacy StartSlave method from agent code which caused problems on some Perl versions
- Added a few fixes to prevent non-exclusive roles from moving. This caused internal status structures to be corrupted.
- Made all mysql checks properly report errors occurring (previously they were resulting in an OK response). Thanks to Phillip Pearson.
- Some memory leaks found in mysql checkers and as a quick fix we’ve added an idea of “Maximum Checks Before Restart” to all checkers. If you want some checker to restart after N checks, simply add “restart after N” to your checker declaration.
- Added some more docs to the project site.
New version can be obtained here or from the project’s SVN repository.