Last few days one of our customers (one of the largest Ruby on Rails sites on the Net) was struggling to solve some really strange problem – once upon a time they were getting an error from ActiveRecord on their site:
(ActiveRecord::StatementInvalid) "Mysql::Error: Lock wait timeout exceeded; try restarting transaction: UPDATE some_table.....
They have innodb_lock_wait_timeout set to 20 seconds. After a few hours of looking for strange transactions we were decided to create s script to dump SHOW INNODB STATUS and SHOW FULL PROCESSLIST commands output to a file every 10 seconds to catch one of those moments when this error occurred.
Today we’ve got next error and started digging in our logs…
Read the rest of this entry »
I’m returned from my 1-week vacation today and want to say – I’ve never been so productive as I was there Blue ocean, hot sun and white sand really helped me to finish my work on the first release of one really awesome project.
Today I’m proud to announce our first public release of the Data Recovery Toolkit for InnoDB – set of tools for checking InnoDB tablespaces and recovering data from damaged tablespaces or from dropped/truncated InnoDB tables.
Read the rest of this entry »
Today I was working on one small consulting task and our client asked for an upgrade from MySQL 5.0 to 5.1. It was pretty easy and task was successfully finished and reported to the customer… But few hours after my report I’ve got an email from customer with something like “WTF? Where is my 5.1?!”. I was shocked when I saw happily running 5.0 on their server w/o anything related to my 5.1 installation…
After some short investigation I’ve found out, that it was cpanel (dumb software for dumb system administrators) – it noticed, that installed mysql version (5.1) is not the same as it thought it should be (5.0), so without any warnings or notices it removed all 5.1 rpms and installed “brand new” 5.0.
Here I’d like to say GREAT THANKS to mysql team for such a great software which did not screwed up user’s data in such situation. But what idiots in cpanel development team decided, that is it appropriate and acceptable to perform such operations?! As an administrator and as a software developer I do not understand them – I just can’t understand such approach….
So, enough complaining – here is a piece of useful information for my readers: If you’re so unlucky to have cpanel installed on your server and you’d like to upgrade your mysql manually, then you can perform following operations:
# touch /etc/mysqlupdisable
# chattr +i /etc/mysqlupdisable
# service cpanel restart
After these small changes your cpanel will forget about mysql upgrades and you’ll be able to do what you want and not what some dumb developers decided you should do.
It’s been a long time since we’ve started this project and it is time to make a checkpoint. So, I’ve decided to release final 1.0 version and make 1.X branch stable while all serious development with deep architectural changes will be done 2.X branch (trunk at this moment).
Changes from previous release:
- Perl semaphores implementation caused huge memory leaks (mmmd_mod).
- Now we do not send any commands to hard offline hosts with dead TCP/IP stack to prevent mointoring problems for other hosts.
- Removed legacy StartSlave method from agent code which caused problems on some Perl versions
- Added a few fixes to prevent non-exclusive roles from moving. This caused internal status structures to be corrupted.
- Made all mysql checks properly report errors occurring (previously they were resulting in an OK response). Thanks to Phillip Pearson.
- Some memory leaks found in mysql checkers and as a quick fix we’ve added an idea of “Maximum Checks Before Restart” to all checkers. If you want some checker to restart after N checks, simply add “restart after N” to your checker declaration.
- Added some more docs to the project site.
New version can be obtained here or from the project’s SVN repository.
One of MMM users reported that they’re experiencing really weird memory leaks in checker processes used by MMM. After a deep investigation I’ve found out that Perl part of the checker and checker modules does not leak (at least I didn’t found these leaks), so I think it could be caused by some problems in MySQL DBD module (client uses Ubuntu server).
So, I’d like to ask all users to check if their checker processes use more memory than expected and if yes, what OS, MySQL libraries versions and Perl version used on their servers.
Thanks in advance for any help.
Recently I had one customer for consulting and aside from mysql optimization, etc they asked me for cacti installation/setup to monitor their pretty generic LAMP application. I’ve started setting up all this stuff and I’ve never thought it could be so painful… lots of different templates for the same tasks, all of them are incompatible with recent cacti releases, etc, etc… So, this post is generally a list of used templates with a fixes I’ve made to make them work on recent cacti release.
Read the rest of this entry »
Thanks to MySQL community members we’ve got really great collection of media files from the recent mysqluc. Thanks to Seeri for all that work he’s done to collect everything in one place so we could watch/listen/read information from this great event.
So, if you did not attended mysqluc07, then you definitely should visit Technocation page, dedicated to this conference.
P.S. I’m going to post links to these videos and descriptions for the sessions on the Best Tech Videos soon, so If you are not sure to watch some video or not, just wait while I’m merging these links with information from mysqluc site.
New alpha release 1.0-pre4 of the MySQL Master-Master Replication Manager. This release has lots of major fixes and I’m glad to announce first sponsored port of mmm to non-linux platform – it has been ported to Solaris 10. So, here are our changes in this version:
- Real checks timeouts – I’ve found and fixed lots of problems in checks timeout code and now if you specified in your mmm_mon.conf, that some check should timeout in 5 sec, it would timeout correctly on all supported platforms.
- External third-party tools using – On all supported non-linux platforms mmm will use system binaries for fping and arp_ping so porting to another platforms would be much easier.
- Agent notifications fix – Now we don’t try to notify dead agents about cluster changes and additionally we have 10 sec timeout on notification sends to prevent monitoring from lagging on network connection timeouts.
- Bundled fping and send_arp – We have both used third-party tools used in mmm bundled in our distribution as separate build trees (you could find and build/install them from contrib directory).
- Flexible perl binary location – We’re using “#!/usr/bin/env perl” as shebang line in our perl scripts so you can use any perl interpreter just by placing it earlier in your PATH variable.
Notice: If you’ll try to install this version, try to run bin/sys/fping and bin/sys/send_arp on your server before installation. If you’ll notice any errors, feel free to build binaries for your platform using contrib/* sources (you’ll need gcc and libnet installed).
So, as you can see, mmm development goes forward and we’re fixing some problems trying to make this software mature. If you want to help us, you can send your comments to mmm-devel mailing list, post your bug reports to our bug tracking or sponsor any changes you need
MySQL Master-Master Replication Manager version 1.0-pre3 has been released today. Changes list is really short now:
- Major fix in multiple clusters support – now you can use many clusters with one monitoring node (details are in mmmd_mon man page)
- Man pages for mmmd_mon, mmmd_agent and mmm_control scripts
- Startup scripts added/fixed for mmmd_mon and mmmd_agent
- Installation script now requires iproute package to be installed on server.
As always, if you have any questions/suggestions, post them here or in mmm-devel mail list.
So, first two days of mysqlconf’07 are finished now. What can I say is that without any doubts: It worth it! If you’re working on some high traffic projects, some high-loaded database driven systems, etc, you definitely should attend such conferences – you’d never be able to get such big amounts of information from the best people in the industry as you can get here.
I’ve been attending mostly practical MySQL scale-out sessions and BOFs and I’ve got really controversial impressions. No – everybody was great, controversial thing is my own level – I never before was so sure that I know nothing at all! I see people here for which most of new things for me (especially in MySQL scaling) are pretty obvious and it is hard to keep myself from some kind of self-beating because I don’t know these things.
Interesting thing happened today – I clearly realized why YouTube guys got their billion and we or someone else didn’t – IMHO – one of the major factors was their ability to handle all these users traffic efficiently. They’ve got one of the best (imho) technical people in their command. I was really impressed by one of their guys (sorry, but I’m too bad in remembering people’s names) – one of the two founders – who participated in MySQL replication and scale out BOF as attendee. He clearly showed me that they’re able to grow almost without any limits because of their skills which is one of the major requirements in today’s fast changung world.
Another interesting thing was that I realized that my spoken English could be really good when people speak clearly and I understand their questions. I spoke with some guys about our mmm project, about maby things and there was no problems at all!
So, its going pretty late and tomorrow going to be a long day so I want to repeat – if you have any chance to get on such conferences, DO IT! If your company can’t help you, spend your own money or find some other solutions, but DO IT! It is the best thing could happen to YOU as professional!
P.S. Just remembered – I saw some women-DBAs today! Really smart girls! I never thought that pretty girl can become such great IT prefessional and now I know – I was really wrong.
P.P.S. Sorry to russian-speaking readers – I will definitely translate my last posts later – just when I’ll come back home from this crazy (but the best) event with 14hrs of load a day