Tag: development
ActiveMQ + Ruby Stomp Client: How to process elements one by one
30 Oct2008

Few months ago I’ve switched one of our internal projects from doing synchronous database saves of analytics data to an asynchronous processing using starling + a pool of workers. This was the day when I really understood the power of specialized queue servers. I was using database (mostly, MySQL) for this kind of tasks for years and sometimes (especially under a highly concurrent load) it worked not so fast… Few times I worked with some queue servers, but those were either some small tasks or I didn’t have a time to really get the idea, that specialized queue servers were created just to do these tasks quickly and efficiently.

All this time (few months now) I was using starling noticed really bad thing in how it works: if workers die (really die, or lock on something for a long time, or just start lagging) and queue start growing, the thing could kill your server and you won’t be able to do something about it – it just eats all your memory and this is it. Since then I’ve started looking for a better solution for our queuing, the technology was too cool to give up. I’ve tried 5 or 6 different popular solutions and all of them sucked… They ALL had the same problem – if your queue grows, this is your problem and not queue broker’s :-/ The last solution I’ve tested was ActiveMQ and either I wasn’t able to push it to its limits or it is really so cool, but looks like it does not have this memory problem. So, we’ve started using it recently.

In this small post I’d like to describe a few things that took me pretty long to figure out in ruby Stomp client: how to make queues persistent (really!) and how to process elements one by one with clients’ acknowledgments.
Read the rest of this entry

MySQL Master-Master Replication Manager 1.0 Released
29 Oct2007

It’s been a long time since we’ve started this project and it is time to make a checkpoint. So, I’ve decided to release final 1.0 version and make 1.X branch stable while all serious development with deep architectural changes will be done 2.X branch (trunk at this moment).

Changes from previous release:

  • Perl semaphores implementation caused huge memory leaks (mmmd_mod).
  • Now we do not send any commands to hard offline hosts with dead TCP/IP stack to prevent mointoring problems for other hosts.
  • Removed legacy StartSlave method from agent code which caused problems on some Perl versions
  • Added a few fixes to prevent non-exclusive roles from moving. This caused internal status structures to be corrupted.
  • Made all mysql checks properly report errors occurring (previously they were resulting in an OK response). Thanks to Phillip Pearson.
  • Some memory leaks found in mysql checkers and as a quick fix we’ve added an idea of “Maximum Checks Before Restart” to all checkers. If you want some checker to restart after N checks, simply add “restart after N” to your checker declaration.
  • Added some more docs to the project site.

New version can be obtained here or from the project’s SVN repository.