Homo-Adminus Blog

Today I’ve managed to finish initial version of our bounces-handler package we use for mailing-related stuff in Scribd.

Bounces-handler package is a simple set of scripts to automatically process email bounces and ISP‘s feedback loops emails, maintain your mailing blacklists and a Rails plugin to use those blacklists in your RoR applications.

This piece of software has been developed as a part of more global work on mailing quality improvement in Scribd.com, but it was one of the most critical steps after setting up reverse DNS records, DKIM and SPF.

The package itself consists of two parts:

• Perl scripts to process incoming email:
  • bounces processor — could be assigned to process all your bounce emails
  • feedback loops messages processor — more specific for Scribd, but still – could be modified for your needs (will be released soon).
• Rails plugin to work with mailing blacklists

For more information, please check our README file. If you have any questions, comments or suggestions, please leave them here as a comments and I’ll try to reply as soon as possible.

If you’ve ever worked in companies with 5-10+ servers and it was your responsibility to install new boxes, change some configuration files and install new software on many boxes you definitely know how painful this work is. Every time you need to change something on 3-5-100 boxes, you go there and make those changes. Most experienced of us used some weird scripts to perform some task on many boxes or used some stuff like dsh. Even with those tricks I’d never wish this work to anyone.

While I was working in Galt, I’ve asked our junior admin to check out puppet and try to use it on our servers. After a week of screaming he’s managed to install and configure it and said “Wow! This is what we definitely need!”. Since then he was using it and was happy :-). But, unfortunately (for me) I’ve never tried it and when I’ve changed my job and started doing the same old (but not good) things like dsh and for+ssh I’ve started thinking: “Stop! Try puppet – maybe it is what you need!”. And what can i say? Yes! This is what every admin needs! Today we’ve deployed our next web server (we do it pretty often) and it was done with puppet – pretty painlessly and quickly.

So, what is puppet? Let me quote their site:

Puppet lets you centrally manage every important aspect of your system using a cross-platform specification language that manages all the separate elements normally aggregated in different files, like users, cron jobs, and hosts, along with obviously discrete elements like packages, services, and files.

Puppet’s simple declarative specification language provides powerful classing abilities for drawing out the similarities between hosts while allowing them to be as specific as necessary, and it handles dependency and prerequisite relationships between objects clearly and explicitly. Puppet is written entirely in Ruby.

Basically, puppet is a tool which lets you describe whatever you’d like about your server(s) and deploy your configuration (configs, packages, files, directories, cron jobs, user accounts, ssh keys, etc) to any number of servers you need. You can describe some simple parts of your system (like “apache”, “nginx”, “rails”, “useful_gems”, “admin-ssh-keys”, etc) and then describe your servers using those terms (like machine X = apache+rails+useful_gems+admin_ssh_keys). When it is done, you’ll be able to deploy all this stuff with one command!

So, if you’re struggling with deploying some changes on many servers or, even, have only a few of them, I’d recommend to try puppet – it could save you a lots of time and nerves.

P. S. When you’ll try to use it – do not give up for a few days – it is hard to change your way of thinking and start writing correct and convenient scripts using puppet’s language.

This will be one of those posts I’d like to publish primarily to be able to coma back later and check it out instead of reading docs again 🙂

So, we have a server with two (or more) network interfaces are we need to be able to use more than one interface in our VDS machines. How do we set it up?

Read the rest of this entry →

It’s been a long time since we’ve started this project and it is time to make a checkpoint. So, I’ve decided to release final 1.0 version and make 1.X branch stable while all serious development with deep architectural changes will be done 2.X branch (trunk at this moment).

Changes from previous release:

• Perl semaphores implementation caused huge memory leaks (mmmd_mod).
• Now we do not send any commands to hard offline hosts with dead TCP/IP stack to prevent mointoring problems for other hosts.
• Removed legacy StartSlave method from agent code which caused problems on some Perl versions
• Added a few fixes to prevent non-exclusive roles from moving. This caused internal status structures to be corrupted.
• Made all mysql checks properly report errors occurring (previously they were resulting in an OK response). Thanks to Phillip Pearson.
• Some memory leaks found in mysql checkers and as a quick fix we’ve added an idea of “Maximum Checks Before Restart” to all checkers. If you want some checker to restart after N checks, simply add “restart after N” to your checker declaration.
• Added some more docs to the project site.

New version can be obtained here or from the project’s SVN repository.

New alpha release 1.0-pre4 of the MySQL Master-Master Replication Manager. This release has lots of major fixes and I’m glad to announce first sponsored port of mmm to non-linux platform – it has been ported to Solaris 10. So, here are our changes in this version:

• Real checks timeouts – I’ve found and fixed lots of problems in checks timeout code and now if you specified in your mmm_mon.conf, that some check should timeout in 5 sec, it would timeout correctly on all supported platforms.
• External third-party tools using – On all supported non-linux platforms mmm will use system binaries for fping and arp_ping so porting to another platforms would be much easier.
• Agent notifications fix – Now we don’t try to notify dead agents about cluster changes and additionally we have 10 sec timeout on notification sends to prevent monitoring from lagging on network connection timeouts.
• Bundled fping and send_arp – We have both used third-party tools used in mmm bundled in our distribution as separate build trees (you could find and build/install them from contrib directory).
• Flexible perl binary location – We’re using “#!/usr/bin/env perl” as shebang line in our perl scripts so you can use any perl interpreter just by placing it earlier in your PATH variable.

Notice: If you’ll try to install this version, try to run bin/sys/fping and bin/sys/send_arp on your server before installation. If you’ll notice any errors, feel free to build binaries for your platform using contrib/* sources (you’ll need gcc and libnet installed).

So, as you can see, mmm development goes forward and we’re fixing some problems trying to make this software mature. If you want to help us, you can send your comments to mmm-devel mailing list, post your bug reports to our bug tracking or sponsor any changes you need 😉