Oleksiy Kovyrin

Yet Another Admin's Blog

Archive for the ‘Networks’ Category

Fwd: Scorching hot Startup Needs Scalability Sorcerer and Optimization Freak

Posted by Oleksiy Kovyrin under Databases, Development, Networks

Question: Do you think you have what it takes to take a service from a few hundred thousand users to tens of millions of users in 1 year flat? If you do read on and perhaps become the next beloved scalability rockstar of our age.

We are looking for a data charmer. A mysql magician. A code hack. A funny man. A mad man. A passionate man. Or perhaps a woman who does all these things and more.

Here’s what you gotta do:

  • Pro-active and reactive performance analysis, monitoring and general database plumbing of all leaky issues.
  • Work with others on the team to help maintain/improve and support the infrastructure for a high traffic, high growth site
  • Optimize and tune the database day to day
  • Algorithmic bent. Develop algos to quicken search times, response times, find shortest paths between various connections on site.
  • Have solid low level networking/protocol/computer security skills
  • Log everything. Usage stats, search stats, user behaviour stats. Draw conclusions. Constantly refine and tinker.
  • Help with periodic large storage migrations
  • Work intimately with operations, development, and strategy team to ensure smooth deployments of new iterations, high availability of database services.
  • Understand capacity planning. Always thinking 10 steps ahead. (Whether it means looking at distributed systems services, cloud computing options, evaluating HA models used in other industries etc)
  • Have a pulse on the state of the web, social media, social networking, different scalability architectures, benefits/negatives of each.
  • Interest in high concurrency, distributed systems architectures.
  • General low level hacking/scripting/optimizations in perl/python.
  • Evaluate changing conditions in the archi
  • Think creatively. No dogmatists.

Ideal skillset:

  • BS in Comp Sci or equivalent
  • 5+ years experiene with Linux/Unix systems
  • 3+ years with MySQL in production environment
  • Knowledge and experience with partitioned architectures and a database sharding techniques
  • Capacity planning/high growth planning/emergency planning experience
  • Passion, bordering on paranoia, for hunting bottlenecks, and optimizing IO operations
  • Experience with MySQL replication
  • Deep experience with MySQL internals
  • Experience with performance analysis tools, storage engines, backup methodologies for MySQL
  • Great perl/shell scripting experience
  • Team player, self motivated, able to handle high stress situations while maintaining a calm disposition
  • Great communication skills, attention for detail, and an interest in the business side of the equation of systems/scale planning
  • Eat/sleep/breathe the web, startups, and the landscape of the social web
  • Insomniac

We’re ready to offer an aggressive salary with tremendous upside by way of stock options, commensurate with your experience, your drive and your results.

Apply directly to:

net ‘dot’ startup ‘at’ googles mail service dot com

by sending us a CV/resume, and optionally, a link to your blog or Linkedin profile.

Puppet – Admin’s Best Friend

Posted by Oleksiy Kovyrin under Admin-tips, Networks

If you’ve ever worked in companies with 5-10+ servers and it was your responsibility to install new boxes, change some configuration files and install new software on many boxes you definitely know how painful this work is. Every time you need to change something on 3-5-100 boxes, you go there and make those changes. Most experienced of us used some weird scripts to perform some task on many boxes or used some stuff like dsh. Even with those tricks I’d never wish this work to anyone.

While I was working in Galt, I’ve asked our junior admin to check out puppet and try to use it on our servers. After a week of screaming he’s managed to install and configure it and said “Wow! This is what we definitely need!”. Since then he was using it and was happy :-) . But, unfortunately (for me) I’ve never tried it and when I’ve changed my job and started doing the same old (but not good) things like dsh and for+ssh I’ve started thinking: “Stop! Try puppet – maybe it is what you need!”. And what can i say? Yes! This is what every admin needs! Today we’ve deployed our next web server (we do it pretty often) and it was done with puppet – pretty painlessly and quickly.

So, what is puppet? Let me quote their site:

Puppet lets you centrally manage every important aspect of your system using a cross-platform specification language that manages all the separate elements normally aggregated in different files, like users, cron jobs, and hosts, along with obviously discrete elements like packages, services, and files.

Puppet’s simple declarative specification language provides powerful classing abilities for drawing out the similarities between hosts while allowing them to be as specific as necessary, and it handles dependency and prerequisite relationships between objects clearly and explicitly. Puppet is written entirely in Ruby.

Basically, puppet is a tool which lets you describe whatever you’d like about your server(s) and deploy your configuration (configs, packages, files, directories, cron jobs, user accounts, ssh keys, etc) to any number of servers you need. You can describe some simple parts of your system (like “apache”, “nginx”, “rails”, “useful_gems”, “admin-ssh-keys”, etc) and then describe your servers using those terms (like machine X = apache+rails+useful_gems+admin_ssh_keys). When it is done, you’ll be able to deploy all this stuff with one command!

So, if you’re struggling with deploying some changes on many servers or, even, have only a few of them, I’d recommend to try puppet – it could save you a lots of time and nerves.

P. S. When you’ll try to use it – do not give up for a few days – it is hard to change your way of thinking and start writing correct and convenient scripts using puppet’s language.

MySQL Master-Master Replication Manager 1.0 Released

Posted by Oleksiy Kovyrin under Databases, Development, My Projects, Networks

It’s been a long time since we’ve started this project and it is time to make a checkpoint. So, I’ve decided to release final 1.0 version and make 1.X branch stable while all serious development with deep architectural changes will be done 2.X branch (trunk at this moment).

Changes from previous release:

  • Perl semaphores implementation caused huge memory leaks (mmmd_mod).
  • Now we do not send any commands to hard offline hosts with dead TCP/IP stack to prevent mointoring problems for other hosts.
  • Removed legacy StartSlave method from agent code which caused problems on some Perl versions
  • Added a few fixes to prevent non-exclusive roles from moving. This caused internal status structures to be corrupted.
  • Made all mysql checks properly report errors occurring (previously they were resulting in an OK response). Thanks to Phillip Pearson.
  • Some memory leaks found in mysql checkers and as a quick fix we’ve added an idea of “Maximum Checks Before Restart” to all checkers. If you want some checker to restart after N checks, simply add “restart after N” to your checker declaration.
  • Added some more docs to the project site.

New version can be obtained here or from the project’s SVN repository.

MMM Release 1.0-pre4

Posted by Oleksiy Kovyrin under Databases, Development, My Projects, Networks

New alpha release 1.0-pre4 of the MySQL Master-Master Replication Manager. This release has lots of major fixes and I’m glad to announce first sponsored port of mmm to non-linux platform – it has been ported to Solaris 10. So, here are our changes in this version:

  • Real checks timeouts – I’ve found and fixed lots of problems in checks timeout code and now if you specified in your mmm_mon.conf, that some check should timeout in 5 sec, it would timeout correctly on all supported platforms.
  • External third-party tools using – On all supported non-linux platforms mmm will use system binaries for fping and arp_ping so porting to another platforms would be much easier.
  • Agent notifications fix – Now we don’t try to notify dead agents about cluster changes and additionally we have 10 sec timeout on notification sends to prevent monitoring from lagging on network connection timeouts.
  • Bundled fping and send_arp – We have both used third-party tools used in mmm bundled in our distribution as separate build trees (you could find and build/install them from contrib directory).
  • Flexible perl binary location – We’re using “#!/usr/bin/env perl” as shebang line in our perl scripts so you can use any perl interpreter just by placing it earlier in your PATH variable.

Notice: If you’ll try to install this version, try to run bin/sys/fping and bin/sys/send_arp on your server before installation. If you’ll notice any errors, feel free to build binaries for your platform using contrib/* sources (you’ll need gcc and libnet installed).

So, as you can see, mmm development goes forward and we’re fixing some problems trying to make this software mature. If you want to help us, you can send your comments to mmm-devel mailing list, post your bug reports to our bug tracking or sponsor any changes you need ;-)

MySQL Conference 2007: First Impressions and Findings

Posted by Oleksiy Kovyrin under Databases, Development, Networks

So, first two days of mysqlconf’07 are finished now. What can I say is that without any doubts: It worth it! If you’re working on some high traffic projects, some high-loaded database driven systems, etc, you definitely should attend such conferences – you’d never be able to get such big amounts of information from the best people in the industry as you can get here.

I’ve been attending mostly practical MySQL scale-out sessions and BOFs and I’ve got really controversial impressions. No – everybody was great, controversial thing is my own level – I never before was so sure that I know nothing at all! I see people here for which most of new things for me (especially in MySQL scaling) are pretty obvious and it is hard to keep myself from some kind of self-beating because I don’t know these things.

Interesting thing happened today – I clearly realized why YouTube guys got their billion and we or someone else didn’t – IMHO – one of the major factors was their ability to handle all these users traffic efficiently. They’ve got one of the best (imho) technical people in their command. I was really impressed by one of their guys (sorry, but I’m too bad in remembering people’s names) – one of the two founders – who participated in MySQL replication and scale out BOF as attendee. He clearly showed me that they’re able to grow almost without any limits because of their skills which is one of the major requirements in today’s fast changung world.

Another interesting thing was that I realized that my spoken English could be really good when people speak clearly and I understand their questions. I spoke with some guys about our mmm project, about maby things and there was no problems at all!

So, its going pretty late and tomorrow going to be a long day so I want to repeat – if you have any chance to get on such conferences, DO IT! If your company can’t help you, spend your own money or find some other solutions, but DO IT! It is the best thing could happen to YOU as professional!

P.S. Just remembered – I saw some women-DBAs today! Really smart girls! I never thought that pretty girl can become such great IT prefessional and now I know – I was really wrong.

P.P.S. Sorry to russian-speaking readers – I will definitely translate my last posts later – just when I’ll come back home from this crazy (but the best) event with 14hrs of load a day

SOS!

Posted by Oleksiy Kovyrin under Blog, Networks · русский

This blog turned 1 year old last month and I think all of my readers were glad to read it. I’ve never asked for help and I offered help to people who needed it. But today I’m forced to ask my readers for help because it is the first time in my long practice when I really don’t know how to solve my problem.

Read the rest of this entry »

MySQL Master-Master replication manager released

Posted by Oleksiy Kovyrin under Databases, Development, Networks · русский

So, I’s been a long time since I wrote my last post here. Lots of work and almost no new interesting technologies in my work caused such delay. But today I’m proud to announce release of really interesting project: MySQL Master-Master replication manager – set of flexible scripts for management different MySQL deployment schemes with master-master replication involved.

More information about this software could be found in detailed announce in Peter Zaitsev blog (actually this software was created by me for his company’s client) or at project page. All your questions and suggestions welcomed in MMM development group on Google Groups. If you’d like to support this software on Digg.com, you’re welcome. ;-)

Looking For Optimal Solution: Ruby On Rails and Mongrel

Posted by Oleksiy Kovyrin under Development, Networks · русский

This article is part of “Looking For Optimal Solution” series, devoted to testing various Ruby On Rails deployment schemes and doing some simple benchmarks on these schemes. General idea of testing is to find subset of most optimal RoR deployment schemes for different situations.

This small article is about Rails+Mongrel setup and its performance. List of other tested deployment schemes, description of testing methodology and, of course, all benchmark results you can find on “Ruby On Rails Benchmark Summary and Findings” page.

Read the rest of this entry »

HAProxy – The Reliable, High Performance TCP/HTTP Load Balancer

Posted by Oleksiy Kovyrin under Networks · русский

While I’ve been doing performance testing of different Ruby on Rails deployment schemes, I came across very interesting software – HAProxy.

HAProxy is a TCP/HTTP reverse proxy which is particularly suited for high
availability environments. Indeed, it can :
– route HTTP requests depending on statically assigned cookies ;
– spread the load among several servers while assuring server persistence
through the use of HTTP cookies ;
– switch to backup servers in the event a main one fails ;
– accept connections to special ports dedicated to service monitoring ;
– stop accepting connections without breaking existing ones ;
– add/modify/delete HTTP headers both ways ;
– block requests matching a particular pattern ;

It needs very little resource. Its event-driven architecture allows it to easily
handle thousands of simultaneous connections on hundreds of instances without
risking the system’s stability.

As for me, I’m really impressed by HAProxy’s performance and I will suggest to try to use it in some high-availability environments because of its performance and set of features. Take a look at this software – it is really good tool!