Archive for the ‘Networks’ Category

Enabling IPv6 Support in nginx

Posted by Scoundrel under Admin-tips, Networks

This is going to be a really short post, but for someone it could save an hour of life.

So, you’ve nothing to do and you’ve decided to play around with IPv6 or maybe you’re happened to be an administrator of a web service that needs to support IPv6 connectivity and you need to make your nginx server work nicely with this protocol.

First thing you need to do is to enable IPv6 in nginx by recompiling it with --with-ipv6 configure option and reinstalling it. If you use some pre-built package, check if your nginx already has this key enabled by running nginx -V.

Read the rest of this entry »

Using SSH tunnel connection as a SOCKS5 proxy

Posted by Scoundrel under Admin-tips, Networks

Month ago I was on a vacation and as usual even though our hotel provided us with an internet connection on a pretty decent speeds, I wasn’t able to work there because they’ve banned all tcp ports but some major ones (like 80, 21, etc) and I needed to be able to use ssh, mysql, IMs and other non-web software.

After a short research I’ve found a pretty simple to set up and easy to use approach to such a connection problems I’d like to describe here.

Read the rest of this entry »

Lighttpd Book from Packt – Great Thanksgiving Present

Posted by Scoundrel under Admin-tips, Networks

Many people know me as a nginx web server evangelist. But as (IMHO) any professional I think that it is really rewarding to know as much as possible about all the tools available on the market so every time you need to make a decision on some technical issue, you’d consider all pros and cons based on my own knowledge.

This is why when I received an email from Packt company asking if I’d like to read and review their book on Lighttpd I decided to give it a shot (I usually do not review any books because I do not always have enough time to read a book thoroughly to be able to write a review). So, here are my impressions from this book.

Read the rest of this entry »

Advanced Squid Caching for Rails Applications: Preface

Posted by Scoundrel under Databases, Development, My Projects, Networks

Since the day one when I joined Scribd, I was thinking about the fact that 90+% of our traffic is going to the document view pages, which is a single action in our documents controller. I was wondering how could we improve this action responsiveness and make our users happier.

Few times I was creating a git branches and hacking this action trying to implement some sort of page-level caching to make things faster. But all the time results weren’t as good as I’d like them to be. So, branches were sitting there and waiting for a better idea.
Read the rest of this entry »

Bounces-handler Released

Posted by Scoundrel under Development, My Projects, Networks

Today I’ve managed to finish initial version of our bounces-handler package we use for mailing-related stuff in Scribd.

Bounces-handler package is a simple set of scripts to automatically process email bounces and ISP‘s feedback loops emails, maintain your mailing blacklists and a Rails plugin to use those blacklists in your RoR applications.

This piece of software has been developed as a part of more global work on mailing quality improvement in Scribd.com, but it was one of the most critical steps after setting up reverse DNS records, DKIM and SPF.

The package itself consists of two parts:

  • Perl scripts to process incoming email:
    • bounces processor — could be assigned to process all your bounce emails
    • feedback loops messages processor — more specific for Scribd, but still – could be modified for your needs (will be released soon).
  • Rails plugin to work with mailing blacklists

For more information, please check our README file. If you have any questions, comments or suggestions, please leave them here as a comments and I’ll try to reply as soon as possible.

Fwd: Scorching hot Startup Needs Scalability Sorcerer and Optimization Freak

Posted by Scoundrel under Databases, Development, Networks

Question: Do you think you have what it takes to take a service from a few hundred thousand users to tens of millions of users in 1 year flat? If you do read on and perhaps become the next beloved scalability rockstar of our age.

We are looking for a data charmer. A mysql magician. A code hack. A funny man. A mad man. A passionate man. Or perhaps a woman who does all these things and more.

Here’s what you gotta do:

  • Pro-active and reactive performance analysis, monitoring and general database plumbing of all leaky issues.
  • Work with others on the team to help maintain/improve and support the infrastructure for a high traffic, high growth site
  • Optimize and tune the database day to day
  • Algorithmic bent. Develop algos to quicken search times, response times, find shortest paths between various connections on site.
  • Have solid low level networking/protocol/computer security skills
  • Log everything. Usage stats, search stats, user behaviour stats. Draw conclusions. Constantly refine and tinker.
  • Help with periodic large storage migrations
  • Work intimately with operations, development, and strategy team to ensure smooth deployments of new iterations, high availability of database services.
  • Understand capacity planning. Always thinking 10 steps ahead. (Whether it means looking at distributed systems services, cloud computing options, evaluating HA models used in other industries etc)
  • Have a pulse on the state of the web, social media, social networking, different scalability architectures, benefits/negatives of each.
  • Interest in high concurrency, distributed systems architectures.
  • General low level hacking/scripting/optimizations in perl/python.
  • Evaluate changing conditions in the archi
  • Think creatively. No dogmatists.

Ideal skillset:

  • BS in Comp Sci or equivalent
  • 5+ years experiene with Linux/Unix systems
  • 3+ years with MySQL in production environment
  • Knowledge and experience with partitioned architectures and a database sharding techniques
  • Capacity planning/high growth planning/emergency planning experience
  • Passion, bordering on paranoia, for hunting bottlenecks, and optimizing IO operations
  • Experience with MySQL replication
  • Deep experience with MySQL internals
  • Experience with performance analysis tools, storage engines, backup methodologies for MySQL
  • Great perl/shell scripting experience
  • Team player, self motivated, able to handle high stress situations while maintaining a calm disposition
  • Great communication skills, attention for detail, and an interest in the business side of the equation of systems/scale planning
  • Eat/sleep/breathe the web, startups, and the landscape of the social web
  • Insomniac

We’re ready to offer an aggressive salary with tremendous upside by way of stock options, commensurate with your experience, your drive and your results.

Apply directly to:

net ‘dot’ startup ‘at’ googles mail service dot com

by sending us a CV/resume, and optionally, a link to your blog or Linkedin profile.

Puppet – Admin’s Best Friend

Posted by Scoundrel under Admin-tips, Networks

If you’ve ever worked in companies with 5-10+ servers and it was your responsibility to install new boxes, change some configuration files and install new software on many boxes you definitely know how painful this work is. Every time you need to change something on 3-5-100 boxes, you go there and make those changes. Most experienced of us used some weird scripts to perform some task on many boxes or used some stuff like dsh. Even with those tricks I’d never wish this work to anyone.

While I was working in Galt, I’ve asked our junior admin to check out puppet and try to use it on our servers. After a week of screaming he’s managed to install and configure it and said “Wow! This is what we definitely need!”. Since then he was using it and was happy :-) . But, unfortunately (for me) I’ve never tried it and when I’ve changed my job and started doing the same old (but not good) things like dsh and for+ssh I’ve started thinking: “Stop! Try puppet – maybe it is what you need!”. And what can i say? Yes! This is what every admin needs! Today we’ve deployed our next web server (we do it pretty often) and it was done with puppet – pretty painlessly and quickly.

So, what is puppet? Let me quote their site:

Puppet lets you centrally manage every important aspect of your system using a cross-platform specification language that manages all the separate elements normally aggregated in different files, like users, cron jobs, and hosts, along with obviously discrete elements like packages, services, and files.

Puppet’s simple declarative specification language provides powerful classing abilities for drawing out the similarities between hosts while allowing them to be as specific as necessary, and it handles dependency and prerequisite relationships between objects clearly and explicitly. Puppet is written entirely in Ruby.

Basically, puppet is a tool which lets you describe whatever you’d like about your server(s) and deploy your configuration (configs, packages, files, directories, cron jobs, user accounts, ssh keys, etc) to any number of servers you need. You can describe some simple parts of your system (like “apache”, “nginx”, “rails”, “useful_gems”, “admin-ssh-keys”, etc) and then describe your servers using those terms (like machine X = apache+rails+useful_gems+admin_ssh_keys). When it is done, you’ll be able to deploy all this stuff with one command!

So, if you’re struggling with deploying some changes on many servers or, even, have only a few of them, I’d recommend to try puppet – it could save you a lots of time and nerves.

P. S. When you’ll try to use it – do not give up for a few days – it is hard to change your way of thinking and start writing correct and convenient scripts using puppet’s language.

Small Tip: How to set up two interface Xen machine

Posted by Scoundrel under Admin-tips, Networks

This will be one of those posts I’d like to publish primarily to be able to coma back later and check it out instead of reading docs again :-)

So, we have a server with two (or more) network interfaces are we need to be able to use more than one interface in our VDS machines. How do we set it up?

Read the rest of this entry »

MySQL Master-Master Replication Manager 1.0 Released

Posted by Scoundrel under Databases, Development, My Projects, Networks

It’s been a long time since we’ve started this project and it is time to make a checkpoint. So, I’ve decided to release final 1.0 version and make 1.X branch stable while all serious development with deep architectural changes will be done 2.X branch (trunk at this moment).

Changes from previous release:

  • Perl semaphores implementation caused huge memory leaks (mmmd_mod).
  • Now we do not send any commands to hard offline hosts with dead TCP/IP stack to prevent mointoring problems for other hosts.
  • Removed legacy StartSlave method from agent code which caused problems on some Perl versions
  • Added a few fixes to prevent non-exclusive roles from moving. This caused internal status structures to be corrupted.
  • Made all mysql checks properly report errors occurring (previously they were resulting in an OK response). Thanks to Phillip Pearson.
  • Some memory leaks found in mysql checkers and as a quick fix we’ve added an idea of “Maximum Checks Before Restart” to all checkers. If you want some checker to restart after N checks, simply add “restart after N” to your checker declaration.
  • Added some more docs to the project site.

New version can be obtained here or from the project’s SVN repository.