Today I was doing some work on one of our database servers (each of them has 4 SAS disks in RAID10 on an Adaptec controller) and it required huge multi-thread I/O-bound read load. Basically it was a set of parallel full-scan reads from a 300Gb compressed innodb table (yes, we use innodb plugin). Looking at the iostat I saw pretty expected results: 90-100% disk utilization and lots of read operations per second. Then I decided to play around with linux I/O schedulers and try to increase disk subsystem throughput. Here are the results:
Read the rest of this entry »
Hello my dear readers.
Today I have a question for all of you. What platforms (32bit or 64 bit) do you use for your servers with more than 4Gb RAM? I’m asking because recently we‘ve hit few really weird bugs in Linux kernels 2.6.18 to 2.6.22 and all those bugs were PAE-related. Now I’d really love to move all machines to 64-bit, but I’m in doubt because we don’t know too much about Rails stack (ruby, mongrel, haproxy) on 64-bit platforms (all our DB boxes are 64-bit of course).
So, please drop me a line if you have any experience (negative or positive) with Rails platform on 64-bit machines. I’d really appreciate your help.
This will be one of those posts I’d like to publish primarily to be able to coma back later and check it out instead of reading docs again 
So, we have a server with two (or more) network interfaces are we need to be able to use more than one interface in our VDS machines. How do we set it up?
Read the rest of this entry »
Today I was working on one small consulting task and our client asked for an upgrade from MySQL 5.0 to 5.1. It was pretty easy and task was successfully finished and reported to the customer… But few hours after my report I’ve got an email from customer with something like “WTF? Where is my 5.1?!”. I was shocked when I saw happily running 5.0 on their server w/o anything related to my 5.1 installation…
After some short investigation I’ve found out, that it was cpanel (dumb software for dumb system administrators) – it noticed, that installed mysql version (5.1) is not the same as it thought it should be (5.0), so without any warnings or notices it removed all 5.1 rpms and installed “brand new” 5.0.
Here I’d like to say GREAT THANKS to mysql team for such a great software which did not screwed up user’s data in such situation. But what idiots in cpanel development team decided, that is it appropriate and acceptable to perform such operations?! As an administrator and as a software developer I do not understand them – I just can’t understand such approach….
So, enough complaining – here is a piece of useful information for my readers: If you’re so unlucky to have cpanel installed on your server and you’d like to upgrade your mysql manually, then you can perform following operations:
1
2
3
| # touch /etc/mysqlupdisable
# chattr +i /etc/mysqlupdisable
# service cpanel restart |
After these small changes your cpanel will forget about mysql upgrades and you’ll be able to do what you want and not what some dumb developers decided you should do.
Recently I had one customer for consulting and aside from mysql optimization, etc they asked me for cacti installation/setup to monitor their pretty generic LAMP application. I’ve started setting up all this stuff and I’ve never thought it could be so painful… lots of different templates for the same tasks, all of them are incompatible with recent cacti releases, etc, etc… So, this post is generally a list of used templates with a fixes I’ve made to make them work on recent cacti release.
Read the rest of this entry »
This weekend I was doing some development for one of our projects and we needed to make screenshots of a web pages (see my next posts about this task). I’ve managed to develop small piece of code which uses GtkMozEmbed component (Mozilla Gecko-based renderer for web pages) to create screenshots of any page, but there was some problem… The problem was a following: GTK+ library can’t work w/o fully-functional X server running on your machine. Obviously I didn’t want to run such software (no monitor/keyboard/mouse, dumb graphics adapter on the server, etc, etc) so I’ve tried to find some solution… And in this tiny article I’ll describe the method I’ve managed to find.
Read the rest of this entry »
Few days ago I worked on some customer’s server and there was a problem – their nfs server went down and we were forced to change some settings on their FC4 clients to prevent shares from dieing because of kernel bug. But when we’ve changed settings in /etc/fstab there was one more step before task was completed – we need to remount this share (I mean unmount/mount). But how to perform this operation if there are some processes in D (non-interruptible sleep) waiting for dead share and prevent it from unmounting? They wait because of hard option on the share and lack of intr option and any unmount request would produce a following results:
1
2
3
| streaming01:~# umount /storages/2
umount: /storages/2: device is busy
umount: /storages/2: device is busy |
So, here is a list of steps you need to do to be able to remount your share.
First of all, you need to send KILL(9) signal to all you processes waiting for share. I’ve used ps axu and filtered all processes in D state. When all processes ‘killed’ (they can’t be killed actually), you’ll need to send the same signal to rpciod processes in your system. After this all your sleeping processes will die and you’ll be able to unmount your share.
That’s it – simple and really useful tip for people using NFS in their systems.
Few days ago I’ve started migration of some of my non-critical servers to Debian Etch (from Sarge). Just after first apt-get update && apt-get dist-upgrade, when apt has been upgraded, I noticed really strange (as for me) error: when I’ve tried to do “apt-get update” it worked fine, but there was annoying message like following:
1
2
3
4
5
6
7
8
| # apt-get update
......
Fetched 5562B in 13s (421B/s)
Reading package lists... Done
W: There are no public key available for the following key IDs:
A70DAF536070D3A1
W: You may want to run apt-get update to correct these problems
# |
UPDATE: Thanks to Kurt Fitzner we know, that:
There is already a mechanism to do this automatically:
This will obtain the necesary keys and import them. No need to go through gpg directly.
After not so long research I figured out, that this problem was caused by change of gpg key used by ftpmaster on Debian official repository servers. Google gave me some information and I found some fix which works fine for me:
1
2
3
4
5
6
| # gpg --keyserver wwwkeys.eu.pgp.net --recv-keys XXXXXXXXXXXXXXXX
...
# apt-key add /root/.gnupg/pubring.gpg
...
# apt-get update
... |
Where XXXXXXXXXXXXXXXX is your missing key (e.g A70DAF536070D3A1).
That’s it! Happy using Debian GNU/Linux!
Recently we’ve got critical load on one of our new free web hosting servers that handles millions of hits per day. When I’ve tried to analyze how to get more performance from that servers, I noted that these servers has ICH7 chipsets and Seagate SATA drives with NCQ support… But in closer view I noticed, that NCQ support is not enabled on our servers with ata_piix IDE driver. So, I decided that it would be great to enable NCQ and take a look what will happen with performance (it was obvious, that performance should increase)…
Read the rest of this entry »
One of the main tasks of any administrator is to create stable environment for different sorts of businesses. Big part of this task is troubleshooting. There are many different tools in UNIX for system monitoring, but, at my mind, one of the most useful tools is lsof- one of the least-talked-about tools in a UNIX sysadmin’s toolkit. Lsof lists information about files opened by processes. But that’s really an understatement.
Most people forget that, in UNIX, (almost) everything is a file. The OS makes hardware available to applications by way of files in /dev. Kernel, system, memory, device etc. information in made available inside files in /proc. TCP/UDP sockets are sometimes represented internally as files. Even directories are really just files containing other filenames.
Lsof works by examining kernel data-structures and provides a variety of information related to files, pipes, sockets and more.
Read the rest of this entry »
