Oleksiy Kovyrin

Yet Another Admin's Blog

Posts Tagged ‘linux’

Found an Ideal I/O Scheduler for my MySQL boxes

Posted by Oleksiy Kovyrin under Admin-tips, Databases

Today I was doing some work on one of our database servers (each of them has 4 SAS disks in RAID10 on an Adaptec controller) and it required huge multi-thread I/O-bound read load. Basically it was a set of parallel full-scan reads from a 300Gb compressed innodb table (yes, we use innodb plugin). Looking at the iostat I saw pretty expected results: 90-100% disk utilization and lots of read operations per second. Then I decided to play around with linux I/O schedulers and try to increase disk subsystem throughput. Here are the results:

Read the rest of this entry »

32bit VS 64bit – what do you use?

Posted by Oleksiy Kovyrin under Admin-tips, Development, General

Hello my dear readers.

Today I have a question for all of you. What platforms (32bit or 64 bit) do you use for your servers with more than 4Gb RAM? I’m asking because recently we‘ve hit few really weird bugs in Linux kernels 2.6.18 to 2.6.22 and all those bugs were PAE-related. Now I’d really love to move all machines to 64-bit, but I’m in doubt because we don’t know too much about Rails stack (ruby, mongrel, haproxy) on 64-bit platforms (all our DB boxes are 64-bit of course).

So, please drop me a line if you have any experience (negative or positive) with Rails platform on 64-bit machines. I’d really appreciate your help.

Why I HATE “smart” Software: Cpanel vs Consultant

Posted by Oleksiy Kovyrin under Admin-tips, Databases

Today I was working on one small consulting task and our client asked for an upgrade from MySQL 5.0 to 5.1. It was pretty easy and task was successfully finished and reported to the customer… But few hours after my report I’ve got an email from customer with something like “WTF? Where is my 5.1?!”. I was shocked when I saw happily running 5.0 on their server w/o anything related to my 5.1 installation…

After some short investigation I’ve found out, that it was cpanel (dumb software for dumb system administrators) – it noticed, that installed mysql version (5.1) is not the same as it thought it should be (5.0), so without any warnings or notices it removed all 5.1 rpms and installed “brand new” 5.0.

Here I’d like to say GREAT THANKS to mysql team for such a great software which did not screwed up user’s data in such situation. But what idiots in cpanel development team decided, that is it appropriate and acceptable to perform such operations?! As an administrator and as a software developer I do not understand them – I just can’t understand such approach….

So, enough complaining – here is a piece of useful information for my readers: If you’re so unlucky to have cpanel installed on your server and you’d like to upgrade your mysql manually, then you can perform following operations:

1
2
3
# touch /etc/mysqlupdisable
# chattr +i /etc/mysqlupdisable
# service cpanel restart

After these small changes your cpanel will forget about mysql upgrades and you’ll be able to do what you want and not what some dumb developers decided you should do.

Useful Cacti Templates to Monitor Your Servers

Posted by Oleksiy Kovyrin under Admin-tips, Databases, Development

Recently I had one customer for consulting and aside from mysql optimization, etc they asked me for cacti installation/setup to monitor their pretty generic LAMP application. I’ve started setting up all this stuff and I’ve never thought it could be so painful… lots of different templates for the same tasks, all of them are incompatible with recent cacti releases, etc, etc… So, this post is generally a list of used templates with a fixes I’ve made to make them work on recent cacti release.

Read the rest of this entry »

How to run GUI-programs on a server without any monitor

Posted by Oleksiy Kovyrin under Admin-tips, Development

This weekend I was doing some development for one of our projects and we needed to make screenshots of a web pages (see my next posts about this task). I’ve managed to develop small piece of code which uses GtkMozEmbed component (Mozilla Gecko-based renderer for web pages) to create screenshots of any page, but there was some problem… The problem was a following: GTK+ library can’t work w/o fully-functional X server running on your machine. Obviously I didn’t want to run such software (no monitor/keyboard/mouse, dumb graphics adapter on the server, etc, etc) so I’ve tried to find some solution… And in this tiny article I’ll describe the method I’ve managed to find.

Read the rest of this entry »

How to unmount NFS share mounted with hard option

Posted by Oleksiy Kovyrin under Admin-tips

Few days ago I worked on some customer’s server and there was a problem – their nfs server went down and we were forced to change some settings on their FC4 clients to prevent shares from dieing because of kernel bug. But when we’ve changed settings in /etc/fstab there was one more step before task was completed – we need to remount this share (I mean unmount/mount). But how to perform this operation if there are some processes in D (non-interruptible sleep) waiting for dead share and prevent it from unmounting? They wait because of hard option on the share and lack of intr option and any unmount request would produce a following results:

1
2
3
streaming01:~# umount /storages/2
umount: /storages/2: device is busy
umount: /storages/2: device is busy

So, here is a list of steps you need to do to be able to remount your share.

First of all, you need to send KILL(9) signal to all you processes waiting for share. I’ve used ps axu and filtered all processes in D state. When all processes ‘killed’ (they can’t be killed actually), you’ll need to send the same signal to rpciod processes in your system. After this all your sleeping processes will die and you’ll be able to unmount your share.

That’s it – simple and really useful tip for people using NFS in their systems.

Small Tip: How to fix “There are no public key available for the following key IDs” Error in Debian

Posted by Oleksiy Kovyrin under Admin-tips

Few days ago I’ve started migration of some of my non-critical servers to Debian Etch (from Sarge). Just after first apt-get update && apt-get dist-upgrade, when apt has been upgraded, I noticed really strange (as for me) error: when I’ve tried to do “apt-get update” it worked fine, but there was annoying message like following:

1
2
3
4
5
6
7
8
# apt-get update
......
Fetched 5562B in 13s (421B/s)
Reading package lists... Done
W: There are no public key available for the following key IDs:
A70DAF536070D3A1
W: You may want to run apt-get update to correct these problems
#

UPDATE: Thanks to Kurt Fitzner we know, that:

There is already a mechanism to do this automatically:

1
$ apt-key update

This will obtain the necesary keys and import them. No need to go through gpg directly.

After not so long research I figured out, that this problem was caused by change of gpg key used by ftpmaster on Debian official repository servers. Google gave me some information and I found some fix which works fine for me:

1
2
3
4
5
6
# gpg --keyserver wwwkeys.eu.pgp.net --recv-keys XXXXXXXXXXXXXXXX
...
# apt-key add /root/.gnupg/pubring.gpg
...
# apt-get update
...

Where XXXXXXXXXXXXXXXX is your missing key (e.g A70DAF536070D3A1).

That’s it! Happy using Debian GNU/Linux!

Включение поддержки NCQ Queing на чипсетах ICH7 Chipsets под Linux

Posted by Oleksiy Kovyrin under Admin-tips · english

Недавно на нескольких из наших серверов, обслуживающих миллионы посещений в сутки, мы достигли предела производительности дисковой подсистемы. Когда я начал анализировать методы, которыми можно исправить ситуацию, я обратил внимание, что на эти сервера имеют чипсеты ICH7 и SATA-диски Seagate с поддержкой NCQ… Но при ближайшем рассмотрении я увидел, что поддержка NCQ не включена на этих серверах с IDE-драйвером ata_piix. Итак, я решил было бы отлично включить поддержку NCQ и взглянуть, как это повлияет на производительность (было очевидно, что она должна увеличиться)…

Read the rest of this entry »

Поиск Неисправностей В UNIX-Системах При Помощи lsof

Posted by Oleksiy Kovyrin under Admin-tips · english

Одной из основных задач любого администратора является создание стабильного окружения для выполнения определенных базнес-процессов. Важным элементом этого процесса является поиск неисправностей или каких-либо проблем в системе. В UNIX существует множество утилит для поиска и устранения проблем в системе, но, на мой взгляд, одна из самых полезных таких утилит – lsof – является одним из инструментов администратора, упоминаемых реже всего. Lsof выводит информацию об открытых файлах и открывших их процессах. но это слишком краткое описание.

Большинство людей не знают или забывают, что в UNIX (практически) все является файлом. ОС делает устройства доступнми для приложений при помощи служебных файлов в каталоге /dev. Информация о ядре, системе, памяти, устройствах и т. д. – все это есть в файлах к каталоге /proc. TCP/UDP сокеты часто представляются в программировании как файлы. Даже каталоги – это просто файлы, содержащие списки других файлов.

Lsof работает анализируя структуры данных ядра ОС и представляет различную информацию, относящуюся к файлам, каналам межпроцессного взаимодействия, сокетам и многому другому.

Read the rest of this entry »