[Comm] Mysql crash

[Alexey Morsov <samurai@ricom.ru>]

[-- Attachment #1: Type: text/plain, Size: 1307 bytes --]

Привет,
Сегодня с утра прихожу а мой mysqld лежит к верху пузом.

Хотелось бы выяснить в чем проблема, но знаний в этой области 
маловато, поэтому обращаюсь к общественности - что могло этому 
способствовать?

Предыстория - mysqld крутиться уже минимум полгода - никогда 
такого не было. База используеться восновном для хранения 
статистики пользователей интернета. Соответственно мой скрипт 
сохраняет в нее определенные данные по крону раз в минуту, раз в 
день и еще раз в день (в другое время - результаты обработки 
логов сквида).
Единственное изменение сделанное вчера в системе - установил и 
запустил zope.
Машинка не шибко крутая - Cel300/256mb/8gb - память тестировалась 
перед установкой memtest86 - проблем найдено не было.

Самая большая таблица в mysql (по объему она процентов на 90% от 
всей базы) содержит чуть меньше 93000 записей (count(*)).

Прилагаю логи messages и mysqld.log - в messages я оставил только 
то где есть проблемы (остальное сплошные arpwatch и общие 
сообщения). Там ядрышко кричит что out of memory - но вроде как 
раньше хватало, а база с тех пор не сильно то выросла.

Буду благодарен за помощь.

-- 
Всего наилучшего,
Системный Администратор ЗАО "ИК "РИКОМ-ТРАСТ"
Алексей Морсов
ICQ: 196766290
Jabber: Samurai@jabber.ru
http://www.ricom.ru
http://www.fondmarket.ru

[-- Attachment #2: mysqld.log --]
[-- Type: text/plain, Size: 7782 bytes --]

040511 12:51:31  mysqld ended

040511 12:51:39  mysqld started
Cannot initialize InnoDB as 'innodb_data_file_path' is not set.
If you do not want to use transactional InnoDB tables, add a line
skip-innodb
to the [mysqld] section of init parameters in your my.cnf
or my.ini. If you want to use InnoDB tables, add to the [mysqld]
section, for example,
innodb_data_file_path = ibdata1:10M:autoextend
But to get good performance you should adjust for your hardware
the InnoDB startup options listed in section 2 at
http://www.innodb.com/ibman.html
/usr/libexec/mysqld: ready for connections
040511 12:52:05  Warning: Found 167 of 0 rows when repairing './traffic/history'
040513 11:13:13  /usr/libexec/mysqld: Normal shutdown

040513 11:13:14  /usr/libexec/mysqld: Shutdown Complete

040513 11:13:14  mysqld ended

040513 11:15:34  mysqld started
Cannot initialize InnoDB as 'innodb_data_file_path' is not set.
If you do not want to use transactional InnoDB tables, add a line
skip-innodb
to the [mysqld] section of init parameters in your my.cnf
or my.ini. If you want to use InnoDB tables, add to the [mysqld]
section, for example,
innodb_data_file_path = ibdata1:10M:autoextend
But to get good performance you should adjust for your hardware
the InnoDB startup options listed in section 2 at
http://www.innodb.com/ibman.html
/usr/libexec/mysqld: ready for connections
040521 14:33:57  /usr/libexec/mysqld: Normal shutdown

040521 14:33:58  /usr/libexec/mysqld: Shutdown Complete

040521 14:33:58  mysqld started
040521 14:33:58  mysqld ended

Cannot initialize InnoDB as 'innodb_data_file_path' is not set.
If you do not want to use transactional InnoDB tables, add a line
skip-innodb
to the [mysqld] section of init parameters in your my.cnf
or my.ini. If you want to use InnoDB tables, add to the [mysqld]
section, for example,
innodb_data_file_path = ibdata1:10M:autoextend
But to get good performance you should adjust for your hardware
the InnoDB startup options listed in section 2 at
http://www.innodb.com/ibman.html
/usr/libexec/mysqld: ready for connections
040521 14:34:49  /usr/libexec/mysqld: Normal shutdown

040521 14:34:49  /usr/libexec/mysqld: Shutdown Complete

040521 14:34:49  mysqld ended

040521 14:35:01  mysqld started
Cannot initialize InnoDB as 'innodb_data_file_path' is not set.
If you do not want to use transactional InnoDB tables, add a line
skip-innodb
to the [mysqld] section of init parameters in your my.cnf
or my.ini. If you want to use InnoDB tables, add to the [mysqld]
section, for example,
innodb_data_file_path = ibdata1:10M:autoextend
But to get good performance you should adjust for your hardware
the InnoDB startup options listed in section 2 at
http://www.innodb.com/ibman.html
/usr/libexec/mysqld: ready for connections
040524 16:25:27  /usr/libexec/mysqld: Normal shutdown

040524 16:25:28  /usr/libexec/mysqld: Shutdown Complete

040524 16:25:29  mysqld ended

040524 16:25:29  mysqld started
Cannot initialize InnoDB as 'innodb_data_file_path' is not set.
If you do not want to use transactional InnoDB tables, add a line
skip-innodb
to the [mysqld] section of init parameters in your my.cnf
or my.ini. If you want to use InnoDB tables, add to the [mysqld]
section, for example,
innodb_data_file_path = ibdata1:10M:autoextend
But to get good performance you should adjust for your hardware
the InnoDB startup options listed in section 2 at
http://www.innodb.com/ibman.html
/usr/libexec/mysqld: ready for connections
040610 10:53:09  /usr/libexec/mysqld: Normal shutdown

040610 10:53:09  /usr/libexec/mysqld: Shutdown Complete

040610 10:53:10  mysqld ended

040610 10:55:22  mysqld started
Cannot initialize InnoDB as 'innodb_data_file_path' is not set.
If you do not want to use transactional InnoDB tables, add a line
skip-innodb
to the [mysqld] section of init parameters in your my.cnf
or my.ini. If you want to use InnoDB tables, add to the [mysqld]
section, for example,
innodb_data_file_path = ibdata1:10M:autoextend
But to get good performance you should adjust for your hardware
the InnoDB startup options listed in section 2 at
http://www.innodb.com/ibman.html
/usr/libexec/mysqld: ready for connections
040714 18:54:53  /usr/libexec/mysqld: Normal shutdown

040714 18:54:55  /usr/libexec/mysqld: Shutdown Complete

040714 18:54:56  mysqld ended

040714 18:57:14  mysqld started
Cannot initialize InnoDB as 'innodb_data_file_path' is not set.
If you do not want to use transactional InnoDB tables, add a line
skip-innodb
to the [mysqld] section of init parameters in your my.cnf
or my.ini. If you want to use InnoDB tables, add to the [mysqld]
section, for example,
innodb_data_file_path = ibdata1:10M:autoextend
But to get good performance you should adjust for your hardware
the InnoDB startup options listed in section 2 at
http://www.innodb.com/ibman.html
/usr/libexec/mysqld: ready for connections

Number of processes running now: 0
040824 00:06:22  mysqld restarted
Cannot initialize InnoDB as 'innodb_data_file_path' is not set.
If you do not want to use transactional InnoDB tables, add a line
skip-innodb
to the [mysqld] section of init parameters in your my.cnf
or my.ini. If you want to use InnoDB tables, add to the [mysqld]
section, for example,
innodb_data_file_path = ibdata1:10M:autoextend
But to get good performance you should adjust for your hardware
the InnoDB startup options listed in section 2 at
http://www.innodb.com/ibman.html
/usr/libexec/mysqld: ready for connections

Number of processes running now: 0
040824 00:07:23  mysqld restarted
Cannot initialize InnoDB as 'innodb_data_file_path' is not set.
If you do not want to use transactional InnoDB tables, add a line
skip-innodb
to the [mysqld] section of init parameters in your my.cnf
or my.ini. If you want to use InnoDB tables, add to the [mysqld]
section, for example,
innodb_data_file_path = ibdata1:10M:autoextend
But to get good performance you should adjust for your hardware
the InnoDB startup options listed in section 2 at
http://www.innodb.com/ibman.html
/usr/libexec/mysqld: ready for connections

Number of processes running now: 0
040824 00:07:41  mysqld restarted
Cannot initialize InnoDB as 'innodb_data_file_path' is not set.
If you do not want to use transactional InnoDB tables, add a line
skip-innodb
to the [mysqld] section of init parameters in your my.cnf
or my.ini. If you want to use InnoDB tables, add to the [mysqld]
section, for example,
innodb_data_file_path = ibdata1:10M:autoextend
But to get good performance you should adjust for your hardware
the InnoDB startup options listed in section 2 at
http://www.innodb.com/ibman.html
040824 00:07:48  mysqld ended

040824 11:33:42  mysqld started
Cannot initialize InnoDB as 'innodb_data_file_path' is not set.
If you do not want to use transactional InnoDB tables, add a line
skip-innodb
to the [mysqld] section of init parameters in your my.cnf
or my.ini. If you want to use InnoDB tables, add to the [mysqld]
section, for example,
innodb_data_file_path = ibdata1:10M:autoextend
But to get good performance you should adjust for your hardware
the InnoDB startup options listed in section 2 at
http://www.innodb.com/ibman.html
/usr/libexec/mysqld: ready for connections

Number of processes running now: 0
040824 11:45:46  mysqld restarted
Cannot initialize InnoDB as 'innodb_data_file_path' is not set.
If you do not want to use transactional InnoDB tables, add a line
skip-innodb
to the [mysqld] section of init parameters in your my.cnf
or my.ini. If you want to use InnoDB tables, add to the [mysqld]
section, for example,
innodb_data_file_path = ibdata1:10M:autoextend
But to get good performance you should adjust for your hardware
the InnoDB startup options listed in section 2 at
http://www.innodb.com/ibman.html
/usr/libexec/mysqld: ready for connections

[-- Attachment #3: messages --]
[-- Type: text/plain, Size: 2504 bytes --]

Aug 24 00:05:45 www arpwatch: bogon 169.254.100.100 0:d:54:ae:c7:40
Aug 24 00:06:05 www kernel: Out of Memory: Killed process 1612 (mysqld).
Aug 24 00:06:05 www kernel: Out of Memory: Killed process 1622 (mysqld).
Aug 24 00:06:05 www kernel: Out of Memory: Killed process 1625 (mysqld).
Aug 24 00:06:05 www kernel: Out of Memory: Killed process 8094 (mysqld).
Aug 24 00:06:15 www arpwatch: bogon 169.254.100.100 0:d:54:ae:c7:40
Aug 24 00:06:28 www kernel: Out of Memory: Killed process 31861 (python2.3).
Aug 24 00:06:28 www kernel: Out of Memory: Killed process 31862 (python2.3).
Aug 24 00:06:28 www kernel: Out of Memory: Killed process 31863 (python2.3).
Aug 24 00:06:28 www kernel: Out of Memory: Killed process 31864 (python2.3).
Aug 24 00:06:28 www kernel: Out of Memory: Killed process 31865 (python2.3).
Aug 24 00:06:45 www arpwatch: bogon 169.254.100.100 0:d:54:ae:c7:40
Aug 24 00:07:15 www arpwatch: bogon 169.254.100.100 0:d:54:ae:c7:40
Aug 24 00:07:22 www kernel: Out of Memory: Killed process 8219 (mysqld).
Aug 24 00:07:22 www kernel: Out of Memory: Killed process 8221 (mysqld).
Aug 24 00:07:22 www kernel: Out of Memory: Killed process 8222 (mysqld).
Aug 24 00:07:40 www kernel: Out of Memory: Killed process 8255 (mysqld).
Aug 24 00:07:40 www kernel: Out of Memory: Killed process 8256 (mysqld).
Aug 24 00:07:40 www kernel: Out of Memory: Killed process 8257 (mysqld).
Aug 24 00:07:45 www arpwatch: bogon 169.254.100.100 0:d:54:ae:c7:40
Aug 24 00:07:48 www kernel: Out of Memory: Killed process 8268 (mysqld).
Aug 24 00:08:02 www kernel: Out of Memory: Killed process 8085 (trafficsql).
Aug 24 00:08:15 www arpwatch: bogon 169.254.100.100 0:d:54:ae:c7:40


Aug 24 04:02:49 www arpwatch: bogon 169.254.100.100 0:d:54:ae:c7:40
Aug 24 04:03:12 www logrotate: ALERT exited abnormally with [1]
Aug 24 04:03:19 www arpwatch: bogon 169.254.100.100 0:d:54:ae:c7:40

Aug 24 11:45:05 www last message repeated 3 times
Aug 24 11:45:08 www kernel: Out of Memory: Killed process 10639 (mysqld).
Aug 24 11:45:09 www kernel: Out of Memory: Killed process 10640 (mysqld).
Aug 24 11:45:09 www kernel: Out of Memory: Killed process 10641 (mysqld).
Aug 24 11:45:09 www kernel: Out of Memory: Killed process 10836 (mysqld).
Aug 24 11:45:35 www arpwatch: bogon 169.254.100.100 0:d:54:ae:c7:40
Aug 24 11:46:05 www arpwatch: bogon 169.254.100.100 0:d:54:ae:c7:40
Aug 24 11:46:15 www kernel: Out of Memory: Killed process 10834 (trafficsql).
Aug 24 11:46:35 www arpwatch: bogon 169.254.100.100 0:d:54:ae:c7:40