From: Slava Dubrovskiy <slava@tangramltd.com>
To: community@lists.altlinux.org
Subject: [Comm] глючит ядро ovz-el
Date: Mon, 04 Jul 2011 15:20:25 -0400
Message-ID: <4E121279.4040608@tangramltd.com> (raw)
[-- Attachment #1: Type: text/plain, Size: 11892 bytes --]
Здравствуйте уважаемая редакция.
Серверов с ядром ovz-el на серверах становится все больше и больше. И
багов соответственно.
Есть один сервер на котором периодически (1-2 раза в неделю) происходит
ребут в результате kernel panic.
Перед ребутом LA уходит в космос - выше 200.
Ядро 2.6.32-ovz-el-alt24, бранч t6.
Стоит 2 SATA диска в raid1 на mdadm на котором установлена система и все
служебные VPS
И 2 SAS диска из которых собран raid1 и с него работает одна рабочая VPS.
Диски по smart живые, ошибок нет.
В логе вот такое перед ребутом вижу:
Jul 4 21:05:51 ua83 kernel: [293760.725062] INFO: task flush-253:1:1294
blocked for more than 120
seconds.
Jul 4 21:05:59 ua83 kernel: [293760.725069] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this
message.
Jul 4 21:06:26 ua83 kernel: [293760.725074] flush-253:1 D
ffff8801dcb82c90 0 1294 2
0x00000000
Jul 4 21:07:33 ua83 kernel: [293760.725083] ffff8801d5a0ba10
0000000000000046 0000000000000000
ffffffffa02cc12c
Jul 4 21:07:33 ua83 kernel: [293760.725091] ffff88013696e5a8
ffffffff811b3e70 ffffffffa00d3c40
00000001117b8ec4
Jul 4 21:07:33 ua83 kernel: [293760.725098] ffff8801dcb83270
ffff8801d5a0bfd8 000000000000f7c8
ffff8801dcb83270
Jul 4 21:07:33 ua83 kernel: [293760.725106] Call
Trace:
Jul 4 21:07:33 ua83 kernel: [293760.725132] [<ffffffffa02cc12c>] ?
dm_table_unplug_all+0x5c/0xd0
[dm_mod]
Jul 4 21:07:33 ua83 kernel: [293760.725155] [<ffffffff811b3e70>] ?
end_buffer_async_write+0x0/0x190
Jul 4 21:07:33 ua83 kernel: [293760.725185] [<ffffffffa00d3c40>] ?
ext4_bh_delay_or_unwritten+0x0/0x30
[ext4]
Jul 4 21:07:33 ua83 kernel: [293760.725195] [<ffffffff813ff403>]
io_schedule+0xa3/0x110
Jul 4 21:07:33 ua83 kernel: [293760.725202] [<ffffffff8111a6e0>] ?
sync_page+0x0/0x50
Jul 4 21:07:45 ua83 kernel: [293760.725207] [<ffffffff8111a71d>]
sync_page+0x3d/0x50
Jul 4 21:08:27 ua83 kernel: [293760.725212] [<ffffffff813ffc8f>]
__wait_on_bit+0x5f/0x90
Jul 4 21:08:29 ua83 kernel: [293760.725218] [<ffffffff8111a8d3>]
wait_on_page_bit+0x73/0x80
Jul 4 21:08:38 ua83 kernel: [293760.725225] [<ffffffff81091a30>] ?
wake_bit_function+0x0/0x50
Jul 4 21:08:38 ua83 kernel: [293760.725234] [<ffffffff81131355>] ?
pagevec_lookup_tag+0x25/0x40
Jul 4 21:08:38 ua83 kernel: [293760.725239] [<ffffffff8111ac9b>]
wait_on_page_writeback_range+0xfb/0x190
Jul 4 21:08:38 ua83 kernel: [293760.725245] [<ffffffff8111ad5f>]
filemap_fdatawait+0x2f/0x40
Jul 4 21:08:38 ua83 kernel: [293760.725252] [<ffffffff811ab418>]
writeback_single_inode+0x238/0x2e0
Jul 4 21:08:38 ua83 kernel: [293760.725258] [<ffffffff811ab706>]
writeback_sb_inodes+0xf6/0x1b0
Jul 4 21:08:38 ua83 kernel: [293760.725263] [<ffffffff811abae0>]
wb_writeback+0x170/0x400
Jul 4 21:08:38 ua83 kernel: [293760.725268] [<ffffffff813feb4c>] ?
thread_return+0x4e/0x862
Jul 4 21:08:38 ua83 kernel: [293760.725276] [<ffffffff8107c7ba>] ?
del_timer_sync+0x2a/0x40
Jul 4 21:08:38 ua83 kernel: [293760.725281] [<ffffffff811abe35>]
wb_do_writeback+0xc5/0x250
Jul 4 21:08:38 ua83 kernel: [293760.725286] [<ffffffff8107bd20>] ?
process_timeout+0x0/0x10
Jul 4 21:08:38 ua83 kernel: [293760.725292] [<ffffffff811ac027>]
bdi_writeback_task+0x67/0x1c0
Jul 4 21:08:38 ua83 kernel: [293760.725297] [<ffffffff810918c7>] ?
bit_waitqueue+0x17/0xc0
Jul 4 21:08:38 ua83 kernel: [293760.725305] [<ffffffff81141556>]
bdi_start_fn+0x86/0x100
Jul 4 21:08:38 ua83 kernel: [293760.725310] [<ffffffff811414d0>] ?
bdi_start_fn+0x0/0x100
Jul 4 21:08:38 ua83 kernel: [293760.725315] [<ffffffff81091680>]
kthread+0xc0/0xe0
Jul 4 21:08:38 ua83 kernel: [293760.725321] [<ffffffff8100c2ca>]
child_rip+0xa/0x20
Jul 4 21:08:38 ua83 kernel: [293760.725326] [<ffffffff810915c0>] ?
kthread+0x0/0xe0
Jul 4 21:08:38 ua83 kernel: [293760.725331] [<ffffffff8100c2c0>] ?
child_rip+0x0/0x20
Jul 4 21:08:38 ua83 kernel: [293760.725336] INFO: task jbd2/dm-1-8:1306
blocked for more than 120
seconds.
Jul 4 21:09:58 ua83 kernel: [293760.725340] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this
message.
Jul 4 21:10:37 ua83 kernel: [293760.725344] jbd2/dm-1-8 D
ffff8801da9ea640 0 1306 2
0x00000000
Jul 4 21:10:38 ua83 kernel: [293760.725351] ffff8801d4951b00
0000000000000046 ffff8801d4951ac0
ffffffffa02cc12c
Jul 4 21:10:38 ua83 kernel: [293760.725358] ffff880119e48808
ffff880119eddb40 ffff8801d4951ab0
ffffffff8109c569
Jul 4 21:10:38 ua83 kernel: [293760.725365] ffff8801da9eac20
ffff8801d4951fd8 000000000000f7c8
ffff8801da9eac20
Jul 4 21:10:38 ua83 kernel: [293760.725372] Call
Trace:
Jul 4 21:10:38 ua83 kernel: [293760.725381] [<ffffffffa02cc12c>] ?
dm_table_unplug_all+0x5c/0xd0
[dm_mod]
Jul 4 21:10:38 ua83 kernel: [293760.725388] [<ffffffff8109c569>] ?
ktime_get_ts+0xa9/0xe0
Jul 4 21:10:38 ua83 kernel: [293760.725394] [<ffffffff8109c569>] ?
ktime_get_ts+0xa9/0xe0
Jul 4 21:10:38 ua83 kernel: [293760.725399] [<ffffffff813ff403>]
io_schedule+0xa3/0x110
Jul 4 21:10:38 ua83 kernel: [293760.725404] [<ffffffff8111a6e0>] ?
sync_page+0x0/0x50
Jul 4 21:10:38 ua83 kernel: [293760.725409] [<ffffffff8111a71d>]
sync_page+0x3d/0x50
Jul 4 21:10:38 ua83 kernel: [293760.725414] [<ffffffff813ffc8f>]
__wait_on_bit+0x5f/0x90
Jul 4 21:10:38 ua83 kernel: [293760.725419] [<ffffffff8111a8d3>]
wait_on_page_bit+0x73/0x80
Jul 4 21:10:38 ua83 kernel: [293760.725425] [<ffffffff81091a30>] ?
wake_bit_function+0x0/0x50
Jul 4 21:10:38 ua83 kernel: [293760.725430] [<ffffffff81131355>] ?
pagevec_lookup_tag+0x25/0x40
Jul 4 21:10:38 ua83 kernel: [293760.725436] [<ffffffff8111ac9b>]
wait_on_page_writeback_range+0xfb/0x190
Jul 4 21:10:38 ua83 kernel: [293760.725443] [<ffffffff811b7c9b>] ?
bio_alloc_bioset+0x5b/0xf0
Jul 4 21:10:38 ua83 kernel: [293760.725449] [<ffffffff8111ad5f>]
filemap_fdatawait+0x2f/0x40
Jul 4 21:10:38 ua83 kernel: [293760.725462] [<ffffffffa00a7ec0>]
jbd2_journal_commit_transaction+0x7f0/0x1490
[jbd2]
Jul 4 21:10:38 ua83 kernel: [293760.725469] [<ffffffff810919f0>] ?
autoremove_wake_function+0x0/0x40
Jul 4 21:10:38 ua83 kernel: [293760.725480] [<ffffffffa00ad978>]
kjournald2+0xb8/0x220
[jbd2]
Jul 4 21:10:38 ua83 kernel: [293760.725486] [<ffffffff810919f0>] ?
autoremove_wake_function+0x0/0x40
Jul 4 21:10:38 ua83 kernel: [293760.725496] [<ffffffffa00ad8c0>] ?
kjournald2+0x0/0x220
[jbd2]
Jul 4 21:10:38 ua83 kernel: [293760.725501] [<ffffffff81091680>]
kthread+0xc0/0xe0
Jul 4 21:10:38 ua83 kernel: [293760.725506] [<ffffffff8100c2ca>]
child_rip+0xa/0x20
Jul 4 21:10:38 ua83 kernel: [293760.725511] [<ffffffff810915c0>] ?
kthread+0x0/0xe0
Jul 4 21:10:38 ua83 kernel: [293760.725516] [<ffffffff8100c2c0>] ?
child_rip+0x0/0x20
и т.д.
Подскажите что делать? Куда еще можно посмотреть?
--
WBR,
Dubrovskiy Viacheslav
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 6229 bytes --]
next reply other threads:[~2011-07-04 19:20 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-07-04 19:20 Slava Dubrovskiy [this message]
2011-07-04 20:16 ` Slava Dubrovskiy
2011-07-04 21:40 ` Arcady V. Ivanov
2011-07-05 6:03 ` Anton Gorlov
2011-07-06 15:41 ` Michael Shigorin
2011-08-04 12:33 ` Sergey
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4E121279.4040608@tangramltd.com \
--to=slava@tangramltd.com \
--cc=community@lists.altlinux.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
ALT Linux Community general discussions
This inbox may be cloned and mirrored by anyone:
git clone --mirror http://lore.altlinux.org/community/0 community/git/0.git
# If you have public-inbox 1.1+ installed, you may
# initialize and index your mirror using the following commands:
public-inbox-init -V2 community community/ http://lore.altlinux.org/community \
mandrake-russian@linuxteam.iplabs.ru community@lists.altlinux.org community@lists.altlinux.ru community@lists.altlinux.com
public-inbox-index community
Example config snippet for mirrors.
Newsgroup available over NNTP:
nntp://lore.altlinux.org/org.altlinux.lists.community
AGPL code for this site: git clone https://public-inbox.org/public-inbox.git