OpenVZ Forum


Home » Mailing lists » Devel » irq0 stops working
irq0 stops working [message #21433] Tue, 09 October 2007 05:26 Go to next message
vaverin is currently offline  vaverin
Messages: 708
Registered: September 2005
Senior Member
On one of our servers timer interrupts (i.e irq0) are stops working. As result
any kernel timers do not triggers and tasks waiting some signals from timers
hangs forever.

Most noticeable effect of this situation is that any write operations to disk
are stalled, and nobody can log in on the node.

At the same time node all existing shells works away. I'm able to read
interrupts statistic from /proc/interrupts file and it shows that all other
interrupts are changed when these devices are accessed: disk on sata controller,
network, cdrom on ide controller, keyboard, serial console, LOC interrupts.

Also I've found that disable of irqbalance service on the node helps to
workaround this issue, however of course it fixes nothing.

All details about hardware/logs could be found in
http://bugzilla.kernel.org/show_bug.cgi?id=8650

I'm able to reproduce this situation, however now I have no ideas how to
continue the investigation of this problem.

Could please anybody advise me any new ways for investigation of this issue?

Thank you,
	Vasily Averin

OpenVZ Linux Kernel Team
Re: irq0 stops working [message #21434 is a reply to message #21433] Tue, 09 October 2007 05:34 Go to previous messageGo to next message
Jan Engelhardt is currently offline  Jan Engelhardt
Messages: 18
Registered: August 2006
Junior Member
On Oct 9 2007 09:26, Vasily Averin wrote:
>
>On one of our servers timer interrupts (i.e irq0) are stops working. As result
>any kernel timers do not triggers and tasks waiting some signals from timers
>hangs forever.

What kernel.. and tried CONFIG_NO_HZ=n?
Re: irq0 stops working [message #21435 is a reply to message #21434] Tue, 09 October 2007 06:12 Go to previous messageGo to next message
vaverin is currently offline  vaverin
Messages: 708
Registered: September 2005
Senior Member
Jan Engelhardt wrote:
> On Oct 9 2007 09:26, Vasily Averin wrote:
>> On one of our servers timer interrupts (i.e irq0) are stops working. As result
>> any kernel timers do not triggers and tasks waiting some signals from timers
>> hangs forever.
> 
> What kernel.. and tried CONFIG_NO_HZ=n?

Originally I've noticed this issue on RHEL5 kernels, but then I've reproduced it
on latest mainstream kernels, in my last attempt it was 2.6.23-rc7.

Thank you for for your tips about  CONFIG_NO_HZ=n, will try to to it.

thank you,
	Vasily Averin

OpenVZ Linux Kernel Team
Re: irq0 stops working [message #21446 is a reply to message #21433] Tue, 09 October 2007 06:00 Go to previous messageGo to next message
Thomas Gleixner is currently offline  Thomas Gleixner
Messages: 14
Registered: October 2007
Junior Member
On Tue, 9 Oct 2007, Vasily Averin wrote:
> On one of our servers timer interrupts (i.e irq0) are stops working. As result
> any kernel timers do not triggers and tasks waiting some signals from timers
> hangs forever.

Which kernel version ?
 
> Most noticeable effect of this situation is that any write operations to disk
> are stalled, and nobody can log in on the node.
> 
> At the same time node all existing shells works away. I'm able to read
> interrupts statistic from /proc/interrupts file and it shows that all other
> interrupts are changed when these devices are accessed: disk on sata controller,
> network, cdrom on ide controller, keyboard, serial console, LOC interrupts.
> 
> Also I've found that disable of irqbalance service on the node helps to
> workaround this issue, however of course it fixes nothing.

Well, it's at least a hint. Can you try the patch below please ?

	tglx

diff --git a/arch/x86_64/kernel/time.c b/arch/x86_64/kernel/time.c
index 6d48a4e..248987a 100644
--- a/arch/x86_64/kernel/time.c
+++ b/arch/x86_64/kernel/time.c
@@ -360,7 +360,7 @@ void stop_timer_interrupt(void)
 
 static struct irqaction irq0 = {
 	.handler	= timer_interrupt,
-	.flags		= IRQF_DISABLED | IRQF_IRQPOLL,
+	.flags		= IRQF_DISABLED | IRQF_IRQPOLL | IRQF_NOBALANCING,
 	.mask		= CPU_MASK_NONE,
 	.name		= "timer"
 };
@@ -403,6 +403,7 @@ void __init time_init(void)
 		cpu_khz / 1000, cpu_khz % 1000);
 	init_tsc_clocksource();
 
+	irq0.mask = cpumask_of_cpu(0);
 	setup_irq(0, &irq0);
 }
Re: irq0 stops working [message #21447 is a reply to message #21435] Tue, 09 October 2007 06:23 Go to previous messageGo to next message
Thomas Gleixner is currently offline  Thomas Gleixner
Messages: 14
Registered: October 2007
Junior Member
On Tue, 9 Oct 2007, Vasily Averin wrote:

> Jan Engelhardt wrote:
> > On Oct 9 2007 09:26, Vasily Averin wrote:
> >> On one of our servers timer interrupts (i.e irq0) are stops working. As result
> >> any kernel timers do not triggers and tasks waiting some signals from timers
> >> hangs forever.
> > 
> > What kernel.. and tried CONFIG_NO_HZ=n?
> 
> Originally I've noticed this issue on RHEL5 kernels, but then I've reproduced it
> on latest mainstream kernels, in my last attempt it was 2.6.23-rc7.
> 
> Thank you for for your tips about  CONFIG_NO_HZ=n, will try to to it.

You run a 64 bit kernel, where this option is not available yet.

	tglx
Re: irq0 stops working [message #22842 is a reply to message #21446] Tue, 06 November 2007 07:15 Go to previous message
vaverin is currently offline  vaverin
Messages: 708
Registered: September 2005
Senior Member
Thomas Gleixner wrote:
> On Tue, 9 Oct 2007, Vasily Averin wrote:
>> On one of our servers timer interrupts (i.e irq0) are stops working. As result
>> any kernel timers do not triggers and tasks waiting some signals from timers
>> hangs forever.
>>
>> Also I've found that disable of irqbalance service on the node helps to
>> workaround this issue, however of course it fixes nothing.
> 
> Well, it's at least a hint. Can you try the patch below please ?

Patch helps, after 1-month-long testing I could not reproduce this issue again,
what are we going to do now?

Unfortunately my test node is still unstable, now it reproduces sata timeout
issue, details are here:
http://bugzilla.kernel.org/show_bug.cgi?id=8650

Thank you,
	Vasily Averin
Previous Topic: [PATCH] Clean the ip_sockglue.c from some ugly ifdefs
Next Topic: [PATCH -net 1/2] Convert /proc/net/ipv6_route to seq_file interface
Goto Forum:
  


Current Time: Sat Jul 20 12:46:57 GMT 2024

Total time taken to generate the page: 0.02354 seconds