SCSI DPT vs I2O problems [was: Will development now move to 2.6.16?]
Hi, now that linux 2.6.16 is released I'm wondering if development will be based on that instead on 2.6.15. I'm asking because 2.6.16 introduces some driver changes that are quite important to me. Regards, Mephisto
Re: Will development now move to 2.6.16?
Yes, we will release a 2.6.16-based kernel Real Soon Now. I have already said that today in OpenVZ blog. Quote: I'm asking because 2.6.16 introduces some driver changes that are quite important to me. If you have problem with a particular hardware driver (not working, missing, etc.) -- do not hesitate to file a bug report. We are already supporting a lot of new hardware in our stable kernel series (by either backporting stuff and/or taking vendor drivers).
Great! I'll look into this. The problem is that I couldn't even get the 2.6.8 kernel to work on my server, yet - 2.6.15 patch was bootin with the same config, but was always crashing after the machine was running for 1-2 days. I'm about 150 miles away, to this is a bit problematic to look into.
Can you please provide the output of lspci and lspci -n? It would help me to understand what hardware you use and which driver is responsible for it. Thanks
Sure: lspci 00:00.0 Host bridge: Intel Corporation E7501 Memory Controller Hub (rev 01) 00:00.1 Class ff00: Intel Corporation E7500/E7501 Host RASUM Controller (rev 01) 00:02.0 PCI bridge: Intel Corporation E7500/E7501 Hub Interface B PCI-to-PCI Bridge (rev 01) 00:03.0 PCI bridge: Intel Corporation E7500/E7501 Hub Interface C PCI-to-PCI Bridge (rev 01) 00:1d.0 USB Controller: Intel Corporation 82801CA/CAM USB (Hub #1) (rev 02) 00:1d.1 USB Controller: Intel Corporation 82801CA/CAM USB (Hub #2) (rev 02) 00:1d.2 USB Controller: Intel Corporation 82801CA/CAM USB (Hub #3) (rev 02) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 42) 00:1f.0 ISA bridge: Intel Corporation 82801CA LPC Interface Controller (rev 02) 00:1f.1 IDE interface: Intel Corporation 82801CA Ultra ATA Storage Controller (rev 02) 00:1f.3 SMBus: Intel Corporation 82801CA/CAM SMBus Controller (rev 02) 01:1c.0 PIC: Intel Corporation 82870P2 P64H2 I/OxAPIC (rev 04) 01:1d.0 PCI bridge: Intel Corporation 82870P2 P64H2 Hub PCI Bridge (rev 04) 01:1e.0 PIC: Intel Corporation 82870P2 P64H2 I/OxAPIC (rev 04) 01:1f.0 PCI bridge: Intel Corporation 82870P2 P64H2 Hub PCI Bridge (rev 04) 03:02.0 Ethernet controller: Intel Corporation 82546EB Gigabit Ethernet Controller (Copper) (rev 01) 03:02.1 Ethernet controller: Intel Corporation 82546EB Gigabit Ethernet Controller (Copper) (rev 01) 04:1c.0 PIC: Intel Corporation 82870P2 P64H2 I/OxAPIC (rev 04) 04:1d.0 PCI bridge: Intel Corporation 82870P2 P64H2 Hub PCI Bridge (rev 04) 04:1e.0 PIC: Intel Corporation 82870P2 P64H2 I/OxAPIC (rev 04) 04:1f.0 PCI bridge: Intel Corporation 82870P2 P64H2 Hub PCI Bridge (rev 04) 06:01.0 RAID bus controller: Adaptec (formerly DPT) SmartRAID V Controller (rev 01) 07:01.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27) lspci -n 00:00.0 0600: 8086:254c (rev 01) 00:00.1 ff00: 8086:2541 (rev 01) 00:02.0 0604: 8086:2543 (rev 01) 00:03.0 0604: 8086:2545 (rev 01) 00:1d.0 0c03: 8086:2482 (rev 02) 00:1d.1 0c03: 8086:2484 (rev 02) 00:1d.2 0c03: 8086:2487 (rev 02) 00:1e.0 0604: 8086:244e (rev 42) 00:1f.0 0601: 8086:2480 (rev 02) 00:1f.1 0101: 8086:248b (rev 02) 00:1f.3 0c05: 8086:2483 (rev 02) 01:1c.0 0800: 8086:1461 (rev 04) 01:1d.0 0604: 8086:1460 (rev 04) 01:1e.0 0800: 8086:1461 (rev 04) 01:1f.0 0604: 8086:1460 (rev 04) 03:02.0 0200: 8086:1010 (rev 01) 03:02.1 0200: 8086:1010 (rev 01) 04:1c.0 0800: 8086:1461 (rev 04) 04:1d.0 0604: 8086:1460 (rev 04) 04:1e.0 0800: 8086:1461 (rev 04) 04:1f.0 0604: 8086:1460 (rev 04) 06:01.0 0104: 1044:a511 (rev 01) 07:01.0 0300: 1002:4752 (rev 27) This is a Supermicro X5DP8-G2 running two SCSI raids (using aic79xx).
Hello mephisto, Unfortunately I cannot understand what kind of problem you have on the node. As far as I see from your lspci, you have I2O zero-channel raid controller. It supported by our kernel and your node should work even on the 2.6.8 kernel. To load our kernel on this hardware you should create correct initrd image. Could you please send me /etc/modprobe.conf file and output of the following command mkinitrd -v -v -f /boot/custom.initrd.img 2.6.8-022stab070.1 thank you, Vasily Averin PS could you probably give me an access to the node?
Hi Vasily, thank you for your answer. I'm getting "-bash: mkinitrd: command not found". I'm using Gentoo and so far made no use of the initrd, because I had all the important modules compiled in. I do have a initramfs-genkernel-x86-2.6.8-022stab070-smp under /boot, I'll check if it helps to include it into my lilo.conf I'm sorry, but I can't give you access to the server. My /etc/modprobe.conf looks like this ### This file is automatically generated by modules-update # # Please do not edit this file directly. If you want to change or add # anything please take a look at /etc/modules.conf and read # the manpage for modules-update. # alias binfmt-204 binfmt_aout alias binfmt-263 binfmt_aout alias binfmt-264 binfmt_aout alias binfmt-267 binfmt_aout alias binfmt-387 binfmt_aout alias binfmt-332 iBCS alias binfmt--310 binfmt_java alias block-major-1-* rd alias block-major-2-* floppy alias block-major-3-* ide-probe-mod alias block-major-7-* loop alias block-major-8-* sd_mod alias block-major-9-* md alias block-major-11-* sr_mod alias block-major-13-* xd alias block-major-15-* cdu31a alias block-major-16-* gscd alias block-major-17-* optcd alias block-major-18-* sjcd alias block-major-20-* mcdx alias block-major-22-* ide-probe-mod alias block-major-23-* mcd alias block-major-24-* sonycd535 alias block-major-25-* sbpcd alias block-major-26-* sbpcd alias block-major-27-* sbpcd alias block-major-29-* aztcd alias block-major-32-* cm206 alias block-major-33-* ide-probe-mod alias block-major-34-* ide-probe-mod alias block-major-37-* ide-tape alias block-major-44-* ftl alias block-major-46-* pcd alias block-major-47-* pf alias block-major-56-* ide-probe-mod alias block-major-57-* ide-probe-mod alias block-major-58-* lvm-mod alias block-major-88-* ide-probe-mod alias block-major-89-* ide-probe-mod alias block-major-90-* ide-probe-mod alias block-major-91-* ide-probe-mod alias block-major-93-* nftl alias block-major-113-* viocd alias char-major-4-* serial alias char-major-5-* serial alias char-major-6-* lp alias char-major-9-* st alias char-major-10-0 busmouse alias char-major-10-2 msbusmouse alias char-major-10-3 atixlmouse alias char-major-10-116 snd alias char-major-10-130 wdt alias char-major-10-131 wdt alias char-major-10-135 rtc alias char-major-10-139 openprom alias char-major-10-144 nvram alias char-major-10-157 applicom alias char-major-10-175 agpgart alias char-major-10-181 toshiba alias char-major-10-183 hw_random alias char-major-10-184 microcode alias char-major-10-187 irnet alias char-major-10-189 ussp alias char-major-10-200 tun alias char-major-10-250 hci_vhci alias char-major-13-* input alias char-major-13-0 joydev alias char-major-13-32 mousedev alias char-major-14-* soundcore alias char-major-19-* cyclades alias char-major-20-* cyclades alias char-major-21-* sg alias char-major-22-* pcxx alias char-major-23-* pcxx alias char-major-27-* zftape alias char-major-34-* scc alias char-major-35-* tclmidi alias char-major-36-* netlink alias char-major-37-* ide-tape alias char-major-48-* riscom8 alias char-major-49-* riscom8 alias char-major-57-* esp alias char-major-58-* esp alias char-major-63-* kdebug alias char-major-90-* mtdchar alias char-major-96-* pt alias char-major-97-* pg alias char-major-99-* ppdev alias char-major-107-* 3dfx alias char-major-108-* ppp_generic alias char-major-109-* lvm-mod alias char-major-161-* ircomm-tty alias char-major-171-* raw1394 alias char-major-195-* NVdriver alias char-major-200-* vxspec alias char-major-206-* osst alias char-major-216-* rfcomm alias dos msdos alias dummy0 dummy alias dummy1 dummy alias iso9660 isofs alias md-personality-1 linear alias md-personality-2 raid0 alias md-personality-3 raid1 alias md-personality-4 raid5 alias md-personality-7 multipath alias net-pf-1 unix alias net-pf-2 ipv4 alias net-pf-4 ipx alias net-pf-5 appletalk alias net-pf-15 af_key alias net-pf-17 af_packet alias net-pf-20 atm alias net-pf-23 irda alias net-pf-24 pppoe alias net-pf-25 wanrouter alias net-pf-26 llc alias net-pf-31 bluez alias netalias-2 ip_alias alias irlan0 irlan alias irda-dongle-0 tekram alias irda-dongle-1 esi alias irda-dongle-2 actisys alias irda-dongle-3 actisys alias irda-dongle-4 girbil alias irda-dongle-5 litelink alias irda-dongle-6 airport alias irda-dongle-7 old_belkin alias bt-proto-0 l2cap alias bt-proto-2 sco alias bt-proto-3 rfcomm alias bt-proto-4 bnep alias bt-proto-5 cmtp alias bt-proto-6 hidp alias bt-proto-7 avdtp alias plip0 plip alias plip1 plip alias tunl0 ipip alias cipcb0 cipcb alias cipcb1 cipcb alias cipcb2 cipcb alias cipcb3 cipcb alias ppp0 ppp_async alias ppp1 ppp_async alias slip0 slip alias slip1 slip alias tty-ldisc-1 slip alias tty-ldisc-3 ppp_async alias tty-ldisc-11 irtty alias tty-ldisc-14 ppp_synctty alias tty-ldisc-15 hci_uart alias ppp-compress-18 ppp_mppe alias ppp-compress-21 bsd_comp alias ppp-compress-24 ppp_deflate alias ppp-compress-26 ppp_deflate alias ppp ppp_async alias parport_lowlevel parport_pc alias usbdevfs usbcore alias xfrm-type-2-50 esp4 alias xfrm-type-2-51 ah4 alias xfrm-type-2-108 ipcomp alias xfrm-type-10-50 esp6 alias xfrm-type-10-51 ah6 alias xfrm-type-10-108 ipcomp6 alias cipher_null crypto_null alias digest_null crypto_null alias compress_null crypto_null alias sha384 sha512 alias char-major-81-* bttv alias /dev/ppp ppp_generic alias loop-xfer-gen-0 loop_gen alias loop-xfer-3 loop_fish2 alias loop-xfer-gen-10 loop_gen alias cipher-2 des alias cipher-3 fish2 alias cipher-4 blowfish alias cipher-6 idea alias cipher-7 serp6f alias cipher-8 mars6 alias cipher-11 rc62 alias cipher-15 dfc2 alias cipher-16 rijndael alias cipher-17 rc5 alias char-major-89-* i2c-dev alias /dev/nvidiactl NVdriver alias binfmt-0064 binfmt_aout options sb io=0x220 irq=7 dma=1 dma16=5 mpu_io=0x330 install binfmt-0000 /bin/true install char-major-10 /bin/true install char-major-10-1 /bin/true install dummy0 /sbin/modprobe -o dummy0 --ignore-install dummy install dummy1 /sbin/modprobe -o dummy1 --ignore-install dummy install eth0 /bin/true install net-pf-10 /bin/true install net-pf-19 /bin/true install net-pf-3 /bin/true install net-pf-6 /bin/true install scsi_hostadapter /bin/true
Hello Mephisto, Usually we tried to include all the required drivers int kernels, but our i2o driver did not work by this way and we should be forced to compile it as module. Therefore you should use initrd or initramfs images for our kernels. I hope our kernel will work when you include initrd image to your lilo.conf file. Unfortunately I'm not familiar with Gentoo. I suppose that the list of requierd modules on this distributive are stores in /etc/modules.conf file. If you will have problems with our kernel, please send me /etc/modules.conf file and your initramfs file. Thank you, Vasily Averin
BTW, do you have any bootup logs/screens with the information how 2.6.8 crashed on your machines?
2.6.8 never crashed, it didn't even start (could not mount root fs). It was 2.6.15. I have the feeling that it was a memory related problem because people complained that the server was getting slower and slower before beeing not available anymore - yes I'm using the node as a server even though it is not recommended.
I just realized that CONFIG_SCSI_DPT_I2O was not compiled into the openvz kernel while it was compiled into the running kernel - might have been my own stupidity all along.
Oh, ok. Give us to know whether it helped or some other efforts are needed from us to resolve it.
hello Mephisto We use a new driver for this hardware: i2o_block (CONFIG_I2O_BLOCK) I would note that dpt_i2o driver also did not work inside kernel. I've tried to build dpt_i2o driver as module and it works OK, But dpt_i2o and i2o_block drivers have a conflicts and cannot be loaded together. Thank you, Vasily Averin
So i2o_block is a replacement for dpt_i2o? What is the advantage of i2o_block? In my running 2.6.14 kernel I have CONFIG_I2O_EXT_ADAPTEC=y - is that a replacement for CONFIG_I2O_BLOCK?
Hello Mephisto, yes, i2o_block is new driver for i2o hardware, it replaced old dpt_i2o driver in 2.6-based Red Hat disributions. CONFIG_I2O_EXT_ADAPTEC is not the same as CONFIG_I2O_BLOCK There are 2 different suboption of new i2o layer. to build new i2o layer you should enable at least following config options in kernel config file: CONFIG_I2O=m CONFIG_I2O_CONFIG=m CONFIG_I2O_BLOCK=m CONFIG_I2O_SCSI=m I would note that i2o_block driver used unusual device names: instead usual scsi-like /dev/sda it uses /dev/i2o/hda device names. If you not use LABEL's on your filesystem, you may have troubles with partition mounting. I would note that the best solution is to use the same driver as on your linux distribution. If Gentoo uses dpt_i2o driver -- that I would recommend you to _disable_ CONFIG_I2O option at all and enable compilation for old dpt_i2o driver: CONFIG_SCSI_DPT_I2O=m Thank you, Vasily Averin
Quote: I just realized that CONFIG_SCSI_DPT_I2O was not compiled into the openvz kernel while it was compiled into the running kernel - might have been my own stupidity all along. mephisto, does it work now? have you managed starting it?
So far not. I converted my fstab to use LABELS for my reiserfs partitions, but couldn't label my root partition. I'm uncertain what to add to my "root = " line in lilo.conf as /dev/i20/hda2 was not accepted by lilo. I recompiled my kernel to use dpt_i2o again and will try it later today.
I've been so far unable to boot the kernel with CONFIG_I2O_BLOCK. The 2.6.15 and 2.6.16 kernels boot with dpt_i2o, but get unstable after the node has been running for about a day. I've copied about 100GB of data using rsync without problems, but once the Samba fileserver on the hardware node was running and under regular use the machine got slower and slower until it wasn't usable anymore. No VPS was running at that time, just the VZ modules were loaded. I don't think it's a memory issue, because I trippled the memory from 512MB to 1,5GB and the time until it goes down is the same. My 2.6.16 kernel config can be found here I will investigate the problem with 2.6.8 once I get it to boot. PS: I've been running 2.6.15-025stab014.1 without problems on another Gentoo system using the same build environment. That hardware node is running LAMP in addition to OpenVZ, so it must either be hardware related or related to Samba.
mephisto, any progress with booting 2.6.8? can I help somehow? maybe you can provide boot messages from 2.6.8 when it doesn't boot (e.g. through serial or net console?)
Re: SCSI DPT vs I2O problems [was: Will development now move to 2.6.16?]
Here is the output of the not booting 2.6.8-022stab072: Loading Adaptec I2O RAID: Version 2.4 Build 5go Detecting Adaptec I2O RAID controllers... ACPI: PCI interrupt 0000:06:01.0[A] -> GSI 72 (level, low) -> IRQ 72 Adaptec I2O RAID controller 0 irq=72 BAR0 f4807000 - size= 100000 BAR1 f4998000 - size= 1000000 dpti: If you have a lot of devices this could take a few minutes dpti0: Reading the hardware ressource table. dpti0: Get status timeout. scsi0: Vendor: Adaptec Model: mice: PS/2 mouse device common for all mice serio: i8042 AUX port at 0x60,0x64 irq 12 input: PS/2 Generic Mouse on isa0060/serio1 serio:i8042 KBD port at 0x60,0x64 irq 1 input: AT Translated Set 2 keyboard on isa0060/serio0 device-mapper: 4.1.0-ioctl (2003-12-10) initialised: dm@uk.sistina.com NET: Registered protocol family 2 IP: routing cache hash table of 16384 buckets, 128Kbytes TCP: Hash tables configured (established 524288 bind 65536) NET: Registered protocol family 1 Starting balanced_irq VFS: Cannot open root device "sda2" or unknown-block(0,0) Please append a correct "root=" boot option Kernel panic: VFS: Unable to mount root fs on unknown-block(0,0) Both kernels have these options set. While the 2.6.16 kernel boots, the 2.6.8 kernel doesn't: CONFIG_SCSI_AIC79XX=y CONFIG_AIC79XX_CMDS_PER_DEVICE=32 CONFIG_AIC79XX_RESET_DELAY_MS=15000 # CONFIG_AIC79XX_BUILD_FIRMWARE is not set CONFIG_AIC79XX_ENABLE_RD_STRM=y CONFIG_AIC79XX_DEBUG_ENABLE=y CONFIG_AIC79XX_DEBUG_MASK=0 CONFIG_AIC79XX_REG_PRETTY_PRINT=y CONFIG_SCSI_DPT_I2O=y # # I2O device support # CONFIG_I2O=m CONFIG_I2O_CONFIG=m # CONFIG_I2O_BLOCK is not set CONFIG_I2O_SCSI=m CONFIG_I2O_PROC=m
mephisto, as vaverin have already said we had faced issues that i2o_block and dpt_i2o drivers do not work properly in 2.6.8 kernel if compiled into the kernel. Please, could you build a 2.6.8 OpenVZ kernel with _disabled_ CONFIG_I2O option and CONFIG_SCSI_DPT_I2O=m (message #2135)? Yes, in that case you should create initrd file to boot, but it looks like this is the only possible way to make dpt_i2o driver working in 2.6.8 kernel.
I also tried that and it did not work. I'm sorry I have no error message but I'm happy the system is running now and I can't reboot it.