OpenVZ Forum


Home » Mailing lists » Users » openvz ploop images on moosefs mount
openvz ploop images on moosefs mount [message #45731] Sat, 31 March 2012 18:44 Go to next message
Corin Langosch is currently offline  Corin Langosch
Messages: 18
Registered: March 2012
Junior Member
From: *parallels.com
Hi,

has anybody tried using the new ploop storage for openvz images together
with moosefs ( http://www.moosefs.org/)?

ploop mount -d /dev/ploop0 /mfs-mount/root.hdd
Adding delta dev=/dev/ploop0 img=/mfs-mount/root.hdd (rw)
PLOOP_IOC_ADD_DELTA /mfs-mount/root.hdd: Invalid argument

In syslog I find:
kernel: File on FS without backing device

I thought it might be because ploop needs direct-io (right?), but
mounting with direct-io enabled seems not to be supported by moosefs:

mfsmount -o direct_io /mfs-mount
mfsmaster accepted connection with parameters: read-write,restricted_ip
; root mapped to root:root
fuse: unknown option `direct_io'

Has anybody got it working somehow? :)

Btw: I'll also post this question to the moosefs mailinglist as I'm
pretty sure the devs there might have interesting answers too.

Corin
Re: openvz ploop images on moosefs mount [message #45732 is a reply to message #45731] Sat, 31 March 2012 18:54 Go to previous messageGo to next message
Kirill Kolyshkin is currently offline  Kirill Kolyshkin
Messages: 9
Registered: October 2006
Junior Member
From: *parallels.com
Ploop only supports ext4 and nfs
31.03.2012 22:50 пользователь "Corin Langosch" <info@corinlangosch.com>
написал:

> Hi,
>
> has anybody tried using the new ploop storage for openvz images together
> with moosefs ( http://www.moosefs.org/)?
>
> ploop mount -d /dev/ploop0 /mfs-mount/root.hdd
> Adding delta dev=/dev/ploop0 img=/mfs-mount/root.hdd (rw)
> PLOOP_IOC_ADD_DELTA /mfs-mount/root.hdd: Invalid argument
>
> In syslog I find:
> kernel: File on FS without backing device
>
> I thought it might be because ploop needs direct-io (right?), but mounting
> with direct-io enabled seems not to be supported by moosefs:
>
> mfsmount -o direct_io /mfs-mount
> mfsmaster accepted connection with parameters: read-write,restricted_ip ;
> root mapped to root:root
> fuse: unknown option `direct_io'
>
> Has anybody got it working somehow? :)
>
> Btw: I'll also post this question to the moosefs mailinglist as I'm pretty
> sure the devs there might have interesting answers too.
>
> Corin
>
>
Re: openvz ploop images on moosefs mount [message #45733 is a reply to message #45732] Sat, 31 March 2012 19:13 Go to previous messageGo to next message
Corin Langosch is currently offline  Corin Langosch
Messages: 18
Registered: March 2012
Junior Member
From: *parallels.com
Hi Kirill,

thank you for your very fast response. :)

Am 31.03.2012 20:54, schrieb Kirill Kolyshkin:
>
> Ploop only supports ext4 and nfs
>
Is there any special reason for this? I'd really like to know the
technical aspects because I then might be able to make moosefs (or
another fuse based fs developed by myself) working with ploop. If I can
read about it in any documents, please redirect me to them :)

Another question: is it (easily) possible make an openvz container use a
normal block devices for storage (which is formatted with ex. ext4)?
Before seeing ploop I thought this won't be possible, but now with ploop
it seems vzctl is only mounting /dev/ploopX somewhere and then using
this mount as the container's root. I'd especially be interested in
using ex. an iscsi backed block device with openvz containers.

Corin

Am 31.03.2012 20:54, schrieb Kirill Kolyshkin:
>
> Ploop only supports ext4 and nfs
>
> 31.03.2012 22:50 ???????????? "Corin Langosch" <info@corinlangosch.com
> <mailto:info@corinlangosch.com>> ???????:
>
> Hi,
>
> has anybody tried using the new ploop storage for openvz images
> together with moosefs ( http://www.moosefs.org/)?
>
> ploop mount -d /dev/ploop0 /mfs-mount/root.hdd
> Adding delta dev=/dev/ploop0 img=/mfs-mount/root.hdd (rw)
> PLOOP_IOC_ADD_DELTA /mfs-mount/root.hdd: Invalid argument
>
> In syslog I find:
> kernel: File on FS without backing device
>
> I thought it might be because ploop needs direct-io (right?), but
> mounting with direct-io enabled seems not to be supported by moosefs:
>
> mfsmount -o direct_io /mfs-mount
> mfsmaster accepted connection with parameters:
> read-write,restricted_ip ; root mapped to root:root
> fuse: unknown option `direct_io'
>
> Has anybody got it working somehow? :)
>
> Btw: I'll also post this question to the moosefs mailinglist as
> I'm pretty sure the devs there might have interesting answers too.
>
> Corin
>
>
> _______________________________________________
> Users mailing list
> Users@openvz.org <mailto:Users@openvz.org>
> https://openvz.org/mailman/listinfo/users
>
>
>
Re: openvz ploop images on moosefs mount [message #45734 is a reply to message #45733] Sun, 01 April 2012 12:11 Go to previous messageGo to next message
kir is currently offline  kir
Messages: 1645
Registered: August 2005
Location: Moscow, Russia
Senior Member

From: *parallels.com
On 03/31/2012 11:13 PM, Corin Langosch wrote:
> Hi Kirill,
>
> thank you for your very fast response. :)
>
> Am 31.03.2012 20:54, schrieb Kirill Kolyshkin:
>>
>> Ploop only supports ext4 and nfs
>>
> Is there any special reason for this? I'd really like to know the
> technical aspects because I then might be able to make moosefs (or
> another fuse based fs developed by myself) working with ploop. If I
> can read about it in any documents, please redirect me to them :)

Ploop developers are now working on fuse I/O module for ploop (and fuse
improvements required for it). When this work will be ready, you might
revisit this issue and see what is required from moosefs to be used for
ploop.

>
> Another question: is it (easily) possible make an openvz container use
> a normal block devices for storage (which is formatted with ex. ext4)?
> Before seeing ploop I thought this won't be possible, but now with
> ploop it seems vzctl is only mounting /dev/ploopX somewhere and then
> using this mount as the container's root. I'd especially be interested
> in using ex. an iscsi backed block device with openvz containers.

Yes it's pretty possible, with some minor modifications to vzctl. I need
to think a bit more about it, but it looks like a trivial case (no
vzquota etc). Let me know if you want to work on that.

>
> Am 31.03.2012 20:54, schrieb Kirill Kolyshkin:
>>
>> Ploop only supports ext4 and nfs
>>
>> 31.03.2012 22:50 пользователь "Corin Langosch"
>> <info@corinlangosch.com <mailto:info@corinlangosch.com>> написал:
>>
>> Hi,
>>
>> has anybody tried using the new ploop storage for openvz images
>> together with moosefs ( http://www.moosefs.org/)?
>>
>> ploop mount -d /dev/ploop0 /mfs-mount/root.hdd
>> Adding delta dev=/dev/ploop0 img=/mfs-mount/root.hdd (rw)
>> PLOOP_IOC_ADD_DELTA /mfs-mount/root.hdd: Invalid argument
>>
>> In syslog I find:
>> kernel: File on FS without backing device
>>
>> I thought it might be because ploop needs direct-io (right?), but
>> mounting with direct-io enabled seems not to be supported by moosefs:
>>
>> mfsmount -o direct_io /mfs-mount
>> mfsmaster accepted connection with parameters:
>> read-write,restricted_ip ; root mapped to root:root
>> fuse: unknown option `direct_io'
>>
>> Has anybody got it working somehow? :)
>>
>> Btw: I'll also post this question to the moosefs mailinglist as
>> I'm pretty sure the devs there might have interesting answers too.
>>
>> Corin
>>


Kir Kolyshkin
http://static.openvz.org/userbars/openvz-developer.png
Re: openvz ploop images on moosefs mount [message #45741 is a reply to message #45731] Mon, 02 April 2012 10:19 Go to previous messageGo to next message
Aleksandar Ivanisevic is currently offline  Aleksandar Ivanisevic
Messages: 34
Registered: April 2011
Member
From: *parallels.com
Corin Langosch
<info@corinlangosch.com> writes:

> Hi,
>
> has anybody tried using the new ploop storage for openvz images
> together with moosefs ( http://www.moosefs.org/)?

Out of interest, have you tried running VEs off of moosefs, directly,
without ploop? How does that work, if at all?

I'm still trying to find a replacement for DRBD for a HA setup with a
lower latency, but all I find when I google "moosefs openvz" is my own
question on this mailing list a year ago ;)
Re: Re: openvz ploop images on moosefs mount [message #45744 is a reply to message #45741] Mon, 02 April 2012 10:54 Go to previous messageGo to next message
Corin Langosch is currently offline  Corin Langosch
Messages: 18
Registered: March 2012
Junior Member
From: *parallels.com
Am 02.04.2012 12:19, schrieb Aleksandar Ivanisevic:
> Out of interest, have you tried running VEs off of moosefs, directly,
> without ploop? How does that work, if at all?

Yes, but only for a quick test and not with any production data:

- setup moosefs as usual and mount it on host
- create (sparse) file for container on moosefs mount
- use this file to setup a loop device using losetup
- make fs on loop device and mount it to the container's private folder
- start the container as usual

I din't really like this approach because it involes a lot of (slow)
layers and double buffering issues and so I'd expect quite bad
performance. If you try it yourself and do some benchmarks I'd be very
happy to know the results.

I didn't try to use moosefs directly (without loop device) as a backing
storage for containers because I made very bad experience (mostly
performance and stability) with all kinds of network storages in the past.

From my past experience having a distributed block device (and not a
distributed file system) is the way to go, as the system then has almost
"native" performance due to the page cache (if not too many layers are
involved as above).

I'm currently evaluating distributed network block devices, but
unluckily there doesn't one to exist yet:
http://serverfault.com/questions/375570/distributed-fault-to lerant-network-block-device/375821#375821

Corin
Re: openvz ploop images on moosefs mount [message #45746 is a reply to message #45734] Mon, 02 April 2012 11:56 Go to previous messageGo to next message
Corin Langosch is currently offline  Corin Langosch
Messages: 18
Registered: March 2012
Junior Member
From: *parallels.com
Hi Kir,

Am 01.04.2012 14:11, schrieb Kir Kolyshkin:
>
> Ploop developers are now working on fuse I/O module for ploop (and
> fuse improvements required for it). When this work will be ready, you
> might revisit this issue and see what is required from moosefs to be
> used for ploop.

Sounds great. Can you please point me to the current git repo of the
kernel stuff? The kernel stuff at http://git.openvz.org/ seems to be
outdated and I couldn't find the new address.

>
> Yes it's pretty possible, with some minor modifications to vzctl. I
> need to think a bit more about it, but it looks like a trivial case
> (no vzquota etc). Let me know if you want to work on that.

Yes I could try, but I cannot guarentee how much time I can invest.
Please let me know how to proceed if anything special is needed.
Otherwise I'd just clone the git repo and send you the patch when
finished. Or I'd clone the repo at github and create a new branch for
the feature?

Corin
Re: openvz ploop images on moosefs mount [message #45751 is a reply to message #45746] Mon, 02 April 2012 18:23 Go to previous message
kir is currently offline  kir
Messages: 1645
Registered: August 2005
Location: Moscow, Russia
Senior Member

From: *parallels.com
On 04/02/2012 03:56 PM, Corin Langosch wrote:
> Hi Kir,
>
> Am 01.04.2012 14:11, schrieb Kir Kolyshkin:
>>
>> Ploop developers are now working on fuse I/O module for ploop (and
>> fuse improvements required for it). When this work will be ready, you
>> might revisit this issue and see what is required from moosefs to be
>> used for ploop.
>
> Sounds great. Can you please point me to the current git repo of the
> kernel stuff? The kernel stuff at http://git.openvz.org/ seems to be
> outdated and I couldn't find the new address.

Since we haven't yet tried to push ploop upstream yet its git repo is
not available (and internally we use something like stgit but based on
CVS and lots of shell code and *.patch files).

>
>>
>> Yes it's pretty possible, with some minor modifications to vzctl. I
>> need to think a bit more about it, but it looks like a trivial case
>> (no vzquota etc). Let me know if you want to work on that.
>
> Yes I could try, but I cannot guarentee how much time I can invest.
> Please let me know how to proceed if anything special is needed.
> Otherwise I'd just clone the git repo and send you the patch when
> finished. Or I'd clone the repo at github and create a new branch for
> the feature?

A patch (or, better, a patchset) is fine.

I suggest you start with using --layout option, introduce a new CT
layout (say 'device'... can't think of any good name right now) and
patch the mount/umount code to support it. VE_PRIVATE can be a block
device name in that case I guess, VE_ROOT remains the same (ie mount point).


Kir Kolyshkin
http://static.openvz.org/userbars/openvz-developer.png
Re: Re: openvz ploop images on moosefs mount [message #45830 is a reply to message #45741] Mon, 02 April 2012 10:49 Go to previous message
Corin Langosch is currently offline  Corin Langosch
Messages: 18
Registered: March 2012
Junior Member
From: *parallels.com
Am 02.04.2012 12:19, schrieb Aleksandar Ivanisevic:
> Out of interest, have you tried running VEs off of moosefs, directly,
> without ploop? How does that work, if at all?

Yes, but only for a quick test and not with any production data:

- setup moosefs as usual and mount it on host
- create (sparse) file for container on moosefs mount
- use this file to setup a loop device using losetup
- make fs on loop device and mount it to the container's private folder
- start the container as usual

I din't really like this approach because it involes a lot of (slow)
layers and double buffering issues and so I'd expect quite bad
performance. If you try it yourself and do some benchmarks I'd be very
happy to know the results.

I didn't try to use moosefs directly (without loop device) as a backing
storage for containers because I made very bad experience (mostly
performance and stability) with all kinds of network storages in the past.

From my past experience having a distributed block device (and not a
distributed file system) is the way to go, as the system then has almost
"native" performance due to the page cache (if not too many layers are
involved as above).

I'm currently evaluating distributed network block devices, but
unluckily there doesn't one to exist yet:
http://serverfault.com/questions/375570/distributed-fault-to lerant-network-block-device/375821#375821

Corin
Previous Topic: openvz oracle linux unbreakable kernel
Next Topic: Missing changelogs in kernel.spec
Goto Forum:
  


Current Time: Wed Nov 21 03:29:31 GMT 2018