OpenVZ Forum


Home » Mailing lists » Devel » [PATCH 1/4] Virtualization/containers: introduction
Re: [PATCH 1/4] Virtualization/containers: introduction [message #1425 is a reply to message #1411] Wed, 08 February 2006 14:13 Go to previous messageGo to next message
Hubertus Franke is currently offline  Hubertus Franke
Messages: 16
Registered: February 2006
Junior Member
Eric W. Biederman wrote:
> Hubertus Franke <frankeh@watson.ibm.com> writes:
>
>>Agreed.. here are some issued we learned from other projects that had
>>similar interception points.
>>
>>Having a central umbrella object (let's stick to the name container)
>>is useful, but being the only object through which every access has to
>>pass may have drawbacks..
>>
>>task->container->pspace->pidmap[offset].page implies potential
>>cachemisses etc.
>>
>>If overhead becomes too large, then we can stick (cache) the pointer
>>additionally in the task struct. But ofcourse that should be carefully
>>examined on a per subsystem base...
>
>
> Ok. After talking with the vserver guys on IRC. I think grasp the
> importance. The key feature is to have a place to put limits and the
> like for your entire container. Look at all of the non-signal stuff
> in struct signal for an example. The nested namespaces seem to
> be just an implementation detail.
>
> For OpenVZ having the other namespaces nested may have some
> importance. I haven't gotten their yet.
>
> The task->container->pspace->.... thing feels very awkward to me,
> and feels like it increases our chance getting a cache miss.
>
> So I support the concept of a place to put all of the odd little
> things like rlimits for containers. But I would like to flatten
> it in the task_struct if we can.
>

My point was to mainly identify the performance culprits and provide
an direct access to those "namespaces" for performance reasons.
So we all agreed on that we need to do that..

Question now (see other's note as well), should we provide
a pointer to each and every namespace in struct task.
Seem rather wasteful to me as certain path/namespaces are not
exercise heavily.

Having one object "struct container" that still embodies all
namespace still seems a reasonable idea.
Otherwise identifying the respective namespace of subsystems will
have to go through container->init->subsys_namespace or similar.
Not necessarily bad either..

-- Hubertus
Re: The issues for agreeing on a virtualization/namespaces implementation. [message #1427 is a reply to message #1421] Wed, 08 February 2006 14:38 Go to previous messageGo to next message
serue is currently offline  serue
Messages: 750
Registered: February 2006
Senior Member
Quoting Herbert Poetzl (herbert@13thfloor.at):
> > 3) How do we refer to namespaces and containers when we are not members?
> > - Do we refer to them indirectly by processes or other objects that
> > we can see and are members?
>
> the process will be an unique identifier to the
> namespace, but it might not be easy to use it, so
> IMHO it might at least make sense to ...

Especially from userspace. If I want to start a checkpoint on a
container, but I have to use the process to identify the
container/namespace, well I can't uniquely specify the process by pid
anymore...

-serge
Re: The issues for agreeing on a virtualization/namespaces implementation. [message #1428 is a reply to message #1412] Wed, 08 February 2006 14:40 Go to previous messageGo to next message
Hubertus Franke is currently offline  Hubertus Franke
Messages: 16
Registered: February 2006
Junior Member
Eric W. Biederman wrote:
> Hubertus Franke <frankeh@watson.ibm.com> writes:
>
>
>>Eric W. Biederman wrote:
>>

>>>2) What is the syscall interface to create these namespaces?
>>> - Do we add clone flags? (Plan 9 style)
>>
>>Like that approach .. flexible .. particular when one has well specified
>>namespaces.
>>
>>
>>> - Do we add a syscall (similar to setsid) per namespace?
>>> (Traditional unix style)?
>>
>>Where does that approach end .. what's wrong with doing it at clone() time ?
>>Mainly the naming issue. Just providing a flag does not give me name.
>
>
> It really is a fairly even toss up. The usual argument for doing it
> this way is that you will get a endless stream of arguments added to
> fork+exec other wise. Look of posix_spawn or the windows version if
> you want an example. Bits to clone are skirting the edge of a slippery
> slope.
>

So it seems the clone( flags ) is a reasonable approach to create new
namespaces. Question is what is the initial state of each namespace?
In pidspace we know we should be creating an empty pidmap !
In network, someone suggested creating a loopback device
In uts, create "localhost"
Are there examples where we rather inherit ? Filesystem ?
Can we iterate the assumption for each subsystem what people thing is right?

IMHO, there is only a need to refer to a namespace from the global context.
Since one will be moving into a new container, but getting out of one
could be prohibitive (e.g. after migration)
It does not make sense therefore to know the name of a namespace in
a different container.

The example you used below by using the pid comes natural, because
that already limits visibility.

I am still struggling with why we need new sys_calls.
sys_calls already exist for changing certain system parameters (e.g. utsname )
so to me it boils down to identifying a proper initial state when the
namespace is created.

>
>>>3) How do we refer to namespaces and containers when we are not members?
>>> - Do we refer to them indirectly by processes or other objects that
>>> we can see and are members?
>>> - Do we assign some kind of unique id to the containers?
>>
>>In containers I simply created an explicite name, which ofcourse colides with
>>the
>>clone() approach ..
>>One possibility is to allow associating a name with a namespace.
>>For instance
>>int set_namespace_name( long flags, const char *name ) /* the once we are using
>>in clone */
>>{
>> if (!flag)
>> set name of container associated with current.
>> if (flag())
>> set the name if only one container is associated with the
>>namespace(s)
>> identified .. or some similar rule
>>}
>>
>
>
> What I have done which seems easier than creating new names is to refer
> to the process which has the namespace I want to manipulate.

Is then the idea to only allow the container->init to manipulate
or is there need to allow other priviliged processes to perform namespace
manipulation?
Also after thinking about it.. why is there a need to have an external name
for a namespace ?

>
>
>>>6) How do we do all of this efficiently without a noticeable impact on
>>> performance?
>>> - I have already heard concerns that I might be introducing cache
>>> line bounces and thus increasing tasklist_lock hold time.
>>> Which on big way systems can be a problem.
>>
>>Possible to split the lock up now.. one for each pidspace ?
>
>
> At the moment it is worth thinking about. If the problem isn't
> so bad that people aren't actively working on it we don't have to
> solve the problem for a little while, just be aware of it.
>

Agree, just need to be sure we can split it up. But you already keep
a task list per pid-namespace, so there should be no problem IMHO.
If so let's do it now and take it of the table it its as simple as

task_list_lock ::= pspace->task_list_lock

>
>>>7) How do we allow a process inside a container to create containers
>>> for it's children?
>>> - In general this is trivial but there are a few ugly issues
>>> here.
>>
>>Speaking of pids only here ...
>>Does it matter, you just hang all those containers hang of init.
>>What ever hierarchy they form is external ...
>
>
> In general it is simple. For resource accounting, and for naming so
> you can migrate a container with a nested container it is a question
> you need to be slightly careful with.

Absolutely, that's why it is useful to have an "external" idea of how
containers are constructed of basic namespaces==subsystems.
The it "simply" becomes a policy. E.g. one can not migrate a container
that has shared subsystems.
Resource accounting I agree, that might required active aggregation
at request time.

-- Hubertus
Re: The issues for agreeing on a virtualization/namespaces implementation. [message #1429 is a reply to message #1427] Wed, 08 February 2006 14:51 Go to previous messageGo to next message
Hubertus Franke is currently offline  Hubertus Franke
Messages: 16
Registered: February 2006
Junior Member
Serge E. Hallyn wrote:
> Quoting Herbert Poetzl (herbert@13thfloor.at):
>
>>>3) How do we refer to namespaces and containers when we are not members?
>>> - Do we refer to them indirectly by processes or other objects that
>>> we can see and are members?
>>
>>the process will be an unique identifier to the
>>namespace, but it might not be easy to use it, so
>>IMHO it might at least make sense to ...
>
>
> Especially from userspace. If I want to start a checkpoint on a
> container, but I have to use the process to identify the
> container/namespace, well I can't uniquely specify the process by pid
> anymore...
>
> -serge
>

Well we first can agree that througout all processes/tasks of a container
the namespaces used are the same.
Hence looking at the init_task of each container is sufficient.

Restricting visibility to the default container makes sense to me,
because one is not supposed to be able to look into another container.

However, in the global context we do have pid that we can use.

-- Hubertus
Re: The issues for agreeing on a virtualization/namespaces implementation. [message #1430 is a reply to message #1396] Wed, 08 February 2006 15:34 Go to previous messageGo to next message
dev is currently offline  dev
Messages: 1693
Registered: September 2005
Location: Moscow
Senior Member

>>Eric W. Biederman wrote:
>>So it seems the clone( flags ) is a reasonable approach to create new
>>namespaces. Question is what is the initial state of each namespace?
>>In pidspace we know we should be creating an empty pidmap !
>>In network, someone suggested creating a loopback device
>>In uts, create "localhost"
>>Are there examples where we rather inherit ? Filesystem ?
> Of course filesystem is already implemented, and does inheret a full
> copy.

why do we want to use clone()? Just because of its name and flags?
I think it is really strange to fork() to create network context. What
has process creation has to do with it?

After all these clone()'s are called, some management actions from host
system are still required, to add these IPs/routings/etc.
So? Why mess it up? Why not create a separate clean interface for
container management?

Kirill
Re: [PATCH 1/4] Virtualization/containers: introduction [message #1431 is a reply to message #1386] Wed, 08 February 2006 15:35 Go to previous messageGo to next message
dev is currently offline  dev
Messages: 1693
Registered: September 2005
Location: Moscow
Senior Member

> The pid-namespace (pspace) provides an approach of fully separate
> the allocation and maintenance of the pids and treating the <pspace,pid>
> tuple as an entity to uniquely identify a task and vice versa.
> As a result the logic of lookup can be pushed down the find_task_by_pid()
> lookup. There are specific cases where the init_task of a container or
> pspace needs to be checked to ensure that signals/waits and alike are
> properly
> handled across pspace boundaries. I think this is an intuitive and clean
> way.
> It also completely avoids the problem of having to think about all the
> locations
> at the user/kernel boundary where a vpid/pid conversion needs to take
> place.
> It also avoids the problems that logically vpids and pids are different
> types and
> therefore it would have been good to have type checking automatically
> identify
> problem spots.
> On the negative side, it does require to maintain a pidmap per pidspace.
Additional negative sides:
- full isolation can be inconvinient from containers management point of
view. You will need to introduce new modified tools such as top/ps/kill
and many many others. You won't be able to strace/gdb processes from the
host also.
- overhead when virtualization is off, result is not the same.
- additional args everywhere (stack usage, etc.)

> The vpid approach has the drawbacks of having to identify the conversion
> spots
> of all vpid vs. pid semantics. On the otherhand it does take advantage
> of the fact that no virtualization has to take place until a "container"
> has been migrated, thus rendering most of the vpid<->pid calls to be
> noops.
It has some other additional advantages:
- flexible: you can select full isolation or weak is required. I really
believe this is very important.

> The container is just an umbrella object that ties every "virtualized"
> subsystem
> together.
Yep. And containers were what I wanted to start with actually. Not VPIDs.

Kirill
Re: [PATCH 1/4] Virtualization/containers: introduction [message #1432 is a reply to message #1425] Wed, 08 February 2006 15:42 Go to previous messageGo to next message
dev is currently offline  dev
Messages: 1693
Registered: September 2005
Location: Moscow
Senior Member

> My point was to mainly identify the performance culprits and provide
> an direct access to those "namespaces" for performance reasons.
> So we all agreed on that we need to do that..
After having looked at Eric's patch, I can tell that he does all these
dereferences in quite the same amount.

For example, lot's of skb->sk->host->...
while in OpenVZ it would be econtainer()->... which is essentially
current->container->...

So until someone did measurements it looks doubtfull that one solution
is better than the another.

> Question now (see other's note as well), should we provide
> a pointer to each and every namespace in struct task.
> Seem rather wasteful to me as certain path/namespaces are not
> exercise heavily.

> Having one object "struct container" that still embodies all
> namespace still seems a reasonable idea.
> Otherwise identifying the respective namespace of subsystems will
> have to go through container->init->subsys_namespace or similar.
> Not necessarily bad either..

why not simply container->subsys_namespace?

Kirill
Re: The issues for agreeing on a virtualization/namespaces implementation. [message #1436 is a reply to message #1430] Wed, 08 February 2006 15:57 Go to previous messageGo to next message
Hubertus Franke is currently offline  Hubertus Franke
Messages: 16
Registered: February 2006
Junior Member
Kirill Korotaev wrote:
>>> Eric W. Biederman wrote:
>>> So it seems the clone( flags ) is a reasonable approach to create new
>>> namespaces. Question is what is the initial state of each namespace?
>>> In pidspace we know we should be creating an empty pidmap !
>>> In network, someone suggested creating a loopback device
>>> In uts, create "localhost"
>>> Are there examples where we rather inherit ? Filesystem ?
>>
>> Of course filesystem is already implemented, and does inheret a full
>> copy.
>
>
> why do we want to use clone()? Just because of its name and flags?
> I think it is really strange to fork() to create network context. What
> has process creation has to do with it?
>
> After all these clone()'s are called, some management actions from host
> system are still required, to add these IPs/routings/etc.
> So? Why mess it up? Why not create a separate clean interface for
> container management?
>
> Kirill
>

We need a "init" per container, which represents the context of the
system represented by the container.
If that is the case, then why not create the container such that
we specify what namespaces need to be new for a container at
the container creation time and initialize them to a well understood
state that makes sense (e.g. copy namespace (FS, uts) , new fresh state (pid) ).

Then use the standard syscall to modify state (now "virtualized" through
the task->xxx_namespace access ).

Do you see a need to change the namespace of a container after it
has been created. I am not referring to the state of the namespace
but truely moving to a completely different namespace after the
container has been created.

Obviously you seem to have some other usage in mind, beyond what my
limited vision can see. Can you share some of those examples, because
that would help this discussion along ...

Thanks a 10^6.

-- Hubertus
Re: [PATCH 1/4] Virtualization/containers: introduction [message #1438 is a reply to message #1432] Wed, 08 February 2006 16:39 Go to previous messageGo to next message
ebiederm is currently offline  ebiederm
Messages: 1354
Registered: February 2006
Senior Member
Kirill Korotaev <dev@sw.ru> writes:

>> My point was to mainly identify the performance culprits and provide
>> an direct access to those "namespaces" for performance reasons.
>> So we all agreed on that we need to do that..
> After having looked at Eric's patch, I can tell that he does all these
> dereferences in quite the same amount.
>
> For example, lot's of skb->sk->host->...
> while in OpenVZ it would be econtainer()->... which is essentially
> current->container->...

Except at that point in time I cannot use current, because it
does not have necessarily have the appropriate context. So doubt
you could correctly use econtainer there.

> So until someone did measurements it looks doubtfull that one solution is better
> than the another.

I agree, exactly where we look is a minor matter, unless we can find
an instance where there are good reasons for preferring one representation
over another. Performance currently does not look a sufficient reason.


My basis for preferring a flat layout is essentially because
that is how other similar interfaces are currently implemented
in the kernel.

In linux we have a plan9 inspired system call called clone.
One of the ideas with clone is that when you create a new
task you either share or you don't share resources. Namespaces
are one of those resources.

Since we are dealing with concepts that appear to fit the
existing model beautifully my feeling is to use the existing
model of how things are currently implemented in the kernel.


The only other reason I can think of is namespace implementation
independence. If we have an additional structure that we collect
up the pointers it feels like it causes unnecessary tying.



The best justification I can think of for putting it all in
one structure seems like the current->container current->econtainer
thing that comes out of OpenVZ. Currently I have followed the
threads enough to know it exists but I yet understand it.

If there is a good reason for the container/econtainer thing I can
certainly seem some benefit.

Eric
Re: The issues for agreeing on a virtualization/namespaces implementation. [message #1439 is a reply to message #1430] Wed, 08 February 2006 16:48 Go to previous messageGo to next message
ebiederm is currently offline  ebiederm
Messages: 1354
Registered: February 2006
Senior Member
Kirill Korotaev <dev@sw.ru> writes:

>>>Eric W. Biederman wrote:
>>>So it seems the clone( flags ) is a reasonable approach to create new
>>>namespaces. Question is what is the initial state of each namespace?
>>>In pidspace we know we should be creating an empty pidmap !
>>>In network, someone suggested creating a loopback device
>>>In uts, create "localhost"
>>>Are there examples where we rather inherit ? Filesystem ?
>> Of course filesystem is already implemented, and does inheret a full
>> copy.
>
> why do we want to use clone()? Just because of its name and flags?
> I think it is really strange to fork() to create network context. What has
> process creation has to do with it?

Agreed. Although clones brother unshare takes process creation out of the
picture, but otherwise preserves the same interface.

> After all these clone()'s are called, some management actions from host system
> are still required, to add these IPs/routings/etc.
> So? Why mess it up? Why not create a separate clean interface for container
> management?

If we need additional arguments besides create the thing. We have a clear
argument that clone is completely the wrong interface.

However. So far I have not seen an instance where using the existing
standard configuration mechanisms from inside the namespace is not the
proper way to set things up. The only thing I know that needs to happen from
outside is to pass the container a network interface. And if it is a physical
interface that is all that must happen.

Eric
Re: [PATCH 1/4] Virtualization/containers: introduction [message #1443 is a reply to message #1431] Wed, 08 February 2006 17:16 Go to previous messageGo to next message
ebiederm is currently offline  ebiederm
Messages: 1354
Registered: February 2006
Senior Member
Kirill Korotaev <dev@sw.ru> writes:

> Additional negative sides:
> - full isolation can be inconvinient from containers management point of
> view. You will need to introduce new modified tools such as top/ps/kill and many
> many others. You won't be able to strace/gdb processes from the host also.

Ok. I have to pick a nit here. Needing all new tools top/ps/kill was an artifact
of your implementation. Mine does not suffer from it.

> - overhead when virtualization is off, result is not the same.
> - additional args everywhere (stack usage, etc.)

Agreed. When using a PID namespace the code always behaves the same.
Which is with more arguments than the code used to have.
However the code always behaving the same is a tremendous advantage
for maintainability.

>> The vpid approach has the drawbacks of having to identify the conversion spots
>> of all vpid vs. pid semantics. On the otherhand it does take advantage
>> of the fact that no virtualization has to take place until a "container"
>> has been migrated, thus rendering most of the vpid<->pid calls to be
>> noops.
> It has some other additional advantages:
> - flexible: you can select full isolation or weak is required. I really believe
> this is very important.

I am willing to be convinced but so far there seem to be other solutions
like running gdb_stubs inside your container. To address most of the issues.

In a fairly real sense mucking with a container from the outside appears
to be a sysadmin layering violation.

>> The container is just an umbrella object that ties every "virtualized"
>> subsystem
>> together.
> Yep. And containers were what I wanted to start with actually. Not VPIDs.

The historical linux approach is to build things out of tasks sharing
resources that give the impression of containers not out containers
and their children. I am still trying to understand why that model
does not work in this instance.

Eric
Re: The issues for agreeing on a virtualization/namespaces implementation. [message #1444 is a reply to message #1428] Wed, 08 February 2006 17:46 Go to previous messageGo to next message
ebiederm is currently offline  ebiederm
Messages: 1354
Registered: February 2006
Senior Member
Hubertus Franke <frankeh@watson.ibm.com> writes:

>
> So it seems the clone( flags ) is a reasonable approach to create new
> namespaces. Question is what is the initial state of each namespace?
> In pidspace we know we should be creating an empty pidmap !
> In network, someone suggested creating a loopback device
> In uts, create "localhost"
"localhost" is wrong. Either copy or set it to the value from
system initialization time.

> Are there examples where we rather inherit ? Filesystem ?
> Can we iterate the assumption for each subsystem what people thing is right?

I think this needs to happen on a pure subsystem basis as we merge the
patches.

> IMHO, there is only a need to refer to a namespace from the global context.
> Since one will be moving into a new container, but getting out of one
> could be prohibitive (e.g. after migration)
> It does not make sense therefore to know the name of a namespace in
> a different container.
>
> The example you used below by using the pid comes natural, because
> that already limits visibility.
>
> I am still struggling with why we need new sys_calls.
> sys_calls already exist for changing certain system parameters (e.g. utsname )
> so to me it boils down to identifying a proper initial state when the
> namespace is created.

Agreed. But we can't always count on everything having a useful state
that we can modify from the inside. So it is important to leave the
option open at least for those case.

And again there is always the idea that by adding flags we will
transform fork or fork+exec into a spaghetti system call. I think
that is a reasonable concern.

Also there is always the danger that we run out of clone flags.

>> What I have done which seems easier than creating new names is to refer
>> to the process which has the namespace I want to manipulate.
>
> Is then the idea to only allow the container->init to manipulate
> or is there need to allow other priviliged processes to perform namespace
> manipulation?
> Also after thinking about it.. why is there a need to have an external name
> for a namespace ?

Largely it connects to the super chroot usage where you have one
sysadmin he has multiple daemons each running in their own environment
for isolation purposes. Nothing is installed in the chroot so an
attacker that gets in cannot do anything.

>>>>6) How do we do all of this efficiently without a noticeable impact on
>>>> performance?
>>>> - I have already heard concerns that I might be introducing cache
>>>> line bounces and thus increasing tasklist_lock hold time.
>>>> Which on big way systems can be a problem.
>>>
>>>Possible to split the lock up now.. one for each pidspace ?
>> At the moment it is worth thinking about. If the problem isn't
>> so bad that people aren't actively working on it we don't have to
>> solve the problem for a little while, just be aware of it.
>>
>
> Agree, just need to be sure we can split it up. But you already keep
> a task list per pid-namespace, so there should be no problem IMHO.
> If so let's do it now and take it of the table it its as simple as
>
> task_list_lock ::= pspace->task_list_lock

Actually I don't although that could be trivial. But it is the
wrong split. The problem is that it is a lock with global effect.

Eric
Re: The issues for agreeing on a virtualization/namespaces implementation. [message #1446 is a reply to message #1396] Wed, 08 February 2006 18:31 Go to previous messageGo to next message
Hubertus Franke is currently offline  Hubertus Franke
Messages: 16
Registered: February 2006
Junior Member
Serge E. Hallyn wrote:
> Quoting Hubertus Franke (frankeh@watson.ibm.com):
>
>>IMHO, there is only a need to refer to a namespace from the global context.
>>Since one will be moving into a new container, but getting out of one
>>could be prohibitive (e.g. after migration)
>>It does not make sense therefore to know the name of a namespace in
>>a different container.
>
>
> Not sure I agree. What if we are using a private namespace for a
> vserver, and then we want to create a private namespace in there for a
> mobile application. Since we're talking about nested namespaces, this
> should be possible.
>
> Now I believe Eric's code so far would make it so that you can only
> refer to a namespace from it's *creating* context. Still restrictive,
> but seems acceptable.
>

That's what I meant .. as usually used the wrong word..
s/global context/spawing context/g .. because that's the only
place where you have a pid to refer to the newly created container !

> (right?)
>

Yes, seriii .. ahmm serue
> -serge
>
Re: The issues for agreeing on a virtualization/namespaces implementation. [message #1447 is a reply to message #1436] Wed, 08 February 2006 19:02 Go to previous messageGo to next message
Herbert Poetzl is currently offline  Herbert Poetzl
Messages: 239
Registered: February 2006
Senior Member
On Wed, Feb 08, 2006 at 10:57:24AM -0500, Hubertus Franke wrote:
>Kirill Korotaev wrote:
>>>>Eric W. Biederman wrote:
>>>>So it seems the clone( flags ) is a reasonable approach to create new
>>>>namespaces. Question is what is the initial state of each namespace?
>>>>In pidspace we know we should be creating an empty pidmap !
>>>>In network, someone suggested creating a loopback device
>>>>In uts, create "localhost"
>>>>Are there examples where we rather inherit ? Filesystem ?
>>>
>>>Of course filesystem is already implemented, and does inheret a full
>>>copy.

I try to comment on both mails here because I thing that
clone() is basically a good interface, but will require
some redesign and/or extension ...

>> why do we want to use clone()?

because it is a natural and existing interface for this purpose
at least in Linux-VServer it would work (to some extend). why?

because we aleady use tools like chbind and chcontext which
do similar things as chroot, and chroot could, in theory, use
clone() and rbind to do it's job ...

>> Just because of its name and flags?

extending the flags seems natural to me, but the problem might
actually be that there are not enough of them left

>> I think it is really strange to fork() to create network context.

if you look at it as network namespace and sharing existing
spaces and/or creating new ones, then clone() and unshare()
make pretty much sense there ...

>> What has process creation has to do with it?

it is a natural interface where you can decide whether to
share a space or acquire a new one ... IMHO it would make
sense to get trivial userspace tools to create those
new spaces in one go, so that the user can use those
'building blocks' to create new spaces whenever she needs

>> After all these clone()'s are called, some management actions
>> from host system are still required, to add these IPs/routings/etc.

not necessarily, for example Linux-VServer uses some kind
of 'priviledged' mode, in which the initial guest process
can modify various things (like in this case the networking)
and setup whatever is required, then, shortly after, giving
up those priviledges ...

>> So? Why mess it up?
>> Why not create a separate clean interface for container management?

I'm not against a clean interface at all, but how would
such a 'clean' interface look like?

- a complicated sysfs interface where you write strange
values into even stranger places?
- 40 different syscalls to do stuff like adding or removing
various parts from the spaces?
- a new ioctl for processes?

>> Kirill
>
> We need a "init" per container, which represents the context of the
> system represented by the container.

that's only partially true, for example Linux-VServer also
allows for light-weight guests/containers which do not
have a separate init process, just a 'fake' one, so we can
save the resources consumed by a separate init process ...

it turns out that this works perfectly fine, even without
that fake init, if you teach a few tools like pstree that
they should not blindly assume that there is a pid=1 :)

> If that is the case, then why not create the container such that
> we specify what namespaces need to be new for a container at the
> container creation time and initialize them to a well understood
> state that makes sense (e.g. copy namespace (FS, uts) , new fresh
> state (pid) ).

agreed, but now comes the interesting part, how does such
a well understood state look like for all contexts?

obviously the name space makes a complete copy, leading to
various issues when you 'try' to get rid of the 'copied'
data, like host filesystem mount points and such ...

removing the mounts 'above' a certain chroot() path might
seem like a good solution here, but actually it will cause
issues when you want to maintain/access a guest/container
from the host/parent

leaving all mounts in place will require to either inherit
changes from the host/parent to all guests/containers, just
to avoid having e.g. /mnt/cdrom mounted in process A, which
does not even see it (as it's accessible space starts
somewhere else) and therefore being unable to eject the
thing, although it is obviously unused

> Then use the standard syscall to modify state (now "virtualized"
> through the task->xxx_namespace access ).

works as long as you have a handle for that, and actually
you do not need one init per guest/container, you need
one 'uniquely identified' task 'outside' the container

which in the typical case already makes two of them, the
'handle' task outside and the 'init' task inside ...

> Do you see a need to change the namespace of a container after it
> has been created. I am not referring to the state of the namespace
> but truely moving to a completely different namespace after the
> container has been created.

container itself, probably not, task, definitely yes ...
i.e. you definitely want to move between all those spaces
given that you are sufficiently priviledged, which is a
completely different can of worms ...

> Obviously you seem to have some other usage in mind, beyond what my
> limited vision can see. Can you share some of those examples, because
> that would help this discussion along ...

I guess one case where the separate container setup is
desired is when you want to keep a container alive even
after the processes have seized to exist. for example to
visit a 'stopped' guest/context just to do some emergency
operations or install some files/packages/etc

IMHO this can easily be solved by keeping the 'handle'
process (or whatever handle may be used for the complete
context consisting of all spaces) around, even if no
process is using those spaces ...

the context as 'collection' of certain namespaces is
definitely something which will be required to allow to
'move' into a specific container from the host/parent
side, as the parent process obviously does not hold that
information.

best,
Herbert

> Thanks a 10^6.
>
> -- Hubertus
>
>
>
Re: The issues for agreeing on a virtualization/namespaces implementation. [message #1449 is a reply to message #1396] Wed, 08 February 2006 20:21 Go to previous messageGo to next message
Dave Hansen is currently offline  Dave Hansen
Messages: 240
Registered: October 2005
Senior Member
On Wed, 2006-02-08 at 12:03 -0600, Serge E. Hallyn wrote:
> Now I believe Eric's code so far would make it so that you can only
> refer to a namespace from it's *creating* context. Still restrictive,
> but seems acceptable.

The same goes for filesystem namespaces. You can't see into random
namespaces, just the ones underneath your own. Sounds really reasonable
to me.

-- Dave
Re: [PATCH 1/4] Virtualization/containers: introduction [message #1451 is a reply to message #1431] Wed, 08 February 2006 20:43 Go to previous messageGo to next message
Dave Hansen is currently offline  Dave Hansen
Messages: 240
Registered: October 2005
Senior Member
On Wed, 2006-02-08 at 18:36 +0300, Kirill Korotaev wrote:
> - full isolation can be inconvinient from containers management point of
> view. You will need to introduce new modified tools such as top/ps/kill
> and many many others. You won't be able to strace/gdb processes from the
> host also.

I'd like to put a theory out there: the more isolation we perform, the
easier checkpointing and migration become to guarantee.

Agree? Disagree?

But, full isolation is hard to code. The right approach is very likely
somewhere in the middle where we require some things to happen
underneath us. For instance, requiring that the filesystem be made
consistent if a container is moved across systems.

-- Dave
Re: [PATCH 1/4] Virtualization/containers: introduction [message #1452 is a reply to message #1451] Wed, 08 February 2006 21:04 Go to previous messageGo to next message
ebiederm is currently offline  ebiederm
Messages: 1354
Registered: February 2006
Senior Member
Dave Hansen <haveblue@us.ibm.com> writes:

> On Wed, 2006-02-08 at 18:36 +0300, Kirill Korotaev wrote:
>> - full isolation can be inconvinient from containers management point of
>> view. You will need to introduce new modified tools such as top/ps/kill
>> and many many others. You won't be able to strace/gdb processes from the
>> host also.
>
> I'd like to put a theory out there: the more isolation we perform, the
> easier checkpointing and migration become to guarantee.
>
> Agree? Disagree?

Agree. But that does not address the reasons OpenVZ and Vserver exist.

> But, full isolation is hard to code.

Disagree. If you limit your self to just changing the code that
translates from names to objects it is a very narrow slice of code,
and there are very few surprises. There is a lot of grunt work involved
but it easy to tell if you got everything and did it correctly.

Other approaches are more adhoc, take short cuts, and seem prone to missing
the corner cases.

> The right approach is very likely
> somewhere in the middle where we require some things to happen
> underneath us. For instance, requiring that the filesystem be made
> consistent if a container is moved across systems.

Possibly. That is very out from where we are at the moment.
Let's get the isolation and see where we are at.

Eric
Re: The issues for agreeing on a virtualization/namespaces implementation. [message #1454 is a reply to message #1428] Wed, 08 February 2006 22:28 Go to previous messageGo to next message
ebiederm is currently offline  ebiederm
Messages: 1354
Registered: February 2006
Senior Member
Hubertus Franke <frankeh@watson.ibm.com> writes:

> Eric W. Biederman wrote:
>> Hubertus Franke <frankeh@watson.ibm.com> writes:
>>
>>>Eric W. Biederman wrote:
>>>
>>
>>>>3) How do we refer to namespaces and containers when we are not members?
>>>> - Do we refer to them indirectly by processes or other objects that
>>>> we can see and are members?
>>>> - Do we assign some kind of unique id to the containers?
>>>
>>>
>> What I have done which seems easier than creating new names is to refer
>> to the process which has the namespace I want to manipulate.
>
> Is then the idea to only allow the container->init to manipulate
> or is there need to allow other priviliged processes to perform namespace
> manipulation?
> Also after thinking about it.. why is there a need to have an external name
> for a namespace ?

There are several cases.

Passing network devices to a childs namespace, as usually
the loopback interface is not enough.

Monitoring the namespace from outside, so among other things
you aren't required to checkpoint and migrate your monitoring
daemon.

There are several other control and monitoring operations
that I am not quite as familiar. One of them is the
vserver idea of entering a guest.

To expand on things a little bit. If we have interfaces
that take strings we can refer to an arbitrary child process
as pid/pid/pid/.... So we should not be limited to what
is at the init of the container. If that proves desirable.

Permissions checks for most of these operations require some
serious thinking before they are merged.

Eric
Re: [PATCH 1/4] Virtualization/containers: introduction [message #1457 is a reply to message #1331] Thu, 09 February 2006 00:24 Go to previous messageGo to next message
ebiederm is currently offline  ebiederm
Messages: 1354
Registered: February 2006
Senior Member
Kirill Korotaev <dev@openvz.org> writes:

> Hello,
>
> I tried to take into account all the comments from you guys (thanks a lot for
> them!) and prepared a new version of virtualization patches. I will send only 4
> patches today, just not to overflow everyone and keep it clear/tidy/possible to
> review.
>
> This patch introduces some abstract container kernel structure and a number of
> operations on it.
>
> The important properties of the proposed container implementation:
> - each container has unique ID in the system
> - each process in the kernel can belong to one container only
> - effective container pointer (econtainer()) is used on the task to avoid
> insertion of additional argument "container" to all functions where it is
> required.
> - kernel compilation with disabled virtualization should result in old good
> linux kernel
>
> Patches following this one will be used for virtualization of the kernel
> resources based on this container infrastructure, including those VPID patches I
> sent before. Every virtualized resource can be given separate config option if
> needed (just give me to know if it is desired).

After having digested this I think there is something sane with regard to
this container idea. I still think the implementation is totally wrong but
there is a potential problem the basic idea solves.

In the traditional linux/plan9 style of namespaces the question is
which resources different tasks share, and we pass clone bits to determine
what we want to share and what we don't want to share. Semantically this
is very clean and allows for a great deal of flexibility. However
as the flexibility increases we get more code in do_fork more
reference counts and ultimately the performance decreases. In addition
it is not common for us to change which resources we share.

So our traditional route is flexible but it does not optimize the common
case where we share all of the same things. Containers can potentially
optimize that case. So long as anyone can create a new container even
someone inside a container we do not loose flexibility.

To deal with networking there are currently a significant number of
variables with static storage duration. Making those variables global
and placing them in structures is neither as efficient as it could be
nor is it as maintainable as it should be. Other subsystems have
similar problems.

So if we put econtainer in struct thread_info and optimize it like we
do current, create an interface to variables similar to
DEFINE_PER_CPU, create a syscall interface similar to clone, and
show that by doing this we don't loose flexibility, then it looks like
a good idea.

If we can't do better than the current clone model of shared resources
then this model feels like gratuitous change.

Eric
Re: The issues for agreeing on a virtualization/namespaces implementation. [message #1462 is a reply to message #1428] Wed, 08 February 2006 15:17 Go to previous messageGo to next message
serue is currently offline  serue
Messages: 750
Registered: February 2006
Senior Member
Quoting Hubertus Franke (frankeh@watson.ibm.com):
> Eric W. Biederman wrote:
> So it seems the clone( flags ) is a reasonable approach to create new
> namespaces. Question is what is the initial state of each namespace?
> In pidspace we know we should be creating an empty pidmap !
> In network, someone suggested creating a loopback device
> In uts, create "localhost"
> Are there examples where we rather inherit ? Filesystem ?

Of course filesystem is already implemented, and does inheret a full
copy.

-serge
Re: The issues for agreeing on a virtualization/namespaces implementation. [message #1463 is a reply to message #1428] Wed, 08 February 2006 18:03 Go to previous messageGo to next message
serue is currently offline  serue
Messages: 750
Registered: February 2006
Senior Member
Quoting Hubertus Franke (frankeh@watson.ibm.com):
> IMHO, there is only a need to refer to a namespace from the global context.
> Since one will be moving into a new container, but getting out of one
> could be prohibitive (e.g. after migration)
> It does not make sense therefore to know the name of a namespace in
> a different container.

Not sure I agree. What if we are using a private namespace for a
vserver, and then we want to create a private namespace in there for a
mobile application. Since we're talking about nested namespaces, this
should be possible.

Now I believe Eric's code so far would make it so that you can only
refer to a namespace from it's *creating* context. Still restrictive,
but seems acceptable.

(right?)

-serge
Re: The issues for agreeing on a virtualization/namespaces implementation. [message #1464 is a reply to message #1449] Wed, 08 February 2006 21:22 Go to previous messageGo to next message
serue is currently offline  serue
Messages: 750
Registered: February 2006
Senior Member
Quoting Dave Hansen (haveblue@us.ibm.com):
> On Wed, 2006-02-08 at 12:03 -0600, Serge E. Hallyn wrote:
> > Now I believe Eric's code so far would make it so that you can only
> > refer to a namespace from it's *creating* context. Still restrictive,
> > but seems acceptable.
>
> The same goes for filesystem namespaces. You can't see into random
> namespaces, just the ones underneath your own. Sounds really reasonable
> to me.

Hmmm? I suspect I'm misreading what you're saying, but to be clear:

Let's say I start a screen session. In one of those shells, I clone,
specify CLONE_NEWNS, and exec a shell. now i do a bunch of mounting.
Other shells in the screen session won't see the results of those
mounts, and if i ctrl-d, the shell which started the screen session
can't either. Each of these is in the "parent filesystem namespace".

OTOH, shared subtrees specified in the parent shell could make it such
that the parent ns, but not others, see the results. Is that what
you're referring to?

thanks,
-serge
Re: [PATCH 1/4] Virtualization/containers: introduction [message #1465 is a reply to message #1457] Thu, 09 February 2006 02:18 Go to previous messageGo to next message
Jeff Dike is currently offline  Jeff Dike
Messages: 4
Registered: February 2006
Junior Member
On Wed, Feb 08, 2006 at 05:24:16PM -0700, Eric W. Biederman wrote:
> To deal with networking there are currently a significant number of
> variables with static storage duration. Making those variables global
> and placing them in structures is neither as efficient as it could be
> nor is it as maintainable as it should be. Other subsystems have
> similar problems.

BTW, there is another solution, which you may or may not consider to
be clean.

That is to load a separate copy of the subsystem (code and data) as a
module when you want a new instance of it. The code doesn't change,
but you probably have to move it around some and provide some sort of
interface to it.

I did this to the scheduler last year - see
http://marc.theaimsgroup.com/?l=linux-kernel&m=111404726 721747&w=2

Jeff
Re: [PATCH 1/4] Virtualization/containers: introduction [message #1468 is a reply to message #1465] Thu, 09 February 2006 03:16 Go to previous messageGo to next message
ebiederm is currently offline  ebiederm
Messages: 1354
Registered: February 2006
Senior Member
Jeff Dike <jdike@addtoit.com> writes:

> On Wed, Feb 08, 2006 at 05:24:16PM -0700, Eric W. Biederman wrote:
>> To deal with networking there are currently a significant number of
>> variables with static storage duration. Making those variables global
>> and placing them in structures is neither as efficient as it could be
>> nor is it as maintainable as it should be. Other subsystems have
>> similar problems.
>
> BTW, there is another solution, which you may or may not consider to
> be clean.
>
> That is to load a separate copy of the subsystem (code and data) as a
> module when you want a new instance of it. The code doesn't change,
> but you probably have to move it around some and provide some sort of
> interface to it.

There are some real drawbacks to that. Basically as things are structured
you only want multiple copies of the user facing data structures. Pretty
much everything else really does remain the same.

> I did this to the scheduler last year - see
> http://marc.theaimsgroup.com/?l=linux-kernel&m=111404726 721747&w=2

I will take a look but I don't expect to find anything.

Eric
Re: The issues for agreeing on a virtualization/namespaces implementation. [message #1471 is a reply to message #1396] Thu, 09 February 2006 05:41 Go to previous messageGo to next message
ebiederm is currently offline  ebiederm
Messages: 1354
Registered: February 2006
Senior Member
Kyle Moffett <mrmacman_g4@mac.com> writes:

> On Feb 07, 2006, at 17:06, Eric W. Biederman wrote:
>> I think I can boil the discussion down into some of the fundamental questions
>> that we are facing.
>>
>> Currently everyone seems to agree that we need something like my namespace
>> concept that isolates multiple resources.
>>
>> We need these for
>> UIDS
>> FILESYSTEM
>
> I have one suggestion for this (it also covers capabilities to a certain
> extent). Could we use the kernel credentials system to abstract away the
> concept of a single UID/GID? We currently have uid, euid, gid, egid, groups,
> fsid. I'm thinking that there would be virtualized UID tables to determine
> ownership of processes/SHM/etc.
>
> Each process would have a (uid_container,uid) pair (or similar) as its "uid"
> and likewise for gid. Then the ability to send signals to any given
> (uid_container,uid) or (gid_container,gid) pair would be given by keys in the
> kernel keyring indexed by the "uid_container" part and containing the "uid"
> part (or maybe just a pointer).
>
> Likewise the filesystem access could be virtualized by using uid and gid keys
> in the kernel keyring indexed by vfsmount (Not superblock, so that it would be
> possible to have different UID representations on different mounts/parts of the
> same filesystem).
>
> I'm guessing that the performance implications of the above would not be quite
> so nice, as it would put a lot of code in the fastpath, but I would guess that
> it might be possible to use the existing fields for processes without any
> virtualization needs.

At least for signal sending it looks like it would be easier to just compare
the pointers to struct user. At least in that context it looks like it
would be as cheap as what we are doing now. I just don't know where
to find a struct user for the euid, or is it the normal uid.

Eric
Re: The issues for agreeing on a virtualization/namespaces implementation. [message #1473 is a reply to message #1396] Thu, 09 February 2006 04:45 Go to previous messageGo to next message
Kyle Moffett is currently offline  Kyle Moffett
Messages: 4
Registered: February 2006
Junior Member
On Feb 07, 2006, at 17:06, Eric W. Biederman wrote:
> I think I can boil the discussion down into some of the fundamental
> questions that we are facing.
>
> Currently everyone seems to agree that we need something like my
> namespace concept that isolates multiple resources.
>
> We need these for
> UIDS
> FILESYSTEM

I have one suggestion for this (it also covers capabilities to a
certain extent). Could we use the kernel credentials system to
abstract away the concept of a single UID/GID? We currently have
uid, euid, gid, egid, groups, fsid. I'm thinking that there would be
virtualized UID tables to determine ownership of processes/SHM/etc.

Each process would have a (uid_container,uid) pair (or similar) as
its "uid" and likewise for gid. Then the ability to send signals to
any given (uid_container,uid) or (gid_container,gid) pair would be
given by keys in the kernel keyring indexed by the "uid_container"
part and containing the "uid" part (or maybe just a pointer).

Likewise the filesystem access could be virtualized by using uid and
gid keys in the kernel keyring indexed by vfsmount (Not superblock,
so that it would be possible to have different UID representations on
different mounts/parts of the same filesystem).

I'm guessing that the performance implications of the above would not
be quite so nice, as it would put a lot of code in the fastpath, but
I would guess that it might be possible to use the existing fields
for processes without any virtualization needs.

Cheers,
Kyle Moffett

--
There is no way to make Linux robust with unreliable memory
subsystems, sorry. It would be like trying to make a human more
robust with an unreliable O2 supply. Memory just has to work.
-- Andi Kleen
Re: [PATCH 1/4] Virtualization/containers: introduction [message #1480 is a reply to message #1465] Thu, 09 February 2006 16:38 Go to previous messageGo to next message
Hubertus Franke is currently offline  Hubertus Franke
Messages: 16
Registered: February 2006
Junior Member
Jeff Dike wrote:
> On Wed, Feb 08, 2006 at 05:24:16PM -0700, Eric W. Biederman wrote:
>
>>To deal with networking there are currently a significant number of
>>variables with static storage duration. Making those variables global
>>and placing them in structures is neither as efficient as it could be
>>nor is it as maintainable as it should be. Other subsystems have
>>similar problems.
>
>
> BTW, there is another solution, which you may or may not consider to
> be clean.
>
> That is to load a separate copy of the subsystem (code and data) as a
> module when you want a new instance of it. The code doesn't change,
> but you probably have to move it around some and provide some sort of
> interface to it.
>
> I did this to the scheduler last year - see
> http://marc.theaimsgroup.com/?l=linux-kernel&m=111404726 721747&w=2
>
> Jeff
>


Jeff, interesting, but won't that post some serious scalability issue?
Imaging 100s of container/namespace ?

The namespace is mainly there to identify which data needs to be private
when multiple instances of a subsystem are considered and
encapsulate that data in an object/datastructure !

-- Hubertus
Re: [PATCH 1/4] Virtualization/containers: introduction [message #1483 is a reply to message #1480] Thu, 09 February 2006 17:47 Go to previous messageGo to next message
Jeff Dike is currently offline  Jeff Dike
Messages: 4
Registered: February 2006
Junior Member
On Thu, Feb 09, 2006 at 11:38:31AM -0500, Hubertus Franke wrote:
> Jeff, interesting, but won't that post some serious scalability issue?
> Imaging 100s of container/namespace ?

In terms of memory?

Running size on sched.o gives me this on x86_64:
text data bss dec hex filename
35685 6880 28800 71365 116c5 sched.o

and on i386 (actually UML/i386)

text data bss dec hex filename
10010 36 2504 12550 3106 obj/kernel/sched.o

I'm not sure why there's such a big difference, but 100 instances adds
a meg or two (or three) to the kernel. This overstates things a bit
because there are things in sched.c which wouldn't be duplicated, like
the system calls.

How big a deal is that on a system which you plan to have 100s of
containers on anyway?

It's heavier than your namespaces, but does have the advantage that it
imposes no cost when it's not being used.

> The namespace is mainly there to identify which data needs to be private
> when multiple instances of a subsystem are considered and
> encapsulate that data in an object/datastructure !

Sure, and that's a fine approach. It's just not the only one.

Jeff
Re: [PATCH 1/4] Virtualization/containers: introduction [message #1486 is a reply to message #1457] Thu, 09 February 2006 21:56 Go to previous messageGo to next message
ebiederm is currently offline  ebiederm
Messages: 1354
Registered: February 2006
Senior Member
Ok digesting this some more I don't see a glaring case in the code
where it is set_econtainer(...); do stuff ; restore_econtainer(..);
has a tremendous advantage at the moment. Plus it removes the
opportunity to catch old code that deals with multiple contexts at
compile time, and has to be done by pain staking code review.

That aside the truly important thing I realized is that for
the optimization effects it is an implementation detail. Batched
reference counting of shared resources is something we can
go back in and add at any time. It will take a little refactoring
to do but it something that can be done without changing the
user space API.

So since individual namespace pointers in the task_struct
are simpler and have the required flexibility. I think we
can table this part of the discussion for now.

The other detail I brought up which was most interesting was
the marking of specific variables that would go into a
container or namespace. Well as long as it is implemented on
the namespace level that is still possible, and may even
be beneficial.

Eric
Re: [PATCH 1/4] Virtualization/containers: introduction [message #1487 is a reply to message #1483] Thu, 09 February 2006 22:09 Go to previous messageGo to next message
Sam Vilain is currently offline  Sam Vilain
Messages: 73
Registered: February 2006
Member
Jeff Dike wrote:
> On Thu, Feb 09, 2006 at 11:38:31AM -0500, Hubertus Franke wrote:
>>Jeff, interesting, but won't that post some serious scalability issue?
>>Imaging 100s of container/namespace ?
> In terms of memory?
> Running size on sched.o gives me this on x86_64:
> text data bss dec hex filename
> 35685 6880 28800 71365 116c5 sched.o
>
> and on i386 (actually UML/i386)
>
> text data bss dec hex filename
> 10010 36 2504 12550 3106 obj/kernel/sched.o
>
> I'm not sure why there's such a big difference, but 100 instances adds
> a meg or two (or three) to the kernel. This overstates things a bit
> because there are things in sched.c which wouldn't be duplicated, like
> the system calls.
>
> How big a deal is that on a system which you plan to have 100s of
> containers on anyway?

Quite a big deal. You might have 2Gigs of main memory, but your CPU is
unlikely to be more than a Megabyte in close reach. A meg or two of
scheduler data and code means that your L1 and L2 cache will be cycling
every scheduler round; which is OK if you have very short runqueues but
as you get more and more processes it will really start to hurt.

Remember, systems today are memory bound and anything you can do to
reduce the amount of time the system sits around waiting for memory to
fetch, the better.

Compare that to the Token Bucket Scheduler of Linux-VServer; a tiny
struct for each process umbrella, that will generally fit in one or two
cachelines, to which the scheduling support adds four ints and a
spinlock. With this it achieves fair CPU scheduling between vservers.

Sam.
Re: The issues for agreeing on a virtualization/namespaces implementation. [message #1488 is a reply to message #1396] Thu, 09 February 2006 22:25 Go to previous messageGo to next message
ebiederm is currently offline  ebiederm
Messages: 1354
Registered: February 2006
Senior Member
ebiederm@xmission.com (Eric W. Biederman) writes:

> I think I can boil the discussion down into some of the fundamental
> questions that we are facing.
>
> Currently everyone seems to agree that we need something like
> my namespace concept that isolates multiple resources.
>
> We need these for
> PIDS
> UIDS
> SYSVIPC
> NETWORK
> UTSNAME
> FILESYSTEM
> etc.
>
> The questions seem to break down into:
> 1) Where do we put the references to the different namespaces?
> - Do we put the references in a struct container that we reference from
> struct task_struct?
> - Do we put the references directly in struct task_struct?

Answer in the task_struct. It is the simplest and most flexible
route and the other implementations are still possible.

> 2) What is the syscall interface to create these namespaces?
> - Do we add clone flags?
> (Plan 9 style)
> - Do we add a syscall (similar to setsid) per namespace?
> (Traditional unix style)?
> - Do we in addition add syscalls to manipulate containers generically?

The answer seems to be we decide on a per namespace basis with
additional syscalls being mandatory if we have any additional data to
pass.

> 3) How do we refer to namespaces and containers when we are not members?

I have seen no arguments against referring to namespaces or containers
by global ids. So it seems we do not need a container id.

> 4) How do we implement each of these namespaces?
> Besides being maintainable are there other constraints?

Largely quite. But I have not heard additional constraints.

> 5) How do we control the resource inside a namespace starting
> from a process that is outside of that namespace?
> - The filesystem mount namespace gave an interesting answer.
> So it is quite possible other namespaces will give
> equally interesting and surprising answers.

Not yet resolved, but a bit of speculation.

> 6) How do we do all of this efficiently without a noticeable impact on
> performance?
> - I have already heard concerns that I might be introducing cache
> line bounces and thus increasing tasklist_lock hold time.
> Which on big way systems can be a problem.

A little discussion. At the level of the last few cache line I
think this needs to be addressed when we merge. Simply not
messing up existing optimizations sounds like a good initial
target. Basically at this stage trying hard would be a
premature optimization.

> 7) How do we allow a process inside a container to create containers
> for it's children?
> - In general this is trivial but there are a few ugly issues
> here.

This look mostly like something to be discussed when we merge
namespaces. But as long as we keep it in mind it is easy.

Eric
Re: The issues for agreeing on a virtualization/namespaces implementation. [message #1700 is a reply to message #1403] Mon, 20 February 2006 12:08 Go to previous messageGo to next message
dev is currently offline  dev
Messages: 1693
Registered: September 2005
Location: Moscow
Senior Member

>> The questions seem to break down into:
>> 1) Where do we put the references to the different namespaces?
>> - Do we put the references in a struct container that we reference
>> from struct task_struct?
>> - Do we put the references directly in struct task_struct?
>
>
> You "cache" task_struct->container->hotsubsys under
> task_struct->hotsubsys.
> We don't change containers other then at clone time, so no coherency
> issue here !!!!
> Which subsystems pointers to "cache", should be agreed by the experts,
> but first approach should always not to cache and go through the container.
agreed. I see no much reason to cache it and make tons of the same
pointers in all the tasks. Only if needed.
Also, in OpenVZ container has many fields intergrated inside, so there
is no additional dereference, but task->container->subsys_field

>> 2) What is the syscall interface to create these namespaces?
>> - Do we add clone flags? (Plan 9 style)
> Like that approach .. flexible .. particular when one has well specified
> namespaces.
mmm, how do you plan to pass additional flags to clone()?
e.g. strong or weak isolation of pids?

another questions:
how do you plan to meet the dependancies between namespaces?
e.g. conntracks require netfilters to be initialized.
network requires sysctls and proc to be initialized and so on.
do you propose to track all this in clone()? huh...

>> - Do we add a syscall (similar to setsid) per namespace?
>> (Traditional unix style)?
can be so...

>> - Do we in addition add syscalls to manipulate containers generically?
>>
>> I don't think having a single system call to create a container and
>> a new
>> instance of each namespace is reasonable as that does not give us a
>> path into the future when we create yet another namespace.
>>
> Agreed.
why do you think so?
this syscalls will start handling this new namespace and that's all.
this is not different from many syscalls approach.

>> 4) How do we implement each of these namespaces?
>> Besides being maintainable are there other constraints?
>>
> Good question... at least with PID and FS two are there ..
>>
>> 6) How do we do all of this efficiently without a noticeable impact on
>> performance?
>> - I have already heard concerns that I might be introducing cache
>> line bounces and thus increasing tasklist_lock hold time.
>> Which on big way systems can be a problem.
this is nothing compared to hierarchy operations.
BTW, heirarchy also introduces complicated resource accounting,
sometimes making it even impossible.

Kirill
Re: The issues for agreeing on a virtualization/namespaces implementation. [message #1701 is a reply to message #1700] Mon, 20 February 2006 12:40 Go to previous messageGo to next message
Herbert Poetzl is currently offline  Herbert Poetzl
Messages: 239
Registered: February 2006
Senior Member
On Mon, Feb 20, 2006 at 03:11:32PM +0300, Kirill Korotaev wrote:
> >>The questions seem to break down into:
> >>1) Where do we put the references to the different namespaces?
> >> - Do we put the references in a struct container that we reference
> >>from struct task_struct?
> >> - Do we put the references directly in struct task_struct?
> >
> >
> >You "cache" task_struct->container->hotsubsys under
> >task_struct->hotsubsys.
> >We don't change containers other then at clone time, so no coherency
> >issue here !!!!
> >Which subsystems pointers to "cache", should be agreed by the experts,
> >but first approach should always not to cache and go through the container.
> agreed. I see no much reason to cache it and make tons of the same
> pointers in all the tasks. Only if needed.

> Also, in OpenVZ container has many fields intergrated inside, so there
> is no additional dereference, but task->container->subsys_field

as does Linux-VServer currently, but do you have
any proof that putting all the fields together in
one big structure actually has any (dis)advantage
over separate structures?

> >>2) What is the syscall interface to create these namespaces?
> >> - Do we add clone flags? (Plan 9 style)
> >Like that approach .. flexible .. particular when one has well
> >specified namespaces.
> mmm, how do you plan to pass additional flags to clone()?
> e.g. strong or weak isolation of pids?

do you really have to pass them at clone() time?
would shortly after be more than enough?
what if you want to change those properties later?

> another questions:
> how do you plan to meet the dependancies between namespaces?
> e.g. conntracks require netfilters to be initialized.
> network requires sysctls and proc to be initialized and so on.
> do you propose to track all this in clone()? huh...

this is missing isolation/virtualization, and I guess
it has to be done to make those spaces useful ...

> >> - Do we add a syscall (similar to setsid) per namespace?
> >> (Traditional unix style)?
> can be so...
>
> >> - Do we in addition add syscalls to manipulate containers
> >> generically?
> >>
> >> I don't think having a single system call to create a container
> >> and a new instance of each namespace is reasonable as that does
> >> not give us a path into the future when we create yet another
> >> namespace.
> >Agreed.
> why do you think so?

> this syscalls will start handling this new namespace and that's all.
> this is not different from many syscalls approach.

well, let's defer the 'how amny syscalls' issue to
a later time, when we know what we want to implement :)

> >>4) How do we implement each of these namespaces?
> >> Besides being maintainable are there other constraints?
> >>
> >Good question... at least with PID and FS two are there ..
> >>
> >>6) How do we do all of this efficiently without a noticeable impact on
> >> performance?
> >> - I have already heard concerns that I might be introducing cache
> >> line bounces and thus increasing tasklist_lock hold time.
> >> Which on big way systems can be a problem.

> this is nothing compared to hierarchy operations.
> BTW, heirarchy also introduces complicated resource accounting,
> sometimes making it even impossible.

well, depends how you do it ...

best,
Herbert

> Kirill
Re: The issues for agreeing on a virtualization/namespaces implementation. [message #1704 is a reply to message #1701] Mon, 20 February 2006 14:25 Go to previous messageGo to next message
dev is currently offline  dev
Messages: 1693
Registered: September 2005
Location: Moscow
Senior Member

> as does Linux-VServer currently, but do you have
> any proof that putting all the fields together in
> one big structure actually has any (dis)advantage
> over separate structures?
have no proof and don't mind if there are many pointers. Though this
doesn't look helpful to me as well.

>>mmm, how do you plan to pass additional flags to clone()?
>>e.g. strong or weak isolation of pids?
> do you really have to pass them at clone() time?
> would shortly after be more than enough?
> what if you want to change those properties later?
I don't think it is always suiatable to do configuration later.
We had races in OpenVZ on VPS create/stop against exec/enter etc. (even
introduced flag is_running). So I have some experience to believe it
will be painfull place.

>>this syscalls will start handling this new namespace and that's all.
>>this is not different from many syscalls approach.
> well, let's defer the 'how amny syscalls' issue to
> a later time, when we know what we want to implement :)
agreed.

Kirill
Re: The issues for agreeing on a virtualization/namespaces implementation. [message #1706 is a reply to message #1704] Mon, 20 February 2006 15:16 Go to previous messageGo to next message
Herbert Poetzl is currently offline  Herbert Poetzl
Messages: 239
Registered: February 2006
Senior Member
On Mon, Feb 20, 2006 at 05:26:13PM +0300, Kirill Korotaev wrote:
>> as does Linux-VServer currently, but do you have
>> any proof that putting all the fields together in
>> one big structure actually has any (dis)advantage
>> over separate structures?

> have no proof and don't mind if there are many pointers.
> Though this doesn't look helpful to me as well.

well, my point is just that we don't know yet
so we should not favor one over the other, just
because somebody did it like that and it didn't
hurt :)

>>> mmm, how do you plan to pass additional flags to clone()?
>>> e.g. strong or weak isolation of pids?

>> do you really have to pass them at clone() time?
>> would shortly after be more than enough?
>> what if you want to change those properties later?

> I don't think it is always suiatable to do configuration later.
> We had races in OpenVZ on VPS create/stop against exec/enter etc.
> (even introduced flag is_running).
> So I have some experience to believe it will be painfull place.

well, Linux-VServer uses a state called 'setup'
which allows to change all kinds of things before
the guest can be entered, this state is changed
as the last operation of the setup, which in turn
drops all the capabilities and makes the guest
visible to the outside ...

works quite well and seems to be free of those
races you mentioned ...

>>> this syscalls will start handling this new namespace and that's all.
>>> this is not different from many syscalls approach.
>> well, let's defer the 'how amny syscalls' issue to
>> a later time, when we know what we want to implement :)
> agreed.

btw, maybe it's just me, but would it be possible
to do the email quoting like this:

>>> Text

instead of

> >>Text

TIA,
Herbert

> Kirill
Re: The issues for agreeing on a virtualization/namespaces implementation. [message #1707 is a reply to message #1706] Mon, 20 February 2006 15:17 Go to previous message
Andi Kleen is currently offline  Andi Kleen
Messages: 33
Registered: February 2006
Member
If you guys want to continue to argue this point forever please drop
me from cc.

Thanks.

-Andi
Previous Topic: Versioning issue on vzquota-3.0.0-2
Next Topic: [NET][IA64] Unaligned access in sk_run_filter
Goto Forum:
  


Current Time: Sat Aug 02 21:51:42 GMT 2025

Total time taken to generate the page: 0.88836 seconds