OpenVZ Forum - RDF feed
type: channel
items: Re: Network virtualization/isolation
items: RE: Network virtualization/isolation
items: Re: Re: Network virtualization/isolation
items: Network virtualization/isolation

Re: Network virtualization/isolation
type: item
description: jamal <hadi@cyberus.ca> writes: > On Fri, 2006-27-10 at 11:10 +0200, Daniel Lezcano wrote: > >> No, it uses virtualization at layer 2 and I had already mention it >> before (see the first email of the thread), but thank you for the email >> thread pointer. > > > What would be really useful is someone takes the time and creates a > matrix of the differences between the implementations. > It seems there are quiet a few differences but without such comparison > (to which all agree to) it is hard to form an opinion without a document > of some form. > > For one, I am puzzled by the arguements about L2 vs L3 - Is this the > host side or inside the VE? > > If it is a discussion of the host side: > To me it seems it involves the classification of some packet header > arriving on a physical netdevice on the host side (irrelevant whether > they are L2 or L7) and reaching a decision to select some redirected to > virtual netdevice. There are two techniques in real use. - Bind/Accept filtering Which layer 3 addresses a socket can bind/accept are filtered, but otherwise the network stack remains unchanged. When your container/VE only has a single IP address this works great. When you get multiple IPs this technique starts to fall down because it is not obvious how to make this correctly handle wild card ip addresses. This technique also falls down because it is very hard to support raw IP packets. The major advantage of this approach is that it is insanely simple and cheap. When the discussion started this is what I called Layer 3, bind filtering is probably a more precise name. - Network object tagging Every network device, each socket, each routing table, each net filter table, everything but the packets themselves is associated with a single VE/container. In principle the network stack doesn't change except everything that currently access global variables gets an additional pointer indirection. To find where a packet is you must look at it's network device on ingress, and y…
creator: ebiederm
date: 2006-11-25T08:21:39-00:00

Re: Network virtualization/isolation
type: item
description: Daniel Lezcano <dlezcano@fr.ibm.com> writes: >> Then a matrix of how each requires what modifications in the network >> code. Of course all players need to agree that the description is >> accurate. >> Is there such a document? >> cheers, >> jamal > > Hi, > > the attached document describes the network isolation at the layer 2 and at the > layer 3, it presents the pros and cons of the different approaches, their common > points and the impacted network code. > I hope it will be helpful :) Roughly it is correctly but I the tradeoffs you describe are incorrect. > Isolating and virtualizing the network > -------------------------------------- > > Some definitions: > ----------------- > > isolation : This is a restrictive technique which divides a set of the > available system objects to smaller subsets assigned to a group of > processes. This technique ensures an application will use only a > subset of the system resources and will never access other > resources. > > virtualization : This technique gives the illusion to an application > that its owns all the system resources instead of a subset of them > provided by the isolation. > > container: it is the name of the base element which brings the > isolation and the virtualization where applications are running into. > > system container: operating system running inside a container. > > application container : application running inside a container. > > checkpoint/restart: take a snapshot of a container at a given time > and recreate the container from this snapshot. > > mobility: checkpoint/restart used to move a container to one host to > another host. > > ---------------------------- > > Actually, containers are being developed in the kernel with the > following functions : > > * separate the system resources between containers in order > to avoid an application, running into a container, to > access the resources outside the container. That > facilitates the resources management, ensures the > application is jailed…
creator: ebiederm
date: 2006-11-25T09:09:20-00:00

RE: Network virtualization/isolation
type: item
description: > -----Original Message----- > From: netdev-owner@vger.kernel.org > [mailto:netdev-owner@vger.kernel.org] On Behalf Of Eric W. Biederman > Then the question is how do we reduce the overhead when we > don't have enough physical network interfaces to go around. > My feeling is that we could push the work to the network > adapters and allow single physical network adapters to > support multiple network interfaces, each with a different > link-layer address. At which point the overhead is nearly > nothing and newer network adapters may start implementing > enough filtering in hardware to do all of the work for us. Correct, to a degree. There will be always a limit on the number of physical "channels" that a NIC can support, while keeping these channels fully independent and protected at the hw level. So, you will probably still need to implement the sw path, with the assumption that some containers (that care about performance) will get a separate NIC interface and avoid the overhead, and other containers will have to use the sw path. There are some multi-channel NICs shipping today so it would be possible to see the overhead between the two options (I suspect it will be quite noticeable), but for a general idea about what work could be pushed down to network adapters in the near future you can look at the pcisig.com I/O Virtualization Workgroup. Once the single root I/O Virtualization spec is completed, it is likely to be supported by several NIC vendors to provide multiple network interfaces on a single NIC that you are looking for. _______________________________________________ Containers mailing list Containers@lists.osdl.org https://lists.osdl.org/mailman/listinfo/containers
creator: Leonid Grossman
date: 2006-11-25T16:35:58-00:00

Re: Network virtualization/isolation
type: item
description: "Leonid Grossman" <Leonid.Grossman@neterion.com> writes: > > >> -----Original Message----- >> From: netdev-owner@vger.kernel.org >> [mailto:netdev-owner@vger.kernel.org] On Behalf Of Eric W. Biederman > >> Then the question is how do we reduce the overhead when we >> don't have enough physical network interfaces to go around. >> My feeling is that we could push the work to the network >> adapters and allow single physical network adapters to >> support multiple network interfaces, each with a different >> link-layer address. At which point the overhead is nearly >> nothing and newer network adapters may start implementing >> enough filtering in hardware to do all of the work for us. > > Correct, to a degree. > There will be always a limit on the number of physical "channels" that a > NIC > can support, while keeping these channels fully independent and > protected at the hw level. > So, you will probably still need to implement the sw path, > with the assumption that some containers (that care about performance) > will get a separate > NIC interface and avoid the overhead, and other containers will have to > use the sw path. > There are some multi-channel NICs shipping today so it would be possible > to see the overhead between the two options (I suspect it will be quite > noticeable), but for a general idea about what work could be pushed down > to network adapters in the near future you can look at the pcisig.com > I/O Virtualization Workgroup. > Once the single root I/O Virtualization spec is completed, it is likely > to be supported by several NIC vendors to provide multiple network > interfaces on a single NIC that you are looking for. Pushing it all of the way into the hardware is an optimization, that while great is likely not necessary. Simply doing a table lookup by link-level address and selecting between several network interfaces is enough to ensure we only traverse the network stack once. To keep overhead down in the container case I don't need the hard…
creator: ebiederm
date: 2006-11-25T19:26:45-00:00

RE: Network virtualization/isolation
type: item
description: > -----Original Message----- > From: Eric W. Biederman [mailto:ebiederm@xmission.com] > Sent: Saturday, November 25, 2006 11:27 AM > To: Leonid Grossman > Cc: hadi@cyberus.ca; Daniel Lezcano; Dmitry Mishin; Stephen > Hemminger; netdev@vger.kernel.org; Linux Containers > Subject: Re: Network virtualization/isolation > > "Leonid Grossman" <Leonid.Grossman@neterion.com> writes: > > > > > > >> -----Original Message----- > >> From: netdev-owner@vger.kernel.org > >> [mailto:netdev-owner@vger.kernel.org] On Behalf Of Eric W. > Biederman > > > >> Then the question is how do we reduce the overhead when we > don't have > >> enough physical network interfaces to go around. > >> My feeling is that we could push the work to the network > adapters and > >> allow single physical network adapters to support multiple network > >> interfaces, each with a different link-layer address. At > which point > >> the overhead is nearly nothing and newer network adapters > may start > >> implementing enough filtering in hardware to do all of the > work for > >> us. > > > > Correct, to a degree. > > There will be always a limit on the number of physical > "channels" that > > a NIC can support, while keeping these channels fully > independent and > > protected at the hw level. > > So, you will probably still need to implement the sw path, with the > > assumption that some containers (that care about > performance) will get > > a separate NIC interface and avoid the overhead, and other > containers > > will have to use the sw path. > > There are some multi-channel NICs shipping today so it would be > > possible to see the overhead between the two options (I suspect it > > will be quite noticeable), but for a general idea about what work > > could be pushed down to network adapters in the near future you can > > look at the pcisig.com I/O Virtualization Workgroup. > > Once the single root I/O Virtualization spec is completed, it is > > likely to be supported by several NIC vendors to provide mult…
creator: Leonid Grossman
date: 2006-11-25T22:17:03-00:00

Re: Network virtualization/isolation
type: item
description: "Leonid Grossman" <Leonid.Grossman@neterion.com> writes: > I did not mean kernel bypass, just L2 hw channels that for > all practical purposes act as separate NICs - > different MAC addresses, no blocking, independent reset, etc. Yes. Nearly all of what you need for safe kernel bypass. >> In the worst case I might need someone to go as far as the >> Grand Unified Lookup to remove all of the overheads. Except >> for distributing the work load more evenly across the machine >> with separate interrupts and the like I see no need for >> separate hardware channels to make things go fast for my needs. >> >> Despite the title of this thread there is no virtualization >> or emulation of the hardware involved. Just enhancements to >> the existing hardware abstractions. > > Right, I was just trying to say that IOV support (likely, from multiple > vendors since > virtualization is expected to be widely used) would provide an option to > export multiple > independent L2 interfaces from a single NIC - even if only a subset of > IOV functionality would be used in this case. Agreed, and I think I understood that. My basic point was that it doesn't look to me like I need the hardware support, just that I can use it when it is there. The core advantage I see of the multiple queues, is in being able to split the processing of network traffic and interrupts among multiple cores. Eric _______________________________________________ Containers mailing list Containers@lists.osdl.org https://lists.osdl.org/mailman/listinfo/containers
creator: ebiederm
date: 2006-11-25T23:16:06-00:00

Re: Network virtualization/isolation
type: item
description: On Sat, Nov 25, 2006 at 01:21:39AM -0700, Eric W. Biederman wrote: > > jamal <hadi@cyberus.ca> writes: > > > On Fri, 2006-27-10 at 11:10 +0200, Daniel Lezcano wrote: > > > >> No, it uses virtualization at layer 2 and I had already mention it > >> before (see the first email of the thread), but thank you for the email > >> thread pointer. > > > > > > What would be really useful is someone takes the time and creates a > > matrix of the differences between the implementations. > > It seems there are quiet a few differences but without such comparison > > (to which all agree to) it is hard to form an opinion without a document > > of some form. > > > > For one, I am puzzled by the arguements about L2 vs L3 - Is this the > > host side or inside the VE? > > > > If it is a discussion of the host side: > > To me it seems it involves the classification of some packet header > > arriving on a physical netdevice on the host side (irrelevant whether > > they are L2 or L7) and reaching a decision to select some redirected to > > virtual netdevice. > > There are two techniques in real use. > - Bind/Accept filtering > > Which layer 3 addresses a socket can bind/accept are filtered, > but otherwise the network stack remains unchanged. When your > container/VE only has a single IP address this works great. > When you get multiple IPs this technique starts to fall down because > it is not obvious how to make this correctly handle wild card ip > addresses. not really, you have to check for a (sub)set of IPs that's quite simple and not hard to get right, I agree that it increases the overhead on those checks, but this only occurs at bind/connect/accept time ... > This technique also falls down because it is very hard to > support raw IP packets. raw ip packets could be header checked in the same way but in general, this type of isolation does not make too much sense for raw ip purposes ... > The major advantage of this approach is that it is insanely simple > and cheap. and don't forge…
creator: Herbert Poetzl
date: 2006-11-26T18:34:02-00:00

Re: Network virtualization/isolation
type: item
description: Herbert Poetzl wrote: > On Sat, Nov 25, 2006 at 01:21:39AM -0700, Eric W. Biederman wrote: > > Then the question is how do we reduce the overhead when we don't have >> enough physical network interfaces to go around. My feeling is that >> we could push the work to the network adapters and allow single >> physical network adapters to support multiple network interfaces, each >> with a different link-layer address. >> > > that would be something interesting, but again, the > number of nics allowing for an arbitrary number of > filters, which also can be identified/correlated to > the network context without adding even more overhead > is probably insignificant ... so IMHO that would: > > - keep all interfaces in promisc mode > - check each packet for the set of MACs > > as the checks would require to identify the interface, > that would immediately result in O(N) overhead for > each packet received, plus the overhead added by > disabling the hardware filters ... but maybe that > changed over the years, I'm definitely no network > stack/device expert ... > This can be implemented similar to how MAC-VLANs are currently done (in my out-of-tree patch). There is a performance hit with lots of virtual interfaces (maybe 10% in some cases), but this is still greater than 500Mbps full-duplex on 2 ports on a modern dual-core machine. I don't even have hashing implemented, but it could be easily added and that should significantly decrease the search time from O(n) to something approaching O(1) in the normal case. This should also be an easy feature for NICs to add, and just as with 802.1Q VLANs, when hardware support is available, the features can migrate into the NIC, with the software mac-vlan logic handling generic hardware. In a switched environment, going into PROMISC mode should not add any significant overhead.. Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com _______________________________________________ Containers mailin…
creator: Ben Greear
date: 2006-11-26T19:41:50-00:00

Re: Network virtualization/isolation
type: item
description: Herbert Poetzl <herbert@13thfloor.at> writes: > On Sat, Nov 25, 2006 at 01:21:39AM -0700, Eric W. Biederman wrote: >> There are two techniques in real use. >> - Bind/Accept filtering >> >> Which layer 3 addresses a socket can bind/accept are filtered, >> but otherwise the network stack remains unchanged. When your >> container/VE only has a single IP address this works great. > >> When you get multiple IPs this technique starts to fall down because >> it is not obvious how to make this correctly handle wild card ip >> addresses. > > not really, you have to check for a (sub)set of IPs > that's quite simple and not hard to get right, I agree > that it increases the overhead on those checks, but > this only occurs at bind/connect/accept time ... The general problem is you get into mental model problems. You think you are isolated but you don't realize you can route packets over the loopback interface for example. But with care yes you can solve it. However while I think there is value in this technique it doesn't solve any of my problems, nor do I think it can be easily stretched to solve my problems. My gut feel for implementation still says this should be a new netfilter table that filters binds and accepts if we implement this. For most of us we need more power than we can get with the simple bind/accept filtering so we I think the network namespace work should concentrate on the general technique that gives us the entire power of the current network stack. At least until we have proved the overheads are unacceptable. >> Given that performance is the primary concern this is something a >> network stack expert might be able to help with. My gut feel is >> the extra pointer indirection for the more general technique is >> negligible and will not affect the network performance. The network >> stack can be very sensitive to additional cache misses so I could be >> wrong. Opinions? > > well, here we are talking about layer2 _isolation_ > if I got that right, i.e. you spl…
creator: ebiederm
date: 2006-11-26T20:52:14-00:00

Re: Network virtualization/isolation
type: item
description: Eric W. Biederman wrote: [ snip ] >> >> The packets arrive to the real device and go through the routes >> engine. From this point, the used route is enough to know to which >> container the traffic can go and the sockets subset assigned to the >> container. > > Note this has potentially the highest overhead of them all because > this is the only approach in which it is mandatory to inspect the > network packets to see which container they are in. If the container is in the route information, when you use the route, you have the container destination with it. I don't see the overhead here. > > My real problem with this approach besides seriously complicating > the administration by not delegating it is that you loose enormous > amounts of power. I don't understand why you say administration is more complicated. unshare -> ifconfig 1 container = 1 IP [ snip ] > So you have two columns that you rate these things that I disagree > with, and you left out what the implications are for code maintenance. > > 1) Network setup. > Past a certainly point both bind filtering and Daniel's L3 use a new > paradigm for managing the network code and become nearly impossible for > system administrators to understand. The classic one is routing packets > between machines over the loopback interface by accident. Huh? What is this new paradigm you are talking about ? > > The L2. Network setup iss simply the cost of setting up a multiple > machine network. This is more complicated but it is well understood > and well documented today. Plus for the common cases it is easy to > get a tool to automate this for you. When you get a complicated > network this wins hands down because the existing tools work and > you don't have to retrain your sysadmins to understand what is > happening. unshare -> (guest) add mac address (host) add mac address (guest) set ip address (host) set ip address (host) setup bridge 1 container = 2 net devices (root + guest), 2 IPs, 2 mac addresses, 1 bridge. 100 conta…
creator: Daniel Lezcano
date: 2006-11-28T14:15:26-00:00

Re: Network virtualization/isolation
type: item
description: I do not want to get into a big debate on the merits of various techniques at this time. We seem to be in basic agreement about what we are talking about. There is one thing I think we can all agree upon. - Everything except isolation at the network device/L2 layer, does not allow guests to have the full power of the linux networking stack. - There has been a demonstrated use for the full power of the linux networking stack in containers.. - There are a set of techniques which look as though they will give us full speed when we do isolation of the network stack at the network device/L2 layer. Is there any reason why we don't want to implement network namespaces without the full power of the linux network stack? If there is a case where we clearly don't want the full power of the linux network stack in a guest but we still need a namespace we can start looking at the merits of the alternatives. > What is this new paradigm you are talking about ? The basic point is this. The less like stock linux the inside of a container looks, and the more of a special case it is the more confusing it is. The classic example is that for a system container routing packets between containers over the loopback interface is completely unexpected. > There is not extra networking data structure instantiation in the > Daniel's L3. Nope just an extra field which serves the same purpose. >> - Bind/Connect/Accept filtering. There are so few places in >> the code this is easy to maintain without sharing code with >> everyone else. > > For isolation too ? Can we build network migration on top of that ? As long as you can take your globally visible network address with you when you migrate you can build network migration on top of it. So yes bind/accept filtering is sufficient to implement migration, if you are only using IP based protocols. Eric _______________________________________________ Containers mailing list Containers@lists.osdl.org https://lists.osdl.org/mailman/listinfo/containers
creator: ebiederm
date: 2006-11-28T16:51:57-00:00

Re: Network virtualization/isolation
type: item
description: On Tue, Nov 28, 2006 at 09:51:57AM -0700, Eric W. Biederman wrote: > > I do not want to get into a big debate on the merits of various > techniques at this time. We seem to be in basic agreement > about what we are talking about. > > There is one thing I think we can all agree upon. > - Everything except isolation at the network device/L2 layer, does not > allow guests to have the full power of the linux networking stack. > > - There has been a demonstrated use for the full power of the linux > networking stack in containers.. - There has been a demonstrated use for the full performance IP layer isolation too, both in BSD and Linux for several years now ... > - There are a set of techniques which look as though they will give > us full speed when we do isolation of the network stack at the > network device/L2 layer. > > Is there any reason why we don't want to implement network namespaces > without the full power of the linux network stack? duplicate negation -> "Is there any reason why we _want_ to implement network namespaces _with_ the full power of the linux network stack?" yes, I think you have some reasons for doing so, especially the migration part seems to depend on it OTOH, we _also_ want IP isolation, as it allows to separate services (and even handle overlapping sets) in a very natural (linux) way, without adding interfaces and virtual switches and bridges at a potentially high overhead just to do simple layer 3 isolation > If there is a case where we clearly don't want the full power of the > linux network stack in a guest but we still need a namespace we can > start looking at the merits of the alternatives. see above, of course, all cases can be 'simulated' by a fully blown layer 2 virtualization, so that's not an argument but OTOH, all this can also be achieved with Xen, so we could as well bring the argument, why have network namespaces at all, if you can get the same functionality (including the migration) with a Xen domU ... > > What is this new pa…
creator: Herbert Poetzl
date: 2006-11-28T17:37:19-00:00

Re: Network virtualization/isolation
type: item
description: Eric W. Biederman wrote: > I do not want to get into a big debate on the merits of various > techniques at this time. We seem to be in basic agreement > about what we are talking about. > > There is one thing I think we can all agree upon. > - Everything except isolation at the network device/L2 layer, does not > allow guests to have the full power of the linux networking stack. Agree. > > - There has been a demonstrated use for the full power of the linux > networking stack in containers.. Agree. > > - There are a set of techniques which look as though they will give > us full speed when we do isolation of the network stack at the > network device/L2 layer. Agree. > Is there any reason why we don't want to implement network namespaces > without the full power of the linux network stack? Don't make me wrong, I never said layer 2 should not be used. I am only arguing a layer 3 should use the mechanism provided by the layer 2 and use a subset of it like the sockets virtualization/isolation. Just IP isolation for lightweight containers, applications containers in order to have mobility. > If there is a case where we clearly don't want the full power of the > linux network stack in a guest but we still need a namespace we can > start looking at the merits of the alternatives. Dmitry and I, we are looking for a l3 based on a subset of the l2 and according with Herbert needs. If we can provide a l3 isolation based on the l2 which: - does not collide with l2 - fit the needs of Herbert - allows the migration - use common code between l2 and l3 Should it not be sufficient to justify to have a l3 with the l2 isolation ? >> What is this new paradigm you are talking about ? > > The basic point is this. The less like stock linux the inside of a > container looks, and the more of a special case it is the more > confusing it is. The classic example is that for a system container > routing packets between containers over the loopback interface is > completely unexpected. Right for …
creator: Daniel Lezcano
date: 2006-11-28T20:26:52-00:00

Re: Network virtualization/isolation
type: item
description: Daniel Lezcano <dlezcano@fr.ibm.com> writes: > Eric W. Biederman wrote: >> I do not want to get into a big debate on the merits of various >> techniques at this time. We seem to be in basic agreement >> about what we are talking about. >> >> There is one thing I think we can all agree upon. >> - Everything except isolation at the network device/L2 layer, does not >> allow guests to have the full power of the linux networking stack. > Agree. >> >> - There has been a demonstrated use for the full power of the linux >> networking stack in containers.. > Agree. >> >> - There are a set of techniques which look as though they will give >> us full speed when we do isolation of the network stack at the >> network device/L2 layer. > Agree. Herbert Poetzl <herbert@13thfloor.at> writes: > correct, don't get me wrong, I'm absolutely not against > layer 2 virtualization, but not at the expense of light- > weight layer 3 isolation, which _is_ the traditional way > 'containers' are built (see BSD, solaris ...) Ok. So on this point we agree. Full isolation at the network device/L2 level is desirable and no one is opposed to that. There is however a strong feeling especially for the case of application containers that something more focused on what a non-privileged process can use and deal with would be nice. The ``L3'' case. I agree that has potential but I worry about 2 things. - Premature optimization. - A poor choice of semantics. - Feature creep leading to insane semantics. I feel there is something in the L3 arguments as well and it sounds like it would be a good idea to flush out the semantics. For full network isolation we have the case that every process, every socket, and every network device belongs to a network namespace. This is enough to derive the network namespace for all other user visible data structures, and to a large extent to define their semantics. We still need a definition of the non-privileged case, that is compatible with the former definition. ..... What …
creator: ebiederm
date: 2006-11-28T21:50:03-00:00

Re: Network virtualization/isolation
type: item
description: On Tue, Nov 28, 2006 at 02:50:03PM -0700, Eric W. Biederman wrote: > Daniel Lezcano <dlezcano@fr.ibm.com> writes: > > > Eric W. Biederman wrote: > >> I do not want to get into a big debate on the merits of various > >> techniques at this time. We seem to be in basic agreement > >> about what we are talking about. > >> > >> There is one thing I think we can all agree upon. > >> - Everything except isolation at the network device/L2 layer, does not > >> allow guests to have the full power of the linux networking stack. > > Agree. > >> > >> - There has been a demonstrated use for the full power of the linux > >> networking stack in containers.. > > Agree. > >> > >> - There are a set of techniques which look as though they will give > >> us full speed when we do isolation of the network stack at the > >> network device/L2 layer. > > Agree. > > Herbert Poetzl <herbert@13thfloor.at> writes: > > correct, don't get me wrong, I'm absolutely not against > > layer 2 virtualization, but not at the expense of light- > > weight layer 3 isolation, which _is_ the traditional way > > 'containers' are built (see BSD, solaris ...) > Ok. So on this point we agree. Full isolation at the network device/L2 > level is desirable and no one is opposed to that. > There is however a strong feeling especially for the case of > application containers that something more focused on what a > non-privileged process can use and deal with would be nice. > The ``L3'' case. > I agree that has potential but I worry about 2 things. > - Premature optimization. > - A poor choice of semantics. > - Feature creep leading to insane semantics. > I feel there is something in the L3 arguments as well and it sounds > like it would be a good idea to flush out the semantics. > For full network isolation we have the case that every process, > every socket, and every network device belongs to a network namespace. > This is enough to derive the network namespace for all other user > visible data structures, and to a la…
creator: Herbert Poetzl
date: 2006-11-29T05:54:32-00:00

Re: Network virtualization/isolation
type: item
description: On Tue, Nov 28, 2006 at 09:26:52PM +0100, Daniel Lezcano wrote: > Eric W. Biederman wrote: > > I do not want to get into a big debate on the merits of various > > techniques at this time. We seem to be in basic agreement > > about what we are talking about. > > > > There is one thing I think we can all agree upon. > > - Everything except isolation at the network device/L2 layer, does not > > allow guests to have the full power of the linux networking stack. > Agree. > > > > - There has been a demonstrated use for the full power of the linux > > networking stack in containers.. > Agree. > > > > - There are a set of techniques which look as though they will give > > us full speed when we do isolation of the network stack at the > > network device/L2 layer. > Agree. > > > Is there any reason why we don't want to implement network namespaces > > without the full power of the linux network stack? > Don't make me wrong, I never said layer 2 should not be used. I am only > arguing a layer 3 should use the mechanism provided by the layer 2 and > use a subset of it like the sockets virtualization/isolation. > > Just IP isolation for lightweight containers, applications containers in > order to have mobility. > > > If there is a case where we clearly don't want the full power of the > > linux network stack in a guest but we still need a namespace we can > > start looking at the merits of the alternatives. > Dmitry and I, we are looking for a l3 based on a subset of the l2 and > according with Herbert needs. > If we can provide a l3 isolation based on the l2 which: > - does not collide with l2 > - fit the needs of Herbert > - allows the migration > - use common code between l2 and l3 > Should it not be sufficient to justify to have a l3 with the l2 > isolation? sounds good to me ... > >> What is this new paradigm you are talking about ? > > > > The basic point is this. The less like stock linux the inside of a > > container looks, and the more of a special case it is the more …
creator: Herbert Poetzl
date: 2006-11-29T05:58:57-00:00

Re: Network virtualization/isolation
type: item
description: Eric W. Biederman wrote: > I think for cases across network socket namespaces it should > be a matter for the rules, to decide if the connection should > happen and what error code to return if the connection does not > happen. > > There is a potential in this to have an ambiguous case where two > applications can be listening for connections on the same socket > on the same port and both will allow the connection. If that > is the case I believe the proper definition is the first socket > that we find that will accept the connection gets the connection. Wouldn't you want to catch this at bind() and/or configuration time and fail? Having overlapping namespaces/rules seems undesirable, since as Herbert said, can get you "unexpected behaviour". > I think with the appropriate set of rules it provides what is needed > for application migration. I.e. 127.0.0.1 can be filtered so that > you can only connect to sockets in your current container. > > It does get a little odd because it does allow for the possibility > that you can have multiple connected sockets with same source ip, > source port, destination ip, destination port. If the rules are > setup appropriately. I don't see that peculiarity being visible on > the outside network so it shouldn't be a problem. So if they're using the same protocol (eg TCP), how is it decided which one gets an incoming packet? Maybe I'm missing something as I don't understand your inside/outside network reference - is that to the loopback address comment in the previous paragraph? Thanks, -Brian _______________________________________________ Containers mailing list Containers@lists.osdl.org https://lists.osdl.org/mailman/listinfo/containers
creator: Brian Haley
date: 2006-11-29T20:21:00-00:00

Re: Re: Network virtualization/isolation
type: item
description: Brian Haley wrote: > Eric W. Biederman wrote: >> I think for cases across network socket namespaces it should >> be a matter for the rules, to decide if the connection should >> happen and what error code to return if the connection does not >> happen. >> >> There is a potential in this to have an ambiguous case where two >> applications can be listening for connections on the same socket >> on the same port and both will allow the connection. If that >> is the case I believe the proper definition is the first socket >> that we find that will accept the connection gets the connection. No. If you try to connect, the destination IP address is assigned to a network namespace. This network namespace is used to leave the listening socket ambiguity. > > Wouldn't you want to catch this at bind() and/or configuration time and > fail? Having overlapping namespaces/rules seems undesirable, since as > Herbert said, can get you "unexpected behaviour". Overlapping is not a problem, you can have several sockets binded on the same INADDR_ANY/port without ambiguity because the network namespace pointer is added as a new key for sockets lookup, (src addr, src port, dst addr, dst port, net ns pointer). The bind should not be forced to a specific address because you will not be able to connect via 127.0.0.1. > >> I think with the appropriate set of rules it provides what is needed >> for application migration. I.e. 127.0.0.1 can be filtered so that >> you can only connect to sockets in your current container. >> >> It does get a little odd because it does allow for the possibility >> that you can have multiple connected sockets with same source ip, >> source port, destination ip, destination port. If the rules are >> setup appropriately. I don't see that peculiarity being visible on >> the outside network so it shouldn't be a problem. > > So if they're using the same protocol (eg TCP), how is it decided which > one gets an incoming packet? Maybe I'm missing something as I don't > unde…
creator: Daniel Lezcano
date: 2006-11-29T22:10:39-00:00

Re: Re: Network virtualization/isolation
type: item
description: Daniel Lezcano wrote: > Brian Haley wrote: >> Eric W. Biederman wrote: >>> I think for cases across network socket namespaces it should >>> be a matter for the rules, to decide if the connection should >>> happen and what error code to return if the connection does not >>> happen. >>> >>> There is a potential in this to have an ambiguous case where two >>> applications can be listening for connections on the same socket >>> on the same port and both will allow the connection. If that >>> is the case I believe the proper definition is the first socket >>> that we find that will accept the connection gets the connection. > No. If you try to connect, the destination IP address is assigned to a > network namespace. This network namespace is used to leave the listening > socket ambiguity. >> >> Wouldn't you want to catch this at bind() and/or configuration time and >> fail? Having overlapping namespaces/rules seems undesirable, since as >> Herbert said, can get you "unexpected behaviour". > > Overlapping is not a problem, you can have several sockets binded on the > same INADDR_ANY/port without ambiguity because the network namespace > pointer is added as a new key for sockets lookup, (src addr, src port, > dst addr, dst port, net ns pointer). The bind should not be forced to a > specific address because you will not be able to connect via 127.0.0.1. So, all this leads to me ask, how to handle 127.0.0.1? For L2 it seems easy. Each namespace gets a tagged lo device. How do you propose to do it for L3, because disabling access to loopback is not a valid option, IMO. I agree that adding a namespace to the (using generic terms) TCB lookup solves the conflict issue. -vlad
creator: Vlad Yasevich
date: 2006-11-30T16:15:11-00:00

Re: Network virtualization/isolation
type: item
description: On Wed, 2006-14-11 at 16:17 +0100, Daniel Lezcano wrote: > The attached document describes the network isolation at the layer 2 > and at the layer 3 .. Daniel, I apologize for taking this long to get back to you. The document (I hope) made it clear to me at least the difference between the two approaches. So thanks for taking the time to put it together. So here are my thoughts ... I havent read the rest of the thread so i may be repeating some of the discussion; i have time today, I will try to catchup with the discussion. * i think the L2 approach is the more complete of the two approaches: It caters to more applications: eg i can have network elements such as virtual bridges and routers. It doesnt seem like i can do that with the L3 approach. I think this in itself is a powerful enough reason to disqualify the L3 approach. Leading from the above, I dont have to make _a single line of code change_ to any of the network element management tools inside the container. i.e i can just run quagga and OSPF and BGP will work as is or the bridge daemon and STP will work as is or tc to control "real" devices or ip to control "real" ip addresses. Virtual routers and bridges are real world applications (if you want more info ask me or ask google, she knows). **** This wasnt clear to me from the doc on the L3 side of things, so please correct me: because of the pid virtualization in the L2 approach(openvz?) I can run all applications as is. They just dont know they are running on a virtual environment. To use an extreme example: if i picked apache as a binary compiled 10 years ago, it will run on the L2 approach but not on the L3 approach. Is this understanding correct? I find it hard to believe that the L3 approach wouldnt work this way - it may be just my reading into the doc. So lets say the approach taken is that of L2 (I am biased towards this because i want to be able to do virtual bridges and routers). The disadvantage of the L2 approach (or is it just the openvz?) appr…
creator: jamal
date: 2006-12-03T12:26:02-00:00

Network virtualization/isolation
type: item
description: I have removed the Re: just to add some freshness to the discussion So i read quickly the rest of the discussions. I was almost suprised to find that i agree with Eric on a lot of opinions (we also agree that vindaloo is good for you i guess);-> The two issues that stood out for me (in addition to what i already said below): 1) the solution must ease the migration of containers; i didnt see anything about migrating them to another host across a network, but i assume that this is a given. 2) the socket level bind/accept filtering with multiple IPs. From reading what Herbert has, it seems they have figured a clever way to optimize this path albeit some challenges (speacial casing for raw filters) etc. I am wondering if one was to use the two level muxing of the socket layer, how much more performance improvement the above scheme provides for #2? Consider the case of L2 where by the time the packet hits the socket layer on incoming, the VE is already known; in such a case, the lookup would be very cheap. The advantage being you get rid of the speacial casing altogether. I dont see any issues with binds per multiple IPs etc using such a technique. For the case of #1 above, wouldnt it be also easier if the tables for netdevices, PIDs etc were per VE (using the 2 level mux)? In any case, folks, i hope i am not treading on anyones toes; i know each one of you has implemented and has users and i am trying to be as neutral as i can (but clearly biased;->). cheers, jamal On Sun, 2006-03-12 at 07:26 -0500, jamal wrote: > On Wed, 2006-14-11 at 16:17 +0100, Daniel Lezcano wrote: > > The attached document describes the network isolation at the layer 2 > > and at the layer 3 .. > > Daniel, > > I apologize for taking this long to get back to you. The document (I > hope) made it clear to me at least the difference between the two > approaches. So thanks for taking the time to put it together. > > So here are my thoughts ... > I havent read the rest of the thread so i may be repeatin…
creator: jamal
date: 2006-12-03T14:13:18-00:00

Re: Network virtualization/isolation
type: item
description: On Sun, Dec 03, 2006 at 07:26:02AM -0500, jamal wrote: > On Wed, 2006-14-11 at 16:17 +0100, Daniel Lezcano wrote: > > The attached document describes the network isolation at the layer 2 > > and at the layer 3 .. > > Daniel, > > I apologize for taking this long to get back to you. The document (I > hope) made it clear to me at least the difference between the two > approaches. So thanks for taking the time to put it together. > > So here are my thoughts ... > I havent read the rest of the thread so i may be repeating some of the > discussion; i have time today, I will try to catchup with the > discussion. > > * i think the L2 approach is the more complete of the two approaches: > > It caters to more applications: eg i can have network elements such as > virtual bridges and routers. It doesnt seem like i can do that with the > L3 approach. I think this in itself is a powerful enough reason to > disqualify the L3 approach. > > Leading from the above, I dont have to make _a single line of code > change_ to any of the network element management tools inside the > container. i.e i can just run quagga and OSPF and BGP will work as is or > the bridge daemon and STP will work as is or tc to control "real" > devices or ip to control "real" ip addresses. Virtual routers and > bridges are real world applications (if you want more info ask me or ask > google, she knows). > > **** This wasnt clear to me from the doc on the L3 side of things, so > please correct me: > because of the pid virtualization in the L2 approach(openvz?) I can run > all applications as is. They just dont know they are running on a > virtual environment. To use an extreme example: if i picked apache as a > binary compiled 10 years ago, it will run on the L2 approach but not on > the L3 approach. Is this understanding correct? I find it hard to > believe that the L3 approach wouldnt work this way - it may be just my > reading into the doc. the 10 year old apache will run with layer 3 isolation as well as wi…
creator: Herbert Poetzl
date: 2006-12-03T16:37:19-00:00

Re: Network virtualization/isolation
type: item
description: On Sun, 2006-03-12 at 17:37 +0100, Herbert Poetzl wrote: > On Sun, Dec 03, 2006 at 07:26:02AM -0500, jamal wrote: > To use an extreme example: if i picked apache as a > > binary compiled 10 years ago, it will run on the L2 approach but not on > > the L3 approach. Is this understanding correct? I find it hard to > > believe that the L3 approach wouldnt work this way - it may be just my > > reading into the doc. > > the 10 year old apache will run with layer 3 isolation > as well as with layer 2 virtualization (probably a little > faster though, we do not know yet :), because what it > does is IP (layer 3) traffic ... > Ok, thanks for clarifying this. > > I think the litmus test for this approach is the answer to the question: > > If i compiled in the containers in and do not use the namespaces, how > > much more overhead is there for the host path? I would hope that it is > > as close to 0 as possible. It should certainly be 0 if i dont compile in > > containers. > > IMHO there are three cases to consider, to get valid > 'performance' numbers: > > - host system with and without containers enabled > - single guest (container) compared to host system _without_ Sound reasonable. > - bunch of guests (e.g. 10) compared to 10 apps/threads on host > Your mileage may vary. For me trying to run virtual routers; this is not an important test. I want to be able to have containers each running quagga and OSPF. I cant achieve my goals with with 10 quaggas without making some major changes to quagga. > one proven feature of the L3 isolation is that those > all end up with the same or even better performance I think it is valuable to reduce the overhead. I think that it may be reasonable to some threshold to trade a little performance for genericity. What the threshold is, i dont know. > > - Manageability from the host side. It seems to be more complex with the > > L2 than with L3. But so what? These tools are written from scratch and > > there is no "backward compatibility" baggag…
creator: jamal
date: 2006-12-03T16:58:30-00:00

Re: Network virtualization/isolation
type: item
description: Hi Jamal, thanks for taking the time read the document. The objective of the document was not to convince one approach is better than other. I wanted to show the pros and the cons of each approach and to point that the 2 approaches are complementary. Currently, there are some resources moved to a namespace relative access, the IPC and the utsname and this is into the 2.6.19 kernel. The work on the pid namespace is still in progress. The idea is to use a "clone" approach relying on the "unshare_ns" syscall. The syscall is called with a set of flags for pids, ipcs, utsname, network ... You can then "unshare" only the network and have an application into its own network environment. For a l3 approach, like a l2, you can run an apache server into a unshared network environment. Better, you can run several apaches server into several network namespaces without modifying the server's network configuration. Some of us, consider l2 as perfectly adapted for some kind of containers like system containers and some kind of application containers running big servers, but find the l2 too much (seems to be a hammer to crush a beetle) for simple network requirements like for network migration, jails or containers which does not take care of such virtualization. For example, you want to create thousands of containers for a cluster of HPC jobs and just to have migration for these jobs. Does it make sense to have l2 approach ? Dmitry Mishin and I, we thought about a l2/l3 solution and we thing we found a solution to have the 2 at runtime. Roughly, it is a l3 based on bind filtering and socket isolation, very similar to what vserver provides. I did a prototype, and it works well for IPV4/unicast. So, considering, we have a l2 isolation/virtualization, and having a l3 relying on the l2 network isolation resources subset. Is it an acceptable solution ? -- Daniel
creator: Daniel Lezcano
date: 2006-12-04T10:18:09-00:00

Re: Network virtualization/isolation
type: item
description: jamal <hadi@cyberus.ca> writes: > I have removed the Re: just to add some freshness to the discussion > > So i read quickly the rest of the discussions. I was almost suprised to > find that i agree with Eric on a lot of opinions (we also agree that > vindaloo is good for you i guess);-> > The two issues that stood out for me (in addition to what i already said > below): > > 1) the solution must ease the migration of containers; i didnt see > anything about migrating them to another host across a network, but i > assume that this is a given. It is mostly a given. It is a goal for some of us and not for others. Containers are a necessary first step to getting migration and checkpoint/restart assistance from the kernel. > 2) the socket level bind/accept filtering with multiple IPs. From > reading what Herbert has, it seems they have figured a clever way to > optimize this path albeit some challenges (speacial casing for raw > filters) etc. > > I am wondering if one was to use the two level muxing of the socket > layer, how much more performance improvement the above scheme provides > for #2? I don't follow this question. > Consider the case of L2 where by the time the packet hits the socket > layer on incoming, the VE is already known; in such a case, the lookup > would be very cheap. The advantage being you get rid of the speacial > casing altogether. I dont see any issues with binds per multiple IPs etc > using such a technique. > > For the case of #1 above, wouldnt it be also easier if the tables for > netdevices, PIDs etc were per VE (using the 2 level mux)? Generally yes. s/VE/namespace/. There is a case with hash tables where it seems saner to add an additional entry because hash it is hard to dynamically allocate a hash table, (because they need something large then a single page allocation). But for everything else yes it makes things much easier if you have a per namespace data structure. A practical question is can we replace hash tables with some variant of …
creator: ebiederm
date: 2006-12-04T12:15:00-00:00

Re: Network virtualization/isolation
type: item
description: Daniel, On Mon, 2006-04-12 at 11:18 +0100, Daniel Lezcano wrote: > Hi Jamal, > Currently, there are some resources moved to a namespace relative > access, the IPC and the utsname and this is into the 2.6.19 kernel. > The work on the pid namespace is still in progress. > > The idea is to use a "clone" approach relying on the "unshare_ns" > syscall. The syscall is called with a set of flags for pids, ipcs, > utsname, network ... You can then "unshare" only the network and have an > application into its own network environment. > Ok, so i take it this call is used by the setup manager on the host side? > For a l3 approach, like a l2, you can run an apache server into a > unshared network environment. Better, you can run several apaches server > into several network namespaces without modifying the server's network > configuration. > ok - as i understand it now, this will be the case for all the approaches taken? > Some of us, consider l2 as perfectly adapted for some kind of containers > like system containers and some kind of application containers running > big servers, but find the l2 too much (seems to be a hammer to crush a > beetle) for simple network requirements like for network migration, > jails or containers which does not take care of such virtualization. For > example, you want to create thousands of containers for a cluster of HPC > jobs and just to have migration for these jobs. Does it make sense to > have l2 approach ? > Perhaps not for the specific app you mentioned above. But it makes sense for what i described as virtual routers/bridges. I would say that the solution has to cater for a variety of applications, no? > Dmitry Mishin and I, we thought about a l2/l3 solution and we thing we > found a solution to have the 2 at runtime. Roughly, it is a l3 based on > bind filtering and socket isolation, very similar to what vserver > provides. I did a prototype, and it works well for IPV4/unicast. > ok - so you guys seem to be reaching at least some consen…
creator: jamal
date: 2006-12-04T13:22:37-00:00

Re: Network virtualization/isolation
type: item
description: On Mon, 2006-04-12 at 05:15 -0700, Eric W. Biederman wrote: > jamal <hadi@cyberus.ca> writes: > > Containers are a necessary first step to getting migration and checkpoint/restart > assistance from the kernel. Isnt it like a MUST have if you are doing things from scratch instead of it being an after thought. > > > 2) the socket level bind/accept filtering with multiple IPs. From > > reading what Herbert has, it seems they have figured a clever way to > > optimize this path albeit some challenges (speacial casing for raw > > filters) etc. > > > > I am wondering if one was to use the two level muxing of the socket > > layer, how much more performance improvement the above scheme provides > > for #2? > > I don't follow this question. if you had the sockets tables being in two level mux, first level to hash on namespace which leads to an indirection pointer to the table to find the socket and its bindings (with zero code changes to the socket code), then isnt this "fast enough"? Clearly you can optimize as in the case of bind/accept filtering, but then you may have to do that for every socket family/protocol (eg netlink doesnt have IP addresses, but the binding to multiple groups is possible) Am i making any more sense? ;-> > > Consider the case of L2 where by the time the packet hits the socket > > layer on incoming, the VE is already known; in such a case, the lookup > > would be very cheap. The advantage being you get rid of the speacial > > casing altogether. I dont see any issues with binds per multiple IPs etc > > using such a technique. > > > > For the case of #1 above, wouldnt it be also easier if the tables for > > netdevices, PIDs etc were per VE (using the 2 level mux)? > > Generally yes. s/VE/namespace/. There is a case with hash tables where > it seems saner to add an additional entry because hash it is hard to dynamically > allocate a hash table, (because they need something large then a > single page allocation). A page to store the namespace indirection …
creator: jamal
date: 2006-12-04T13:44:54-00:00

Re: Network virtualization/isolation
type: item
description: jamal <hadi@cyberus.ca> writes: > On Mon, 2006-04-12 at 05:15 -0700, Eric W. Biederman wrote: >> jamal <hadi@cyberus.ca> writes: >> > >> Containers are a necessary first step to getting migration and > checkpoint/restart >> assistance from the kernel. > > Isnt it like a MUST have if you are doing things from scratch instead of > it being an after thought. Having the proper semantics is a MUST, which generally makes those a requirement to get consensus and to build the general mergeable solution. The logic for serializing the state is totally uninteresting for the first pass at containers. The applications inside the containers simply don't care. There are two basic techniques for containers. 1) Name filtering. Where you keep the same global identifiers as you do now, but applications inside the container are only allowed to deal with a subset of those names. The current vserver layer 3 networking approach is a handy example of this. But this can apply to process ids and just about everything else. 2) Independent namespaces. (Name duplication) Where you allow the same global name to refer to two different objects at the same time, with the context the reference comes being used to resolve which global object you are talking about. Independent namespaces are the only core requirement for migration, because the ensure when you get to the next machine you don't have a conflict with your global names. So at this point simply allowing duplicate names is the only requirement for migration. But yes that part is a MUST. >> > 2) the socket level bind/accept filtering with multiple IPs. From >> > reading what Herbert has, it seems they have figured a clever way to >> > optimize this path albeit some challenges (speacial casing for raw >> > filters) etc. >> > >> > I am wondering if one was to use the two level muxing of the socket >> > layer, how much more performance improvement the above scheme provides >> > for #2? >> >> I don't follow this question. > > if you had the socke…
creator: ebiederm
date: 2006-12-04T15:35:42-00:00

Re: Network virtualization/isolation
type: item
description: On Monday 04 December 2006 18:35, Eric W. Biederman wrote: [skip] > Where and when you look to find the network namespace that applies to > a packet is the primary difference between the OpenVZ L2 > implementation and my L2 implementation. > > If there is a better and less intrusive while still being obvious > method I am all for it. I do not like the OpenVZ thing of doing the > lookup once and then stashing the value in current and the special > casing the exceptions. Why? -- Thanks, Dmitry.
creator: Mishin Dmitry
date: 2006-12-04T16:00:42-00:00

Re: Network virtualization/isolation
type: item
description: Dmitry Mishin <dim@openvz.org> writes: > On Monday 04 December 2006 18:35, Eric W. Biederman wrote: > [skip] >> Where and when you look to find the network namespace that applies to >> a packet is the primary difference between the OpenVZ L2 >> implementation and my L2 implementation. >> >> If there is a better and less intrusive while still being obvious >> method I am all for it. I do not like the OpenVZ thing of doing the >> lookup once and then stashing the value in current and the special >> casing the exceptions. > Why? I like it when things are obvious and not implied. The implementations seems to favor fewer lines of code touched over maintainability of the code. Which if you are maintaining out of tree code is fine. At leas that was my impression last time I looked at the code. I know there are a lot of silly things in the existing implementations because they were initially written without the expectation of being able to merge the code into the main kernel. This resulted in some non-general interfaces, and a preference for patches that touch as few lines of code as possible. Anyway this has bit has been discussed before and we can discuss it seriously in the context of patch review. Eric
creator: ebiederm
date: 2006-12-04T16:52:10-00:00

Re: Network virtualization/isolation
type: item
description: Herbert Poetzl <herbert@13thfloor.at> writes: > On Mon, Dec 04, 2006 at 06:19:00PM +0300, Dmitry Mishin wrote: >> On Sunday 03 December 2006 19:00, Eric W. Biederman wrote: >> > Ok. Just a quick summary of where I see the discussion. >> > >> > We all agree that L2 isolation is needed at some point. > >> As we all agreed on this, may be it is time to send patches >> one-by-one? For the beggining, I propose to resend Cedric's >> empty namespace patch as base for others - it is really empty, >> but necessary in order to move further. >> >> After this patch and the following net namespace unshare >> patch will be accepted, > > well, I have neither seen any performance tests showing > that the following is true: > > - no change on network performance without the > space enabled > - no change on network performance on the host > with the network namespaces enabled > - no measureable overhead inside the network > namespace > - good scaleability for a larger number of network > namespaces Yes all important criteria for selecting the implementation. Eric _______________________________________________ Containers mailing list Containers@lists.osdl.org https://lists.osdl.org/mailman/listinfo/containers
creator: ebiederm
date: 2006-12-04T16:58:04-00:00

Re: Network virtualization/isolation
type: item
description: On Mon, Dec 04, 2006 at 08:02:48PM +0300, Dmitry Mishin wrote: > On Monday 04 December 2006 19:43, Herbert Poetzl wrote: > > On Mon, Dec 04, 2006 at 06:19:00PM +0300, Dmitry Mishin wrote: > > > On Sunday 03 December 2006 19:00, Eric W. Biederman wrote: > > > > Ok. Just a quick summary of where I see the discussion. > > > > > > > > We all agree that L2 isolation is needed at some point. > > > > > > As we all agreed on this, may be it is time to send patches > > > one-by-one? For the beggining, I propose to resend Cedric's > > > empty namespace patch as base for others - it is really empty, > > > but necessary in order to move further. > > > > > > After this patch and the following net namespace unshare > > > patch will be accepted, > > > > well, I have neither seen any performance tests showing > > that the following is true: > > > > - no change on network performance without the > > space enabled > > - no change on network performance on the host > > with the network namespaces enabled > > - no measureable overhead inside the network > > namespace > > - good scaleability for a larger number of network > > namespaces > These questions are for complete L2 implementation, > not for these 2 empty patches. well, I fear that we will have lot of overhead 'sneaking' in via small patches (with almost unnoticeable overhead) making the 2.6 branch slower and slower (regarding networking) so IMHO a complete solution should be drafted, and tested performance wise, we can then adjust it and possibley improve it, untill it shows no measureable overhead ... but IMHO it should be developed 'outside' the kernel, in small and reviewable pieces which are constantly updated to match the recent kernels ... something like stacked git or quilt ... > If you need some data relating to Andrey's > implementation, I'll get it. Which test do you accept? hmm, I think a good mix of netperf, netpipe and iperf would be a good start, probably network folks know better tests to exercise the stack ... a…
creator: Herbert Poetzl
date: 2006-12-04T17:19:26-00:00

Re: Network virtualization/isolation
type: item
description: Dmitry Mishin wrote: > On Monday 04 December 2006 19:43, Herbert Poetzl wrote: >> On Mon, Dec 04, 2006 at 06:19:00PM +0300, Dmitry Mishin wrote: >>> On Sunday 03 December 2006 19:00, Eric W. Biederman wrote: >>>> Ok. Just a quick summary of where I see the discussion. >>>> >>>> We all agree that L2 isolation is needed at some point. >>> As we all agreed on this, may be it is time to send patches >>> one-by-one? For the beggining, I propose to resend Cedric's >>> empty namespace patch as base for others - it is really empty, >>> but necessary in order to move further. >>> >>> After this patch and the following net namespace unshare >>> patch will be accepted, >> well, I have neither seen any performance tests showing >> that the following is true: >> >> - no change on network performance without the >> space enabled >> - no change on network performance on the host >> with the network namespaces enabled >> - no measureable overhead inside the network >> namespace >> - good scaleability for a larger number of network >> namespaces > These questions are for complete L2 implementation, not for these 2 empty > patches. If you need some data relating to Andrey's implementation, I'll get > it. Which test do you accept? tbench ? With the following scenarii: * intra host communication (one time with IP on eth and one time with 127.0.0.1) * inter host communication Each time: - a single network namespace - with 100 network namespace. 1 server communicating and 99 listening but doing nothing. _______________________________________________ Containers mailing list Containers@lists.osdl.org https://lists.osdl.org/mailman/listinfo/containers
creator: Daniel Lezcano
date: 2006-12-04T17:41:33-00:00

Re: Re: Network virtualization/isolation
type: item
description: >>>If there is a better and less intrusive while still being obvious >>>method I am all for it. I do not like the OpenVZ thing of doing the >>>lookup once and then stashing the value in current and the special >>>casing the exceptions. >> >>Why? > > > I like it when things are obvious and not implied. > > The implementations seems to favor fewer lines of code touched over > maintainability of the code. Which if you are maintaining out of > tree code is fine. At leas that was my impression last time > I looked at the code. FYI, when we started doing networking virtualization many years ago we tried both approaches. Over time, context notion looked much more natural and easier for us. Even Alexey Kuznetsov tells that he prefers exec_env as the logic becomes very clear and little mess is introduced. > I know there are a lot of silly things in the existing implementations > because they were initially written without the expectation of being > able to merge the code into the main kernel. This resulted in some > non-general interfaces, and a preference for patches that touch > as few lines of code as possible. Sure, but OpenVZ code is being constantly cleaned from such code and we are open for discussion. No one pretends that code is perferct from the beginning. > Anyway this has bit has been discussed before and we can discuss it > seriously in the context of patch review. Let me explain when explicit context like exec_env IMHO is cleaner: - context is a natural notion of linux kernel. e.g. current. why not pass 'current' to all the functions as an argument starting from entry.S? in_atomic(), in_interrupt() etc. all these functions deal with current context. IMHO when one needs to pass an argument too many times like 'current' it is better to use a notion of the context. - e.g. NFS should set networking context of the mount point or socket. But, ok, it is not the real point to argue so much imho and waste our time instead of doing things. Thanks, Kirill
creator: dev
date: 2006-12-06T11:45:04-00:00

Re: Re: Network virtualization/isolation
type: item
description: On Wed, Dec 06, 2006 at 02:54:16PM +0300, Kirill Korotaev wrote: > >>>If there is a better and less intrusive while still being obvious > >>>method I am all for it. I do not like the OpenVZ thing of doing the > >>>lookup once and then stashing the value in current and the special > >>>casing the exceptions. > >> > >>Why? > > > > > > I like it when things are obvious and not implied. > > > > The implementations seems to favor fewer lines of code touched over > > maintainability of the code. Which if you are maintaining out of > > tree code is fine. At leas that was my impression last time > > I looked at the code. > FYI, when we started doing networking virtualization many years ago > we tried both approaches. > Over time, context notion looked much more natural and easier for us. > Even Alexey Kuznetsov tells that he prefers exec_env as the logic > becomes very clear and little mess is introduced. > > > I know there are a lot of silly things in the existing implementations > > because they were initially written without the expectation of being > > able to merge the code into the main kernel. This resulted in some > > non-general interfaces, and a preference for patches that touch > > as few lines of code as possible. > Sure, but OpenVZ code is being constantly cleaned from such code > and we are open for discussion. No one pretends that code is perferct > from the beginning. > > > Anyway this has bit has been discussed before and we can discuss it > > seriously in the context of patch review. > Let me explain when explicit context like exec_env IMHO is cleaner: > - context is a natural notion of linux kernel. e.g. current. > why not pass 'current' to all the functions as an argument > starting from entry.S? > in_atomic(), in_interrupt() etc. all these functions deal with > current context. IMHO when one needs to pass an argument too many > times like 'current' > it is better to use a notion of the context. > - e.g. NFS should set networking context of the mount poi…
creator: Herbert Poetzl
date: 2006-12-06T18:30:36-00:00

Re: Re: Network virtualization/isolation
type: item
description: Herbert Poetzl <herbert@13thfloor.at> writes: >> But, ok, it is not the real point to argue so much imho and waste our >> time instead of doing things. > > well, IMHO better talk (and think) first, then implement > something ... not the other way round, and then start > fixing up the mess ... Well we need a bit of both. This is thankfully not exported to user space, so as long as our implementation is correct it doesn't much matter. I do agree with the point that context may make sense. I have yet to be convinced though. Eric _______________________________________________ Containers mailing list Containers@lists.osdl.org https://lists.osdl.org/mailman/listinfo/containers
creator: ebiederm
date: 2006-12-08T19:57:49-00:00

Re: Re: Network virtualization/isolation
type: item
description: On Fri, Dec 08, 2006 at 12:57:49PM -0700, Eric W. Biederman wrote: > Herbert Poetzl <herbert@13thfloor.at> writes: > > >> But, ok, it is not the real point to argue so much imho > >> and waste our time instead of doing things. > > well, IMHO better talk (and think) first, then implement > > something ... not the other way round, and then start > > fixing up the mess ... > > Well we need a bit of both. hmm, are 'we' in a hurry here? until recently, 'Linux' (mainline) didn't even want to hear about OS Level virtualization, now there is a rush to quickly get 'something' in, not knowing or caring if it is usable at all? I think there are a lot of 'potential users' for this kind of virtualization, and so 'we' can test almost all aspects outside of mainline, and once we know the stuff works as expected, then we can integrate it ... the UTS namespace was something 'we all' had already implemented in this (or a very similar) way, and in one or two interations, it should actually work as expected. nevertheless, it was one of the simplest spaces ... we do not yet know the details for the IPC namespace, as IPC is not that easy to check as UTS, and 'we' haven't gotten real world feedback on that yet ... so personally I think we should start some serious testing on the upcoming namespaces, and we should continue discussing the various approaches, until 'we' can agree on the (almost) 'perfect' solution > This is thankfully not exported to user space, so as long > as our implementation is correct it doesn't much matter. that's something I do not really agree with, stuff integrated into the kernel should be well designed and it should be tested ... best, Herbert > I do agree with the point that context may make sense. > I have yet to be convinced though. > > Eric _______________________________________________ Containers mailing list Containers@lists.osdl.org https://lists.osdl.org/mailman/listinfo/containers
creator: Herbert Poetzl
date: 2006-12-09T03:50:02-00:00

Re: Re: Network virtualization/isolation
type: item
description: On Sat, 9 Dec 2006 04:50:02 +0100 Herbert Poetzl <herbert@13thfloor.at> wrote: > On Fri, Dec 08, 2006 at 12:57:49PM -0700, Eric W. Biederman wrote: > > Herbert Poetzl <herbert@13thfloor.at> writes: > > > > >> But, ok, it is not the real point to argue so much imho > > >> and waste our time instead of doing things. > > > > well, IMHO better talk (and think) first, then implement > > > something ... not the other way round, and then start > > > fixing up the mess ... > > > > Well we need a bit of both. > > hmm, are 'we' in a hurry here? > > until recently, 'Linux' (mainline) didn't even want > to hear about OS Level virtualization, now there > is a rush to quickly get 'something' in, not knowing > or caring if it is usable at all? It's actually happening quite gradually and carefully. > I think there are a lot of 'potential users' for > this kind of virtualization, and so 'we' can test > almost all aspects outside of mainline, and once > we know the stuff works as expected, then we can > integrate it ... > > the UTS namespace was something 'we all' had already > implemented in this (or a very similar) way, and in > one or two interations, it should actually work as > expected. nevertheless, it was one of the simplest > spaces ... > > we do not yet know the details for the IPC namespace, > as IPC is not that easy to check as UTS, and 'we' > haven't gotten real world feedback on that yet ... We are very dependent upon all stakeholders including yourself to review, test and comment upon this infrastructure as it is proposed and merged. If something is proposed which will not suit your requirements then it is important that we hear about it, in detail, at the earliest possible time. Thanks. _______________________________________________ Containers mailing list Containers@lists.osdl.org https://lists.osdl.org/mailman/listinfo/containers
creator: Andrew Morton
date: 2006-12-09T06:13:48-00:00

Re: Re: Network virtualization/isolation
type: item
description: On Fri, Dec 08, 2006 at 10:13:48PM -0800, Andrew Morton wrote: > On Sat, 9 Dec 2006 04:50:02 +0100 > Herbert Poetzl <herbert@13thfloor.at> wrote: > > > On Fri, Dec 08, 2006 at 12:57:49PM -0700, Eric W. Biederman wrote: > > > Herbert Poetzl <herbert@13thfloor.at> writes: > > > > > > >> But, ok, it is not the real point to argue so much imho > > > >> and waste our time instead of doing things. > > > > > > well, IMHO better talk (and think) first, then implement > > > > something ... not the other way round, and then start > > > > fixing up the mess ... > > > > > > Well we need a bit of both. > > > > hmm, are 'we' in a hurry here? > > > > until recently, 'Linux' (mainline) didn't even want > > to hear about OS Level virtualization, now there > > is a rush to quickly get 'something' in, not knowing > > or caring if it is usable at all? > > It's actually happening quite gradually and carefully. hmm, I must have missed a testing phase for the IPC namespace then, not that I think it is broken (well, maybe it is, we do not know yet) > > I think there are a lot of 'potential users' for > > this kind of virtualization, and so 'we' can test > > almost all aspects outside of mainline, and once > > we know the stuff works as expected, then we can > > integrate it ... > > > > the UTS namespace was something 'we all' had already > > implemented in this (or a very similar) way, and in > > one or two interations, it should actually work as > > expected. nevertheless, it was one of the simplest > > spaces ... > > > > we do not yet know the details for the IPC namespace, > > as IPC is not that easy to check as UTS, and 'we' > > haven't gotten real world feedback on that yet ... > > We are very dependent upon all stakeholders including yourself > to review, test and comment upon this infrastructure as it is > proposed and merged. If something is proposed which will not > suit your requirements then it is important that we hear about > it, in detail, at the earliest possible time. okay,…
creator: Herbert Poetzl
date: 2006-12-09T06:35:17-00:00

Re: Re: Network virtualization/isolation
type: item
description: Herbert Poetzl <herbert@13thfloor.at> writes: > On Fri, Dec 08, 2006 at 12:57:49PM -0700, Eric W. Biederman wrote: >> Herbert Poetzl <herbert@13thfloor.at> writes: >> >> >> But, ok, it is not the real point to argue so much imho >> >> and waste our time instead of doing things. > >> > well, IMHO better talk (and think) first, then implement >> > something ... not the other way round, and then start >> > fixing up the mess ... >> >> Well we need a bit of both. > > hmm, are 'we' in a hurry here? We need to talk about code, and particular patches not just talk. There are two sides to what we are building. - The user interface, and semantics. - The kernel implementation. For the user interface getting it as close to perfect as we can the first time is extremely important. Because we won't be able to change it. For the kernel implementation we don't have to be perfect we have to have something that is good enough. We can change the implementation every release if we find better ways of implementing our user space semantics. > until recently, 'Linux' (mainline) didn't even want > to hear about OS Level virtualization, now there > is a rush to quickly get 'something' in, not knowing > or caring if it is usable at all? > > I think there are a lot of 'potential users' for > this kind of virtualization, and so 'we' can test > almost all aspects outside of mainline, and once > we know the stuff works as expected, then we can > integrate it ... We should do this as part of the linux kernel community. There is no outside of mainline development. We need to get feedback from other developers whose code we may effect. This is particularly true of the kernel networking stack. > that's something I do not really agree with, stuff > integrated into the kernel should be well designed > and it should be tested ... Yes but you can break a problem into reasonable chunks and solve each of those pieces individually. Eric _______________________________________________ Containers mailing lis…
creator: ebiederm
date: 2006-12-09T08:07:22-00:00

Re: Re: Network virtualization/isolation
type: item
description: On Sat, Dec 09, 2006 at 04:50:02AM +0100, Herbert Poetzl wrote: > On Fri, Dec 08, 2006 at 12:57:49PM -0700, Eric W. Biederman wrote: > > Herbert Poetzl <herbert@13thfloor.at> writes: > > > > >> But, ok, it is not the real point to argue so much imho > > >> and waste our time instead of doing things. > > > > well, IMHO better talk (and think) first, then implement > > > something ... not the other way round, and then start > > > fixing up the mess ... > > > > Well we need a bit of both. > > hmm, are 'we' in a hurry here? > > until recently, 'Linux' (mainline) didn't even want > to hear about OS Level virtualization, now there > is a rush to quickly get 'something' in, not knowing > or caring if it is usable at all? Maybe beacuse other Operating Systems have it? For example Solaris' Crossbow... -- Tomasz Torcz RIP is irrevelant. Spoofing is futile. zdzichu@irc.-nie.spam-.pl Your routes will be aggreggated. -- Alex Yuriev _______________________________________________ Containers mailing list Containers@lists.osdl.org https://lists.osdl.org/mailman/listinfo/containers
creator: Tomasz Torcz
date: 2006-12-09T11:27:34-00:00

Re: Re: Network virtualization/isolation
type: item
description: On Sat, Dec 09, 2006 at 12:27:34PM +0100, Tomasz Torcz wrote: > On Sat, Dec 09, 2006 at 04:50:02AM +0100, Herbert Poetzl wrote: > > On Fri, Dec 08, 2006 at 12:57:49PM -0700, Eric W. Biederman wrote: > > > Herbert Poetzl <herbert@13thfloor.at> writes: > > > > > > >> But, ok, it is not the real point to argue so much imho > > > >> and waste our time instead of doing things. > > > > > > well, IMHO better talk (and think) first, then implement > > > > something ... not the other way round, and then start > > > > fixing up the mess ... > > > > > > Well we need a bit of both. > > > > hmm, are 'we' in a hurry here? > > > > until recently, 'Linux' (mainline) didn't even want > > to hear about OS Level virtualization, now there > > is a rush to quickly get 'something' in, not knowing > > or caring if it is usable at all? > > Maybe beacuse other Operating Systems have it? well, that wasn't a good enough reason four years ago, when Linux-VServer tried to push a 'jail' implementation into mainline (was called security contexts back then, and maintained by Jacques Gelinas) > For example Solaris' Crossbow... yes, but the technology isn't really new, not even on Linux and not even in the Open Source community but don't get me wrong here, I'm absolutely for having virtualization (or virtualization elements) in mainline, I just don't want to see a Q&D hack 'we' have to suffer from the next two years :) HTC, Herbert > -- > Tomasz Torcz RIP is irrevelant. Spoofing is futile. > zdzichu@irc.-nie.spam-.pl Your routes will be aggreggated. -- Alex Yuriev > _______________________________________________ Containers mailing list Containers@lists.osdl.org https://lists.osdl.org/mailman/listinfo/containers
creator: Herbert Poetzl
date: 2006-12-09T19:04:14-00:00

Re: Re: Network virtualization/isolation
type: item
description: On Saturday 09 December 2006 09:35, Herbert Poetzl wrote: > On Fri, Dec 08, 2006 at 10:13:48PM -0800, Andrew Morton wrote: > > On Sat, 9 Dec 2006 04:50:02 +0100 > > Herbert Poetzl <herbert@13thfloor.at> wrote: > > > > > On Fri, Dec 08, 2006 at 12:57:49PM -0700, Eric W. Biederman wrote: > > > > Herbert Poetzl <herbert@13thfloor.at> writes: > > > > > > > > >> But, ok, it is not the real point to argue so much imho > > > > >> and waste our time instead of doing things. > > > > > > > > well, IMHO better talk (and think) first, then implement > > > > > something ... not the other way round, and then start > > > > > fixing up the mess ... > > > > > > > > Well we need a bit of both. > > > > > > hmm, are 'we' in a hurry here? > > > > > > until recently, 'Linux' (mainline) didn't even want > > > to hear about OS Level virtualization, now there > > > is a rush to quickly get 'something' in, not knowing > > > or caring if it is usable at all? > > > > It's actually happening quite gradually and carefully. > > hmm, I must have missed a testing phase for the > IPC namespace then, not that I think it is broken > (well, maybe it is, we do not know yet) Herbert, you know that this code is used in our product. And in its turn, our product is tested internally and by a community. We have no reports about bugs in this code. If you have to say more than just "something to say", please, say it. > > > > I think there are a lot of 'potential users' for > > > this kind of virtualization, and so 'we' can test > > > almost all aspects outside of mainline, and once > > > we know the stuff works as expected, then we can > > > integrate it ... > > > > > > the UTS namespace was something 'we all' had already > > > implemented in this (or a very similar) way, and in > > > one or two interations, it should actually work as > > > expected. nevertheless, it was one of the simplest > > > spaces ... > > > > > > we do not yet know the details for the IPC namespace, > > > as IPC is not that easy to check…
creator: Mishin Dmitry
date: 2006-12-09T21:18:08-00:00

Re: Re: Network virtualization/isolation
type: item
description: Herbert Poetzl wrote: > On Fri, Dec 08, 2006 at 10:13:48PM -0800, Andrew Morton wrote: > >> >> It's actually happening quite gradually and carefully. >> > > hmm, I must have missed a testing phase for the > IPC namespace then, not that I think it is broken > (well, maybe it is, we do not know yet) > > You have announced at LKML that Linux-VServer now uses the stuff that was merged in 2.6.19-rc1, haven't you? I suppose that means you are using IPC namespaces from mainstream? Isn't that considered testing? Or you don't test Linux-VServer? Please clarify, I'm a bit lost here. Speaking of OpenVZ, as Kirill Korotaev said before we have backported all that to 2.6.18 back in September and are using it since then. And yes, we found a bug in IPC namespaces, and fix from Pavel Emelyanov has made it to 2.6.19-rc5 (see commit c7e12b838989b0e432c7a1cdf1e6c6fd936007f6 to linux-2.6-git). _______________________________________________ Containers mailing list Containers@lists.osdl.org https://lists.osdl.org/mailman/listinfo/containers
creator: kir
date: 2006-12-09T22:34:14-00:00

Re: Re: Network virtualization/isolation
type: item
description: On Sun, Dec 10, 2006 at 01:34:14AM +0300, Kir Kolyshkin wrote: > Herbert Poetzl wrote: > >On Fri, Dec 08, 2006 at 10:13:48PM -0800, Andrew Morton wrote: > > > >> > >>It's actually happening quite gradually and carefully. > >> > > > >hmm, I must have missed a testing phase for the > >IPC namespace then, not that I think it is broken > >(well, maybe it is, we do not know yet) > > > You have announced at LKML that Linux-VServer now uses the > stuff that was merged in 2.6.19-rc1, haven't you? yes, correct, and we already fixed several issues the changes caused, both in handling as well as functionality > I suppose that means you are using IPC namespaces from > mainstream? yes, we do > Isn't that considered testing? of course it is testing, but it is already in mainstream, and for my part, I wasn't able to provide feedback from testing yet ... > Or you don't test Linux-VServer? we do the same testing you folks do IIRC (i.e. some secret test procedure which takes roughly a week or so, after which we can tell that everything works as expected :) > Please clarify, I'm a bit lost here. > Speaking of OpenVZ, as Kirill Korotaev said before we have > backported all that to 2.6.18 back in September nice, but what relevance has that for 2.6.19? > and are using it since then. cool, how much feedback regarding IPC did you get since then? > And yes, we found a bug in IPC namespaces, and fix from > Pavel Emelyanov has made it to 2.6.19-rc5 (see commit > c7e12b838989b0e432c7a1cdf1e6c6fd936007f6 to linux-2.6-git). it's good that some bugs have been found, but of what relevance is that for testing mainline patches? - typical linux users will only excercise a small fraction of the new code, if at all - virtualization solutions like OpenVZ and Linux-VServer add their custom modifications and/or adjustments, and serve a much smaller userbase - I haven't seen any test suites or similar for the spaces so it all boils down to waiting for somebody to stumble over an issue, which then will get …
creator: Herbert Poetzl
date: 2006-12-10T02:21:00-00:00