Author Archives: jejb

Unprivileged Build Containers

A while ago, a goal I set myself was to be able to maintain my build and test environments for architecture emulation containers without having to do any of the tasks as root and without creating any suid binaries to do this.  One of the big problems here is that distributions get annoyed (and don’t run correctly) if root doesn’t own most of the files … for instance the installers all check to see that the file got installed with the correct ownership and permissions and fail if they don’t.  Debian has an interesting mechanism, called fakeroot, to get around this using a preload library intercepting the chmod and chown system calls, but it’s getting a bit hackish to try to extend this to work permanently for an emulation container.

The correct way to do this is with user namespaces so the rest of this post will show you how.  Before we get into how to use them, lets begin with the theory of how user namespaces actually work.

Theory of User Namespaces

A user namespace is the single namespace that can be created by an unprivileged user.  Their job is to map a set of interior (inside the user namespace) uids, gids and projids1 to a set of exterior (outside the user namespace).

The way this works is that the root user namespace simply has a 1:1 identity mapping of all 2^32 identifiers, meaning it fully covers the space.  However, any new user namespace only need remap a subset of these.  Any id that is not mapped into the user namespace becomes inaccessible to that namespace.  This doesn’t mean completely inaccessible, it just means any resource owned or accessed by an unmapped id treats an attempted access (even from root in the namespace) as though it were completely unprivileged, so if the resource is readable by any id, it can still be read even in a user namespace where its owning id is unmapped.

User namespaces can also be nested but the nested namespace can only map ids that exist in its parent, so you can only reduce but not expand the id space by nesting.  The way the nested mapping works is that it remaps through all the parent namespaces, so what appears on the resource is still the original exterior ids.

User Namespaces also come with an owner (the uid/gid of the process that created the container).  The reason for this is that this owner is allowed to execute setuid/setgid to any id mapped into the namespace, so the owning uid/gid pair is the effective “root” of the container.  Note that this setuid/setgid property works on entry to the namespace even if uid 0 is not mapped inside the namespace, but won’t survive once the first setuid/setgid is issued.

The final piece of the puzzle is that every other namespace also has an owning user namespace, so while I cannot create a mount namespace as unprivileged user jejb, I can as remapped root inside my user namespace

jejb@jarvis:~> unshare --mount
unshare: unshare failed: Operation not permitted
jejb@jarvis:~> nsenter --user=/tmp/userns
root@jarvis:~# unshare --mount
root@jarvis:~#

And once created, I can always enter this mount namespace provided I’m also in my user namespace.

Setting up Unprivileged Namespaces

Any system user can actually create a user namespace.  However a non-root (meaning not uid zero in the parent namespace) user cannot remap any exterior id except their own.  This means that, because a build container needs a range of ids, it’s not possible to set up the intial remapped namespace without the help of root.  However, once that is done, the user can pretty much do every other operation2

The way remap ranges are set up is via the uid_map, gid_map and projid_map files sitting inside the /proc/<pid> directory.  These files may only be written to once and never updated3

As an example, to set up a build container, I need a remapping for every id that would be created during installation.  Traditionally for Linux, these are ids 0-999.  I want to remap them down to something unprivileged, say 100,000 so my line entry for this is

0 100000 1000

However, I also want an identity mapping for my own id (currently I’m at uid 1000), so I can still use my home directory from within the build container.  This also means I can create the roots for the containers within my home directory.  Finally, the nobody user and nobody,nogroup groups also need to be mapped, so the final uid map entries look like

0 100000 1000
1000 1000 1
65534 101001 1

For the groups, it’s even more complex because on openSUSE, I’m a member of the users group (gid 100) which sits in the middle of the privileged 0-999 group range, so the gid_map entry I use is

0 100000 100
100 100 1
101 100100 899
65533 101000 2

Which is almost up to the kernel imposed limit of five separate lines.

Finally, here’s how to set this up and create a binding for the user namespace.  As myself (I’m uid 1000 user name jejb) I do

jejb@jarvis:~> unshare --user
nobody@jarvis:~> echo $$
20211
nobody@jarvis:~>

Note that I become nobody inside the container because currently the map files are unwritten so there are no mapped ids at all.  Now as root, I have to write the mapping files and bind the entry file to the namespace somewhere

jarvis:/home/jejb # echo 1|awk '{print "0 100000 1000\n1000 1000 1\n65534 101001 1"}' > /proc/20211/uid_map
jarvis:/home/jejb # echo 1|awk '{print "0 100000 100\n100 100 1\n101 100100 899\n65533 101000 2"}' > /proc/20211/gid_map
jarvis:/home/jejb # touch /tmp/userns
jarvis:/home/jejb # mount --bind /proc/20211/ns/user /tmp/userns

Now I can exit my user namespace because it’s permanently bound and the next time I enter it I become root inside the container (although with uid 100000 outside)

jejb@jarvis:~> nsenter --user=/tmp/userns
root@jarvis:~# id
uid=0(root) gid=0(root) groups=0(root)
root@jarvis:~# su - jejb
jejb@jarvis:~> id
uid=1000(jejb) gid=100(users) groups=100(users)

Giving me a user namespace with sufficient mapped ids to create a build container.

Unprivileged Architecture Emulation Containers

Essentially, I can use the user namespace constructed above to bootstrap and enter the entire build container and its mount namespace with one proviso that I have to have a pre-created devices directory because I don’t possess the mknod capability as myself, so my container root also doesn’t possess it.  The way I get around this is to create the initial dev directory as root and then change the ownership to 100000.100000 (my unprivileged ids)

jejb@jarvis:~/containers/debian-amd64/dev> ls -l
total 0
lrwxrwxrwx 1 100000 100000 13 Feb 20 09:45 fd -> /proc/self/fd/
crw-rw-rw- 1 100000 100000 1, 7 Feb 20 09:45 full
crw-rw-rw- 1 100000 100000 1, 3 Feb 20 09:45 null
lrwxrwxrwx 1 100000 100000 8 Feb 20 09:45 ptmx -> pts/ptmx
drwxr-xr-x 2 100000 100000 6 Feb 20 09:45 pts/
crw-rw-rw- 1 100000 100000 1, 8 Feb 20 09:45 random
drwxr-xr-x 2 100000 100000 6 Feb 20 09:45 shm/
lrwxrwxrwx 1 100000 100000 15 Feb 20 09:45 stderr -> /proc/self/fd/2
lrwxrwxrwx 1 100000 100000 15 Feb 20 09:45 stdin -> /proc/self/fd/0
lrwxrwxrwx 1 100000 100000 15 Feb 20 09:45 stdout -> /proc/self/fd/1
crw-rw-rw- 1 100000 100000 5, 0 Feb 20 09:45 tty
crw-rw-rw- 1 100000 100000 1, 9 Feb 20 09:45 urandom
crw-rw-rw- 1 100000 100000 1, 5 Feb 20 09:45 zero

This seems to be sufficient of a dev skeleton to function with.  For completeness sake, I placed my bound user namespace into /run/build-container/userns and following the original architecture container emulation post, with modified susebootstrap and build-container scripts.  The net result is that as myself I can now enter and administer and update the build and test architecture emulation container with no suid required

jejb@jarvis:~> nsenter --user=/run/build-container/userns --mount=/run/build-container/ppc64
root@jarvis:/# id
uid=0(root) gid=0(root) groups=0(root)
root@jarvis:/# uname -m
ppc64
root@jarvis:/# su - jejb
jejb@jarvis:~> id
uid=1000(jejb) gid=100(users) groups=100(users)

The only final wrinkle is that root has to set up the user namespace on every boot, but that’s only because there’s no currently defined operating system way of doing this.

Are GPLv2 and CDDL incompatible?

Canonical recently threw this issue into sharp relief by their decision to ship CDDL licensed ZFS as a module of the GPLv2 licensed Linux kernel.  Reading their legal justification for this leaves me somewhat unconvinced because it’s essentially the same “not a derivative work” argument that a number of dubious actors have used to justify binary modules.  So what I’d like to do is look at this issue from a completely different viewpoint.  First by accepting the premise that CDDL and GPLv2 are incompatible (since there’s some debate on this) and secondly by accepting the even more controversial proposition that creating a kernel module is a derivative work.  I don’t want to debate these premises because it’s a worst case assumption I’m using as inputs to make the following analysis possible.

What is compliance?

One of the curious thing about CDDL and GPLv2 is that they’re both copyleft (albeit in differing forms) and the compliance requirements: the release of complete corresponding source code for your binary containing the licensed work.  In fact, the only significant difference is the requirement for build scripts, which is in GPLv2 but not in CDDL.  Therefore you can say that if you follow the compliance regime for GPLv2 on CDDL code, you’ll be in full compliance with the CDDL.  The licences do, in fact, have compatible compliance requirements.  This fact becomes very relevant when you consider the requirements for bringing a copyright lawsuit in the first place.

Where’s the Harm?

Copyright law is something called a tort in law.  That essentially means a branch of law for resolving disputes between individual parties.  However, in order to stem what could be seen as frivolous lawsuits, bringing a claim under tort law requires not just a showing that someone broke the rules of whatever agreement they were under but also that quantifiable harm resulted1.  The essential elements of a tort claim are a showing of a violation, a theory of the harm produced and a request for damages based on the harm2.

All of the bodies which do GPL enforcement recently published a charter in which they make clear that the sole requirement from an enforcement action should be compliance with the terms of the licence.  However, as I showed above, it is perfectly possible to be compatibly in compliance with both the CDDL and the GPLv2.  So the question becomes if the party is already in compliance, even though there’s a technical violation of the terms of the licence produced by the combination, what would our theory of harm be given that we don’t really seem to have anything extra we’d ask of the violating party?

Of course, one can wax lyrical about the “harm to the licence” of allowing incompatible combinations.  However, here we’re on a very sticky wicket because there have been a lot of works published (including by the FSF itself) bemoaning the problems of licence incompatibility.  Indeed, part of the design of the GPLv3 process was to make the licence more amenable to combination with other open source licences.  So suddenly becoming ardent advocates for the benefits of licence incompatibility is probably to be unlikely to fly before a judge as a theory of harm.

Conclusion

What the above analysis shows is that even though we presumed combination of GPLv2 and CDDL works to be a technical violation, there’s no way actually to prosecute such a violation because we can’t develop a convincing theory of harm resulting.  Because this makes it impossible to take the case to court, effectively it must be concluded that the combination of GPLv2 and CDDL, provided you’re following a GPLv2 compliance regime for all the code, is allowable.  This conclusion is the same as the one Canonical reached, but the means by which I got there are very different.

Note that this conclusion would apply to mixing any open source licence with GPLv2: provided the compliance regimes are compatible and the stricter one is followed, it’s difficult to develop a theory of harm and thus the combination is allowable.

Final Thought

The above analysis is all from the point of view of the Linux kernel compliance activities.  However, with ZFS, there is another copyright holder: Oracle.  Nothing prevents Oracle suing for copyright violation with a theory of harm that says something like the CDDL was deliberately designed to be incompatible with GPLv2 to prevent ZFS being shipped in Linux and as the shipper of products base on ZFS, they’ve suffered commercial harm (which would be quantifiable) by this action.

efitools arm 32 bit build fixed

It turned out there was a bug in gnu-efi, in the linker script.  I’ve added a local fix for this to the OBS UEFI project and filed a bug with gnu-efi on sourceforge.  The net result is that the arm 32 bit binaries are now passing most of their tests (the remaining problems may be due to faults in the UEFI environment, still investigating).  I should also have the OVMF image for arm (the ArmVirt package) building for 32 bit arm shortly.

I’ve released version 1.6.1 of efitools to make sure a new version gets rebuilt.

efitools for arm released

Thanks to using architecture emulation containers, I can now build and test efitools for arm (both 64 bit: aarch64 and 32 bit armv7l) on my x86_64 laptop.  To test the actual EFI binaries, you need the arm equivalent of OVMF which is ArmVirt.  I’ve got the build service spitting out the images (still called OVMF because of package naming requirements) for those who want to try this at home.  To run the UEFI image, you need qemu-system-<arch> which, on SUSE, is provided by the qemu-arm package.  The Linaro ARM64 site has detailed instructions, but for the impatient, the required command line (for aarch64) is

qemu-system-aarch64 \
        -m 2048 \
        -cpu cortex-a57 \
        -M virt \
        -bios /home/jejb/ovmf-aarch64.bin \
        -serial stdio \
        -drive if=none,file=fat:.,id=hd0 \
        -device virtio-blk-device,drive=hd0

Which brings up the UEFI bios image and mounts the current directory as an additional fat volume.  The Linaro site recommends 1024 for the memory size, but if you want to boot a kernel or do anything fancy, you’ll find 2048 is the minimum.  The same thing will work for 32 bit arm (except that you want to remove the -cpu line).

To get efitools to build, I had to get the arm versions of sbsigntools up and running, which I have.  The source tree I used for this is here.  There were a few modifications necessary to build arm, but nothing drastic.  Since the Ubuntu version of the git tree appears to be dead, I’ve merged back all the ubuntu patches and will maintain this going forwards.  I’ll bump the release version to 0.8 when I’m sure everything is stable.

Status

Using this, the current status is that the aarch64 binaries are all running fine and have verified nicely.  Unfortunately, there’s still some problem with generating efi binaries for the 32 bit arm systems.  Most of the simple binaries are running OK, but the more complex ones (like KeyTool) are showing symptoms of data relocation problems which I’m still investigating.  I’ll push a new release of efitools when I can get this sorted out, although it looks like a more generic problem inside gnu-efi itself.  Edk2 builds of am32 efi binaries seem to be working, so I’ll see if I can deduce what’s missing from gnu-efi.

Constructing Architecture Emulation Containers

Usually container related stuff goes on $EMPLOYER blog, but this time, I had a container need for my hobbies. the problem: how to build and test efitools for arm and aarch64 while not possessing any physical hardware.  The solution is to build an architecture emulation container using qemu and mount namespaces such that when its entered you find yourself in your home directory but with the rest of Linux running natively (well emulated natively via qemu) as a new architecture.  Binary emulation in Linux is nothing new: the binfmt_misc kernel module does it, and can execute anything provided you’ve told it what header to expect and how to do the execution.  Most distributions come with a qemu-linux-user package which will usually install the necessary binary emulators via qemu to run non-native binaries.  However, there’s a problem here: the installed binary emulator usually runs as /usr/bin/qemu-${arch}, so if you’re running a full operating system container, you can’t install any package that would overwrite that.  Unfortunately for me, the openSUSE Build Service package osc requires qemu-linux-user and would cause the overwrite of the emulator and the failure of the container.  The solution to this was to bind mount the required emulator into the / directory, where it wouldn’t be overwritten and to adjust the binfmt_misc paths accordingly.

Aside about binfmt_misc

The documentation for this only properly seems to exist in the kernel Documentation directory as binfmt_misc.txt.  However, very roughly, the format is

:name:type:offset:magic:mask:interpreter:flags

name is just a handle which will appear in /proc/sys/fs/binfmt_misc, type is M for magic or E for extension (Magic means recognise the type by the binary header, the usual UNIX way and E means recognise the type by the file extension, the Windows way). offset is where in the file to find the magic header to recognise;  magic and mask are the mask to and the binary string with and the magic to find once the masking is done.  Interpreter is the name of the interpreter to execute and flags tells binfmt_misc how to execute the interpreter.  For qemu, the flags always need to be OC meaning open the binary and generate credentials based on it (this can be seen as a security problem because the interpreter will execute with the same user and permissions as the binary, so you have to trust it).

If you’re on a systemd system, you can put all the above into /etc/binfmt.d/file.conf and systemd will feed it to binfmt_misc on boot.  Here’s an example of the aarch64 emulation file I use.

Bootstrapping

To bring up a minimal environment that’s fully native, you need to bootstrap it by installing just enough binaries using your native system before you can enter the container.  At a minimum, this is enough shared libraries and binaries to run the shell.  If you’re on a debian system, you probably already know how to use debootstrap to do this, but if you’re on openSUSE, like me, this is a much harder proposition because persuading zypper to install non native binaries isn’t easy.  The first thing you need to know is that you need to install an architectural override for libzypp in the file pointed to by the ZYPP_CONF environment variable. Here’s an example of a susebootstrap shell script that will install enough of the architecture to run zypper (so you can install all the packages you actually need).  Just run it as (note, you must have the qemu-<arch> binary installed because the installer will try to run pre and post scripts which may fail if they’re binary unless the emulation is working):

susebootstrap --arch <arch> <location>

And the bootstrap image will be build at <location> (I usually choose somewhere in my home directory, but you can use /var/tmp or anywhere else in your filesystem tree).  Note this script must be run as root because zypper can’t change ownership of files otherwise.  Now you are ready to start the architecture emulation container with <location> as the root.

Building an Architecture Emulation Container

All you really need now is a mount namespace with <location> as the real root and all the necessary Linux filesystems like /sys and /proc mounted.  Additionally, you usually want /home and I also mount /var/tmp so there’s a standard location for all my obs build directories.  Building a mount namespace is easy: simply unshare –mount and then bind mount everything you need.  Finally you use pivot_root to swap the new and old roots and unmount -l the old root (-l is necessary because the mount point is in-use outside the mount namespace as your real root, so you just need it unbinding, you don’t need to wait until no-one is using it).

All of this is easily scripted and I created this script to perform these actions.  As a final act, the script binds the process and creates an entry link in /run/build-container/<arch>.  This is the command line I used for the example below:

build-container --arch arm --location /home/jejb/tmp/arm

Now entering the build container is easy (you still have to enter the namespace as root, but you can exec su – <user> to become whatever your non-root user is).

jejb@jarvis:~> sudo -s
jarvis:/home/jejb # uname -m
x86_64
jarvis:/home/jejb # nsenter --mount=/run/build-container/arm
jarvis:/ # uname -m
armv7l
jarvis:/ # exec su - jejb
jejb@jarvis:~> uname -m
armv7l
jejb@jarvis:~> pwd
/home/jejb

And there you are, all ready to build binaries and run them on an armv7 system.

Aside about systemd and Shared Subtrees

On a normal linux system, you wouldn’t need to worry about any of this, but if you’re running systemd, you do, because systemd has some very inimical properties (to mount namespaces) you need to be aware of.

In Linux, a bind mount creates a subtree.  Because you can bind mount from practically anywhere to anywhere, you can have many such subtrees that are substantially related.  The default way to create subtrees is “private” this means that even if the subtrees are effectively the same set of files, a mount operation on one isn’t seen by any of the others.  This is great, because it’s precisely what you want for containers.  However, if a subtree is set to shared (with the mount –make-shared command) then all mount and unmount operations a propagated to every shared copy.  The reason this matters for systemd is because systemd at start of day sets every mount point in the system to shared.  Unless you re-privatise the bind mounts as you create the architecture emulation container, you’ll notice some very weird effects.  Firstly, because pivot_root won’t pivot to a shared subtree, that call will fail but secondly, you’ll notice that when you umount -l /old-root it will propagate to the real root and unmount everything (like your root /proc /dev and /sys) effectively rendering your system unusable.  the mount –make-rprivate /old-root recursively descends the /old-root and sets all the mounts to private so the umount -l simply detached the /old-root instead of propagating all the umount events.

A Modest Proposal on the DCO

In this post, I discussed why corporations are having trouble regarding the DCO as sufficient for contributions to projects using licences which require patent grants.  The fear being that rogue corporations could legitimately claim that under the DCO they were authorizing their developers as agents for copyrights but not for patents.  Rather than argue about the legality of this trick, I think it will be much more productive to move the environment forwards to a place where it simply won’t work.  The key to doing this is to change the expectations of the corporate players which moves them to the point where they expect that a corporate signoff under the DCO gives agency for both patents and copyrights because once this happens for most of them (the good actors), the usual estoppal rules would make it apply to all.

The fact is that even though corporate lawyers fear that agency might not exist for patent grants via DCO signoffs in contributions, all legitimate corporate entities who make bona fide code contributions wish to effect this anyway; that’s why they go to the additional lengths of setting up Contributor Licence Agreements and signing them.  The corollary here is that really only a bad actor in the ecosystem wishes to perpetuate the myth that patents aren’t handled by the DCO.  So if all good actors want the system to work correctly anyway, how do we make it so?

The lever that will help to make this move is a simple pledge, which can be published on a corporate website,  that allows corporations expecting to make legitimate contributions to patent binding licences under the DCO to do so properly without needing any additional Contributor Licence Agreements.  Essentially it would be an explicit statement that when their developers submit code to a project under the DCO using a corporate signoff, they’re acting as agents for the necessary patent and copyright grants, meaning you can always trust a DCO signoff from that corporation.  When enough corporations do this, it becomes standard practice and thus expectations on the DCO have moved to the point we originally assumed they were at, so here’s the proposal for what such a statement would look like.


 

Corporate Contribution Pledge

Preamble

It is our expectation that any DCO signoff from a corporate email address binds that corporation to grant all necessary copyright and, where required, patent rights to satisfy the terms of the licence.  Accordingly, we are publishing this pledge to illustrate how, as a matter of best practice, we implement this expectation.

For the purposes of this pledge, our corporate email domain is @bigcorp.com and its subdomains.

Limitations

  1. This pledge only applies to projects which use an OSI accepted Open Source  licence and which also use a developer certificate of origin (DCO).
  2. No authority is given under this pledge to sign contribution agreements on behalf of the company or otherwise bind it except by contributing code under an OSI approved licence and DCO process.
  3. No authority is given under this pledge if a developer, who may be our employee, posts patches under an email address which is not our corporate email domain above.
  4. No trademarks of this corporation may ever be bound under this pledge.
  5. Except as stated below, no other warranty, express or implied, is made on behalf of the contribution, including, but not limited to, fitness of the code for a specific purpose or merchantability.  The entire risk of the quality and performance of this contribution rests with the recipient.

Warranties

  1. Our corporation trains its Open Source contributors carefully to understand when they may and may not post patches from our corporate email domain and to obtain all necessary internal clearances according to our processes before making such a posting.
  2. When one of our developers posts a patch to a project under an OSI approved licence with a DCO Signed-off-by: from our corporate email domain, we authorise that developer to be our agent in the minimum set of patent and copyright grants that are required to satisfy the terms of the OSI approved licence for the contribution.

The DCO, Patents and OpenStack

Historically, the Developer Certificate of Origin originally adopted by the Linux Kernel in 2005 has seen widespread use within a large variety of Open Source projects.  The DCO is designed to replace a Contributor Licence Agreement with a simple Signed-off-by attestation which is placed into the commit message of the source repository itself, thus meaning that all the necessary DCO attestations are automatically available to anyone who downloads the source code repository.  It also allows (again, through the use of a strong source control system) the identification of who changed any given line of code within the source tree and all their DCO signoffs.

The legal basis of the DCO is that it is an attestation by an individual developer that they have sufficient rights in the contribution to submit it under the project (or file) licence.

The DCO and Corporate Contributions

In certain jurisdictions, particularly the United States of America, when you work as a software developer for a Corporation, they actually own, exclusively, the copyright of any source code you produce under something called the Work for Hire doctrine.  So the question naturally arises: if the developer who makes the Signed-off-by attestation  doesn’t actually own any rights in the code, how is that attestation valid and how does the rights owning entity (the corporation) actually license the code correctly to make the contribution?

The answer to that question resides in something called the theory of agency.  Agency is the mechanism by which individuals give effect to actions of a corporation.  For example, being a nebulous entity with no actual arms or legs, a corporation cannot itself sign any documents.  Thus, when a salesman signs a contract to supply widgets on behalf of a corporation, he is acting as the agent of that corporation.  His signature on the sales contract becomes binding on the corporation as if the corporation itself had made it.  However, there’s a problem here: how does the person who paid for and is expecting the delivery of widgets know that the sales person is actually authorised to be an agent of the corporation?  The answer here is again in the theory of agency: as long as the person receiving the widgets had reasonable cause to think that the salesperson signing the contract is acting as an agent of the corporation.  Usually all that’s required is that the company gave the salesperson a business card and a title which would make someone think they were authorised to sign contracts (such as “Sales Manager”).

Thus, the same thing applies to developers submitting patches on behalf of a corporation.  They become agents of that corporation when making DCO attestations and thus, even if the contribution is a work for hire and the copyright owned by the corporation, the DCO attestation the developer makes is still binding on the corporation.

Email addresses matter

Under the theory of agency, it’s not sufficient to state “I am an agent”, there must be some sign on behalf of the corporation that they’re granting agency (in the case of the salesperson, it was a business card and checkable title).  For developers making contributions with a Signed-off-by, the best indication of agency is to do the signoff using a corporate email address.  For this reason, the Linux kernel has adopted the habit of not accepting controversial patches without a corporate signoff.

Patents and the DCO

The Linux Kernel uses GPLv2 as its licence.  GPLv2 is solely a copyright licence and has nothing really to say about patents, except that if you assert against the project, you lose your right to distribute under GPLv2.  This is what is termed an implied patent licence, but it means that the DCO signoff for GPLv2 only concerns copyrights.  However, there are many open source licences (like Apache-2 or GPLv3) which require explicit patent grants as well as copyright ones, so can the DCO give all the necessary rights, including patent ones, to make the contribution on behalf of the corporation?  The common sense answer, which is if the developer is accepted as an agent for copyright, they should also be an agent for patent grants, isn’t as universally accepted as you might think.

The OpenStack problem

OpenStack has been trying for years to drop its complex contributor licence infrastructure in favour of a simple DCO attestation.  Most recently, the Technical Committee made that request of the board in 2014 and it was finally granted in a limited fashion on November 2015.  The recommendation of the OpenStack counsel was accepted and the DCO was adopted for individuals only, keeping the contributor licence agreements for corporations.  The given reason for this is that the corporate members of OpenStack want more assurance that corporations are correctly granting their patents in their contributions than they believe the DCO gives (conversely, individuals aren’t expected to have any patents, so, for them, the DCO applies just fine since it’s effectively only a copyright attestation they’re giving).

Why are Patents such an Issue?

Or why do lots of people think developers aren’t agents for patents in contributions unlike for copyrights?  The essential argument (as shown here) is that corporations as a matter of practise, do not allow developers (or anyone else except specific title holders) to be agents for patent transactions and thus there should not be an expectation, even when they make a DCO attestation using a corporate email signoff, that they are.

One way to look at this is that corporations have no choice but to make developers agents for the copyright because without that, the DCO attestation is false since the developers themselves has no rights to a work for hire piece of code.  However, some corporations think they can get away with not making developers agents for patents because the contribution and the licence do not require this to happen.  The theory here is that the developer is making an agency grant for the copyright, but an individual grant of the patents (and, since developers don’t usually own patents, that’s no grant at all).  Effectively this is a get out of jail free card for corporations to cheat on the patent requirements of the licence.

Does this interpretation really hold water?  Well, I don’t think so, because it’s deceptive.  It’s deliberately trying to evade the responsibilities for patents that the licences require.  Usually under the theory of agency, deceptive practises are barred.  However, the fear that a court might be induced to accept this viewpoint is sufficient to get the OpenStack board to require that corporations sign a CLA to ensure that patents are well and truly bound.  The problem with this viewpoint is that, if it becomes common enough, it ends up being de facto what the legal situation actually is (because the question courts most often ask in cases about agency is what would the average person think, so a practise that becomes standard in the industry ipso facto becomes what the average reasonable person would think).  Effectively therefore, the very act of OpenStack acting on its fear causes the thing they fear eventually to become true.  The only way to reverse this now is to change the current state of how the industry views patents and the DCO … and that will be the subject of another post.

Respect and the Linux Kernel Mailing Lists

I recently noticed that Sarah Sharp resigned publicly from the kernel giving a failure to impose a mandatory code of conduct as the reason and citing interaction problems, mainly on the mailing lists.  The net result of this posting, as all these comments demonstrate, is to imply directly that nothing has ever changed.  This implication is incredibly annoying, firstly because it is actually untrue, secondly because it does more to discourage participation than the behaviour that is being complained about and finally because it totally disrespects and ignores the efforts of hundreds of people who, over the last decade or so, have been striving to improve all interactions around Linux … a rather nice irony given that “respect” is listed as one of the issues for the resignation.  I’d just like to remind everyone of the history of these efforts and what the record shows they’ve achieved.

The issue of respect on the Mailing lists goes way back to the beginnings of Linux itself, but after the foundation of the OSDL (precursor to the Linux Foundation) Technical Advisory Board (TAB), one of its first issues from OSDL member companies was the imbalance between Asian and European/American contributions to the kernel.  The problems were partly to do with Management culture and partly because the lack of respect on the various mailing lists was directly counter to the culture of respect in a lot of Asian countries and disproportionately discouraged contributions from that region.  The TAB largely works behind the scenes, but some aspects of the effort filtered into the public domain as can be seen with a session on developer relations at the 2007 kernel summit (and, in fact, at a lot of other kernel summits since then).  Progress was gradual, and influenced by a large number of people, but the climate did improve.  I have to confess that I don’t follow LKML (not because of the flame war issues, simply because it’s too much of a firehose); however, the lists I do participate in (linux-scsi, linux-ide, linux-mm, linux-fsdevel, linux-efi, linux-arch, linux-parisc) haven’t seen any flagrantly disrespectful and personally insulting posts for several years now.  Indeed, when an individual came along who could almost have been flame bait for this with serial efforts to get incorrect and badly thought out patches into the kernel (I won’t give cites here to avoid stigmatising individuals) they met with a large reserve of patience and respectful and helpful advice before finally being banned from the lists for being incorrigible … no insults or flames at all.

Although I’d love to take credit for some of this, I’ve got to say that I think the biggest influencer towards civility is actually the “professionalisation”  of Linux: Employers pay people to work on Linux but the statements of those people become identified with their employers (no matter how many disclaimers they have) … in many ways, Open Source engineers are the new corporate spokespeople.  All employers bear this in mind when they hire and they certainly look over the mailing lists to see how people behave.  The net result is really that the only people who can afford to be rude or abusive are those who don’t think they have much chance of a long term career in Linux.

So, by and large, I’m proud of the achievements we’ve made in civility and the way we have improved over the years.  Are we perfect? by no means (but then perfection in such a large community isn’t a realistic goal).  However, we have passed our stress test: that an individual with bad patches to several mailing lists was met with courtesy and helpful advice, in spite of serially repeating the behaviour.

In conclusion, I’d just like to note that even the thread that gave rise to Sarah’s desire to pursue a code of conduct is now over two years old and try as they might, no-one’s managed to come up with a more recent example and no-one has actually invoked the voluntary code of conflict, which was the compromise for not having a mandatory code of conduct.  If it were me, I’d actually take that as a sign of success …

DNSSEC, DANE and the failure of X.509

As a few people have noticed, I’m a bit of an internet control freak: In an age of central “cloud based” services, I run pretty much my own everything (blog, mail server, DNS, OpenID, web page etc.).  That doesn’t make me anti-cloud; I just believe in federation instead of centralisation.  In particular, I believe in owning my own content and obeying my own rules rather than those of $BIGCLOUDPROVIDER.

In the modern world, this is perfectly possible: I rent a co-lo site and I have a DNS delegation so I can run and tune my own services how I wish.  I need a secure web server for a few things (OpenID, an email portal, secure log in for my blog etc) but I’ve always used a self-signed certificate.  Why?  well having to buy one from a self appointed X.509 root of trust always really annoyed me.  Firstly because they do very little for the money; secondly because it means effectively giving my security to some self appointed entity; and thirdly, as all the compromises and misuse attests, the X.509 root of trust model is fundamentally broken.

In the ordinary course of events, none of this would affect me.  However, recently, curl, which is used as the basis of most OpenID implementations took to verifying X.509 certificate chains, meaning that OpenID simply stopped working for ID providers with self signed certificates and at a stroke I was locked out of quite a few internet sites.  The only solution is to give up on OpenID or swallow pride and get a chained X.509 certificate.  Fortunately startssl will issue free certificates and the Linux Foundation is also getting into the game, so the first objection is overcome but not the other two.

So, what’s the answer?  As a supporter of cloud federation, I really like the monkeysphere approach which links ssl certificate verification directly to the user’s personal pgp web of trust.  Unfortunately, that also means that monkeysphere suffers from all the usual web of trust problems, the biggest being that it’s pretty much inaccessible to non-techies who just don’t understand why they should invest time in building up their own trust contacts.  That’s not to say that the web of trust can’t be made accessible in a simple fashion to everyone and indeed google is working on a project along these lines; however, today the reality is that today it isn’t.

Enter DANE.  At is most basic, DANE is a protocol that links certificate verification to the DNS.  It means that because anyone who owns a domain must have a DNS entry somewhere and the ability to modify it, they can directly link their certificate verification to this ability.  To my mind, this represents a nice compromise between making the system simple for end users and the full federation of the web of trust.  The implementation of DANE relies on DNSSEC (which is a royal pain to set up and get right, but I’ll do another blog post about that).  This means that effectively DANE has the same operational model as X.509, because DNSSEC is a hierarchically rooted trust model.  It also means that the delegation record to your domain is managed by your registrar and could be compromised if your registrar is.  However, as long as you trust the DNSSEC root and your registrar, the ability to generate trusted certificates for your domain is delegated to you.  So how is this different from X509?  Surely abusive registrars could cause similar problems as abusive or negligent X.509 roots?  That’s true, but an abusive registrar can only affect their own domain and delegates, they can’t compromise everyone else (unlike X.509), so to give an example of recent origin: the Chinese registrar could falsify the .cn domain, but wouldn’t be able to issue false certificates for the .com one.  The other reason for hope is that DNSSEC is at the root of the scheme to protect the DNS infrastructure itself from attack.  This makes the functioning and administration of DNSSEC a critical task for ICANN itself, so it’s a fair bet to assume that any abuse by a registrar won’t just result in a light slap on the wrist and a vague threat to delist their certificate in some browsers, but will have ICANN threatening to revoke their accreditation and with it, their entire business model as a domain registrar.

In many ways, the foregoing directly links the business model of the registrars to making DNSSEC work correctly for you.  In theory, the same is true of the X.509 CA roots of trust, of course, but there’s no one sitting at the top making sure they behave (and the fabric of the internet isn’t dependent on securing this behaviour, even if there were).

Details of DANE

In spite of the statements above, DANE is designed to complement X.509 as well as replace it.  Dane has four separate certificate verification styles, two of which integrate with X.509 and solve specific threats in its model (the actual solution is called pinning, a way of protecting yourself from the proliferation of X509 CAs all of whom could issue certificates for your site):

  • Mode 0 – X.509 CA pinning: The TLSA record identifies the exact CA the TLS Certificate must chain to.  This certificate must also be a trusted root in the X.509 certificate database.
  • Mode 1 – Certificate Contstraint: The TLSA record identifies the site certificate and that certificate must also pass X.509 validation
  • Mode 2 – Trust Anchor Assertion: The TLSA record specifies the certificate to which the  TLS Certificate must chain to under standard X.509 validation, but this certificate is not expected to be an X.509 root of trust.
  • Mode 3 – Domain Issued Certificates: The TLSA record specifies exactly the TLS certificate which the service must use.  This allows domains securely to specify verifiable self signed certificates.

Mode 3 is most commonly used to specify an exact certificate outside of the X.509 chain.  Mode 2 can be useful, but the site must have access to an external certificate store (using the DNS CERT records) or the TLSA record must specify the full certificate for it to work.

Who Supports DANE?

This is the big problem:  For certificates distributed via DANE to be useful, there must be support for them in browsers.  For Mozilla, there is the DANE validator extension but in spite of several attempts, nothing actually built into the browser certificate verifier itself.  The most complete patch set is from the DNSSEC people and there’s also a Mozilla inspired one about how they will add it one day but right at the moment it isn’t a priority.  The Chromium browser has had a similar attitude.  The conspiracy theorists are quick to point out that this is because the browser companies derive considerable revenue from the CA system, which is in itself a multi-billion dollar industry and thus there’s active lobbying against anything that would dilute the power, and hence perceived value, of the CA roots.  There is some evidence for this position in that Google recognises that certificate pinning, which DANE supports, can protect against recent fake google certificate attacks, but, while supporting DNSSEC (at least for validation, the google DNS doesn’t secure itself via DNSSEC), they steadfastly ignore DANE certificate pinning and instead have a private arrangement with Mozilla.

I learned long ago: never to ascribe to malice (or conspiracy) what can be easily explained by incompetence (or political problems).  In this case, the world was split long ago into using openssl for security (in spite of the problematic licence) or using nss (the Netscape Security Services).  Mozilla, of course, uses the latter but every implementation of DANE for mozilla (including the patches in the bugzilla) use openssl.   I actually have an experimental build of mozilla with DANE, but incorporating the two separate SSL systems is a real pain.  I think it’s safe to say that until someone comes up with a nss based DANE verifier, the DANE patches for mozilla still aren’t yet up to the starting blocks, and thus conspiracy allegations are somewhat premature.  Unfortunately, the same explanation applies to chromium: for better or worse, it’s currently using nss for certificate validation as well.

Getting your old Sync Server to work with New Firefox

Much has been written about Mozilla trying to force people to use their new sync service.  If, like me, you run your own sync server for Firefox, you’ve mostly been ignoring this because there’s still no real way of running your own sync server for the new service (and if you simply keep upgrading, Firefox keeps working with your old server).

However, recently I had cause to want to connect my old sync server to a new installation of firefox without just copying over all the config files (one of the config settings broke google docs and I couldn’t figure out which one it was, so I figured I’d just blow the entire config away and restore from sync).  Long ago Mozilla disabled the ability to connect newer Firefoxes to an old sync server, so this is an exposé of how to do it.  I did actually search the internet for this one, but no-one else seems to have figured it out (or if they have, they’re not known to the search engines).

There are two config files you need to update get new Firefox to connect to sync (note, I did this with Firefox 37; I’ve not tested it with a different version, but I’m pretty sure it will work).  The first is that you need to put your sync key and weave user login into logins.json.  Since the password and user are encypted in this file, the easiest way is to use a password manager extension, like Saved Password Editor add on.  Then you need two new password entries of type “Annotated” under the host chrome://weave.  For each, your username is your weave username.  For the first, you’re going to add your weave password under the annotation “Mozilla Services Password”.  For the second, add the Firefox  key with all the dashes removed as the password under the annotation “Mozilla Services Encryption Passphrase”.  If you’ve got all this right, password manager will show this (my username is jejb):

tmpNext you’re going to close firefox and manually edit the prefs.js file.  To sync completely from scratch, this just needs three entries, so firstly strip out every preference that begins ‘services.sync.’ and then add three new lines

user_pref("services.sync.account", "<my account>");
user_pref("services.sync.serverURL", "<my weave URL>");
user_pref("services.sync.username", "<my weave user name>");

For most people, the account and weave user name are the same.  Now start Firefox and it should just sync on its own.  To check that you got this right, go to the Sync tab of preferences and you should see something like this

tmp

And that’s it.  You’re all done.