Commit 75b02146 authored by Jonathan Corbet's avatar Jonathan Corbet

Add the development process document

This is an extended document intended to help interested developers, their
managers, and their employers work with the kernel development process.

This work was supported by the Linux Foundation.
Signed-off-by: default avatarJonathan Corbet <>
parent d86f4bc4
......@@ -21,6 +21,9 @@ Changes
- list of changes that break older software packages.
- how the boss likes the C code in the kernel to look.
- An extended tutorial on how to work with the kernel development
- DMA API, pci_ API & extensions for non-consistent memory machines.
This diff is collapsed.
This diff is collapsed.
When contemplating a Linux kernel development project, it can be tempting
to jump right in and start coding. As with any significant project,
though, much of the groundwork for success is best laid before the first
line of code is written. Some time spent in early planning and
communication can save far more time later on.
Like any engineering project, a successful kernel enhancement starts with a
clear description of the problem to be solved. In some cases, this step is
easy: when a driver is needed for a specific piece of hardware, for
example. In others, though, it is tempting to confuse the real problem
with the proposed solution, and that can lead to difficulties.
Consider an example: some years ago, developers working with Linux audio
sought a way to run applications without dropouts or other artifacts caused
by excessive latency in the system. The solution they arrived at was a
kernel module intended to hook into the Linux Security Module (LSM)
framework; this module could be configured to give specific applications
access to the realtime scheduler. This module was implemented and sent to
the linux-kernel mailing list, where it immediately ran into problems.
To the audio developers, this security module was sufficient to solve their
immediate problem. To the wider kernel community, though, it was seen as a
misuse of the LSM framework (which is not intended to confer privileges
onto processes which they would not otherwise have) and a risk to system
stability. Their preferred solutions involved realtime scheduling access
via the rlimit mechanism for the short term, and ongoing latency reduction
work in the long term.
The audio community, however, could not see past the particular solution
they had implemented; they were unwilling to accept alternatives. The
resulting disagreement left those developers feeling disillusioned with the
entire kernel development process; one of them went back to an audio list
and posted this:
There are a number of very good Linux kernel developers, but they
tend to get outshouted by a large crowd of arrogant fools. Trying
to communicate user requirements to these people is a waste of
time. They are much too "intelligent" to listen to lesser mortals.
The reality of the situation was different; the kernel developers were far
more concerned about system stability, long-term maintenance, and finding
the right solution to the problem than they were with a specific module.
The moral of the story is to focus on the problem - not a specific solution
- and to discuss it with the development community before investing in the
creation of a body of code.
So, when contemplating a kernel development project, one should obtain
answers to a short set of questions:
- What, exactly, is the problem which needs to be solved?
- Who are the users affected by this problem? Which use cases should the
solution address?
- How does the kernel fall short in addressing that problem now?
Only then does it make sense to start considering possible solutions.
When planning a kernel development project, it makes great sense to hold
discussions with the community before launching into implementation. Early
communication can save time and trouble in a number of ways:
- It may well be that the problem is addressed by the kernel in ways which
you have not understood. The Linux kernel is large and has a number of
features and capabilities which are not immediately obvious. Not all
kernel capabilities are documented as well as one might like, and it is
easy to miss things. Your author has seen the posting of a complete
driver which duplicated an existing driver that the new author had been
unaware of. Code which reinvents existing wheels is not only wasteful;
it will also not be accepted into the mainline kernel.
- There may be elements of the proposed solution which will not be
acceptable for mainline merging. It is better to find out about
problems like this before writing the code.
- It's entirely possible that other developers have thought about the
problem; they may have ideas for a better solution, and may be willing
to help in the creation of that solution.
Years of experience with the kernel development community have taught a
clear lesson: kernel code which is designed and developed behind closed
doors invariably has problems which are only revealed when the code is
released into the community. Sometimes these problems are severe,
requiring months or years of effort before the code can be brought up to
the kernel community's standards. Some examples include:
- The Devicescape network stack was designed and implemented for
single-processor systems. It could not be merged into the mainline
until it was made suitable for multiprocessor systems. Retrofitting
locking and such into code is a difficult task; as a result, the merging
of this code (now called mac80211) was delayed for over a year.
- The Reiser4 filesystem included a number of capabilities which, in the
core kernel developers' opinion, should have been implemented in the
virtual filesystem layer instead. It also included features which could
not easily be implemented without exposing the system to user-caused
deadlocks. The late revelation of these problems - and refusal to
address some of them - has caused Reiser4 to stay out of the mainline
- The AppArmor security module made use of internal virtual filesystem
data structures in ways which were considered to be unsafe and
unreliable. This code has since been significantly reworked, but
remains outside of the mainline.
In each of these cases, a great deal of pain and extra work could have been
avoided with some early discussion with the kernel developers.
When developers decide to take their plans public, the next question will
be: where do we start? The answer is to find the right mailing list(s) and
the right maintainer. For mailing lists, the best approach is to look in
the MAINTAINERS file for a relevant place to post. If there is a suitable
subsystem list, posting there is often preferable to posting on
linux-kernel; you are more likely to reach developers with expertise in the
relevant subsystem and the environment may be more supportive.
Finding maintainers can be a bit harder. Again, the MAINTAINERS file is
the place to start. That file tends to not always be up to date, though,
and not all subsystems are represented there. The person listed in the
MAINTAINERS file may, in fact, not be the person who is actually acting in
that role currently. So, when there is doubt about who to contact, a
useful trick is to use git (and "git log" in particular) to see who is
currently active within the subsystem of interest. Look at who is writing
patches, and who, if anybody, is attaching Signed-off-by lines to those
patches. Those are the people who will be best placed to help with a new
development project.
If all else fails, talking to Andrew Morton can be an effective way to
track down a maintainer for a specific piece of code.
If possible, posting your plans during the early stages can only be
helpful. Describe the problem being solved and any plans that have been
made on how the implementation will be done. Any information you can
provide can help the development community provide useful input on the
One discouraging thing which can happen at this stage is not a hostile
reaction, but, instead, little or no reaction at all. The sad truth of the
matter is (1) kernel developers tend to be busy, (2) there is no shortage
of people with grand plans and little code (or even prospect of code) to
back them up, and (3) nobody is obligated to review or comment on ideas
posted by others. If a request-for-comments posting yields little in the
way of comments, do not assume that it means there is no interest in the
project. Unfortunately, you also cannot assume that there are no problems
with your idea. The best thing to do in this situation is to proceed,
keeping the community informed as you go.
If your work is being done in a corporate environment - as most Linux
kernel work is - you must, obviously, have permission from suitably
empowered managers before you can post your company's plans or code to a
public mailing list. The posting of code which has not been cleared for
release under a GPL-compatible license can be especially problematic; the
sooner that a company's management and legal staff can agree on the posting
of a kernel development project, the better off everybody involved will be.
Some readers may be thinking at this point that their kernel work is
intended to support a product which does not yet have an officially
acknowledged existence. Revealing their employer's plans on a public
mailing list may not be a viable option. In cases like this, it is worth
considering whether the secrecy is really necessary; there is often no real
need to keep development plans behind closed doors.
That said, there are also cases where a company legitimately cannot
disclose its plans early in the development process. Companies with
experienced kernel developers may choose to proceed in an open-loop manner
on the assumption that they will be able to avoid serious integration
problems later. For companies without that sort of in-house expertise, the
best option is often to hire an outside developer to review the plans under
a non-disclosure agreement. The Linux Foundation operates an NDA program
designed to help with this sort of situation; more information can be found
This kind of review is often enough to avoid serious problems later on
without requiring public disclosure of the project.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
At this point, hopefully, you have a handle on how the development process
works. There is still more to learn, however! This section will cover a
number of topics which can be helpful for developers wanting to become a
regular part of the Linux kernel development process.
The use of distributed version control for the kernel began in early 2002,
when Linus first started playing with the proprietary BitKeeper
application. While BitKeeper was controversial, the approach to software
version management it embodied most certainly was not. Distributed version
control enabled an immediate acceleration of the kernel development
project. In current times, there are several free alternatives to
BitKeeper. For better or for worse, the kernel project has settled on git
as its tool of choice.
Managing patches with git can make life much easier for the developer,
especially as the volume of those patches grows. Git also has its rough
edges and poses certain hazards; it is a young and powerful tool which is
still being civilized by its developers. This document will not attempt to
teach the reader how to use git; that would be sufficient material for a
long document in its own right. Instead, the focus here will be on how git
fits into the kernel development process in particular. Developers who
wish to come up to speed with git will find more information at:
and on various tutorials found on the web.
The first order of business is to read the above sites and get a solid
understanding of how git works before trying to use it to make patches
available to others. A git-using developer should be able to obtain a copy
of the mainline repository, explore the revision history, commit changes to
the tree, use branches, etc. An understanding of git's tools for the
rewriting of history (such as rebase) is also useful. Git comes with its
own terminology and concepts; a new user of git should know about refs,
remote branches, the index, fast-forward merges, pushes and pulls, detached
heads, etc. It can all be a little intimidating at the outset, but the
concepts are not that hard to grasp with a bit of study.
Using git to generate patches for submission by email can be a good
exercise while coming up to speed.
When you are ready to start putting up git trees for others to look at, you
will, of course, need a server that can be pulled from. Setting up such a
server with git-daemon is relatively straightforward if you have a system
which is accessible to the Internet. Otherwise, free, public hosting sites
(Github, for example) are starting to appear on the net. Established
developers can get an account on, but those are not easy to come
by; see for more information.
The normal git workflow involves the use of a lot of branches. Each line
of development can be separated into a separate "topic branch" and
maintained independently. Branches in git are cheap, there is no reason to
not make free use of them. And, in any case, you should not do your
development in any branch which you intend to ask others to pull from.
Publicly-available branches should be created with care; merge in patches
from development branches when they are in complete form and ready to go -
not before.
Git provides some powerful tools which can allow you to rewrite your
development history. An inconvenient patch (one which breaks bisection,
say, or which has some other sort of obvious bug) can be fixed in place or
made to disappear from the history entirely. A patch series can be
rewritten as if it had been written on top of today's mainline, even though
you have been working on it for months. Changes can be transparently
shifted from one branch to another. And so on. Judicious use of git's
ability to revise history can help in the creation of clean patch sets with
fewer problems.
Excessive use of this capability can lead to other problems, though, beyond
a simple obsession for the creation of the perfect project history.
Rewriting history will rewrite the changes contained in that history,
turning a tested (hopefully) kernel tree into an untested one. But, beyond
that, developers cannot easily collaborate if they do not have a shared
view of the project history; if you rewrite history which other developers
have pulled into their repositories, you will make life much more difficult
for those developers. So a simple rule of thumb applies here: history
which has been exported to others should generally be seen as immutable
So, once you push a set of changes to your publicly-available server, those
changes should not be rewritten. Git will attempt to enforce this rule if
you try to push changes which do not result in a fast-forward merge
(i.e. changes which do not share the same history). It is possible to
override this check, and there may be times when it is necessary to rewrite
an exported tree. Moving changesets between trees to avoid conflicts in
linux-next is one example. But such actions should be rare. This is one
of the reasons why development should be done in private branches (which
can be rewritten if necessary) and only moved into public branches when
it's in a reasonably advanced state.
As the mainline (or other tree upon which a set of changes is based)
advances, it is tempting to merge with that tree to stay on the leading
edge. For a private branch, rebasing can be an easy way to keep up with
another tree, but rebasing is not an option once a tree is exported to the
world. Once that happens, a full merge must be done. Merging occasionally
makes good sense, but overly frequent merges can clutter the history
needlessly. Suggested technique in this case is to merge infrequently, and
generally only at specific release points (such as a mainline -rc
release). If you are nervous about specific changes, you can always
perform test merges in a private branch. The git "rerere" tool can be
useful in such situations; it remembers how merge conflicts were resolved
so that you don't have to do the same work twice.
One of the biggest recurring complaints about tools like git is this: the
mass movement of patches from one repository to another makes it easy to
slip in ill-advised changes which go into the mainline below the review
radar. Kernel developers tend to get unhappy when they see that kind of
thing happening; putting up a git tree with unreviewed or off-topic patches
can affect your ability to get trees pulled in the future. Quoting Linus:
You can send me patches, but for me to pull a git patch from you, I
need to know that you know what you're doing, and I need to be able
to trust things *without* then having to go and check every
individual change by hand.
To avoid this kind of situation, ensure that all patches within a given
branch stick closely to the associated topic; a "driver fixes" branch
should not be making changes to the core memory management code. And, most
importantly, do not use a git tree to bypass the review process. Post an
occasional summary of the tree to the relevant list, and, when the time is
right, request that the tree be included in linux-next.
If and when others start to send patches for inclusion into your tree,
don't forget to review them. Also ensure that you maintain the correct
authorship information; the git "am" tool does its best in this regard, but
you may have to add a "From:" line to the patch if it has been relayed to
you via a third party.
When requesting a pull, be sure to give all the relevant information: where
your tree is, what branch to pull, and what changes will result from the
pull. The git request-pull command can be helpful in this regard; it will
format the request as other developers expect, and will also check to be
sure that you have remembered to push those changes to the public server.
Some readers will certainly object to putting this section with "advanced
topics" on the grounds that even beginning kernel developers should be
reviewing patches. It is certainly true that there is no better way to
learn how to program in the kernel environment than by looking at code
posted by others. In addition, reviewers are forever in short supply; by
looking at code you can make a significant contribution to the process as a
Reviewing code can be an intimidating prospect, especially for a new kernel
developer who may well feel nervous about questioning code - in public -
which has been posted by those with more experience. Even code written by
the most experienced developers can be improved, though. Perhaps the best
piece of advice for reviewers (all reviewers) is this: phrase review
comments as questions rather than criticisms. Asking "how does the lock
get released in this path?" will always work better than stating "the
locking here is wrong."
Different developers will review code from different points of view. Some
are mostly concerned with coding style and whether code lines have trailing
white space. Others will focus primarily on whether the change implemented
by the patch as a whole is a good thing for the kernel or not. Yet others
will check for problematic locking, excessive stack usage, possible
security issues, duplication of code found elsewhere, adequate
documentation, adverse effects on performance, user-space ABI changes, etc.
All types of review, if they lead to better code going into the kernel, are
welcome and worthwhile.
There are numerous sources of information on Linux kernel development and
related topics. First among those will always be the Documentation
directory found in the kernel source distribution. The top-level HOWTO
file is an important starting point; SubmittingPatches and
SubmittingDrivers are also something which all kernel developers should
read. Many internal kernel APIs are documented using the kerneldoc
mechanism; "make htmldocs" or "make pdfdocs" can be used to generate those
documents in HTML or PDF format (though the version of TeX shipped by some
distributions runs into internal limits and fails to process the documents
Various web sites discuss kernel development at all levels of detail. Your
author would like to humbly suggest as a source;
information on many specific kernel topics can be found via the LWN kernel
index at:
Beyond that, a valuable resource for kernel developers is:
Information about the linux-next tree gathers at:
And, of course, one should not forget, the definitive
location for kernel release information.
There are a number of books on kernel development:
Linux Device Drivers, 3rd Edition (Jonathan Corbet, Alessandro
Rubini, and Greg Kroah-Hartman). Online at
Linux Kernel Development (Robert Love).
Understanding the Linux Kernel (Daniel Bovet and Marco Cesati).
All of these books suffer from a common fault, though: they tend to be
somewhat obsolete by the time they hit the shelves, and they have been on
the shelves for a while now. Still, there is quite a bit of good
information to be found there.
Documentation for git can be found at:
Congratulations to anybody who has made it through this long-winded
document. Hopefully it has provided a helpful understanding of how the
Linux kernel is developed and how you can participate in that process.
In the end, it's the participation that matters. Any open source software
project is no more than the sum of what its contributors put into it. The
Linux kernel has progressed as quickly and as well as it has because it has
been helped by an impressively large group of developers, all of whom are
working to make it better. The kernel is a premier example of what can be
done when thousands of people work together toward a common goal.
The kernel can always benefit from a larger developer base, though. There
is always more work to do. But, just as importantly, most other
participants in the Linux ecosystem can benefit through contributing to the
kernel. Getting code into the mainline is the key to higher code quality,
lower maintenance and distribution costs, a higher level of influence over
the direction of kernel development, and more. It is a situation where
everybody involved wins. Fire up your editor and come join us; you will be
more than welcome.
Markdown is supported
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment