What is the best way to learn Linux kernel development?

This is a question a I got a lot. Today I saw the question What's the best way to learn device driver development on Linux? on Quora  so I spent a few minutes answering it.

The answer finally was longer than I thought so I decide that it was worth a post. I don't know if this is the best approach to learn Linux kernel development and get involved, but is the one I recommend nowadays.

I for example didn't follow these steps, I got involved first watching Greg KH's Write and Submit your first Linux kernel Patch excellent video. After that I realized that anyone (even me) could contribute to the Linux kernel so I cloned the linux-next integration tree, compiled and started sending trivial one-liners patches.

But, after getting a job as a Linux Kernel Engineer and being contributing to the kernel for some time now I realized that there is a better and more fun way to get involved. Obviously device drivers is the best way to get involved for two reasons.

First, unless you are a PhD student doing research on operating systems. It is very unlikely that you will came with a brilliant idea an start hacking the Linux kernel virtual memory or block device subsystem. The Linux kernel has 20 years of development and its core is very, very complex. Second, it is more probable that your first job as a Linux kernel developer will be to write device drivers.

But, whatever is the way you chose to get involved, there is a step you can't skip and is to learn the theory behind operating systems in general and the Linux kernel in particular. So if you don't know about operating systems, I recommend you to read Willam Stalling's OS book. This book has a more hardware oriented approach unlike other OS books that focus more on the algorithms and data structures used in operating systems than explaining with examples how things work in the real world.

After having a high level overview of operating systems, you have to learn about Linux kernel development. Robert Love's Linux Kernel Development book is for me the best one on the subject. You can learn not only about Linux kernel development, but also the motivations behind the technical decisions that lead to the design.

Then you have to learn about device drivers development on Linux, two books that are very good are Linux Device Drivers and Essential Linux Device Drivers. This two books are complementary. The former teaches how to write virtual device drivers for memory based devices, so you can try the examples without the need of special hardware. While the latter shows you how device drives for real devices are actually written. I maintain an up-to-date repository with all LDD3 examples so you can compile and test on recent kernels.

Once you have the theory you can begin with the practice by writing real device drivers. You need a hardware that still is not supported on Linux. The good news is that you probably already have on of this devices.

With the popularity of the Android platform, is very likely that you own an Android device. Even when Android is a Linux-based operating system, Google by a design decision forked the Linux kernel and added some APIs that don't exist on the Linux kernel. These APIs are used by device drivers and for that reason, a developer has to choose whether to write a device driver for Android or Linux.

So, porting Android device drivers to Linux is an excellent opportunity to learn device driver development skills, the Linux kernel development process and how to work with the Linux community while doing something useful.

A final comment, when asking questions to kernel developers on public mailing lists, remember to do your homework and researched first. They tend to be very polite and collaborative with people that show signs that they tried to understand the problem first before to ask, but can be a little grumpy with people that ask first before even tried to understand the problem.

Also, don't be afraid to contact the developers that wrote the Android version device driver. They usually are working for the company that build the hardware of were contracted by them. Usually both the company and the original developer also want the driver to be merged on the Linux kernel but they don't have neither the time nor the resources to do it.

So, if you offer them to help and your time to modify the driver to be in an merge-able state, they will be more than happy to help you finish the task. This is important because you probably will need the hardware datasheet or at least information about the people that know well the hardware.

Happy hacking!

Building your own Nook Color Android kernel

On the last post I told that I'm working on Cypress TrueTouch(tm) touchscreen driver for mainline inclusion.

So to work in a driver first you need to compile a Linux (or Android) kernel. Since I don't have a fully functional Linux kernel for the Nook Color, I'm developing against an Android kernel.

I want to document how to build your own Android kernel for two reasons. First someone could find it useful. Second, I wanted to document the process for myself. I context switch more than I would like and when I comeback to something I didn't do in a few days, I have to remember what are the exact commands to build the uImage, what is the defconfig file I use to build each kernel, etc. This happens even more with my hobbyists projects since I don't work with them on a daily basis.

So these are the steps to build a custom Android kernel for your Nook Color:

1- Installing CyangenMod (CM7) on an SD card:

First you have to decide if you want to install your kernel on your boot partition of your nand memory or using an external SD card. I prefer to only install stable software on my nand so I did an CM7 installation on an SD card and used that for development.

To install a CM7 on your SD card first download and gunzip the generic sd card image.

$ wget http://nook.handhelds.ru/sdimage/generic-sdcard-v1.3.img.gz

$ gunzip generic-sdcard-v1.3.img.gz

Then insert your SD card and copy the image to your SD card:

$ dd if=generic-sdcard-v1.3.img of=/dev/sdX bs=1024

Where X is the block device name for your SD card. To know which block device is associated with your SD card, insert your card and look the kernel log messages with dmesg for example.
After done with writing eject and insert again your SD card on your computer. Download a nightly build of CM7 for the Nook from http://download.cyanogenmod.com/?device=encore I'm using -153 but I guess that a newer or older should work either.

$ wget http://download.cyanogenmod.com/get/cm_encore_full-153.zip

One the file has been downloaded, copy it to the SD card. There should be only one partition on the SD.

$ mount /dev/sdX1 /media/boot

$ cp cm_encore_full-153 /media/boot

That's all, now unmount the SD card, insert it on the Nook and power off. The generic image will find and update file and automatically start CM7 installation. The process will re-partition your SD card and copy all the necessary files in each of them (x-loader MLO, uBoot u-boot.bin, kernel image and ramdisk in the boot partition, binaries and libraries in the rootfs partition, etc).
Once the installation is complete the Nook will power off and you have a fully functional CM7 installation on your SD card. The Nook Color has boot sequence when the SD card has higher precedence than the nand memory. If the SD card is inserted, the Nook will attempt to boot from there. So with the SD inserted, just turn on again the Nook an CM7 will boot from there.
That is, you now have a system to test your newer kernel.

2- Instal a ARM cross-compiling tool-chain

Since the Nook has an ARM processor and it is unlikely that your host machine is ARM base, you will need to install an ARM cross-compiling tool-chain to compile ARM Android images.
You can install different cross tool-chains both from source code and using pre-compiled packages for your host machine distribution. For example to install Linaro's GCC cross tool-chain you can use the following command on Debian and Debian-based distros (such as Ubuntu):

$ apt-get install gcc-arm-linux-gnueabi

3- Obtaining and compiling an Android kernel

Now you need CM7 Android kernel, this can be found in Dalingrin github repository.

$ git clone https://github.com/dalingrin/nook_kernel.git

Now you have to checkout a local branch for your work:

$ cd nook_kernel

$ git checkout -b devel origin/encore-32

Now you have a local branch devel to do your kernel development.
Do all the modifications you wish to the Android kernel and to compile your kernel first generate a .config file from the Nook Color defconfig file:

$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi-  omap3621_evt1a_defconfig

This will generate a .config file that contains all the symbols needed to compile Nook Color's board file and needed drivers. You can customize the build with:

$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi-  menuconfig

Once your kernel compilation is customized you can generate the kernel image with the following command:

$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi-  uImage

Once it finish compiling, the uImage is located in arch/arm/boot/uImage.
The last step is to copy the uImage on the SD card. I prefer to mount the SD boot partition in the Nook Color using ssh or adb commands and copy the uImage using scp, but this setup will be explained in other article. Another option is to remove the SD card from the Nook, insert on your host machine, mounting the boot partition and copy the image:

$ mount /dev/sdX1 /media/boot

$ cp arch/arm/boot/uImage /media/boot

That is, now you can insert the SD card on your Nook Color and boot your customized Android kernel.

Happy Hacking!!

The power of open source

I own an Barnes & Nobles Nook Color e-reader. It is a fantastic device that can be rooted easily and used as a tablet pc.

Besides using with Android I wanted to install Linux on it, but there isn't a fully functional Linux kernel for the Nook. So I contacted Oleg Drokin (a.k.a verygreen) and William Marone (a.k.a wmarone) that were working in porting Nook Color's Android drivers to Linux.

I pick the task to work on the Cypress TrueTouch (tm) touchscreen driver for upstream inclusion. I knew that Kevin McNeely from Cypress submitted a patch-set for a newer and better designed driver for the TrueTouch family. I contacted Kevin asking if I could take that patch-set as a starting point for the driver and what modifications were needed for the driver to be accepted.

Kevin kindly told me that a requirement was to modify the driver to use a different multi-touch protocol type from the one the driver had. So I modified the driver to use the newer multi-touch protocol type B. Kevin also gave me guidance, advice and code when I didn't understand the driver internals.

I posted a patch-set which was reviewed and a few issues were found. Again Kevin point me out what modifications are necessaries to fix these issues.

So that is what I'm working on in my free time. I'll keep modifying the driver and resubmitting the code until the patch is merged or Kevin and the folks from linux-input get tired of me.

This post is now longer than I would like, but I wanted to tell the full story to show the power of open source. I own a hardware that doesn't meet all my needs, I'm able to hack the device, modify it at my will and find help both from hobbyists and companies like Cypress and its engineers.

I think this is the best moment in history to be an engineer, 20 years ago one could only dream of having a modern operating system such as Linux or Android with its complete source code to read, understand and modify.

I know I want to be a FOSS hacker all my life. I don't think there could be a better way to build high quality software such as Linux.

10 years anniversary with computers

It is incredible how time flies. I just realized that I have been hanging with computers for ten years now. Looking back I'm kind of happy with my achievements. I got two academic degrees, a B.S in Computer Engineering and a M.S in High Performance Computer.

I have worked in many sectors doing different things. I have administered Linux servers, developed ERP systems for wholesale companies, design and developed enterprise mobile applications and location based services for a telco and even build a data-warehouse and a GIS for a distribution company.

Also, I've worked in Academia as a researcher, developing a performance analysis and estimation tool for parallel applications running in Linux clusters and published papers in international conferences.

I also contributed to my favorite open source project, the Linux kernel with 65+ patches pushed to mainline.

But the best of all is that I have finally realized what I want to do, I think I've found my path. I realized that I want to become a full-time FOSS hacker. My experience contributing to open source and working both in the industry and the academia, has showed me that it is near impossible to build proprietary (or academic) systems with the same quality of their open source counterparts.

As Eric Raymond said, "With many eyes, all bugs become shallow". But not only the eyes but also the passion that open source developers put to their work and the high level of scrutiny to the code you post.

Also I see open source as a vehicle to do research, as Dirk Hohndel of Intel said, "open source is the shortest path to innovation".

Also, the possibility that your work will be used by millions of people really inspire me. The good news is that because of my previous work with the Linux kernel, a company offered me a job to work full time as an Linux embedded engineer. That means I will be doing not only Linux kernel development but also Linux user space development and even boot-loader development!

I had to make a huge decision. Today I quit as a researcher at the Universitat Autònoma de Barcelona and give up my scholarship for PhD studies. I had the pleasure to work with very smart people while doing research. I'd like to thank my advisors, Dr Emilio Luque and Dr. Dolores Rexachs and fellows at the Computer Architecture and Operating Systems department at UAB.
But I think my decision is correct given the path I choose.

I will work hard and do my best to achieve all my goals in this new adventure. As Linus Torvalds said "Its no the great ideas that bring innovation, its sweat and hard work"

So from now on expect me to post more about Linux kernel development, embedded systems, C programming and things like that!

Emacs configuration for Linux kernel development

Linux kernel programming is a hard task. Mostly due the size of the Linux kernel as a project. Fortunately there are some tools that can help us and make the task easier. One of those tools is the excellent Emacs editor.

In this blog entry I will show you how to make the best of Emacs for Linux kernel development. We will see how to use from Emacs the cscope and tags symbol databases and to set the code indentation according to the Linux kernel coding style.

The first problem I had when starting doing kernel development (and in any medium size project that I was involved BTW) was browsing the code. The first attempt and a method I still use for find complicate expressions is to use the commands grep, frep and the grep git option (git grep). Also you can use the Linux Cross Reference (LXR).

Another approach is to use tools that were created to make it easy to browse source code. Two very used utilities are cscope and ctags. They both create a symbol database known as the index. This index have all global symbols, that in the case of C programs, this will be global and member functions, structures, enums, typedefs and #defines.

etags is a custom flavor of ctags that generates a database that can be used by Emacs.

The tools can be used both to create the index and also to navigate the symbols in the index, for example to find all the places in the code that a symbol is declared or used.

To manually create the index for cscope, first you have to list all the files that have symbols that you want to add to the index, you can do this using the find or whatever command line utility is best for you:

$ find . -name "*.c" -o -name "*.h" > cscope.files

then to build the database:

$ cscope -qkRv

and finally to browse the database:

$ cscope -d

A similar procedure is needed to build the database for etags:

$ etags `find . -name "*.c" -o -name "*.h"`

Create cscope and etags symbol databases for the Linux kernel

Creating the databases for both cscope and etags in the kernel source directory is straightforward since the kernel main Makefile has custom targets for doing this.

So, just go to the directory where the kernel source code is:

$ cd /path/to/linux/source/code

and to create the cscope database execute:

$ make cscope

and to create the etags database

$ make TAGS

Use cscope and etags from Emacs

Once the databases are created, to use it from Emacs just open a source file that is inside the kernel source directory (whose symbols are already in the databases).

To use cscope, you have to add this line to your ~/.emacs config file

(require 'xcscope)

Note: Remember has to special keys that can be used to create key sequences. These keys are the Control and Meta key. When specifying key sequences these key are usually refer as Ctrl and Meta or C and M. These can be confusing for a new Emacs user.

So the key sequences:

Ctrl+c s s
C-c s s

are the same, and so are:

Meta+x
M-x

The Control and Meta keys are usually mapped to the Control and Alt keys, but this is not necessary true and in fact most people remap the Control key to the Caps Lock keyboard key.

Then to navigate trough the symbols just put your cursor over a symbol and use one of these commands:

C-c s s Find symbol.
C-c s d Find global definition.
C-c s g Find global definition (alternate binding).
C-c s G Find global definition without prompting.
C-c s c Find functions calling a function.
C-c s C Find called functions (list functions called from a function).
C-c s t Find text string.
C-c s e Find egrep pattern.
C-c s f Find a file.
C-c s i Find files #including a file.

For a complete list of commands refer to the file: xcscope.el.

To use the etags database, first you have to tell emacs where the symbol database is, for example:

M-x visit-tag-table /path/to/linux/source/code/TAGS

to navigate, go to a function an do:

M-.

to go back

M-*

Configure Emacs indentation according to kernel coding style

The Linux kernel coding style is very strict. This ensures that anyone can modify the source code and it will always look consistent. To configure Emacs so its standard indentation uses the kernel coding style, add this line to your ~/.emacs config file:

(setq c-default-style "linux")

Also I have a few tweaks that maybe you find helpful, I list here a portion of my .emacs configuration relevant to development (since I have other uses for Emacs such as LaTex editing):

; Get rid of the startup message
(setq inhibit-startup-message t)
; Show file full path in title bar
(setq-default frame-title-format
   (list '((buffer-file-name " %f"
             (dired-directory
              dired-directory
              (revert-buffer-function " %b"
              ("%b - Dir:  " default-directory)))))))
; Shows parenthesis 
(show-paren-mode 1)
; Shows column number
(column-number-mode 1)
; Change default colors
(set-background-color "grey14")
(set-foreground-color "white")
(set-cursor-color "white")
; No toolbar
(progn
  (if (fboundp 'tool-bar-mode) (tool-bar-mode -1))
;  (menu-bar-mode -1)
  (scroll-bar-mode -1)
)

Linux checkpoint restart mechanisms

I work in a research group in the HPC field. Our group develops many
tools that use process checkpoint/restart (CR). Basically the people here have found
three CR mechanism that actually works:

1- Berkeley lab's checkpoint/restart (BLCR)

Pros:

- Probably the most robust framework to CR in Linux. Is a hybrid
kernel-space/user-space implementation.
- You can compile OpenMPI message passing library to checkpoint distributed applications using BLCR. This is very convenient in HPC.

Cons:

- It looks that they are slowing down its development. The last
official release is 0.82 (June 16, 2009) and support kernel 2.6.30
(pretty old). To compile with newer kernels there are some patches
flowing in the development mailing list but I think only to give
support until 2.6.34 I think.
- You need root permissions to insert the blcr kernel module. One of
our tools used BLCR and we couldn't run in many clusters because the
sysadmins were skeptical about inserting a kernel module with a few
random patches published in a mailing list.

2- DMTCP: Distributed MultiThreaded CheckPointing

Pros:

- A completely user-space solution. You don't need to bother the
sysadmins to install kernel modules.
- Can checkpoint distributed computation (we already tried with
OpenMPI and it also checkpoints the orte daemon).
- There is current development to add DMTCP to OpenMPI for parallel applications checkpointing as a alternative to BLCR.

Cons:

- Since it is implemented in user-space it has a lot of workarounds to
maintain process state in userspace.
- Duplicates kernel-space process information.
- Only works with socket-based communications (it doesn't work with
proprietary infiniband protocols for example).

3- Linux-cr checkpoint/restart mechanism

Pros:

- The checkpoint/restart mechanism is implemented in the kernel as
syscalls and some user-space tools.
- Their intention is to push the mechanism upstream for kernel inclusion.
- Since their implementation is kernel based it is very robust.

Cons:

- The patch-set still didn't make for kernel inclusion. And the the
whole subject is complicated . Not all kernel developers agree that
implement CR in the kernel is a good idea.
- You need a custom kernel that has linux-cr support.

So which CR mechanism you choose will depend on many factors (if you have
control over the machine, use sockets for communications, can use a custom kernel, etc).

Speeding up Linux kernel compilation

Compiling a Linux kernel can take a long time to finish. Fortunately there are some utilities that can help us to speedup this process.

These tools are:

1. make -jn: The easier way to speed-up Linux kernel compilation is executing make with the option -j (jobs). This option specifies the number of parallel jobs that make will use to split the build process. Because the kernel Makefile have correct dependency information, one doesn't have to use make's default single process execution.
2. ccache: When compiling it is common to use make clean to create a clean build of the project. Unfortunately this deletes all the information from previous compilations. The compiler cache (ccache) detects when the same compilation is doing again and looks in its cache for objects files so you don't have to generate them again.
3. distcc: If you have more than one machine available, you can distribute your builds using distcc. 

I ran a few experiments to measure compilation times with different configurations and compare performance improvement in each scenario.

The experiments where made in my development machine (Intel Core 2 Duo P870 2.53GHz - 4 GB DDR3 with Debian 6 Squeeze). Since I only had one machine I didn't try distcc.

Here are the results:

Using default make (with just one compilation job):

$ time make M=drivers/staging/

real    6m12.030s
user    5m55.002s
sys     0m30.818s

Using make with four compilation jobs:

$ time make -j4 M=drivers/staging/

real    3m39.378s
user    6m27.852s
sys     0m31.614s

Using make with one compilation job and ccache:

$ time make -M=drivers/staging/

real    0m46.364s
user    0m35.042s
sys     0m11.913s

Using make with four compilation jobs and ccache:

$ time make -j4 M=drivers/staging/

real    0m27.513s
user    0m36.374s
sys     0m11.465s

Using make with four compilation jobs and ccache we obtain a speedup of 8X which is really impressive. Also ccache statistics shows that direct cache hit was about 32%.

$ ccache -s
cache directory                     /home/javier/.ccache
cache hit (direct)                  1091
cache hit (preprocessed)              24
cache miss                             1
unsupported compiler option           52
no input file                         25
files in cache                      3455
cache size                          43.7 Mbytes
max cache size                       1.0 Gbytes

Keep in mind that the first time you compile with ccache, your compilation will not be faster. It will even be slower because ccache has to populate its cache (i.e: its cache hit will be zero).

Linux Device Drivers 3 examples updated

One of the books about Linux kernel development that I like the most is Linux Device Drivers 3. In my opinion the best thing about the book is that all the example drivers covered in the book are implemented as memory devices. So one doesn't need special hardware in order to use the example drivers.

With this approach one can understand the ABIs, interfaces and helpers that the kernel offers to driver authors while having drivers for (pseudo) devices to play with.

A different approach is used in the book Essential Linux Device Drivers where the authors explain how drivers for real devices works. I think both approaches and books are complementary.

The problem is that LDD3 book is now a few years old and most of the example drivers do not compile in recent kernels. Also, many new changes were introduced in the kernel since 2.6.10. So examples for many ABIs are not covered in the book, like netlink sockets, threaded interrupt handlers, debugfs, etc.

I started a project to keep the LDD3 examples up-to-date with recent kernels and have plans to include more examples that cover more recent kernel features like the listed before.

The project can be found here.