Nn embedded linux notes
Wiki Main Page | User Page of Ted | Google search engine
Embedded Linux Notes
this wiki article created 2017-12-11 Monday
Overview
This wiki document is a collection of Ted's notes on embedded Linux studies, with a starting point of building a recipe for a Linux system to run on a RaspberryPi 2 target hardware platform. Near term goal is to build a customized Linux 4.x kernel and test that it runs on a RaspberryPi dev board. Long term goal has two parts: (1) build a more useful Linux system with working C and C++ toolchain, networking stack, web server, e-mail and database server support, an X server or comparable graphics and desktop support, and (2) target other hardware platforms beyond RaspberryPi 2 board.
Contents
- 1 Rpi project commands quick reference
- 2 These Rpi QEMU notes rough!!!
- 3 ^ New section - Trouble points
- 4 ^ RaspberryPi Kernel Compilation
- 4.1 ^ edit point - custom Rpi-targeted kernel config issues
- 4.2 ^ RaspberryPi kernel configuration references
- 4.3 ^ Finding kernel version
- 4.4 ^ Main Line Kernel versus RaspberryPi Kernel Project
- 4.5 ^ general kernel configuration and compile reference
- 4.6 ^ Finding kernel configurations - extract-ikconfig and flag CONFIG_IKCONFIG
- 4.7 ^ edit point - drhuvvyas and bcm2709_defconfig comparison
- 4.8 ^ linux/arch/arm/configs and bcm2709_defconfig versus versatile_defconfig
- 4.9 ^ kernel - specific configuration build results
- 5 ^ Buildroot
- 6 ^ Putting The Pieces Together - QEMU, kernel, rootfs
- 7 ^ QEMU references
- 8 Extra Notes on QEMU
- 9 References
Rpi project commands quick reference
Here are some commands we're using often to configure and to compile the Linux kernel for RaspberryPi 2 target board:
# Run interactive ncurses-based kernel config script:
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- menuconfig
# Run kernel config script pulling up latest config to modify a few things:
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- menuconfig oldconfig
# Build kernel, kernel modules and device tree binary as per RaspberryPi kernel compilation tutorial:
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- zImage modules dtbs
# Same command as previous but use two processor cores on host system and capture time taken to compile everything:
$ time { make -j 2 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- zImage modules dtbs; }
These Rpi QEMU notes rough!!!
As of 2017-12-11 these notes are a mess, so Ted reviewing work so far and here starting summary of what's important, what has worked, what best steps known today to take going forward. First the major pieces we are finding and using:
- cross-compiling toolchain, e.g. Linaro gcc
- QEMU system hardware emulator
The cross-compiler and related tools we use to build our kernel and 'target' a.k.a. 'guest' system softwares. The guest system in our case is a Linux kernel and additional programs compiled to run together as a complete system on an ARM based processor, in the target hardware which is the RaspberryPi 2 board. We cross compile in order to get the much faster compilation times available on a typical cerca 2010 Intel desktop computer. The QEMU software allows us to experiment, jumping through some hoops, with our ARM-targeted kernels and softwares, without a physical RaspberryPi board present. We also get to skip the step of creating a system image and flashing that to SD card. Knowing how to create an SD-card-ready image, however, is important. We'll return to this we hope sooner than later, once we have seen our cross-compiled system pieces run in normal expected ways in QEMU's target hardware emulating environment.
Our target system software pieces for a basic Linux operating system include:
- Linux kernel 4.x
- device tree binary (dtb file)
- kernel modules
- root file system
Kernel, device trees and kernel modules are all part of the official operating system space. These three pieces get compiled when configuring Linux kernel for a RaspberryPi target board, using the kernel's arch/arm/configs/bcm2709_defconfig default configuration file. Our needed root file system is a separate matter, which we must built separately. We also have a number of options for how to configure and specify what we want to build in this file system.
As of this Monday, Ted finds good notes at a page of Azeria Labs and also a post by graduate student MedicineYeh. Ted finds a little more success from Azeria Labs article, getting QEMU invocations to bring up an already-compiled QEMU Linux kernel for ARM. But Ted's custom compiled 4.x kernels haven't booted yet, and no diagnostics messages visible save four lines relating to some kind of sound card module. MedicineYeh's blog post goes into more details about root file systems and a couple of ways to build them quickly for basic testing and practice, but so far none of his example calls to QEMU have worked. These all hang and time-out without any diagnostics feedback. MedicineYeh's blog post seems to imply in its QEMU output captures that QEMU can put guest system message to the shell which invoked QEMU. But it's not clear to Ted how this would happen, and it doesn't appear to happen by default. QEMU will present the ARM-targeted kernel with a device tree that's expressed in the device tree binary file that the Linux kernel arch' ARM default configuration describes somewhere under the hood . . . that device tree is different and separate from the host's device tree, including the host system's.
Regarding QEMU passing guest kernel message to standard out, there's mention in Azeria Labs article about QEMU option and argument -serial stdio. Ted to try this once he has kernel, dtb file, kernel modules and rootfs all cross-compiled for ARM on Ubuntu host . . .
^ New section - Trouble points
Two trouble points as of 2017-12-11 Monday. One, we're not seeing guest system messages in any of invoking shell, QEMU window, nor Remmina VNC client window. Ted wants to see guest system diagnostics at QEMU-invoking shell prompt. Two, Ted unsure whether we're mounting a rootfs properly in our call to QEMU.
Figure x - problems building and running Linux for ARM under QEMU
- guest kernel diagnostics not visible
- questions about passing rootfs to QEMU properly
- 2017-12-14 THU -
Over the past couple days Ted found that most call to QEMU are passing the same parameters, and between the working qemu invocation on hand and a non-working one, narrowed the differing piece (argument to QEMU) down to the kernel. Ted's compiled kernels don't boot up, and give few or no diagnostics. The pre-compiled RPi kernels of drhuvvyas90 boot fully. Ted noting also that drhuvvyas90 supplies a kernel config-and-build script along side the pre-compiled kernels, in a diretory named 'tools'. Ted has tried to execute this script on 2017-12-13 Wednesday and 2017-12-14 Thursday but script fails at the point of applying patches to multiple Kconfig files among the Linux kernel's supported processor and system architectures. Here is the latest output from script build-kernel-qemu:
# In /var/local/raspberrypi/qemu-rpi-kernel/tools, where qemu-rpi-kernel is a cloned git repository: $ ./build-kernel-qemu Cloning into 'linux'... p11-kit: invalid config filename, will be ignored in the future: /etc/pkcs11/modules/gnome-keyring-module WARNING: gnome-keyring:: couldn't connect to: /run/user/1000/keyring-cUT3JX/pkcs11: No such file or directory p11-kit: skipping module 'gnome-keyring-module' whose initialization failed: An error occurred on the device remote: Counting objects: 6064925, done. remote: Compressing objects: 100% (1312/1312), done. remote: Total 6064925 (delta 1730), reused 1519 (delta 1240), pack-reused 6062373 Receiving objects: 100% (6064925/6064925), 1.69 GiB | 2.27 MiB/s, done. Resolving deltas: 100% (5054327/5054327), done. Checking connectivity... done. Checking out files: 100% (56781/56781), done. patching file arch/arm/mach-versatile/Kconfig Hunk #1 FAILED at 4. Hunk #2 FAILED at 12. 2 out of 2 hunks FAILED -- saving rejects to file arch/arm/mach-versatile/Kconfig.rej patching file arch/arm/mm/Kconfig Hunk #1 FAILED at 71. Hunk #2 FAILED at 89. Hunk #3 FAILED at 127. Hunk #5 FAILED at 359. 4 out of 5 hunks FAILED -- saving rejects to file arch/arm/mm/Kconfig.rej patching file drivers/mmc/host/Kconfig Hunk #1 FAILED at 6. 1 out of 1 hunk FAILED -- saving rejects to file drivers/mmc/host/Kconfig.rej $
Ted may have to forego any automated RPi kernel configuration in drhuvvyas90's helper script, but before moving on here are the contents of the three mentioned patch rejection files:
# In file ../qemu-rpi-kernel/tools/linux/arch/arm/mach-versatile/Kconfig.rej
--- arch/arm/mach-versatile/Kconfig
+++ arch/arm/mach-versatile/Kconfig
@@ -4,7 +4,6 @@ menu "Versatile platform type"
config ARCH_VERSATILE_PB
bool "Support Versatile Platform Baseboard for ARM926EJ-S"
default y
- select CPU_ARM926T
select MIGHT_HAVE_PCI
help
Include support for the ARM(R) Versatile Platform Baseboard
@@ -12,7 +11,6 @@ config ARCH_VERSATILE_PB
config MACH_VERSATILE_AB
bool "Support Versatile Application Baseboard for ARM926EJ-S"
- select CPU_ARM926T
help
Include support for the ARM(R) Versatile Application Baseboard
for the ARM926EJ-S.
In file "linux/arch/arm/mm/Kconfig.rej" 38L, 1463C
--- arch/arm/mm/Kconfig
+++ arch/arm/mm/Kconfig
@@ -71,7 +71,7 @@ config CPU_ARM9TDMI
# ARM920T
config CPU_ARM920T
- bool "Support ARM920T processor" if (ARCH_MULTI_V4T && ARCH_INTEGRATOR)
+ bool "Support ARM920T processor" if ARCH_INTEGRATOR || ARCH_VERSATILE_PB || ARCH_VERSATILE_AB
select CPU_32v4T
select CPU_ABRT_EV4T
select CPU_CACHE_V4WT
@@ -89,7 +89,7 @@ config CPU_ARM920T
# ARM922T
config CPU_ARM922T
- bool "Support ARM922T processor" if (ARCH_MULTI_V4T && ARCH_INTEGRATOR)
+ bool "Support ARM922T processor" if ARCH_INTEGRATOR || ARCH_VERSATILE_PB || ARCH_VERSATILE_AB
select CPU_32v4T
select CPU_ABRT_EV4T
select CPU_CACHE_V4WT
@@ -127,7 +127,7 @@ config CPU_ARM925T
# ARM926T
config CPU_ARM926T
- bool "Support ARM926T processor" if (!ARCH_MULTIPLATFORM || ARCH_MULTI_V5) && (ARCH_INTEGRATOR || MACH_REALVIEW_EB)
+ bool "Support ARM926T processor" if ARCH_INTEGRATOR || MACH_REALVIEW_EB || ARCH_VERSATILE_PB || ARCH_VERSATILE_AB
select CPU_32v5
select CPU_ABRT_EV5TJ
select CPU_CACHE_VIVT
@@ -359,7 +360,7 @@ config CPU_PJ4B
# ARMv6
config CPU_V6
- bool "Support ARM V6 processor" if (!ARCH_MULTIPLATFORM || ARCH_MULTI_V6) && (ARCH_INTEGRATOR || MACH_REALVIEW_EB || MACH_REALVIEW_PBX || MACH_BCM2708)
+ bool "Support ARM V6 processor" if ARCH_INTEGRATOR || MACH_REALVIEW_EB || MACH_REALVIEW_PBX || MACH_BCM2708 || ARCH_VERSATILE_PB || ARCH_VERSATILE_AB
select CPU_32v6
select CPU_ABRT_EV6
select CPU_CACHE_V6
In file "linux/drivers/mmc/host/Kconfig.rej" 11L, 392C
--- drivers/mmc/host/Kconfig
+++ drivers/mmc/host/Kconfig
@@ -6,7 +6,7 @@ comment "MMC/SD/SDIO Host Controller Drivers"
config MMC_BCM2835
tristate "MMC support on BCM2835"
- depends on MACH_BCM2708 || MACH_BCM2709 || ARCH_BCM2835
+ depends on MACH_BCM2708 || MACH_BCM2709 || ARCH_BCM2835 || ARCH_VERSATILE_PB || ARCH_VERSATILE_AB
help
This selects the MMC Interface on BCM2835.
Before applying these patches the helper script cloned git repositories, two of them between one and two GB in size, into the path from which script build-kernel-qemu is called. Ted noting that some of the paths and file referenced in the script may be incorrect or fixed enough to keep the script from finding what it needs to reference and to modify. Not sure whether this observation is accurate or how many broken issues there are in the script, could be fine. But for example Ted needed to create a symlink to file linux-arm.patches initially to get the Rpi kernel configuring script to find that patch file . . . a patch file which in turn hasn't yet worked this 2017 December.
Another path to try is in drhuvvyas90's script to comment out the call to patch. What happens then? - TMH
- 2017-12-14 . . . Ok looks like the file ./linux/arch/arm/mm/Kconfig must be patched in order to avoid the two kernel compile errors we see today and saw yesterday.
^ RaspberryPi Kernel Compilation
- 2017-11-25 SAT - These notes are of first efforts to cross-compile Linux kernel and device tree blob, on Intel i386/i686 host system, for target ARM system specifically RaspberryPi 2 dev board. As of 2017-DEC-09 Saturday, four or five custom kernel configurations with rpi target chosen / in mind have failed, for reasons such as "enermation or pound define value not an integer" and "alignment type of variable x not available [on target processor]". Ted noting that the Linux kernel stable release with all its options and supported target hardware platforms and peripherals is heavyweight, not by default configured to be small or simple as possible. Now also looking into Tiny Kernel wiki page and home page of Tiny Core Linux.
Here are instructions for compiling and for cross-compiling a Linux kernel for RaspberryPi target system:
Here are limited instructions on how to configure the Linux kernel for RaspberryPi target systems:
Back in 2017 February this year, February 24th just after Embedded Linux Conference, looks like Ted stepped through some of the instructions at this Raspberry Pi dot org documentation page. Ted noting there are two directories on compiling host in /mnt with this timestamp:
ted@localhost:/mnt$ ls -l make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- menuconfig total 44 drwxr-xr-x 2 root root 4096 Oct 24 2016 dvd-rom drwxr-xr-x 2 root root 4096 Feb 24 2017 ext4 drwxr-xr-x 2 root root 4096 Feb 24 2017 fat32 . . .
When issuing the cross-compile command that task seemed to complete quickly, giving the following messages which total less than two hundred lines:
ted@rangari:~/projects/raspberrypi/linux$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcm2709_defconfig # # configuration written to .config # ted@rangari:~/projects/raspberrypi/linux$ make -j 2 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- zImage modules dtbs scripts/kconfig/conf --silentoldconfig Kconfig CHK include/config/kernel.release CHK include/generated/uapi/linux/version.h CHK include/generated/utsrelease.h HOSTCC scripts/dtc/dtc.o HOSTCC scripts/dtc/flattree.o HOSTCC scripts/dtc/fstree.o HOSTCC scripts/dtc/data.o HOSTCC scripts/dtc/livetree.o HOSTCC scripts/dtc/treesource.o HOSTCC scripts/dtc/srcpos.o make[1]: 'include/generated/mach-types.h' is up to date. CHK include/generated/timeconst.h CHK include/generated/bounds.h CHK include/generated/asm-offsets.h CALL scripts/checksyscalls.sh HOSTCC scripts/dtc/checks.o make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- menuconfig HOSTCC scripts/dtc/util.o HOSTCC scripts/dtc/dtc-lexer.lex.o HOSTCC scripts/dtc/dtc-parser.tab.o HOSTLD scripts/dtc/dtc HOSTCC scripts/genksyms/genksyms.o HOSTCC scripts/genksyms/parse.tab.o HOSTCC scripts/mod/mk_elfconfig HOSTCC scripts/genksyms/lex.lex.o MKELF scripts/mod/elfconfig.hhttps://www.raspberrypi.org/forums/viewtopic.php?f=66&t=178806 HOSTCC scripts/mod/modpost.o HOSTLD scripts/genksyms/genksyms HOSTCC scripts/kallsyms HOSTCC scripts/mod/file2alias.o HOSTCC scripts/pnmtologo HOSTCC scripts/conmakehash make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- menuconfig HOSTCC scripts/mod/sumversion.o HOSTCC scripts/recordmcount HOSTLD scripts/mod/modpost HOSTCC scripts/sortextable HOSTCC usr/gen_init_cpio CHK include/generated/compile.h GEN usr/initramfs_data.cpio.gz HOSTCC arch/arm/vdso/vdsomunge AS usr/initramfs_data.o LD usr/built-in.o MUNGE arch/arm/vdso/vdso.so.dbg OBJCOPY arch/arm/vdso/vdso.so AS arch/arm/vdso/vdso.o LD arch/arm/vdso/built-in.o GZIP kernel/config_data.gz make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- menuconfig CHK kernel/config_data.h HOSTCC lib/gen_crc32table HOSTCC lib/raid6/mktables TABLE lib/raid6/tables.c CC [M] lib/raid6/tables.o LD [M] lib/raid6/raid6_pq.o GEN lib/crc32table.h CC lib/crc32.o LD lib/built-in.o LOGO drivers/video/logo/logo_linux_clut224.c LOGO drivers/video/logo/logo_linux_mono.c LOGO drivers/video/logo/logo_superh_mono.c make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- menuconfig LOGO drivers/video/logo/logo_superh_vga16.c make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- menuconfig LOGO drivers/video/logo/logo_linux_vga16.c LOGO drivers/video/logo/logo_blackfin_vga16.c LOGO drivers/video/logo/clut_vga16.c LOGO drivers/video/logo/logo_spe_clut224.c LOGO drivers/video/logo/logo_mac_clut224.c LOGO drivers/video/logo/logo_superh_clut224.c LOGO drivers/video/logo/logo_sun_clut224.c LOGO drivers/video/logo/logo_parisc_clut224.c LOGO drivers/video/logo/logo_blackfin_clut224.c LOGO drivers/video/logo/logo_dec_clut224.c LOGO drivers/video/logo/logo_m32r_clut224.c LOGO drivers/video/logo/logo_sgi_clut224.c CC drivers/video/logo/logo_linux_clut224.o LD drivers/video/logo/built-in.o LD drivers/video/built-in.o make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- menuconfig LD drivers/built-in.o LINK vmlinux LD vmlinux.o MODPOST vmlinux.o GEN .version CHK include/generated/compile.h UPD include/generated/compile.h CC init/version.o LD init/built-in.o KSYM .tmp_kallsyms1.o KSYM .tmp_kallsyms2.o LD vmlinux SORTEX vmlinux SYSMAP System.map OBJCOPY arch/arm/boot/Image Building modules, stage 2. Kernel: arch/arm/boot/Image is ready Kernel: arch/arm/boot/Image is ready GZIP arch/arm/boot/compressed/piggy.gzip AS arch/arm/boot/compressed/piggy.gzip.o LD arch/arm/boot/compressed/vmlinux OBJCOPY arch/arm/boot/zImage Kernel: arch/arm/boot/zImage is ready MODPOST 1540 modules LD [M] lib/raid6/raid6_pq.ko ted@rangari:~/projects/raspberrypi/linux$
At this point it looks like we may have enough to attempt the qemu invocation which was partially working off site west side, since couple days ago we were missing what appeared to be only the kernel when invoking qemu on build box rangari . . .
So we can launch Raspbian for the RaspberryPi 2 board, using a kernel we've built on an i386 architecture host server, using the default bcmp_2907 (mis-spelled) kernel config file. But we can't fully compile a kernel for an ARM / Rpi target architecture when we make any kernel configuration changes via `make menuconfig`. The script which successfully starts our qemu-system-arm version 2.10.0 is:
#!/bin/bash qemu-system-arm -kernel ./kernel-qemu-4.4.34-jessie -cpu arm1176 -m 256 -M versatilepb -serial stdio -append "root=/dev/sda2 rootfstype=ext4 rw" -hda ./2017-09-07-raspbian-stretch-lite.img -redir tcp:5022::22 -no-reboot exit 0
Right now this script is located along side several already compiled, ARM-targeted kernels which Ted believes are from the git available cross-compile Rpi project by one 'dhruvvyas90' . . . Dhruvvyas90 qemu-rpi-kernel.
ted@localhost:/var/local/ted/projects/raspberrypi/qemu_vms$ ls 2017-09-07-raspbian-stretch-lite.img kernel-qemu-4.1.7-jessie kernel-qemu-4.4.21-jessie README.md kernel-qemu-3.10.25-wheezy kernel-qemu-4.4.12-jessie kernel-qemu-4.4.26-jessie start-qemu.sh kernel-qemu-4.1.13-jessie kernel-qemu-4.4.13-jessie kernel-qemu-4.4.34-jessie tools ted@localhost:/var/local/ted/projects/raspberrypi/qemu_vms$
- Summary rpi kernel compilations -
As of 2017 Dec 2 we can using QEMU 2.10.0 boot and use Raspbian image and kernel from Raspberry Pi on-line store. We can also substitute alternate kernel in call to QEMU system emulator, a kernel from . . .
^ edit point - custom Rpi-targeted kernel config issues
- 2017-11-30 -
Took a couple days but Ted learned that following errors arose from running `make menuconfig` on a given set of kernel sources without specifying the ARM architecture nor the GNU abiehf cross-compiler:
In file included from ./include/uapi/linux/stddef.h:1:0,
from ./include/linux/stddef.h:4,
from ./include/uapi/linux/posix_types.h:4,
from ./include/uapi/linux/types.h:13,
from ./include/linux/types.h:5,
from fs/xfs/xfs_linux.h:21,
from fs/xfs/xfs.h:32,
from fs/xfs/xfs_super.c:19:
In function ‘xfs_check_ondisk_structs’,
inlined from ‘init_xfs_fs’ at fs/xfs/xfs_super.c:1974:2:
./include/linux/compiler.h:518:38: error: call to ‘__compiletime_assert_119’ declared with attribute error: XFS: sizeof(xfs_dir2_sf_entry_t) is wrong, expected 3
_compiletime_assert(condition, msg, __compiletime_assert_, __LINE__)
^
./include/linux/compiler.h:501:4: note: in definition of macro ‘__compiletime_assert’
prefix ## suffix(); \
^
./include/linux/compiler.h:518:2: note: in expansion of macro ‘_compiletime_assert’
_compiletime_assert(condition, msg, __compiletime_assert_, __LINE__)
^
./include/linux/bug.h:54:37: note: in expansion of macro ‘compiletime_assert’
#define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
^
fs/xfs/xfs_ondisk.h:22:2: note: in expansion of macro ‘BUILD_BUG_ON_MSG’
BUILD_BUG_ON_MSG(sizeof(structname) != (size), "XFS: sizeof(" \
^
fs/xfs/xfs_ondisk.h:119:2: note: in expansion of macro ‘XFS_CHECK_STRUCT_SIZE’
XFS_CHECK_STRUCT_SIZE(xfs_dir2_sf_entry_t, 3);
^
scripts/Makefile.build:293: recipe for target 'fs/xfs/xfs_super.o' failed
make[2]: *** [fs/xfs/xfs_super.o] Error 1
scripts/Makefile.build:544: recipe for target 'fs/xfs' failed
make[1]: *** [fs/xfs] Error 2
Makefile:991: recipe for target 'fs' failed
make: *** [fs] Error 2
real 17m33.542s
user 16m10.988s
sys 0m53.000s
Ok as of 2017-12-08 and a few days ago Ted realizing that the `make menuconfig` type call to configure a Linux kernel for a RaspberryPi/Arm target architecture must be of the form as follows below . . . and before starting a fresh kernel configuration it is a good idea to clean up all old object files, intermediate files, old config information using:
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- mrproper $ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- menuconfig
Now again calling the version of this command with options to compile kernel, device tree binary, and modules:
$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- zImage modules dtbs
^ RaspberryPi kernel configuration references
Ted found these references in the README.md file of QEMU Rpi Tools project. Local directory is qemu-rpi-kernel/. Invoking `git status .` shows this to be a git repository. Invoking `git remote -v` shows this repository to be a clone of:
ted@localhost:~/projects/qemu/qemu-rpi-kernel$ git remote -v origin https://github.com/dhruvvyas90/qemu-rpi-kernel.git (fetch) origin https://github.com/dhruvvyas90/qemu-rpi-kernel.git (push) ted@localhost:~/projects/qemu/qemu-rpi-kernel$ #
In the READ.md file the Rpi kernel project author who doesn't identify him or herself lists the following four references together, and there are a couple more refs beyond these mentioned near the end of READ.md:
https://web.archive.org/web/20131210001638/http://xecdesign.com/compiling-a-kernel/ https://web.archive.org/web/20131209235952/http://xecdesign.com/compiling-qemu/ https://web.archive.org/web/20131210001407/http://xecdesign.com/working-with-qemu/ https://web.archive.org/web/20131210001526/http://xecdesign.com/qemu-emulating-raspberry-pi-the-easy-way/
- 2017-12-14 THU -
Sysprogs site with multiple kernel related tutorial in left hand column:
^ Finding kernel version
Use of `dpkg-query -l` to find kernel version details on Linux system . . .
Quote from following article: "In a kernel source directory [invoke] `$ head Makefile -n 3` . . .
Looks like there is also kernel version information in a file in an include directory which is part of the given kernel sources:
/var/local/raspberrypi/rpi-linux/include/config$ cat kernel.release 4.4.50-v7+ $
^ Main Line Kernel versus RaspberryPi Kernel Project
A number of the latest kernel releases, both stable and testing releases are not yet supported by the Raspberry Pi team -- if there is a team -- which develops the patches and detailed necessary configurations to build 4.x Linux kernels for the fullest Rpi board support possible:
Quoting one Rpi user Dougie Lawson, "You'll find 4.1.x and 4.4.x are the best supported kernels currently . . ."
^ general kernel configuration and compile reference
- 2017-12-05 - Ted adding section on general articles, some non-Rpi articles and reference materials which explain how to configure a kernel and then how to invoke the build process and point `make` to the new custom configuration file. This so far not clear from the instructions at Raspberry Pi's articles https://www.raspberrypi.org/documentation/linux/kernel/building.md and https://www.raspberrypi.org/documentation/linux/kernel/configuring.md. These references found at Google via search phrase "invoking make after custom kernel configuration":
There are thousands of settable options in the Linux kernel sources. Ted not sure whether all the flags are visible and accessible from `make menuconfig`. This forum post discusses <code>menuconfig's search feature . . .
^ Finding kernel configurations - extract-ikconfig and flag CONFIG_IKCONFIG
A booted Linux kernel which exports its configuration settings to /proc/config.gz can be studied in detail from its configuration standpoint. Tools to use in this case include fdisk on the unmounted root file system which got booted up to a working system with the given kernel. During that run session the kernel will populate /proc/config.gz. On the unmounted root file system image use fdisk to determine partition count, file system type and block size, and starting block of the rootfs. Then mount that partition within the rootfs image file with an offset equal to (block size * starting block). Example of this at Azeria Labs RaspberryPi under Qemu tutorial.
Alternately depending on a kernel configuration flag a script named extract-ikconfig can be used on the compressed kernel image, sometimes named zImage to extract the given kernel's configuration. According to,
the kernel config flag to enable full configuration extraction from kernel image files is named CONFIG_IKCONFIG.
From our rpi-4.4.y based kernel build today, to which Ted applied bcm2709_defconfig configuration we find no kernel configuration info extractable. Nor is there any config info extractable from the host system's kernel /boot/vmlinuz-3.2.0-4-686-pae. However the latest of drhuvvyas90's already compiled kernels produces full configuration settings:
$ /var/local/raspberrypi/rpi-linux/scripts/extract-ikconfig $D21/kernel-qemu-4.4.34-jessie | wc 2736 8998 70148 $
and we saw this a couple days ago. Now going through this output while using menuconfig we should be able to see, albeit slow and tediously, differences in the working Rpi kernel configuration and our own . . .
- 2017-12-15 FRI ~13:50 -
Ok we enabled one new kernel configuration as a built-in part of the kernel rather than its apparent default module [M] setting from the bcm2709_defconfig settings. Recompiled the rpi-4.4.y kernel sources and in a short time just a minute or so have a kernel from which we can extract full configuration using extract-ikconfig. We now have two configuration summary files, one from drhuvvyas90's 4.4.34 kernel for QEMU and Rpi, and one from our kernel based on the rpi-4.4.y branch of RaspberryPi kernel project on Github. The files are a couple thousand lines long or longer . . . it may be also that some kernel config flags in drhuvvyas90's project are not found in . . .
^ edit point - drhuvvyas and bcm2709_defconfig comparison
This is a brute force approach to finding the key differences between two kernel configs, the first one which boots to a fully operational Linux system in QEMU versus a second kernel which fails to boot. First Ted noting the config file sections in each kernel configuration in next table just below these sentences. Easy to see that bcm2709_defconfig enables many more kernel options, and an alternate machine type compared with drhuvvyas90's pre-built rpi kernel for QEMU:
Table x - Comparing kernel configuration sections
|
Table x - Comparing kernel configurations of drhuvvyas90 and bcm2709_defconfig:
<p>
|
Hmm, MediaWiki page rendering engine is causing next section to wrap funny around the table of kernel config comparisons . . .
Looking up some terms, or to look up later:
- RCU subsystem of Linux kernel - https://www.kernel.org/doc/Documentation/RCU/whatisRCU.txt
- CONFIG_COMPAT_BRK=y . . . per menuconfig's help, "disable heap randomization"
^ linux/arch/arm/configs and bcm2709_defconfig versus versatile_defconfig
Today is 2017-12-13 Wednesday, and yesterday we determined that the one difference between bootable QEMU + kernel + rootfs and non-booting QEMU + kernel + rootfs is the compiled kernel file. So somethings are wrong with our kernel configuration. We retrieved the kernel configuration from the drhuvvyas90 pre-compiled Rpi kernel named kernel-qemu-4.4.34-jessie by mounting the "jessie stretch lite" rootfs image from RaspberryPi downloads page, and copying /proc/config.gz from that rootfs. We could not extract any config information from our compiled kernel, but we likely left disabled a kernel config option mentioned in https://stackoverflow.com/questions/14958192/getting-config-from-linux-kernel-image which would permit running script extract-ikconfig on a compressed kernel image.
While studying drhuvvyas90's build-qemu-kernel shell script in file qemu-rpi-kernel/tools/build-kernel-qemu we notice reference to a default configuration file which is different from the bcm2709_defconfig default configuration named in some of our recently found tutorials on building Linux kernel for RaspberryPi 2 development boards . . .
- 2017-12-15 FRI -
From an Rpi forum post yesterday we have word -- not yet confirmed -- that as of mid- to late 2017 the Linux 4.x kernel versions are only a couple of them close to fully supported in their port or configuration works targeting RaspberryPi dev boards. Raspberry Pi kernel releases 4.1.x and 4.4.x are the most suppported kernel versions, just two out of the many 4.x kernel release candidates. With this in mind Ted returning to question of which default ARM configuration is appropriate to invoke when calling `make ... menuconfig`. Ted has seen at RaspberryPi organizations tutorial to set KERNEL=kernel7 and use bmc2709_defconfig. But yesterday in drhuvvyas90's build_kernel_qemu script Ted saw reference to use versatile_defconfig. There is a similarly named bcm2835_defconfig . . . which default ARM targeted config recipes will work to build a functional kernel that runs in QEMU system emulator and on a real world Raspberry Pi 2 board? A further forum post may help, though advice here repeats what's on Raspberry Pi's kernel building tutorial . . .
^ kernel - specific configuration build results
- 2017-12-13 WED -
Kernel build results 1
Errors run into when building kernel following configuration 'versatile_defconfig', after issuing command make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- versatile_defconfig . . .
CC lib/win_minmax.o AR lib/lib.a EXPORTS lib/lib-ksyms.o LD lib/built-in.o LD virt/lib/built-in.o LD virt/built-in.o LD vmlinux.o MODPOST vmlinux.o GEN .version CHK include/generated/compile.h UPD include/generated/compile.h CC init/version.o LD init/built-in.o arch/arm/mm/built-in.o:(___ksymtab+arm926_dma_clean_range+0x0): undefined reference to `arm926_dma_clean_range' arch/arm/mm/built-in.o:(___ksymtab+arm926_dma_inv_range+0x0): undefined reference to `arm926_dma_inv_range' Makefile:972: recipe for target 'vmlinux' failed make: *** [vmlinux] Error 1 real 23m22.245s user 21m5.491s sys 1m21.729s veris@alta-spare-6:/var/local/raspberrypi/linux$
- 2017-12-15 FRI -
Kernel build results 2 - below are tail end results of building kernel version rpi-4.4.y, after exporting KERNEL=kernel7 and invoking make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- zImage modules dtbs:
DTCO arch/arm/boot/dts/overlays/w1-gpio.dtbo DTCO arch/arm/boot/dts/overlays/w1-gpio-pullup.dtbo DTCO arch/arm/boot/dts/overlays/wittypi.dtbo DTC arch/arm/boot/dts/bcm2709-rpi-2-b.dtb DTC arch/arm/boot/dts/bcm2710-rpi-3-b.dtb DTC arch/arm/boot/dts/bcm2710-rpi-cm3.dtb real 74m3.399s user 68m30.589s sys 5m9.499s veris@alta-spare-6:/var/local/raspberrypi/rpi-linux$ exit exit Script done, file is z--kernel-config-using-bcm2709-defconfig--try-2.txt veris@alta-spare-6:/var/local/raspberrypi/rpi-linux$
^ Buildroot
Per Medicine Yeh's tutorial setting out to compile and use buildroot project, to create Linux based root file system to run under QEMU. Mid-way messages from `make` call on buildroot project downloaded from git repository:
/usr/bin/install -m 0644 support/misc/target-dir-warning.txt /var/local/ted/projects-on-slash-var/buildroot/buildroot/output/target/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM >>> skeleton-init-sysv Extracting >>> skeleton-init-sysv Patching >>> skeleton-init-sysv Configuring >>> skeleton-init-sysv Building >>> skeleton-init-sysv Installing to target rsync -a --ignore-times --exclude .svn --exclude .git --exclude .hg --exclude .bzr --exclude CVS --chmod=u=rwX,go=rX --exclude .empty --exclude '*~' package/skeleton-init-sysv//skeleton/ /var/local/ted/projects-on-slash-var/buildroot/buildroot/output/target/ >>> skeleton Extracting >>> skeleton Patching >>> skeleton Configuring >>> skeleton Building >>> skeleton Installing to target >>> linux-headers 4.14.4 Downloading --2017-12-09 23:11:28-- https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.14.4.tar.xz Resolving cdn.kernel.org (cdn.kernel.org)... 151.101.53.176, 2a04:4e42:d::432 Connecting to cdn.kernel.org (cdn.kernel.org)|151.101.53.176|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 100784788 (96M) [application/x-xz] Saving to: ‘/var/local/ted/projects-on-slash-var/buildroot/buildroot/output/build/.linux-4.14.4.tar.xz.VsSH6c/output’
Tail message from `make` on buildroot project:
chmod a+x /var/local/ted/projects-on-slash-var/buildroot/buildroot/output/build/buildroot-fs/fakeroot.fs rm -f /var/local/ted/projects-on-slash-var/buildroot/buildroot/output/target/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM PATH="/var/local/ted/projects-on-slash-var/buildroot/buildroot/output/host/bin:/var/local/ted/projects-on-slash-var/buildroot/buildroot/output/host/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/sbin:/usr/sbin:/opt/cross/bin:/opt/cross/x-tools/arm-unknown-linux-gnueabi/bin:/home/ted/bin:/usr/local/mysql/bin:/usr/lib/xscreensaver:/etc/init.d:/var/opt/sam-ba_cdc_cdc_linux" /var/local/ted/projects-on-slash-var/buildroot/buildroot/output/host/bin/fakeroot -- /var/local/ted/projects-on-slash-var/buildroot/buildroot/output/build/buildroot-fs/fakeroot.fs rootdir=/var/local/ted/projects-on-slash-var/buildroot/buildroot/output/target table='/var/local/ted/projects-on-slash-var/buildroot/buildroot/output/build/buildroot-fs/device_table.txt' /usr/bin/install -m 0644 support/misc/target-dir-warning.txt /var/local/ted/projects-on-slash-var/buildroot/buildroot/output/target/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM real 48m56.055s user 53m27.352s sys 4m47.896s ted@rangari:/var/local/ted/projects/buildroot/buildroot$
^ Putting The Pieces Together - QEMU, kernel, rootfs
On our day host system we need and have a cross-compiler, a guest system emulator, a cross-compiled Linux 4.x kernel, a corresponding device tree binary and a root file system . . .
cross-compiler ~/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin qemu /usr/local/bin/qemu-system-arm kernel /var/local/raspberrypi/linux/arch/arm/boot/zImage device tree binary /var/local/raspberrypi/linux/arch/arm/boot/dts/[ bcm2709-rpi-2-b.dtb | bcm2836-rpi-2-b.dtb | ... ] not sure which dtb to use! - TMH root file system ...
One root file system we have which is part of a successful QEMU boot of a guest Raspbian system is the ~1.85GB raspbian stretch lite image, md5 check sum noted here:
projects-on-slash-var/raspberrypi/qemu_vms$ md5sum /var/local/ted/downloads/raspberrypi/2017-09-07-raspbian-stretch-lite.img 2d5f26a5c75ea7e5677542a3411851e3 /var/local/ted/downloads/raspberrypi/2017-09-07-raspbian-stretch-lite.img
And after copying and uncompressing, in the rpi image /proc/config.gz to /home/pi the image file is yet the same size byte-wise but its md5 checksum has changed:
/var/local/ted/projects-on-slash-var/raspberrypi/qemu_vms$ md5sum /var/local/ted/downloads/raspberrypi/2017-09-07-raspbian-stretch-lite.img 779a317911ccee73c8a63ff62f670977 /var/local/ted/downloads/raspberrypi/2017-09-07-raspbian-stretch-lite.img
Per Azeria Labs tutorial on emulating RaspberryPi operating system and hardware, a prerequisite step which modifies a system "reach run level" time must be modified. To modify a file in a disk image file we mount the image file in a manner like,
$ sudo mount -v -o offset=47185920 -t ext4 /var/local/raspberrypi/file-system-images/2017-04-10-raspbian-jessie.img /mnt/raspbian [sudo] password for user: mount: /dev/loop0 mounted on /mnt/raspbian. $
^ QEMU references
- https://lists.gnu.org/archive/html/qemu-discuss/2014-09/msg00060.html . . . example qemu-system-arm invocations
# qemu-system-arm -kernel arch/arm/boot/zImage -dtb rtsm_ve-cortex_a15x4.dtb -m 512 -M vexpress-a15 -serial stdio -append "console=ttyAMA0"
Extra Notes on QEMU
Gosh frustrating, we were just practicing with calling QEMU in the following way (figure x) which follows the form of invocations we have seen in multiple tutorials and forum posts. Nearly all calls fail with a two lines "audio: Could not init `oss' audio driver . . . Could not read keymap file: 'en-us'". Ted suspected these might be errors coming from the kernel starting up in QEMU, and thought to change the kernel to an earlier release in the pre-compiled kernels of Dhruvvyas. This thought led to choosing a non-existent kernel filename and trying that out in call to QEMU. This made no difference! Same error messages, looks like QEMU doesn't get to the point of sanity checking or needing a valid kernel file name. So how can we debug this failure to boot a kernel plus root filesystem for RaspberryPi "guest" platform? What program or executing code are we actually wanting to debug?
Figure x - QEMU invocation which fails with or without kernel . . .
$ qemu-system-arm -M versatilepb -cpu arm1176 -m 256 -serial stdio -kernel /var/local/raspberrypi/qemu-rpi-kernel/kernel-qemu-4.4.34-jessiez -dtb /var/local/raspberrypi/linux/arch/arm/boot/dts/bcm2709-rpi-2-b.dtb -append "root=/dev/sda2 rootfstype=ext4 rw init=/bin/bash console=ttyAMA0" -drive format=raw,file=/home/veris/Downloads/raspberrypi/2017-08-16-raspbian-stretch-lite.img audio: Could not init `oss' audio driver Could not read keymap file: 'en-us' $
References
