When I first started writing Linux user space applications, I made the classic mistake of mixing up kernel headers with libc headers. It broke my build environment in a very confusing way. In this post, I’ll walk through the error, why it happened, and how I fixed it.
My Original Makefile
I began with this very simple Makefile:
CROSS_COMPILE ?=
KERNEL_DIR ?= /usr/src/linux
CC := $(CROSS_COMPILE)gcc
KERNEL_INCLUDE := -I/usr/include
CFLAGS := -W -Wall -g $(KERNEL_INCLUDE)
LDFLAGS := -g
all: finder-drv
finder-drv: finder.o
$(CC) $(LDFLAGS) -o $@ $^
clean:
rm -f *.o
rm -f finder
This looked fine at first glance. I was just trying to compile a program called finder
.
The Error I Hit
When I ran make
, I got this nasty error:
/usr/include/arm-linux-gnueabi/sys/ioctl.h:22:22: fatal error: features.h: No such file or directory
At first I thought something was wrong with my cross-compiler. But after digging deeper, I realized the problem was self-inflicted.
What Went Wrong (Why features.h
Vanished)
Here’s what I had done earlier:
make headers_install INSTALL_HDR_PATH=/usr
By doing that, I overwrote /usr/include
with kernel headers. This destroyed part of my libc development headers. And features.h
is actually a glibc header, not a kernel header. Once it was gone, any attempt to compile a user-space app failed.
The key lesson:
Kernel headers and libc headers are not interchangeable.
- User-space applications should normally use libc headers (from
libc6-dev
on Debian/Ubuntu, orglibc-devel
on Fedora/CentOS). - Kernel headers are only needed when you’re working with low-level kernel interfaces (like
ioctl
s or netlink). And even then, they should live in their own directory, not/usr/include
.
How I Fix My System
To recover, I had to reinstall the libc development headers:
On Debian/Ubuntu:
sudo apt-get install --reinstall libc6-dev
# If I was cross-compiling:
sudo apt-get install --reinstall libc6-dev:armhf
On Fedora/RHEL:
sudo dnf reinstall glibc-headers glibc-devel
Then I made sure to never again install kernel headers into /usr
. Instead, if I need them:
cd /usr/src/linux
make mrproper # optional cleanup
make headers_install INSTALL_HDR_PATH=/opt/kernel-uapi
Now, kernel UAPI headers live safely in /opt/kernel-uapi/include
.
A Minimal Safe Makefile
Once I understood the issue, I simplified my Makefile. For most user-space apps, I don’t need kernel headers at all:
CROSS_COMPILE ?=
CC := $(CROSS_COMPILE)gcc
CFLAGS := -Wall -Wextra -g
LDFLAGS := -g
all: finder
finder: finder.o
$(CC) $(LDFLAGS) -o $@ $^
clean:
rm -f *.o finder
.PHONY: all clean
This worked perfectly, and I learned that I didn’t need -I/usr/include
in the first place—the compiler already knows where to find libc headers.
When I Do Need Kernel UAPI Headers
Sometimes I do need kernel UAPI headers—for example, when defining custom ioctl
s. In that case, I explicitly point my Makefile at the separate UAPI include path:
CROSS_COMPILE ?=
CC := $(CROSS_COMPILE)gcc
KERNEL_UAPI_PREFIX ?= /opt/kernel-uapi
KERNEL_UAPI_CFLAGS := -isystem $(KERNEL_UAPI_PREFIX)/include
CFLAGS := -Wall -Wextra -g $(KERNEL_UAPI_CFLAGS)
LDFLAGS := -g
all: finder
finder: finder.o
$(CC) $(LDFLAGS) -o $@ $^
clean:
rm -f *.o finder
.PHONY: all clean
Using -isystem
is nice because it marks UAPI headers as “system headers,” which suppresses warnings from them.
My Practice Makefile
To get more comfortable, I built a “practice” Makefile with extra functionality:
- Auto dependency generation
- Debug vs release builds
- Static analysis with
cppcheck
- Linting with
clang-tidy
- Optional cross-compilation with
SYSROOT
- A
run
target to execute the program directly
# ---------- Config ----------
CROSS_COMPILE ?=
SYSROOT ?=
CC := $(CROSS_COMPILE)gcc
BUILD ?= debug
WARNINGS := -Wall -Wextra -Wshadow -Wpointer-arith -Wcast-qual -Wwrite-strings
DEFS :=
ifeq ($(BUILD),release)
OPTFLAGS := -O2 -DNDEBUG
else
OPTFLAGS := -O0 -g
DEFS += -DDEBUG
endif
CFLAGS := $(OPTFLAGS) $(WARNINGS) $(DEFS) -MMD -MP
LDFLAGS := $(OPTFLAGS)
BIN := finder
SRCS := finder.c
OBJS := $(SRCS:.c=.o)
DEPS := $(OBJS:.o=.d)
all: $(BIN)
$(BIN): $(OBJS)
$(CC) $(LDFLAGS) -o $@ $^
%.o: %.c
$(CC) $(CFLAGS) -c $< -o $@
run: $(BIN)
./$(BIN)
asan: CFLAGS += -fsanitize=address -fno-omit-frame-pointer
asan: LDFLAGS += -fsanitize=address
asan: clean $(BIN)
scan:
cppcheck --enable=all --std=c11 --quiet --inline-suppr --suppress=missingIncludeSystem .
lint:
clang-tidy $(SRCS) -- -I.
clean:
rm -f $(OBJS) $(DEPS) $(BIN)
-include $(DEPS)
Final Thoughts
Looking back, the whole experience was frustrating at first but turned out to be a great learning moment. I realized that features.h
belongs to glibc, not the kernel, and that mixing kernel headers with libc headers is a recipe for disaster. The key is to always keep kernel UAPI headers in a separate prefix and rely on the system’s libc headers for most user space applications. I also learned that a Makefile can start simple and gradually grow with useful features like debug/release modes, static analysis, and sanitizers.