How to Fix a Error Linux Device Driver Open

When I first started experimenting with Linux USB drivers, I hit one of the most frustrating issues my userspace program kept throwing “can’t open device” even though I had compiled my driver successfully. At first, I thought I had messed up the open() implementation inside my kernel driver. But after digging deeper, I realized the real problem: my userspace application and kernel driver were speaking two different languages.

The Short Answer

• My userspace program was calling rt_dev_open(), which belongs to Xenomai RTDM (Real-Time Driver Model).
• My kernel driver was a plain Linux USB character driver using file_operations -> .open = skel_open.

Those two don’t match. Unless the driver is implemented as an RTDM driver, rt_dev_open() will always fail with -ENODEV or -ENOENT, because there is no RTDM device to open.

The fix: use plain open() in userspace, or rewrite the kernel driver as an RTDM driver if I really want to stay in the Xenomai world.

A Minimal Working Pair

Here’s the tiny first working version I got running after fixing my mistake.

Userspace Code (use open(), not rt_dev_open())

// app_open.c
#define _GNU_SOURCE
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>

#define DEVICE_PATH "/dev/usb_skel0"   // adjust to your device node

int main(void) {
    int fd = open(DEVICE_PATH, O_RDWR | O_CLOEXEC);
    if (fd < 0) {
        fprintf(stderr, "ERROR: can't open %s (%s)\n",
                DEVICE_PATH, strerror(errno));
        return 1;
    }

    puts("Device opened!");

    // simple round-trip test
    const char msg[] = "ping\n";
    if (write(fd, msg, sizeof msg) < 0) {
        fprintf(stderr, "write failed: %s\n", strerror(errno));
    } else {
        char buf[64];
        ssize_t n = read(fd, buf, sizeof buf);
        if (n < 0) fprintf(stderr, "read failed: %s\n", strerror(errno));
        else write(STDOUT_FILENO, buf, n);
    }

    if (close(fd) < 0) {
        fprintf(stderr, "ERROR: close failed (%s)\n", strerror(errno));
        return 1;
    }

    puts("Closed cleanly.");
    return 0;
}

This was the moment when I finally saw “Device opened!” instead of my dreaded error message.

Kernel Driver Open/Release Functions (tightened up)

static int skel_open(struct inode *inode, struct file *file)
{
    int retval;
    int subminor = iminor(inode);
    struct usb_interface *interface = usb_find_interface(&skel_driver, subminor);
    struct usb_skel *dev;

    if (!interface) {
        pr_err("%s: can't find interface for minor %d\n", __func__, subminor);
        return -ENODEV;
    }

    dev = usb_get_intfdata(interface);
    if (!dev)
        return -ENODEV;

    kref_get(&dev->kref);

    mutex_lock(&dev->io_mutex);
    retval = usb_autopm_get_interface(interface);
    if (retval) {
        mutex_unlock(&dev->io_mutex);
        kref_put(&dev->kref, skel_delete);
        return retval;
    }

    file->private_data = dev;
    mutex_unlock(&dev->io_mutex);
    return 0;
}

static int skel_release(struct inode *inode, struct file *file)
{
    struct usb_skel *dev = file->private_data;

    if (dev && dev->intf)
        usb_autopm_put_interface(dev->intf);

    kref_put(&dev->kref, skel_delete);
    return 0;
}

Key improvements I made:

• Unlocking the mutex on all paths.
• Balancing every get with a put.
• Dropping my kref cleanly in both error and release paths.

Why the Original Code Fail

Once I pieced it together, the reasons became clearer:

• Wrong API: I was using rt_dev_open() for a non-RTDM driver.
• No /dev node: Without a real USB device plugged in, no node is created under /dev/.
• Permissions: Even when the node existed, my normal user couldn’t open it without fixing udev rules or running with sudo.
• Driver not bound: If my device’s VID/PID didn’t match the id_table in the driver, probe() never got called.
• Minor mismatch: I was trying /dev/usb_skel0 but the system actually created /dev/usb_skel1.

And yes this simple driver does require a real USB device attached. Without the device, nothing gets bound, and there’s simply nothing to open.

Practice Features I Added

After I got the basic open/close working, I wanted to explore more. Here are some fun practice add-ons:

• Echo read/write: whatever I wrote, the driver echoed back.
• Simple ioctl: I exposed the VID/PID to userspace with an _IOR ioctl.
• poll()/select() support: I added a wait queue to wake readers when new data arrived.
• Sysfs attributes: I created a /sys/bus/usb/.../my_attr entry just to see how sysfs works.
• Error handling polish: I cleaned up duplicated unlock/put code by using proper error labels.

These little projects helped me build confidence in driver development.

My Quick Debug Checklist

Here’s the list I now always follow when I see “can’t open device”:

1. Decide the model
   • If I’m writing a plain Linux driver → use open().
   • If I’m writing an RTDM driver → stick with rt_dev_open().
2. Confirm device exists dmesg -w # watch for probe() logs when I plug it in lsusb # check my VID:PID ls -l /dev | grep skel
3. Check permissions
   • Quick fix: sudo chmod a+rw /dev/usb_skel0
   • Long-term: proper udev rules.
4. See exact errno
   • Switch to open() and print strerror(errno) to know the real cause.

Final Thoughts

When I look back, the lesson is simple: match your userspace API with your kernel driver type. I wasted hours debugging my open() implementation, when the real culprit was that I was mixing Xenomai’s RTDM world with a vanilla Linux USB char driver. Once I switched to open(), things started to click, and I could finally move on to experimenting with read(), write(), ioctl(), and more.