How Do I Fix x264 M128/__m128 Undefined Error During Make on Linux

I was trying to compile x264 on my Linux machine (x86_64), and everything was going fine until I hit a strange set of compile errors. I had disabled assembly because I didn’t have yasm installed, and that’s when things broke.

Here’s the error I saw during make:

common/rectangle.h:84: error: ‘v4si’ undeclared
common/rectangle.h:86: error: ‘__m128’ undeclared
common/rectangle.h:86: error: expected ‘;’ before ‘v16’
...

And here’s the snippet of code that was failing:

#if HAVE_VECTOREXT && defined(__SSE__)
    v4si v16 = {v, v, v, v};

    M128(d + s*0 + 0) = (__m128)v16;
    M128(d + s*1 + 0) = (__m128)v16;
    if (h == 2) return;
    M128(d + s*2 + 0) = (__m128)v16;
    M128(d + s*3 + 0) = (__m128)v16;
#else

What This Error Actually Means

• __m128 is a special data type provided by SSE intrinsics (from <xmmintrin.h> and related headers).
• v4si in x264 is a GCC vector typedef, representing a vector of four integers.
• The guard #if HAVE_VECTOREXT && defined(__SSE__) tells the compiler, “go ahead, use SSE vector types.”

My problem was that my compiler claimed SSE was available (__SSE__ was defined), but I had disabled assembly support. That left the build in a broken state where SIMD types like __m128 weren’t actually usable.

In short: the build system thought SSE was enabled, but the compiler flags and headers weren’t lining up.

How I Fix It

I found three different ways to solve this problem, depending on what I wanted:

The Proper Fix Install Yasm and Rebuild

The cleanest way was to just install yasm (and optionally nasm) and let x264 build with full assembly optimizations:

# Debian/Ubuntu
sudo apt-get update
sudo apt-get install yasm nasm build-essential

# Fedora/RHEL
sudo dnf install yasm nasm @development-tools

Then I rebuilt from scratch:

cd x264
make distclean 2>/dev/null || true
./configure
make -j$(nproc)

This gave me the fastest, most reliable build.

If I Wanted to Stay Without ASM

I had to force-disable vector code too, otherwise x264 still tried to use __m128:

make distclean 2>/dev/null || true
./configure --disable-asm --host=x86_64-unknown-linux-gnu
make CFLAGS="-O2 -U__SSE__ -UHAVE_VECTOREXT" -j$(nproc)

This way, the #if HAVE_VECTOREXT && defined(__SSE__) block in rectangle.h was skipped entirely.

If I Wanted to Keep Vector Code

I could explicitly tell GCC to enable SSE intrinsics:

make distclean 2>/dev/null || true
./configure --disable-asm CFLAGS="-msse2 -mfpmath=sse"
make -j$(nproc)

This worked, but honestly, enabling asm is the better long-term option.

A Tiny Demo Code to Understand the Error

To make sense of what was happening, I wrote this little file, sse_demo.c:

// sse_demo.c
// Comment OUT the include to reproduce the error
// #include <xmmintrin.h>

int main(void) {
    __m128 x;   // 128-bit SSE vector (4 floats)
    (void)x;
    return 0;
}

When I compiled it:

gcc -O2 -Wall sse_demo.c -o sse_demo

It failed with the exact same ‘__m128’ undeclared error. But if I added SSE support:

gcc -O2 -Wall sse_demo.c -o sse_demo -msse

It worked fine.

That confirmed the real issue: the compiler wasn’t being told to use SSE intrinsics.

More Practice (Hands On Tasks)

Check which macros your compiler defines

echo | gcc -dM -E - | grep -E '__SSE__|__SSE2__|__AVX__'

If __SSE__ shows up but your build fails, it means your flags/headers aren’t aligned.

Try a clean non-SIMD build

./configure --disable-asm
make CFLAGS="-O2 -U__SSE__ -UHAVE_VECTOREXT"

Do a fully optimized build with Yasm

sudo apt-get install yasm nasm
./configure --enable-pic --enable-shared
make -j$(nproc)
./x264 --version

Write your own guarded SIMD code
Here’s guarded_sse.c:

#include <stdio.h>
#if defined(__SSE__)
#  include <xmmintrin.h>
#endif

int main(void) {
#if defined(__SSE__)
    __m128 z = _mm_set1_ps(1.0f);
    float out[4];
    _mm_storeu_ps(out, z);
    printf("SSE OK: %f %f %f %f\n", out[0], out[1], out[2], out[3]);
#else
    printf("SSE not enabled, running scalar path.\n");
#endif
    return 0;
}

Compile and run both with and without -msse and see the difference.

Final Thought

This whole experience taught me something: disabling asm in x264 is usually not worth it. The assembler code is where most of the speed comes from, and without it, you’ll often end up fighting strange compile-time problems like my __m128 error. The best fix is simple install Yasm, rebuild clean, and let x264 detect your CPU features properly. But even if you run into these odd SSE type erors, at least now you know what’s happening and how to work around it.