How to Fix C++ DllMain Not Executing CreateProcess

If you’re struggling with a DLL that refuses to execute CreateProcess in its DllMain function C++, you’re not alone. This issue often stems from subtle mistakes in how DLLs are designed to work, combined with restrictions imposed by the Windows loader. Let’s break down the problem and explore actionable solutions.

Understanding the Problem

The provided code attempts to launch a process (Main.exe) when the DLL is loaded. The DllMain function spawns a thread during DLL_PROCESS_ATTACH, which then calls CreateProcess. Despite seeming correct, the code fails with a cryptic error:

Debug Assertion Failed! (close.cpp Line 56)

Even worse, CreateProcess returns FALSE, triggering the “unsuccessful” message. Here’s why this happens and how to fix it.

Why DllMain Is a Dangerous Place for CreateProcess

1. Loader Lock Limitations:
   When DllMain is invoked (e.g., during DLL injection), Windows holds a loader lock. Certain API calls, including CreateProcess, are unsafe here because they may require the loader lock themselves, leading to deadlocks or undefined behavior.
2. CRT Initialization:
   The C Runtime (CRT) may not be fully initialized during DllMain. Using CRT functions (like MessageBox or file I/O) can trigger assertion failures, especially in non-default contexts (e.g., injected into Chrome).
3. Process Environment:
   If your DLL is loaded into a restricted process (like chrome.exe), sandboxing or security policies might block CreateProcess entirely.

Step-by-Step Fixes

Move Code Out of DllMain

Avoid complex logic in DllMain. Instead, export a function that initializes your process:

// Export a function to trigger process creation
extern "C" __declspec(dllexport) void Init() {
    CreateThread(nullptr, 0, Main, nullptr, 0, nullptr);
}

Call Init() explicitly after loading the DLL (e.g., via LoadLibrary in your host app).

Diagnose CreateProcess Failures

Use GetLastError() to pinpoint why CreateProcess fails:

if (!CreateProcess(...)) {
    DWORD error = GetLastError();
    LPWSTR buffer = nullptr;
    FormatMessage(
        FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM,
        nullptr, error, 0, (LPWSTR)&buffer, 0, nullptr
    );
    MessageBox(nullptr, buffer, L"Error", MB_OK);
    LocalFree(buffer);
}

Common errors include:

• ERROR_FILE_NOT_FOUND: Incorrect executable path.
• ERROR_ACCESS_DENIED: Permissions/sandboxing issues.

Validate the Executable Path

Hardcoded paths are fragile. Verify the path exists and escape backslashes properly. Test with a simple path like C:\\Windows\\System32\\notepad.exe.

Avoid CRT in DllMain

Remove unnecessary CRT dependencies (e.g., <iostream>) and use WinAPI functions like OutputDebugString for logging.

Handle Process Restrictions

If your DLL is injected into a sandboxed process (e.g., Chrome), CreateProcess may be blocked. Consider:

• Using a standalone executable instead of a DLL.
• Adjusting sandbox policies (not recommended for security reasons).

Revised Code Example

#include <windows.h>

DWORD WINAPI Main(LPVOID) {
    STARTUPINFO si = { sizeof(si) };
    PROCESS_INFORMATION pi;

    // Use a simple test path
    if (CreateProcess(
        L"C:\\Windows\\System32\\notepad.exe",
        nullptr, nullptr, nullptr, FALSE, 0, nullptr, nullptr, &si, &pi
    )) {
        WaitForSingleObject(pi.hProcess, INFINITE);
        CloseHandle(pi.hProcess);
        CloseHandle(pi.hThread);
    } else {
        // Log error via GetLastError()
    }
    return 0;
}

// Export an initialization function
extern "C" __declspec(dllexport) void Init() {
    CreateThread(nullptr, 0, Main, nullptr, 0, nullptr);
}

BOOL APIENTRY DllMain(HMODULE, DWORD reason, LPVOID) {
    return TRUE; // Keep DllMinimal
}

Final Thoughts

While DllMain seems like a convenient place to run code, it’s fraught with hidden constraints. By decoupling process creation from DLL initialization, validating paths, and diagnosing errors with GetLastError, you’ll avoid most pitfalls. Always test your DLL in a controlled environment (e.g., a simple host app) before deploying it to complex processes like Chrome. When in doubt, keep DllMain minimal and defer risky operations to exported functions.