Fix resolving global symbols when LoadLibrary() is called after dlopen()

Usage of first_automatic_object cache is wrong. This cache is filled by all
loaded DLL files (either implicitly or explicitly with LoadLibrary() call)
by EnumProcessModules() call at first usage of dlopen(). So dlsym() can
resolve global symbols only if they were loaded prior to dlopen() call. Any
future usage of LoadLibrary() does not include newly loaded DLLs into
first_automatic_object cache.

To fix this problem, first_automatic_object cache is fully removed and
EnumProcessModules() call is issued directly in dlsym() call.

As EnumProcessModules() returns all DLLs, included those which were loaded
by dlopen() with RTLD_LOCAL, it may break RTLD_LOCAL support. To address
this problem switch linked-list of all loaded DLLs with RTLD_GLOBAL to
linked-list of all loaded DLLs with RTLD_LOCAL flag. And then skip modules
from EnumProcessModules() which are in linked-list.

Also in WinAPI all DLLs loaded by LoadLibrary() behaves like RTLD_GLOBAL.
So above change is compatible with this behavior.

There may be another problem. Before retrieving HMODULE for DLL filename
(which is done by LoadLibrary()), it is not possible to detect if DLL was
already loaded by RTLD_LOCAL or not. And after calling LoadLibrary() it is
not possible to know if DLL was loaded either by dlsym() with RTLD_LOCAL or
by LoadLibrary() (which is equivalent to RTLD_GLOBAL). To address this
problem, compare number of loaded modules (counted by EnumProcessModules())
before and after LoadLibrary() called from dlsym(). If number does not
change it means that DLL was already loaded. So based on this result either
add or remove HMODULE from linked-list of RTLD_LOCAL modules.

Added test demonstrate usage of:

global = dlopen(NULL, RTLD_GLOBAL);   /* global handle */
LoadLibrary("library.dll");           /* this provides function */
function = dlsym(global, "function"); /* resolve function from library.dll */

1 files changed, 27 insertions, 0 deletions

diff --git a/test.c b/test.c
index ba440da..10c655e 100644
--- a/test.c
+++ b/test.c
@@ -2,6 +2,7 @@
  * dlfcn-win32

  * Copyright (c) 2007-2009 Ramiro Polla

  * Copyright (c) 2014      Tiancheng "Timothy" Gu

+ * Copyright (c) 2019      Pali Rohár <pali.rohar@gmail.com>

  *

  * dlfcn-win32 is free software; you can redistribute it and/or

  * modify it under the terms of the GNU Lesser General Public

@@ -25,6 +26,7 @@
 #endif

 #include <stdio.h>

 #include <string.h>

+#include <windows.h>

 #include "dlfcn.h"

 

 /* If these dlclose's fails, we don't care as the handles are going to be

@@ -77,6 +79,7 @@ int main()
     size_t (*fwrite_local) ( const void *, size_t, size_t, FILE * );

     int (*nonexistentfunction)( void );

     int ret;

+    HMODULE library3;

 

 #ifdef _DEBUG

     _CrtSetReportMode(_CRT_WARN, _CRTDBG_MODE_FILE);

@@ -335,6 +338,15 @@ int main()
         printf("SUCCESS\tGot symbol from global handle: %p\n", function);

     

 

+    library3 = LoadLibraryA("testdll3.dll");

+    if (!library3)

+    {

+        printf( "ERROR\tCould not open library3 via WINAPI\n" );

+        RETURN_ERROR;

+    }

+    else

+        printf( "SUCCESS\tOpened library3 via WINAPI: %p\n", library3 );

+

     ret = dlclose( library );

     if( ret )

     {

@@ -346,6 +358,21 @@ int main()
     else

         printf( "SUCCESS\tClosed library.\n" );

 

+    function = dlsym(global, "function3");

+    if (!function)

+    {

+        error = dlerror();

+        printf("ERROR\tCould not get symbol from global handle: %s\n",

+            error ? error : "");

+        CLOSE_LIB;

+        CLOSE_GLOBAL;

+        RETURN_ERROR;

+    }

+    else

+        printf("SUCCESS\tGot symbol from global handle: %p\n", function);

+

+    RUNFUNC;

+

     ret = dlclose( global );

     if( ret )

     {