Fix resolving global symbols when LoadLibrary() is called after dlopen()

Usage of first_automatic_object cache is wrong. This cache is filled by all
loaded DLL files (either implicitly or explicitly with LoadLibrary() call)
by EnumProcessModules() call at first usage of dlopen(). So dlsym() can
resolve global symbols only if they were loaded prior to dlopen() call. Any
future usage of LoadLibrary() does not include newly loaded DLLs into
first_automatic_object cache.

To fix this problem, first_automatic_object cache is fully removed and
EnumProcessModules() call is issued directly in dlsym() call.

As EnumProcessModules() returns all DLLs, included those which were loaded
by dlopen() with RTLD_LOCAL, it may break RTLD_LOCAL support. To address
this problem switch linked-list of all loaded DLLs with RTLD_GLOBAL to
linked-list of all loaded DLLs with RTLD_LOCAL flag. And then skip modules
from EnumProcessModules() which are in linked-list.

Also in WinAPI all DLLs loaded by LoadLibrary() behaves like RTLD_GLOBAL.
So above change is compatible with this behavior.

There may be another problem. Before retrieving HMODULE for DLL filename
(which is done by LoadLibrary()), it is not possible to detect if DLL was
already loaded by RTLD_LOCAL or not. And after calling LoadLibrary() it is
not possible to know if DLL was loaded either by dlsym() with RTLD_LOCAL or
by LoadLibrary() (which is equivalent to RTLD_GLOBAL). To address this
problem, compare number of loaded modules (counted by EnumProcessModules())
before and after LoadLibrary() called from dlsym(). If number does not
change it means that DLL was already loaded. So based on this result either
add or remove HMODULE from linked-list of RTLD_LOCAL modules.

Added test demonstrate usage of:

global = dlopen(NULL, RTLD_GLOBAL);   /* global handle */
LoadLibrary("library.dll");           /* this provides function */
function = dlsym(global, "function"); /* resolve function from library.dll */

1 files changed, 38 insertions, 0 deletions

diff --git a/testdll3.c b/testdll3.c
new file mode 100644
index 0000000..3906e48
--- /dev/null
+++ b/testdll3.c
@@ -0,0 +1,38 @@
+/*

+ * dlfcn-win32

+ * Copyright (c) 2007 Ramiro Polla

+ * Copyright (c) 2019 Pali Rohár <pali.rohar@gmail.com>

+ *

+ * dlfcn-win32 is free software; you can redistribute it and/or

+ * modify it under the terms of the GNU Lesser General Public

+ * License as published by the Free Software Foundation; either

+ * version 2.1 of the License, or (at your option) any later version.

+ *

+ * dlfcn-win32 is distributed in the hope that it will be useful,

+ * but WITHOUT ANY WARRANTY; without even the implied warranty of

+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU

+ * Lesser General Public License for more details.

+ *

+ * You should have received a copy of the GNU Lesser General Public

+ * License along with dlfcn-win32; if not, write to the Free Software

+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA

+ */

+

+#ifdef _DEBUG

+#define _CRTDBG_MAP_ALLOC

+#include <stdlib.h>

+#include <crtdbg.h>

+#endif

+#include <stdio.h>

+

+#if defined(_WIN32)

+#define EXPORT __declspec(dllexport)

+#else

+#define EXPORT

+#endif

+

+EXPORT int function3( void )

+{

+    printf( "Hello, world!\n" );

+    return 0;

+}