Bug 98930 - Integer values passed to a C++ function, appear to corrupt stack
Summary: Integer values passed to a C++ function, appear to corrupt stack
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: BASIC (show other bugs)
(earliest affected) release
Hardware: x86 (IA32) Windows (All)
: medium normal
Assignee: Not Assigned
Depends on:
Reported: 2016-03-28 07:54 UTC by Erik
Modified: 2017-07-12 20:51 UTC (History)
3 users (show)

See Also:
Crash report or crash signature:

C++ project (6.32 KB, application/x-zip-compressed)
2016-03-28 07:54 UTC, Erik

Note You need to log in before you can comment on or make changes to this bug.
Description Erik 2016-03-28 07:54:29 UTC
Created attachment 123896 [details]
C++ project

May be related to: https://bugs.documentfoundation.org/show_bug.cgi?id=88953

I am using a macro to pass values to an existing third party C++ library that does further processing. For bug-reporting purposes I made the sample as small as I could. 

Steps to reproduce: 
1) Build the attached C++ project (the project file is VS2015), place the resulting DLL in the DLL search path (for example: syswow64 or system32). 
2) Create a LibreOffice macro with the following contents: 
REM  *****  BASIC  *****
Declare Function fnlotest Lib "lotest.dll"  (integer, integer, integer, integer) As Integer
Declare Function fnlotest2 Lib "lotest.dll"  (String, String, String, String, String) As Integer
declare sub MyMessageBeep Lib "user32.dll" Alias "MessageBeep" (Long)

Sub Main
  MyMessageBeep(5000) ' Test
  Dim szBuffer as string 
  Dim ret as integer 
  szBuffer = Space(512)
  ret = fnlotest(1,2,3,4) ' Expected result: 1+2+3+4 = 10, actual result: 3
  fnlotest2(szBuffer, "foo", "bar", "hello", "world") 
End Sub
### END SAMPLE ###
3) Run the sample and verify the return value of fnlotest(1,2,3,4). It should add all numbers.

If you run the sample while having the C++ debugger attached, you will see that the parameters that come through are: 1, 0, 2, 0 instead of 1,2,3,4. Explaining why we get 3 as a result. 

Interestingly, the second function with string values is fine. 

The third-party DLL uses functions with a signature like: 
foo(int, char*)

Because the stack appears to be corrupted after the first integer passed to a function, we get an access violation when accessing the char *. 

We tested with older versions, and the last working version is:
Currently we are running
Comment 1 Erik 2016-03-28 09:14:40 UTC
Verified that it's still an issue in, updated the version field to reflect that the last version that is known to work is
Comment 2 Oliver Specht (CIB) 2016-04-14 12:06:18 UTC
The align method seems to create too many or wrong objects that are later converted to function parameters.

Only the first parameter is actually used.

>	sblo.dll!`anonymous namespace'::align(std::vector<char,std::allocator<char> > & blob, unsigned int alignment, unsigned int offset, unsigned int add) Line 140	C++
 	sblo.dll!`anonymous namespace'::add<SbxValues>(std::vector<char,std::allocator<char> > & blob, const SbxValues & data, unsigned int alignment, unsigned int offset) Line 150	C++
 	sblo.dll!`anonymous namespace'::marshal(bool outer, SbxVariable * variable, bool special, std::vector<char,std::allocator<char> > & blob, unsigned int offset, `anonymous-namespace'::MarshalData & data) Line 333	C++
 	sblo.dll!`anonymous namespace'::call(const rtl::OUString & dll, const `anonymous-namespace'::ProcData & proc, SbxArray * arguments, SbxVariable & result) Line 497	C++
 	sblo.dll!SbiDllMgr::Call(const rtl::OUString & function, const rtl::OUString & library, SbxArray * arguments, SbxVariable & result, bool cdeclConvention) Line 719	C++
 	sblo.dll!SbiRuntime::DllCall(const rtl::OUString & aFuncName, const rtl::OUString & aDLLName, SbxArray * pArgs, SbxDataType eResType, bool bCDecl) Line 1292	C++
 	sblo.dll!SbiRuntime::StepCALL(unsigned long nOp1, unsigned long nOp2) Line 4164	C++
Comment 3 Erik 2017-01-04 12:48:51 UTC
Verified with the following build: 
Build ID: 3b800451b1d0c48045de03b5b3c7bbbac87f20d9
CPU Threads: 4; OS Version: Windows 6.2; UI Render: GL; Layout Engine: new; 
Locale: da-DK (da_DK); Calc: group

Right now when running the sample below, instead of getting 1,0,2,0, I get the pointer addresses to the right variables. Tested by debugging and running the following code in the immediate window: 
? first
? *((int*)first)
? *((int*)second)
? *((int*)third)
? *((int*)fourth)

While I now at least would be able to get to the actual values, I think the issue is still not completely resolved.
Comment 4 Erik 2017-01-04 13:10:47 UTC
Additional finding: If I declare the parameters 'byval' in the Macro, as such: 
Declare Function fnlotest Lib "lotest.dll"  (byval integer, byval integer, byval integer, byval integer) As Integer

It DOES work as expected. To verify, I also executed the same test code in MS Word VBA, and it has exactly the same result when using byval. When not using byval, I still get the addresses. 

LibreOffice now behaves exactly as VBA when calling the external DLL. So we can close this issue.