Description: When inserting large texts in a CLOB field, only part of the text is saved. I tested this with inserting the whole German base handbook text into a CLOB field. Only 20480 bit are saved; query: BIT_LENGTH. CLOB_FB.odb The same test in HSQLDB results in no data loss; query BIT_LENGTH 19696736 mem_testHSQLDB.odb Version: 7.3.3.2 (x64) / LibreOffice Community Build ID: d1d0ea68f081ee2800a922cac8f79445e4603348 CPU threads: 4; OS: Windows 10.0 Build 22000; UI render: Skia/Raster; VCL: win Locale: de-DE (de_DE); UI: de-DE Calc: CL Steps to Reproduce: 1.inserting large texts in a CLOB field 2. 3. Actual Results: only part of the text is saved Expected Results: full text is saved Reproducible: Always User Profile Reset: Yes Additional Info: no other information
Created attachment 180606 [details] CLOB_FB.odb
Created attachment 180607 [details] mem_testHSQLDB.odb
Tested with internal Firebird database, external Firebird file and Firebird server database connected directly and by JDBC. With Firebird server (JDBC) I got a BIT_LENGTH 9782968 (whole text, Firebird and HSQLDB won't count the same way) Connecting through direct(internal) driver I got a BIT_LENGTH 345784 (only a part of the text) Wonder why I got much more than reported, but it is still only a part of the content. I will set this one to NEW. Tested with OpenSUSE 15.3 and LO 7.3.4.2.
Value of the clob is added in OPreparedStatement::setClob here: https://opengrok.libreoffice.org/xref/core/connectivity/source/drivers/firebird/PreparedStatement.cxx?r=5200a736#646 I gave it a try and had weird values. Indeed, I applied this patch locally: diff --git a/connectivity/source/drivers/firebird/PreparedStatement.cxx b/connectivity/source/drivers/firebird/PreparedStatement.cxx index 9cd9157a7d6d..cb669ca05821 100644 --- a/connectivity/source/drivers/firebird/PreparedStatement.cxx +++ b/connectivity/source/drivers/firebird/PreparedStatement.cxx @@ -19,6 +19,7 @@ #include <sal/config.h> #include <cmath> +#include <iostream> #include "Connection.hxx" #include "PreparedStatement.hxx" @@ -661,6 +662,7 @@ void OPreparedStatement::setClob( sal_Int32 nParameterIndex, const OUString& rSt OString sData = OUStringToOString( rStr, RTL_TEXTENCODING_UTF8); + std::cerr << "sData.getLength()=" << sData.getLength() << "\n"; ISC_STATUS aErr = isc_put_segment( m_statusVector, &aBlobHandle, sData.getLength(), with a length of 200647, I got 32312 with a length of 100323, I got 278296 (yes I got a higher value result with a lower input length). Then I searched in Firebird code and found: src/common/classes/BlobWrapper.cpp:30:static const USHORT SEGMENT_LIMIT = 65535; 65535 bytes, so 524280 bits. Then when putting exactly a text of 65535 bytes, I obtain 524280 but if I put 65536 bytes, I get 0 and if I put 65537 bytes, I get 8. So if input is more than 65535 bytes, it seems we must find a way to split the input and call n times isc_put_segment
I gave a try with https://gerrit.libreoffice.org/c/core/+/135480
When I insert large texts in CLOB fields, it works up to ~ 517952 bit. From larger texts only parts are saved, varying from ~ 20488 bit to ~ 69808. I just tested a few texts, results with even more different values may be produced. If a patch is available, LibreOfficeDevXXXalpha1 versions are the earliest with which on can test it?
If the patch is pushed, you'll find a LO dev version 24/48 hours after in https://dev-builds.libreoffice.org/daily/master/ Also, it can be cherry-picked too on 7.3 branch (so for 7.3.4 or rather 7.3.5) but let's wait first if the patch is ok for master sources (future 7.4.0).
Julien Nabet committed a patch related to this issue. It has been pushed to "master": https://git.libreoffice.org/core/commit/a943e7ddd13315b18d7b33cd1b2f852144f54344 tdf#149470: Firebird, Clob may need several segments to store a very long input It will be available in 7.4.0. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
Patch waiting for review for 7.3 here: https://gerrit.libreoffice.org/c/core/+/135392
Julien Nabet committed a patch related to this issue. It has been pushed to "libreoffice-7-3": https://git.libreoffice.org/core/commit/605298a6e06ccb02b206f90fb9d0b8f831349fd8 tdf#149470: Firebird, Clob may need several segments to store a very long input It will be available in 7.3.5. The patch should be included in the daily builds available at https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More information about daily builds can be found at: https://wiki.documentfoundation.org/Testing_Daily_Builds Affected users are encouraged to test the fix and report feedback.
bugReportLOm@protonmail.com: so now the first version available with the patch will be the release 7.3.5. It should be available mid July according https://wiki.documentfoundation.org/ReleasePlan/7.3. Of course, you can give a try to the daily master build in 24/48 hours, but have in mind that it's a dev/alpha version so be sure to make backups of your files before testing.
@Julien Nabet Thank you for solving this bug quickly, like several others before! Also I am grateful for the information about the daily builds, I did not know that. I checked large CLOBs from my HSQHLB in FB, everything works fine with the Version: 7.4.0.0.alpha1+ (x64) from today. Thanks again. Version: 7.4.0.0.alpha1+ (x64) / LibreOffice Community Build ID: 66b1ebd4ddc7127a923bf81eb569e7f99dd52022 CPU threads: 4; OS: Windows 10.0 Build 22000; UI render: Skia/Vulkan; VCL: win Locale: de-DE (de_DE); UI: de-DE Calc: CL
You're welcome! :-) Let's put this one to VERIFIED then.