Bug 163614 - LOOKUP does only work sporadically when referring to an external file
Summary: LOOKUP does only work sporadically when referring to an external file
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
6.3.0.4 release
Hardware: Other All
: medium normal
Assignee: Eike Rathke
URL:
Whiteboard: target:25.8.0 target:25.2.5 target:25...
Keywords: bibisected, bisected, regression
Depends on:
Blocks: Cell-Formula Calc-External-Datalink
  Show dependency treegraph
 
Reported: 2024-10-25 09:36 UTC by Tomas
Modified: 2025-06-03 10:38 UTC (History)
5 users (show)

See Also:
Crash report or crash signature:


Attachments
file referring (42.60 KB, application/vnd.oasis.opendocument.spreadsheet)
2024-10-26 18:05 UTC, Tomas
Details
file that is referred to (23.83 KB, application/vnd.oasis.opendocument.spreadsheet)
2024-10-26 18:06 UTC, Tomas
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Tomas 2024-10-25 09:36:33 UTC
Description:
For instance:
=LOOKUP(A38;'file:///M:/produktion/ElectrodevArtNr.ods'#$Artikelregister.A$7:A$110;'file:///M:/produktion/ElectrodevArtNr.ods'#$Artikelregister.E$7:E$110)
This can work in one cell but not in another.
I have tried to enter references to local cells and that works also.


Steps to Reproduce:
1.Use the formula in several cells.
2.
3.

Actual Results:
#N/A

Expected Results:
The text in the file that is referred to.


Reproducible: Sometimes


User Profile Reset: No

Additional Info:
No other info.
Comment 1 m_a_riosv 2024-10-25 15:06:44 UTC
Please attach a sample file, reduce the size as much as possible without private information, and paste the information in Menu/Help/About LibreOffice, there is a copy icon.
Comment 2 Tomas 2024-10-26 18:05:59 UTC
Created attachment 197254 [details]
file referring
Comment 3 Tomas 2024-10-26 18:06:16 UTC
Created attachment 197255 [details]
file that is referred to
Comment 4 Tomas 2024-10-26 18:07:12 UTC
This time it was only one value that did not work. The last line, line 38.
Sometimes it seems to work and sometimes it does not.
Thanks
/Tomas
Comment 5 m_a_riosv 2024-10-27 00:00:11 UTC
Reproducible
Version: 25.2.0.0.alpha0+ (X86_64) / LibreOffice Community
Build ID: 9517639bc3189e3ea4dc4d2f7004d4b33d754d47
CPU threads: 16; OS: Windows 11 X86_64 (10.0 build 26100); UI render: Skia/Raster; VCL: win
Locale: es-ES (es_ES); UI: en-US
Calc: CL threaded

No matter if OpenCL or Threaded are enabled or not.

And if fails searching for values greater than 14801

Latest version that works on the ones I have installed.
Version: 4.4.7.2
Build ID: f3153a8b245191196a4b6b9abd1d0da16eead600
Locale: es_ES
Comment 6 raal 2024-10-27 08:50:15 UTC
 Note: Tested on Linux (various versions from 4.1 to 25.2) and when updating external links, I get error: "The following external file could not be loaded. Data linked from this file did not get updated." 
I can open file ElectrodevArtNr_copy.ods without problems.
Comment 7 m_a_riosv 2024-10-28 00:19:30 UTC
Hi @raal, I think updating the links to the external file with the proper name of download files ([Ctrl+H] find & replace), should make it works, if not pls let me know to redo the samples.
Comment 8 Tomas 2024-10-28 07:13:50 UTC
The data source file needs to be added to the trusted source files in  the options/security/macro security section. That may be why you cannot find the source file.
/Tomas
Comment 9 raal 2024-10-28 07:21:38 UTC
(In reply to m_a_riosv from comment #7)
> Hi @raal, I think updating the links to the external file with the proper
> name of download files ([Ctrl+H] find & replace), should make it works, if
> not pls let me know to redo the samples.

Thank you, Miguel. After copy ElectrodevArtNr_copy.ods to ElectrodevArtNr.ods (now I have both files in the directory) I have not any error message.
Comment 10 raal 2024-10-28 07:25:15 UTC
This seems to have begun at the below commit in bibisect repository/OS bibisect-linux-64-6.2.
Adding Cc: to Eike Rathke ; Could you possibly take a look at this one?
Thanks
 976208d734d6ab40d2e7e1d8b7aee8c69b1d3f08 is the first bad commit
commit 976208d734d6ab40d2e7e1d8b7aee8c69b1d3f08
Author: Jenkins Build User <tdf@pollux.tdf>
Date:   Fri Aug 24 15:06:56 2018 +0200

    source 1d0d5734ffaf6f56699c5315b6427ac832af6a13

59548: Eliminate unnecessary per element copy of matrix' first col/row | https://gerrit.libreoffice.org/c/core/+/59548
Comment 11 Tomas 2025-05-27 09:45:07 UTC Comment hidden (noise)
Comment 12 Eike Rathke 2025-05-27 09:49:47 UTC Comment hidden (noise)
Comment 13 Tomas 2025-05-27 09:51:28 UTC Comment hidden (noise)
Comment 14 Eike Rathke 2025-05-27 19:53:30 UTC
The underlying problem is that data in ElectrodevArtNr_copy.ods is in Artikelregister.A7:E102 but the lookup uses $Artikelregister.A$7:A$110 i.e. encounters 8 empty rows at the end. That worked by accident with the old code that copied to an interim vector, where empty elements were implicitly converted to empty strings and strings compare greater than numeric, whereas the non-copied elements have the original empty elements and empty compares less than anything else, which then confuses the binary search that suddenly encounters a less-than value where a greater-or-equal value is expected.

To fit and fix this, specifically for LOOKUP() empty elements must compare greater than anything else as is the case when sorting a range that contains empty cells, which sorts them to the end.
Comment 15 Commit Notification 2025-05-27 22:04:16 UTC
Eike Rathke committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/b16dd0fb306944a02e5d73d4a2bbc68cf80b0b8f

Resolves: tdf#163614 empty compares greater-than for LOOKUP()

It will be available in 25.8.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 16 Eike Rathke 2025-05-27 22:05:59 UTC
Pending review
https://gerrit.libreoffice.org/c/core/+/185933 for 25-2
https://gerrit.libreoffice.org/c/core/+/185934 for 24-8
Comment 17 Commit Notification 2025-05-27 22:52:29 UTC
Eike Rathke committed a patch related to this issue.
It has been pushed to "libreoffice-25-2":

https://git.libreoffice.org/core/commit/5b933213384796552650489a17b0af899eb03a0c

Resolves: tdf#163614 empty compares greater-than for LOOKUP()

It will be available in 25.2.5.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 18 Tomas 2025-05-28 06:18:05 UTC
Hi,
Thank you so much guys!
You are great!
/Tomas
Comment 19 Xisco Faulí 2025-05-28 09:07:35 UTC
Hi Eike,
I would like to create a test for this issue. Could you please confrim whether this issue is only reproducible when referring to another document ? I tried to move the referring sheet to the same document and it seems to work fine...
Comment 20 Commit Notification 2025-05-30 12:47:52 UTC
Eike Rathke committed a patch related to this issue.
It has been pushed to "libreoffice-25-2-4":

https://git.libreoffice.org/core/commit/70fb1c2b91aa500bf1ae47eba3ff113ffbfb9b7b

Resolves: tdf#163614 empty compares greater-than for LOOKUP()

It will be available in 25.2.4.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 21 Eike Rathke 2025-06-02 22:05:19 UTC
(In reply to Xisco Faulí from comment #19)
> Could you please confrim
> whether this issue is only reproducible when referring to another document ?
> I tried to move the referring sheet to the same document and it seems to
> work fine...
The bug occurred only with arrays, like it is the case when an external range reference is read from the document cache. It should also manifest if an inline array was constructed containing empty elements, but I didn't try; like in
{1;2;3;;}
where the trailing fourth and fifth elements are empty. (you may need a larger array or different values to trigger the binary search's failure).
Comment 22 Xisco Faulí 2025-06-03 09:28:46 UTC
(In reply to Eike Rathke from comment #21)
> (In reply to Xisco Faulí from comment #19)
> > Could you please confrim
> > whether this issue is only reproducible when referring to another document ?
> > I tried to move the referring sheet to the same document and it seems to
> > work fine...
> The bug occurred only with arrays, like it is the case when an external
> range reference is read from the document cache. It should also manifest if
> an inline array was constructed containing empty elements, but I didn't try;
> like in
> {1;2;3;;}
> where the trailing fourth and fifth elements are empty. (you may need a
> larger array or different values to trigger the binary search's failure).

Hi Eike,
Thanks a lot for your input. It helped a lot
Comment 23 Commit Notification 2025-06-03 10:38:37 UTC
Xisco Fauli committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/56771bc864465d22083e6eacf8de08ab3b9e0ffb

tdf#163614: sc_spreadsheet_functions_test: Add test

It will be available in 25.8.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.