Bug 128329 - WORK: 'incomplete autocalculations' - cooperation of autocalculate and shared formulas / formulae still broken in LO calc ver. 6.2.7.1 with *threaded* calculation on
Summary: WORK: 'incomplete autocalculations' - cooperation of autocalculate and shared...
Status: UNCONFIRMED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
6.2.7.1 release
Hardware: All Windows (All)
: medium normal
Assignee: Not Assigned
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: AutoCorrect-Complete
  Show dependency treegraph
 
Reported: 2019-10-22 16:28 UTC by b.
Modified: 2024-02-26 10:38 UTC (History)
6 users (show)

See Also:
Crash report or crash signature:


Attachments
a anonymized screenshot showing a stuck cell not being autocalculated (238.48 KB, image/jpeg)
2019-10-22 16:34 UTC, b.
Details

Note You need to log in before you can comment on or make changes to this bug.
Description b. 2019-10-22 16:28:23 UTC
Description:
hello to all, 

may be it's a duplicate?, may be it's claimed as 'fixed' in other threads? may be it's left over from an incomplete patch. 

6.2.7.1 win(x64) does still produce errors or 'incomplete autocalculations' while you work in threaded mode. 

thus, despite all plastering, this area isn't 'clean' yet. 

see attached file in next comment, the upper value in the red rectangle ist calculated from the number below and the two values in the red ellipsoids. check the formula bar. the result is obviously wrong. 

it resulted from pasting a line two lines below that line where the error shows up, which reduced the value in the cell below by 10.000, that's obviously not taken into account for the display of cell BA1436, despite the fact that autocalculate is! on and works fine for nearly all cells in the sheet. 

it's happening randomly and not very often, i've seen it six times in the last month and everytime instantly switched back to *unthreaded* calculation. 

the error can be 'corrected' by forced recalculation (srtg-shift-F9), but imho that's not how spreadsheets are used normally, and if a wrong result is noted and a weak bridge is build it's  not affected nor stabilisized by corrections in the sheet. 

i think it would be a good idea if someone with programming and mathematical skills looks in the code, and doesn't only plaster until he doesn't find errors anymore, but tries an analysis for correctness. 

until that's done it's a good idea to either disable *treaded* mode, or at least issue a warning to the users, 'careful, you are working in threaded mode, results may be incorrect until you press <ctrl-shift-F9>'

Steps to Reproduce:
not yet tried due to lack of time, work in large sheets with multiple columns calculating stacked on the previous line, such as financial sheets, implement checksum or similar to detect errors ... 

Actual Results:
errors with cells not 'autoclculated' despite autocalculation is switched on, 

Expected Results:
correct calculation results for all cells, 


Reproducible: Sometimes


User Profile Reset: No



Additional Info:
Version: 6.2.7.1 (x64)
Build ID: 23edc44b61b830b7d749943e020e96f5a7df63bf
CPU threads: 8; OS: Windows 6.1; UI render: default; VCL: win; 
Locale: de-DE (de_DE); UI-Language: en-US
Calc: CL

!the info was copied after! switching back to unthreaded! 

hardware 'all' doesnt mean i tested all, windows 'all' doesn't mean i tested all, earliest affected doesn't mena earlier versions are proper, it's opposite, all earlier versions have been worse, this bug is just to pinpoint that this version - despite being the cleanest version i've ever seen - is also 'not really clean'!
Comment 1 b. 2019-10-22 16:34:29 UTC
Created attachment 155241 [details]
a anonymized screenshot showing a stuck cell not being autocalculated

a anonymized screenshot showing a stuck cell not being autocalculated, 

explanation in previous post,
Comment 2 bzdf.babri 2020-02-10 01:37:29 UTC
Can confirm I am seeing this problem on:

Version: 6.3.3.2.0+
Build ID: 30(Build:2)
CPU threads: 4; OS: Linux 5.3; UI render: default; VCL: kde5; 
Locale: en-US (en_US.UTF-8); UI-Language: en-US
Calc:


Description:
The autocalculations seem to get stuck using threaded mode on a large spreadsheet. All calculations complete successfully when threaded mode is turned off.


Steps to Reproduce:

1. Retrieve this google sheet and save as xlsx format using File->Download->Microsoft Excel (.xlsx):

https://docs.google.com/spreadsheets/d/1QGrMm6XSGWBVLI8I_DOAeJV5whoCnSdmaR8toQB2Jz8/edit#gid=0

2) Open the downloaded xlsx file with LibreOffice having "Enable multi-threaded calculation" turned on under Tools->Options->LibreOffice->Calc

3) On the "StockBond Returns" sheet, see that cell T3465 displays "Err:522". Click on cell T3465 and press F9 to recalculate. See that it's value is recalculated correctly. Cells in the same column above still display "Err:522"

4) Click on cell T3464 and press F9 to recalculate. See that it's value and many cells above it in the same column which previously showed "Err:522" are now recalculated correctly.

5) See that cell T1742 and above still show "Err:522". Click on cell T1742 and press F9 to recalculate. See that a few more rows above are recalculated correctly. We can continue selecting cells and pressing F9 where the calculations seem to get stuck and each time a few more rows get recalculated.

6) Turn off "Enable multi-threaded calculation" under Tools->Options->LibreOffice->Calc and restart. Open same file and see that all cells are calculated correctly. No cells display "Err:522".
Comment 3 b. 2020-02-10 19:58:30 UTC
@bzdf c#2: 

on download and test with ver: 

Version: 6.4.0.3 (x64)
Build ID: b0a288ab3d2d4774cb44b62f04d5d28733ac6df8
CPU threads: 8; OS: Windows 6.1 Service Pack 1 Build 7601; UI render: default; VCL: win; 
Locale: de-DE (de_DE); UI-Language: en-US
Calc: threaded

cell "StockBond Returns".T3465 is empty, the last filled cell in "StockBond Returns".T is 2477, all cells above show values, all below are empty. 

on first glance i have no err:522 anywhere in the map, 

can you pls. recheck, or estimate where i went wrong, 

my error reappears sometimes, also with above ver., last event was that a vlookup
which worked for years was blocked and needed hard recalc ... ??? 

reg. 



b.
Comment 4 b. 2020-06-15 08:40:23 UTC
@Telesto: why did you change from new to unconfirmed?
Comment 5 Telesto 2020-06-15 10:30:33 UTC
(In reply to b. from comment #4)
> @Telesto: why did you change from new to unconfirmed?

Not seeing any bug doc. Comment 2 point to a file, but followed by your comment 3..

I really think you're right.. but there needs to be something to work with. I can't bibisect a screenshot :-). I could even set it to NEEDINFO..  

It makes no sense to have NEW bug report of something without steps to reproduce.. even if it actually happens.. and people do confirm it in general terms.. It can't be resolved.. QA can't work with it. I won't reach developers.. 

If there should be a list of 'known bugs' without steps, maybe.. but not sure if the bug tracker is the right place.

BTW, thanks for attempting to make LibreOffice better! And yes, the mores in the bug tracker is sometimes frustrating.. Happens to me too..
Comment 6 b. 2020-06-17 04:52:02 UTC
@Telesto: 

to be or not to be ... a bug ... that's the question ... 

imho a bug is when the results of a program deviate from intended and defined calculation, in ancient times this was sometimes caused by physical bugs between contacts of relays, thus this behaviour got its nickname, but the nickname is used for wrong (buggy) results 'in general'!

we don't use open relays anymore, thus there are no real bugs to point to, 

some bugs are easy, some sporadic and hard to spot, but they are! 

it is difficult to ensure the quality of a program if you limit yourself in the processing of errors. 

the requirement for an example (file) excludes runtime errors, the requirement for repeatability excludes sporadic errors, the requirement for 'bibisectibility' excludes all problems introduced from OO ... and so on... 

i see weaknesses in the scheme of how LO currently operates, it is a good way to avoid too much deterioration, but not to produce real quality: 
- in a timely manner, 
- in general, and 
- verifiable

if such an error occurs and is documented... screenshot... someone who can do that would have to look at the corresponding code and analyse if it can make mistakes ... or not. it's not easy, but it has to be, otherwise we won't get rid of all these 'shared formula' - 'autocalculate' errors, and they undermine calc since version 4.2, for over 6 years. 

the whole thing might not have happened if someone had written enough unittests back then, but that (probably) didn't happen. https://www.slideshare.net/kohei101/life-after-calc-core-change - slide 25. 

an idea how to check against - some - mistakes that got unnoticed: 

a routine that extracts a 'fingerprint' (hash value?) from a complete calc map, each cell is somehow calculated so that changes in it change the result, 

a routine that takes this hash value, triggers a hard recalc, takes the hash again and compares it with the first one. hard recalc is quite reliable according to @Eike and my experience (not absolute, i mean i had a case where it didn't help the other day) but you could easily distinguish 'probably error-free' against 'contains calculation errors'. if errors ... go after them ...
Comment 7 Telesto 2020-06-17 06:39:03 UTC
(In reply to b. from comment #6)
The testing environment needs to be improved, yes. It's still far from ideal. There are bugs which really shouldn't be.. Calc issues introduced in 4.2 still around... bugs from OOO days, still around 

OTOH, the code base is large and old. unit tests should ideally be written at the point someone is working on he code change.. to test the 'new' or changed functionality.. 

Lack of volunteer, developers, $$$, different priority's. Unit tests and uiTest are not really sexy.. and a lot of them are needed.. however not appearing out of nowhere 

Feel free to help out...

And most flaws can be proven by a bug doc, takes a while to get the hands on something.. but after 4 year finally found proof for the bugs bothering me 4 years ago.. however in the mean time new bugs have been introduced..
Comment 8 Buovjaga 2020-10-10 15:37:24 UTC
(In reply to b. from comment #6)
> it is difficult to ensure the quality of a program if you limit yourself in
> the processing of errors. 
> 
> the requirement for an example (file) excludes runtime errors, the
> requirement for repeatability excludes sporadic errors, the requirement for
> 'bibisectibility' excludes all problems introduced from OO ... and so on... 

The only requirement is steps and/or an example file.
Comment 9 b. 2020-10-10 18:05:33 UTC
hello @Buovjaga, 

steps: as the fail is sporadic and need not appear in multiple redos of similar steps but only the one or other occasion i can't tell exact steps for reproducing, just play around with data and check all results, 

file: you find one sample with a similar error i was able to catch in tdf#125320, there with threaded of, i can't find more clue than 'wrong values saved' in it, 125320 also has some 'steps' but redraws failed - to fail - ... 

i only catched the file with fail, not a version from before with detailed steps until the fail :-( mostly such errors are spotted too late (by me), 

i spotted the errors during 'normal work', but in the meantime i've adjusted that to avoid errors, actually with ver. 6.1.6.3, openCL and 'threaded' off, thus it's unlikely i'll see one in the next time, 

(i cannot! work 4 weeks with the risk of data corruption in hope to probably see a new error - which wouldn't give any more clue to the issue than 'autocalculate failed at this point' - )

for better forensics ... what about implementing an activity protocol and a re-roll functionality, once somebody spottes an error he can save the file with it's original values, and a 'script' about his actions, then that script can be rolled through step by step, if the error shows up you have 'steps', if not you have clue about 'weak hardware' or a sporadic fail in software ... 

as @erAck said: 'it always fails if sin(moonphase)+cos(mercury*round(sun)) touches a certain value' ... this is! rare but clearly too often! for a reliable program ...
Comment 10 Buovjaga 2020-10-11 06:42:25 UTC
(In reply to b. from comment #9)
> for better forensics ... what about implementing an activity protocol and a
> re-roll functionality, once somebody spottes an error he can save the file
> with it's original values, and a 'script' about his actions, then that
> script can be rolled through step by step, if the error shows up you have
> 'steps', if not you have clue about 'weak hardware' or a sporadic fail in
> software ... 

Well, we have a logger which can even automatically create UI tests:
https://ahmedelshreif.blogspot.com/2019/08/gsoc-final-report.html
https://ahmedelshreifgsoc20.blogspot.com/2020/08/gsoc20-final-report-project.html
https://www.youtube.com/watch?v=nnng_er2mro
Comment 11 pitrej1 2020-10-27 12:58:59 UTC
I am having a similar problem: when updating values on spreadsheet not all values are being recalcualted.  Autocalculate is on.  Doing a manual recalculate appears to resolve the problem.  But until then, some values are recalculated and others are not.

This appeared with latest update - I was not having this issue at all before installing 7.0.2.2 yesterday.

An affected spreadsheet available on request.
Comment 12 Buovjaga 2020-10-27 13:02:42 UTC
(In reply to pitrej1 from comment #11)
> I am having a similar problem: when updating values on spreadsheet not all
> values are being recalcualted.  Autocalculate is on.  Doing a manual
> recalculate appears to resolve the problem.  But until then, some values are
> recalculated and others are not.
> 
> This appeared with latest update - I was not having this issue at all before
> installing 7.0.2.2 yesterday.
> 
> An affected spreadsheet available on request.

You are probably conflating this with bug 137248. Please check with 7.0.3 or latest master
Comment 13 b. 2020-11-25 21:12:31 UTC
got one catched, in ver. 6.1.6.3 'unparallelized', 

(i reported the bug as 'threaded' problem as i exprienced plenty fails in that mode, work without threading or openCL since then, and now spotted that typical bug without parallelizing) 

have the file open and can rollback - rollforward with ctrl-z, ctrl-y and see the error evolving, 

will try to keep that file open and answer questions you have, can't provide it for privacy reasons, and it's useless as the fail isn't reproducible in copies, 

failing region: in col AF about 300 formulas totaling from their respective row, e.g. in AF70: '=SUM(D70:AD70)' - that's the affected cell -, theese formulas are grouped in blocks, sometimes single, most 2 to 30 identic, one big block over 100, the blocks are separated by empty cells, in two positions are additional formulas which total from a part of  the column below it, e.g. in AF62: '=SUM(AF64:AF282)' - which includes AF70, the block where AF70 is in is old-24, new-25 'shared formulas', 

all other formulas are 'inside' their respective row, no cross row calculations, 

steps the error evolved: 

- inserted one row at row 69 with 'alt-s r a', 'insert row above', making the former row 69 to now row 70, 

- copied 'now row 70' to row 69 with ctrl-c - ctrl-v to get the needed formulas, 

- cell AF70 with the formula above is totaling '1' from cell M70 and '1' from cell U70 to '2', 

- pasting a '1' in cell I70 failed to update AF70, still '2', expected '3', 

as far as i understand: a group of 24 shared formulas was split between position 1 and 2, formula from position 3 (former 2) is copied to position 2, position 3 lost it's 'autocalc capability' (the property listening to updates of referenced cells), 

doe's that help to catch the bug?
Comment 14 Kevin Suo 2024-02-26 10:38:41 UTC
This old bug report gets 13 comments, but I do not see any clear steps to reproduce.

Could someone who can still reproduce it on the newest version to simply provide:

STEPS TO REPRODUCE: (to speed up our test, please use numeric lists, do not use large paragraphs)
1. XXX
2. XXX
3. XXX

EXPECTED RESULT:
XXXXXX

CURRENT RESULT:
XXX

TEST FILE
Please privide, if any.

MORE INFO:
(If this is a duplicate, please provide the latest version it ever worked, and the oldest known version it was broken.)

Also, revise the summary line of this bug report to include the wording like "steps in comment xx", so that we can easily go to the steps by clicking the hyberlink in the summary line.