Bug 137527 - FILESAVE LibreOffice Calc advertised as 1024 columns and 1048576 rows appears to be a limit of between 11 534 336 and 13 037 568 active cells that will save to disc.
Summary: FILESAVE LibreOffice Calc advertised as 1024 columns and 1048576 rows appears...
Status: RESOLVED WORKSFORME
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: Calc (show other bugs)
Version:
(earliest affected)
6.4.6.2 release
Hardware: x86-64 (AMD64) Linux (All)
: medium normal
Assignee: Not Assigned
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-10-16 11:33 UTC by Willim
Modified: 2022-11-23 09:37 UTC (History)
3 users (show)

See Also:
Crash report or crash signature:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Willim 2020-10-16 11:33:33 UTC
Description:
Linux Mint 20 16GiB Memory 2.0GiB Swap
Calc Build ID 1:6.4.6-0ubuntu0.20.04.1

From my tests there seems to be a limit of between 11 534 336 and 13 037 568 active cells that will save to disc.  More can be displayed but don't save - Calc is terminated and file has to be recovered

From variation in file size 56.7Mb 120.1Mb I doubt if it is a file size restriction, I could be wrong.

if A -AMJ filled only 12732 rows 120.1Mb  13_037_568 cells
A1=1 B1=A1+10 C1=B1+10    ... AMJ1=AMI1+10
A2=A1+1 B2=B1+1 C2=C1+1 ... AMJ2=AMJ1+1
row 2 copied to rows 3 - 12732

if A1 - A1048576 filled only to column K 56.7Mb 11_534_336 cells
column B copied to columns C to K 
A1=1                          B1=A1+1
A2=A1+1                   B2=B2+1 
A3=A2+1                  B3=A3+1
A1048576=A1048575+1   B1048576=A1048576+1

A1 can be changed to say 10 file saves and can be changed to 1 again 
with no issues if 12732 rows or K columns

Further tests and summary

12732  AMJ (1024) 13_037_568 120.1MB
25347  SR  (512)  12_973_056 117.8MB
50601  IV  (256)  12_956_160 117.3MB
100810 DX  (128)  12_903_680 108.7MB
198896 BL  ( 64)  12_729_344 102.5MB

Working downwards Columns halved and Rows duplicated 


262144  AV (48)   12_582_912  61.0MB
524288  X  (24)   12_582_912  61.6MB
1048576 K  (11)   11_534_336  56.5MB

Working upwards Rows halved and Columns duplicated


Cannot copy row 50601 to 50602 or column AV to AW ...

Additional rows/columns are calculated properly but do NOT save to disc.

When document is recovered it is only to last "Save"

Steps to Reproduce:
First Spreadsheet
if A -AMJ filled only 12732 rows 120.1Mb  13_037_568 cells
A1=1 B1=A1+10 C1=B1+10    ... AMJ1=AMI1+10
A2=A1+1 B2=B1+1 C2=C1+1 ... AMJ2=AMJ1+1
row 2 copied to rows 3 - 12732

Failure when try to save row 12733 ..


on second spreadsheet

if A1 - A1048576 filled only to column K 56.7Mb 11_534_336 cells
column B copied to columns C to K 
A1=1                          B1=A1+1
A2=A1+1                   B2=B2+1 
A3=A2+1                  B3=A3+1
A1048576=A1048575+1   B1048576=A1048576+1

A1 can be changed to say 10 file saves and can be changed to 1 again 
with no issues if 12732 rows or K columns

Failure when trying to save column K

Actual Results:
On attempting to save additional row / column
LibreOffice 6.4 Document Recovery
Due to an error, LibreOffice crashed. All the files you were working on will now be saved. The next time LibreOffice is launched, you files will be recovered automatically.

Only recovers to last "Save" no point in "Recovery"

Expected Results:
File saved and continue 


Reproducible: Always


User Profile Reset: No



Additional Info:
Version: 6.4.6.2
Build ID: 1:6.4.6-0ubuntu0.20.04.1
CPU threads: 12; OS: Linux 5.4; UI render: default; VCL: gtk3; 
Locale: en-GB (en_GB.UTF-8); UI-Language: en-GB
Calc: threaded

OpenGL vendor string: X.Org
OpenGL renderer string: Radeon RX 570 Series (POLARIS10, DRM 3.35.0, 5.4.0-51-generic, LLVM 10.0.0)
OpenGL core profile version string: 4.6 (Core Profile) Mesa 20.0.8
OpenGL core profile shading language version string: 4.60
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile
OpenGL core profile extensions:
OpenGL version string: 4.6 (Compatibility Profile) Mesa 20.0.8
OpenGL shading language version string: 4.60
OpenGL context flags: (none)
OpenGL profile mask: compatibility profile
OpenGL extensions:
OpenGL ES profile version string: OpenGL ES 3.2 Mesa 20.0.8
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20
OpenGL ES profile extensions:
Comment 1 Roman Kuznetsov 2020-10-19 20:59:11 UTC
I don't see problems with filled cells in range A1:AMJ20000. I could save it to disk easy. Yes, it took many time but it ended fully correct and then I could open that ~200mb ODS file

Calc took ~6Gb of memory in that case

Possibly you just have little free space on your HDD/SSD for saving that huge file?

My build is

Version: 7.1.0.0.alpha0+ (x64)
Build ID: 7aaa9ef2e5edaf468f116449776433e98fb1a2f3
CPU threads: 4; OS: Windows 10.0 Build 18363; UI render: Skia/Raster; VCL: win
Locale: ru-RU (ru_RU); UI: ru-RU
Calc: threaded

I have 4 core Intel with 12Gb of memory here and Windows 10
Comment 2 Willim 2020-10-22 11:16:36 UTC
Saving to 1Tb drive 700+Gb free ... No disc issues 

I have done more tests using text a1="a"  b1=a1 c1=b1 .. amj1=ami1
a2=a1 b2=b1 ... amj2=amj1 etc
I varied a1= a ab abc abcdefghij abcdefghijklmnopqrst

a 12747 rows, ab 12597 abc 12448 abcdefghij 11497 & abcdefghijklmnopqrst 10367 rows.

All Test Summary:-
    	                Rows	Columns		Active Cells		%	Size
                    	1048576	1024		1073741824			    MB
							
A1=1	                12732	1024	AMJ	13037568		1.21	120.1
    	                25347	512	SR	    12977664		1.21	117.8
    	                50601	256	IV	    12953856		1.21	117.3
							
	                    100810	128	DX	    12903680		1.20	108.7
	                    198896	64	BL	    12729344		1.19	102.5
							
	                    262144	48	AV	    12582912		1.17	61.0
	                    524288	24	X	    12582912		1.17	61.6
	                    1048576	11	K	    11534336		1.07	56.5
							
							
a	                    12749	1024	AMJ	13054976		1.22	40.0
ab	                    12597	1024	AMJ	12899328		1.20	39.3
abc	                    12448	1024	AMJ	12746752		1.19	38.8
							
abcdefghij          	11497	1024	AMJ	11772928		1.10	35.9
abcdefghijklmnopqrst	10367	1024	AMJ	10615808		0.99	32.4
Comment 3 Xisco Faulí 2021-11-23 11:12:05 UTC
Hello Willim,
Could you please try to reproduce it with the latest version of LibreOffice from https://www.libreoffice.org/download/libreoffice-fresh/ ?
I have set the bug's status to 'NEEDINFO'. Please change it back to 'UNCONFIRMED' if the bug is still present in the latest version.
Comment 4 Willim 2021-11-24 11:06:42 UTC
I am happy knowing the limitations and the much improved "Error Saving Document" system. 

When 2.4% of cells are used the file size is approximately 240MB by extrapolation for all cells to used the maximum required file size could be in the order of 10GB.

The "240MB" file takes 4 minutes to open on my system if the file was 10GB it may take in the order of 160 minutes to open. Perhaps not very practical 

Versions 7.0.6.2 and 7.2.2.2 now can use about 2.4% of the available cells on a sheet this is up from 1.2% of Version 6.4.6.2

I have tested both Versions 7.0.6.2 and 7.2.2.2 the results are the same for both. Details below:-

                Rows    Cols        Cells Used                   %       MB      File
23 Nov 2021	25338	1024	   AMJ	25946112		2.42	238.2
23 Nov 2021	50637	512	    SR	25926144		2.41	235.6
23 Nov 2021	395858	64	    BL	25334912		2.36    204.0
23 Nov 2021	1048576	24	    K	25165824		2.34	122.3

When the file cannot be saved the following message is now displayed:-

        Error saving the document ?_file_name_?:
        General Error.
        General input/output error.
                                [ OK ]

Press OK and one can delete rows or columns until the file saves.

This new message and use of a temporary file much better than the "recovery method" of 6.4.6.2. Thank you coders/ programmers.

Version Information
Version         7.0.6.2
Environment     CPU threads 12: OS: Linux 5.4
User Interface  UI render: default; VCL: gtk3
Locale          en-GB (en.GB.UTF-8); UI en-US
Misc.           Ubuntu package version:
                1:7.0.6-0ubuntu0.20.04.01_lo1
                Calc: threaded
24 Nov 2021
Version Information
Version         7.2.2.2 / LibreOffice Community 
Build           02b2acce88a210515b4a5bb2e46cbfb63fe97d56
Environment     CPU threads 12: OS: Linux 5.4
User Interface  UI render: default; VCL: gtk3
Locale          en-GB (en.GB.UTF-8); UI en-US
Misc.           Calc: threaded

cat /etc/os-release
NAME="Linux Mint"
VERSION="20.2 (Uma)"
ID=linuxmint
ID_LIKE=ubuntu
PRETTY_NAME="Linux Mint 20.2"
VERSION_ID="20.2"
HOME_URL="https://www.linuxmint.com/"
SUPPORT_URL="https://forums.linuxmint.com/"
BUG_REPORT_URL="http://linuxmint-troubleshooting-guide.readthedocs.io/en/latest/"
PRIVACY_POLICY_URL="https://www.linuxmint.com/"
VERSION_CODENAME=uma
UBUNTU_CODENAME=focal

uname -svro
Linux 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 GNU/Linux

lsb_release
LSB Version:	core-11.1.0ubuntu2-noarch:printing-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch
Comment 5 Buovjaga 2022-11-22 11:07:22 UTC
(In reply to Willim from comment #4)
> When the file cannot be saved the following message is now displayed:-
> 
>         Error saving the document ?_file_name_?:
>         General Error.
>         General input/output error.
>                                 [ OK ]
> 
> Press OK and one can delete rows or columns until the file saves.
> 
> This new message and use of a temporary file much better than the "recovery
> method" of 6.4.6.2. Thank you coders/ programmers.

It sounds like we can close this.

If you are concerned by the performance of opening large files, you can open a new report, but first I recommend to test with 7.4 or even with the unreleased 7.5. Many performance improvements have been made in that department.
Comment 6 Willim 2022-11-23 09:36:29 UTC
Happy to mark as resolved
running Linux Mint
Version Information
Version         7.3.2.2
Build           49f2b1bff42cfccbd8f788c8dc32c1c309559be0
Environment     CPU threads 12: OS: Linux 5.4
User Interface  UI render: default; VCL: gtk3
Locale          en-GB (en.GB.UTF-8); UI en-US
Misc.           Calc: threaded

Latest version for Linux "Check for Updates" >" LibreOffice 7.3 is up to date"
Comment 7 Willim 2022-11-23 09:37:41 UTC
Resolved