Bug 118639 - Editing password-protected non-ODF files creates multiple empty files in backup directory
Summary: Editing password-protected non-ODF files creates multiple empty files in back...
Status: RESOLVED FIXED
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: LibreOffice (show other bugs)
Version:
(earliest affected)
4.3.0.4 release
Hardware: x86-64 (AMD64) All
: high major
Assignee: Mike Kaganski
URL:
Whiteboard: target:6.5.0 target:6.4.0.1 target:6.3.5
Keywords: bibisected, bisected, dataLoss, regression
: 124537 (view as bug list)
Depends on:
Blocks: AutoSave-AutoRecovery-Backup
  Show dependency treegraph
 
Reported: 2018-07-09 15:32 UTC by Stuart Edwards
Modified: 2022-09-26 05:37 UTC (History)
7 users (show)

See Also:
Crash report or crash signature:


Attachments
Screen shot from screen capture showing intermittent file appearance (202.87 KB, image/png)
2018-09-07 19:08 UTC, Stuart Edwards
Details
problem file (55.50 KB, text/xls)
2018-09-17 15:03 UTC, Stuart Edwards
Details
registrymodifications.xcu (1.33 MB, text/plain)
2018-09-18 18:42 UTC, Stuart Edwards
Details
Sampling trace of LO process (2.29 MB, text/plain)
2019-02-07 10:21 UTC, Alex Thurgood
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Stuart Edwards 2018-07-09 15:32:55 UTC
Just updated LO to 6.0.5.2 on Mac 10.13.5 and it's completely unusable; every key stroke induces a 5 second (est) wait while the colored ball swirls.  The fan runs and LO is using 100% CPU.  It runs cleanly on Ubuntu and W7 in a VM on the same machine.

This seems to have been a long standing problem that I for one have experienced for a number of years now.  It's never been this bad though.
Comment 1 Xisco Faulí 2018-07-09 16:01:02 UTC
Thank you for reporting the bug. To be certain the reported issue is not
related to corruption in the user profile, could you please reset your
Libreoffice profile ( https://wiki.documentfoundation.org/UserProfile ) and
re-test?

I have set the bug's status to 'NEEDINFO'. Please change it back to
'UNCONFIRMED' if the issue is still present
Comment 2 Stuart Edwards 2018-07-09 18:43:24 UTC
Thanks for the prompt response.  That worked!
Comment 3 Xisco Faulí 2018-07-09 18:48:15 UTC
Glad it helped.
Closing as RESOLVED NOTABUG
Comment 4 Stuart Edwards 2018-07-27 13:08:12 UTC
The plot thickens..... The problem has reoccurred after about 3 weeks.  So I am wondering why the profile file becomes corrupt.  It seems that if it's a recurrent phenomenon, it's likely to be related to my use of LO and -- therefore -- a bug? I don't do anything complex - just write and edit a few technical reports.

So looking in Library/ApplicationSupport/LibreOffice and LibreOfficeDev I find a directory at 4/user/backup that contains multiple copies of a spreadsheet file that is password protected and which I keep open for long periods.  There's over 7000 and 21000 copies in those two directories respectively.  And more are being produced all the time - but all contain zero bytes.  Maybe this has something to do with the performance issue.

So I'm taking the liberty to reopen this bug in the hope that an explanation and fix can be figured out.  Resetting the profile to factory settings once a month is extremely inconvenient - Thanks for your consideration and efforts.  Let me know if there's additional information I can provide.
Comment 5 Alex Thurgood 2018-07-27 14:01:14 UTC
@Sturat : in view of your last comment, you appear to have the automatic backup option ticked in your preferences 

If you try turning this off, do things improve ?
Comment 6 Stuart Edwards 2018-07-27 14:59:58 UTC
I do have the auto recovery setting turned on - with a 10 min interval, but not the 'always create a backup copy'. When I delete all 20000 files, LO functions normally - at least for the short term.

I suspect that while LO has been non responsive it was keeping busy creating empty backup files for this one spreadsheet.  This morning for example it was generating one every 3 seconds (21/min).  I have had the offending file open for an hour or so now and no new backups have been created - so it seems to be an intermittent 'feature'.

I did see in a 4/user/backup-old directory for LO that there were another 17000 copies dating back to 2016. This has been occurring with both LO and LODev.
Comment 7 Telesto 2018-07-27 15:53:24 UTC
Thanks for the investigation :-). A sample document would be nice.
(Please note that the attachment will be public, remove any sensitive information before attaching it. 
See https://wiki.documentfoundation.org/QA/FAQ#How_can_I_eliminate_confidential_data_from_a_sample_document.3F for help on how to do so.)
Comment 8 Stuart Edwards 2018-07-27 16:56:47 UTC
The backup files are not too enlightening with no content.  The original used to be an ods file and suffered from the same problem.  The backup file names have the style:
test_171109.xls_7436.ods
The only distinguishing feature is that it's the one file I use that's password protected.  So it's as if LO tries to make a backup, can't access the content so opens a new file and tries again.

The original is 7 columns, ~350 rows of mixed text and numerical data - sorry can't share that with you. It's quite unremarkable - no formulae or macros.  I've occasionally sorted it but that's about it.
Comment 9 Stuart Edwards 2018-08-06 12:30:19 UTC
After a further one week of usage, LO has not created any new backups of the password protected file - or any others.  The 'always create backup copy' is not turned on - neither was it while the problem was at its worst.  The 'save auto recovery information ' is and always has been active.  I'll continue to monitor this and if there's a recurrence let you know....

thx
Comment 10 Xisco Faulí 2018-08-07 15:38:19 UTC
Happy to hear LibreOffice is working fine now. Anyway, could you please attach the document which is creating so many backups ?
Comment 11 Stuart Edwards 2018-09-07 19:08:09 UTC
Created attachment 144745 [details]
Screen shot from screen capture showing intermittent file appearance

see comment discussing this graphic
Comment 12 Stuart Edwards 2018-09-07 19:11:05 UTC
The saga continues.  Noticing that LO was getting a little sluggish I went back to the user support files and sure enough, the backup directory was full of zero byte copies of my protected file (140,000).  I deleted them and then was able to watch them being re-created in batches of 3.

I also noticed a file name flash in the adjacent finder window - it appears every 30 seconds or so and is hard to read as it's only there for a split second. I started a screen record and managed to catch it and then took a screen shot (attached - iYNegh).  The file name is always different but of the same style - 6 alpha-numeric characters.

An incidental finding is that the user directory contained about 850 Mb of files - so I hunted down the source and found that is was in 'temp' files.  200 Mb in the user and  in user-old/temp another 600 Mb.  Should these files self-delete if they are 'temps'?  Can I safely delete them?
Comment 13 Stuart Edwards 2018-09-07 19:21:50 UTC
Interesting that the replication of this file does not occur in the LODev files and neither is there a mysterious flashing file.

There is however 500 Mb in the temp file that is probably unnecessary?
Comment 14 Xisco Faulí 2018-09-13 10:53:47 UTC
(In reply to Xisco Faulí from comment #10)
> Happy to hear LibreOffice is working fine now. Anyway, could you please
> attach the document which is creating so many backups ?

Any change you could attached the problematic document ?
Comment 15 Stuart Edwards 2018-09-17 15:03:12 UTC
Created attachment 144946 [details]
problem file
Comment 16 Stuart Edwards 2018-09-17 15:06:54 UTC
The problem file is uploaded.  It has been redacted and the password changed to LObug.
When I opened a new calc file today, ~4000 new backups of the password protected problem file were created.
Comment 17 Xisco Faulí 2018-09-18 12:05:50 UTC
Putting back to UNCONFIRMED...
Comment 18 Stuart Edwards 2018-09-18 14:09:37 UTC
Misc observations

The file was created in an earlier version of LO - maybe 5451 or before.

I am now using 6052 and the file is open most of the time in this version.  However, the duplicate backups are accumulating in the backup file associated with 5451 which is not even open.  

The mysterious file name that flashes on briefly is also in the user directory of 5451.

There appears to be no unusual activity in 6052, but LO 6052 becomes sluggish when the backups are being produced.
Comment 19 Stuart Edwards 2018-09-18 18:42:09 UTC
Created attachment 144994 [details]
registrymodifications.xcu

Further misc comments

I deactivated the backup file in 5451 but of course LO created a new one and this time the rogue backups are accumulating in the temp file (~5000 already). LO 5451 has not been opened.

Of possible interest is the registrymodifications.xcu file in users that was created new this morning.  (see attached)
Comment 20 Stuart Edwards 2018-09-18 19:34:05 UTC
The production of multiple zero byte backups appears to occur only when there are unsaved changes in the problem file.  When the document is saved, production ceases.
Comment 21 Alex Thurgood 2018-10-17 07:44:34 UTC
Possible duplicate of bug 115897 ?
Comment 22 Alex Thurgood 2019-02-07 09:47:08 UTC
Hmm, so I tried playing around with the test file, which is an XLS file, in LODev master in my build tree :

Version: 6.3.0.0.alpha0+
Build ID: e9db8eceff48290be72591f7422b4fc45e5752fc
CPU threads: 4; OS: Mac OS X 10.14.2; UI render: default; VCL: osx; 
Locale: fr-FR (fr_FR.UTF-8); UI-Language: en-US
Calc: threaded

Unfortunately, I couldn't see any slowdown when changing cell values and leaving them unsaved, nor even after saving. I couldn't find any evidence of 0-byte file creation in any /tmp folders either, so inconclusive on my part, sorry.

I'm not sure where, or if, LODev creates a /tmp or /backup folders when running from the build tree, so will try again with a locally installed version of LODev to see if I can spot anything.
Comment 23 Alex Thurgood 2019-02-07 09:55:32 UTC
I can confirm the slowdown when using 

Version: 6.3.0.0.alpha0+
Build ID: e9db8eceff48290be72591f7422b4fc45e5752fc
CPU threads: 4; OS: Mac OS X 10.14.2; UI render: default; VCL: osx; 
Locale: fr-FR (fr_FR.UTF-8); UI-Language: en-US
Calc: threaded


installed in my local Applications folder and using a LODev user profile.
It is really quite impressive, to the point where typing in a cell causes a several second delay and spinning beachball to appear on screen until I can move to another cell.

@Stuart: thanks for providing more detailed information !

Why this is happening, I don't know, and am not sure, how to pinpoint the cause further.
Comment 24 Alex Thurgood 2019-02-07 10:21:00 UTC
Created attachment 148975 [details]
Sampling trace of LO process

Enclosing a sampling trace of the LO process whilst it occupies 100% processor resources and is in effect, "not responding". The only way out of this is a forced kill.


Once again, the major culprit appears to be gazillions of CoreGraphic calls.
Comment 25 Alex Thurgood 2019-02-07 10:37:09 UTC
Hmm, my build is made with XCode 10.1, so probably a duplicate of bug 121778.
Comment 26 Xisco Faulí 2019-02-07 14:01:37 UTC
(In reply to Alex Thurgood from comment #25)
> Hmm, my build is made with XCode 10.1, so probably a duplicate of bug 121778.

Talked with bearon about in IRC. I'll update the wiki to show how to build with Xcode 9 on Mojave
Comment 27 Stuart Edwards 2019-02-07 16:33:22 UTC
(In reply to Alex Thurgood from comment #23)
> I can confirm the slowdown when using 
> 
> Version: 6.3.0.0.alpha0+
> Build ID: e9db8eceff48290be72591f7422b4fc45e5752fc
> CPU threads: 4; OS: Mac OS X 10.14.2; UI render: default; VCL: osx; 
> Locale: fr-FR (fr_FR.UTF-8); UI-Language: en-US
> Calc: threaded
> 
> 
> installed in my local Applications folder and using a LODev user profile.
> It is really quite impressive, to the point where typing in a cell causes a
> several second delay and spinning beachball to appear on screen until I can
> move to another cell.
> 
> @Stuart: thanks for providing more detailed information !
> 
> Why this is happening, I don't know, and am not sure, how to pinpoint the
> cause further.

I opened the problem file (PF) this morning to try and get better information for you.  

1. Open Finder to  Library/Application Support/LibreOffice/4/user/ . No /backup present as previously deleted
2. Open PF which is an xls file created in LO
3. Type 'xyz' in empty cell
4. /backup created and copies of PF created at rate of one every 10 seconds (exactly).  Once this directory contains several thousand copies, performance of LO declines to 'unusable'. At that point, each cell entry causes multi-second response delay.
5. Note the creation of files in /user  concurrent with copies in /backup which flash momentarily and then disappear.  Files have names in the style ' G49NIF' as captured using screen video.
6. Search all directories for this file - non found
7. SaveAs  PF.xlsx and close / reopen
8. Type 'abc' in empty cell
9. No copies created, but mystery files are still appearing whenever a cell value is altered.

Any idea what the transitory files are?  Seems this problem will go away as xls is used less and less.....
Comment 28 Mike Kaganski 2019-04-15 07:05:01 UTC
*** Bug 124537 has been marked as a duplicate of this bug. ***
Comment 29 Buovjaga 2019-04-15 12:29:40 UTC
Tweaking metadata per Mike's information.

Bibisected with win32-4.3 to https://gerrit.libreoffice.org/plugins/gitiles/core/+/ef87ff6680f79362a431db6e7ef2f40cfc576219%5E!/
fdo#51819: autorecovery: fix saving password in protected documents.
Comment 30 Mike Kaganski 2019-11-29 10:36:31 UTC
https://gerrit.libreoffice.org/84052
Comment 31 Commit Notification 2019-11-29 13:13:24 UTC
Mike Kaganski committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/commit/63634738dd03cc74806ce6843c16ff5e51a371a0

tdf#118639: store ODF encryption data for autorecovery

It will be available in 6.5.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 32 Commit Notification 2019-12-02 14:59:27 UTC
Mike Kaganski committed a patch related to this issue.
It has been pushed to "libreoffice-6-4":

https://git.libreoffice.org/core/commit/e569dc9824e95617d921bb8f115d243aea0125b9

tdf#118639: store ODF encryption data for autorecovery

It will be available in 6.4.0.1.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 33 Commit Notification 2019-12-05 07:06:16 UTC
Mike Kaganski committed a patch related to this issue.
It has been pushed to "libreoffice-6-3":

https://git.libreoffice.org/core/commit/d1450f5bddd0f108078e0dfb11c9f130175fafe7

tdf#118639: store ODF encryption data for autorecovery

It will be available in 6.3.5.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.