Bug Hunting Session
Bug 95505 - Dump usage stats to text file in user profile rather than std error
Summary: Dump usage stats to text file in user profile rather than std error
Status: NEW
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: UI (show other bugs)
Version:
(earliest affected)
5.1.0.0.alpha1
Hardware: Other All
: medium enhancement
Assignee: Not Assigned
URL:
Whiteboard: target:5.2.0
Keywords:
Depends on:
Blocks: User-Profile Usage-Metrics
  Show dependency treegraph
 
Reported: 2015-11-01 18:37 UTC by Yousuf Philips (jay) (retired)
Modified: 2018-02-14 10:21 UTC (History)
7 users (show)

See Also:
Crash report or crash signature:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Yousuf Philips (jay) (retired) 2015-11-01 18:37:43 UTC
Kendy did preliminary work on gather usage stats, but unfortunately this stats are being dumped to standard error, where other errors are also being dumped, meaning that even capturing the standard error (e.g. `soffice 2> lo_stats.txt`) would result in unnecessary data being added to the file.

So as i has suggested to kendy before, it would be useful to pushed this data to a text file in the user profile (e.g. .config/libreofficedev/4/user/stats/usage.txt ), so it is easier to access and obtain from users.

http://opengrok.libreoffice.org/xref/core/sfx2/source/control/unoctitm.cxx#637
Comment 1 aybuke 2015-11-07 12:18:12 UTC
Hi everyone,

I'm using this "#include <boost/filesystem.hpp>" for creating folder(.config/libreofficedev/4/user/stats/). 

But though I had done nothing in the code compilation output (make sfx2): http://pastebin.com/yVdFNvxu

Compilation happening, but is this normal?

Thanks.
Comment 2 Maxim Monastirsky 2015-11-08 10:36:51 UTC
(In reply to aybuke from comment #1)
> I'm using this "#include <boost/filesystem.hpp>" for creating
> folder(.config/libreofficedev/4/user/stats/). 
> 
> But though I had done nothing in the code compilation output (make sfx2):
> http://pastebin.com/yVdFNvxu
> 
> Compilation happening, but is this normal?
Do you mean the -Wshadow warnings? Yes - it's normal (you should blame boost headers for this). But please - don't introduce yet another boost dependency. LO should have internally everything you need to manipulate files/folders. Never did it myself, but include/osl/file.hxx seems to be a good place to start from.
Comment 3 aybuke 2015-12-05 15:05:03 UTC
Hi everyone,

I have one question about writing in file. I sent this patch: https://gerrit.libreoffice.org/#/c/20413/ 

How do I use the "RC osl::File::write" method here? Can I get advice?

Thank you.
Comment 4 jani 2015-12-10 10:33:46 UTC
Hi

I am here to help getting patches aproved and added to trunk, in short mentoring new people.

Kendy made me aware of your good work, it does however have some smaller problems like:

Writing should be done in utf8 format using OUString utf8() output, remember to for check number of bytes written,

Hardcoding ~/.config/libreofficedev/4/user/stats/, instead use what is currently the user's config dir.

Feel free to mail me directly, if you need/want help.

rgds
jan i
Comment 5 aybuke 2015-12-10 18:27:12 UTC
(In reply to jan iversen from comment #4)
> Hi

Hi jan, thanks to for reply, I'm verry happy :)
> 
> I am here to help getting patches aproved and added to trunk, in short
> mentoring new people.
> 
> Kendy made me aware of your good work, it does however have some smaller
> problems like:
> 
> Writing should be done in utf8 format using OUString utf8() output, remember
> to for check number of bytes written,
> 
> Hardcoding ~/.config/libreofficedev/4/user/stats/, instead use what is
> currently the user's config dir.

One question about hardcode part: I don't need to create new directory "stats", i guess. Should I create file "usage.txt" in ".config/libreofficedev/4/user/config/" (using GetUserConfigPath() method.)?
> 
> Feel free to mail me directly, if you need/want help.
> 
> rgds
> jan i
Comment 6 jani 2015-12-10 18:48:17 UTC
(In reply to aybuke from comment #5)
> One question about hardcode part: I don't need to create new directory
> "stats", i guess. Should I create file "usage.txt" in
> ".config/libreofficedev/4/user/config/" (using GetUserConfigPath() method.)?

That is a good way to do it, may qualify with date "usage-12-10.txt
Comment 7 Yousuf Philips (jay) (retired) 2015-12-11 10:18:04 UTC
(In reply to aybuke from comment #5)
> One question about hardcode part: I don't need to create new directory
> "stats", i guess. Should I create file "usage.txt" in
> ".config/libreofficedev/4/user/config/" (using GetUserConfigPath() method.)?

Yes you should create the stats folder as it leaves room for further improvements to the stats to all be in one folder. Also regarding the name of the file, it should default as 'usage.txt' but if a user/admin wishes to name it something else (e.g. libreoffice is run by multiple users from one instance), then it should take the filename from an environment variable.

@Dave: Do you have any input on how best to work with your multiple user libreoffice instance, as i would assume that each user would need their own stats text file.
Comment 8 Dave Richards 2015-12-11 12:57:42 UTC
Each user has their own home directory, so writing to their .config/libreofficedev/4/user/config/ folder would work on a multi-user server.  This feature would give you the best data in a large enterprise deployment with regular office workers...and for that reason the ability to turn this on easily by an admin should be part of the final feature set.

Something like ./soffice --usage-stats

There is no way the end users would be able to turn this on and off at will, nor would they.  Also, a command line flag would allow me to turn this feature off it it caused some kind of scaling issue or other bug.

I would be interested to hear too about the performance hit with something like this running. 100+ people all dumping usage data as they are clicking through the buttons and menus.  

That would make me almost think about putting this data in a RAM drive, so I might need something like:

./soffice --usage-stats --usage-stats-file=/ramdrive/$USER.usage.txt
Comment 9 jani 2015-12-11 13:04:41 UTC
(In reply to Dave Richards from comment #8)
> Each user has their own home directory, so writing to their
> .config/libreofficedev/4/user/config/ folder would work on a multi-user
> server.  This feature would give you the best data in a large enterprise
> deployment with regular office workers...and for that reason the ability to
> turn this on easily by an admin should be part of the final feature set.
> 
> Something like ./soffice --usage-stats
> 
> There is no way the end users would be able to turn this on and off at will,
> nor would they.  Also, a command line flag would allow me to turn this
> feature off it it caused some kind of scaling issue or other bug.
> 
> I would be interested to hear too about the performance hit with something
> like this running. 100+ people all dumping usage data as they are clicking
> through the buttons and menus.  
> 
> That would make me almost think about putting this data in a RAM drive, so I
> might need something like:
> 
> ./soffice --usage-stats --usage-stats-file=/ramdrive/$USER.usage.txt



Let me suggest to keep this bug simple, once that is solved we can find n ways to expand on it. If we start by expanding the scope we might end up with nothing.

just my opinion
Comment 10 Dave Richards 2015-12-11 13:06:48 UTC
I agree with Jan, just letting you know what it would take to get it working here.  I would be able to provide usage data for 600+ users.
Comment 11 Yousuf Philips (jay) (retired) 2015-12-12 06:13:36 UTC
(In reply to Dave Richards from comment #8)
> Each user has their own home directory, so writing to their
> .config/libreofficedev/4/user/config/ folder would work on a multi-user
> server.

Okay.

> This feature would give you the best data in a large enterprise
> deployment with regular office workers...and for that reason the ability to
> turn this on easily by an admin should be part of the final feature set.
> 
> Something like ./soffice --usage-stats
> 
> There is no way the end users would be able to turn this on and off at will,
> nor would they.  Also, a command line flag would allow me to turn this
> feature off it it caused some kind of scaling issue or other bug.

This issue is now in bug 96434.

> I would be interested to hear too about the performance hit with something
> like this running. 100+ people all dumping usage data as they are clicking
> through the buttons and menus.

The usage data isnt dumped when LO is closed, so there shouldnt be any performance hit.

> That would make me almost think about putting this data in a RAM drive, so I
> might need something like:
> 
> ./soffice --usage-stats --usage-stats-file=/ramdrive/$USER.usage.txt

The --usage-stats-file flag seems good for an alternative file location.
Comment 12 Yousuf Philips (jay) (retired) 2015-12-12 07:07:29 UTC
Did some more research and we shouldnt be dumping it all into a single usage.txt file, but instead to have a time stamped file with the time and date, so we have a file per session - e.g. usage-2015-12-12T06_51_40.csv. This is quite important as we dont output the time stamp in the actual usage data.

This would also mean that we would need a --usage-stats-folder flag to handle putting the files else where.
Comment 13 Jan Holesovsky 2015-12-14 20:13:06 UTC
Adding that as a param to the soffice is I'm afraid not the preferred; we have quite a few of these already :-)

Let's do an env. variable instead - see bug 96434, it contains also the code pointer.  Thank you!
Comment 14 Commit Notification 2016-01-10 12:25:04 UTC
Aybuke Ozdemir committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=d61c16966b017abdbebf5ec0c2131de5a91c67f8

tdf#95505 Dump usage stats to text file

It will be available in 5.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 15 Yousuf Philips (jay) (retired) 2016-01-19 07:34:57 UTC
Aybuke,

Thanks for the patch, but you shouldnt define a location that is likely only available on linux.

#define USAGE "file:///~/.config/libreofficedev/4/user/usage.txt"

Jan mentioned this.

(In reply to jan iversen from comment #4)
> Hardcoding ~/.config/libreofficedev/4/user/stats/, instead use what is
> currently the user's config dir.
Comment 16 aybuke 2016-01-19 08:02:43 UTC
(In reply to Yousuf (Jay) Philips from comment #15)
> Aybuke,
> 
> Thanks for the patch, but you shouldnt define a location that is likely only
> available on linux.
> 
> #define USAGE "file:///~/.config/libreofficedev/4/user/usage.txt"
> 
> Jan mentioned this.
> 
> (In reply to jan iversen from comment #4)
> > Hardcoding ~/.config/libreofficedev/4/user/stats/, instead use what is
> > currently the user's config dir.

Hi Yousuf,

I'm trying to fix this part, using SvtPathOptions::GetUserConfigPath() but I can't finish yet. I'll send patch about hardcoded part.
Comment 17 Yousuf Philips (jay) (retired) 2016-01-19 08:51:10 UTC
(In reply to aybuke from comment #16)
> Hi Yousuf,
> 
> I'm trying to fix this part, using SvtPathOptions::GetUserConfigPath() but I
> can't finish yet. I'll send patch about hardcoded part.

Hi Aybuke,

Thanks for the update. Any input on the possibility of putting it in separate files per session (comment 12) and in a /user_profile/usage/ folder?
Comment 18 aybuke 2016-01-19 10:52:29 UTC
(In reply to Yousuf (Jay) Philips from comment #17)
> (In reply to aybuke from comment #16)
> > Hi Yousuf,
> > 
> > I'm trying to fix this part, using SvtPathOptions::GetUserConfigPath() but I
> > can't finish yet. I'll send patch about hardcoded part.
> 
> Hi Aybuke,
> 
> Thanks for the update. Any input on the possibility of putting it in
> separate files per session (comment 12) and in a /user_profile/usage/ folder?

Sure, the patch i'll send includes what you said. (Directory "usage" and filename contains time information.)
Comment 19 aybuke 2016-01-19 23:15:51 UTC
(In reply to Yousuf (Jay) Philips from comment #17)
> (In reply to aybuke from comment #16)
> > Hi Yousuf,
> > 
> > I'm trying to fix this part, using SvtPathOptions::GetUserConfigPath() but I
> > can't finish yet. I'll send patch about hardcoded part.
> 
> Hi Aybuke,
> 
> Thanks for the update. Any input on the possibility of putting it in
> separate files per session (comment 12) and in a /user_profile/usage/ folder?

I sent this patch: https://gerrit.libreoffice.org/#/c/21609/
I have a question in patch(draft). Can I add you to reviewers part, Yousuf?
Comment 20 Yousuf Philips (jay) (retired) 2016-01-20 10:32:26 UTC
(In reply to aybuke from comment #19)
> I sent this patch: https://gerrit.libreoffice.org/#/c/21609/
> I have a question in patch(draft). Can I add you to reviewers part, Yousuf?

I've commented in the patch of stuff i noticed. :D
Comment 21 Commit Notification 2016-01-27 15:59:56 UTC
aybuke committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=3563d8dde1068094a7b79aad4aa864301583006d

tdf#95505 Dump usage stats to text file in user profile.

It will be available in 5.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 22 Yousuf Philips (jay) (retired) 2016-01-28 08:32:40 UTC
Hey aybuke,

Glad the patch got in and look forward to testing it out when the next build comes out. I noticed in the patch that you didnt change semi-colons to colons, like in the following line, so i'm assuming the output wont be valid CSV when opened (e.g. TextDocument;.uno:DesignerDialog;4 ).

aUsageInfoMsg += "\n" + it->first.toUtf8() + ";" + OString::number(it->second);

I'm assuming the semi-colon that appears between the document type and command name must be coded somewhere else in the code.
Comment 23 aybuke 2016-01-30 12:31:52 UTC
(In reply to Yousuf (Jay) Philips from comment #22)
> Hey aybuke,
> 
> Glad the patch got in and look forward to testing it out when the next build
> comes out. I noticed in the patch that you didnt change semi-colons to
> colons, like in the following line, so i'm assuming the output wont be valid
> CSV when opened (e.g. TextDocument;.uno:DesignerDialog;4 ).
> 
> aUsageInfoMsg += "\n" + it->first.toUtf8() + ";" +
> OString::number(it->second);
> 
> I'm assuming the semi-colon that appears between the document type and
> command name must be coded somewhere else in the code.

Hi Yousuf,

I send this patch for you say: https://gerrit.libreoffice.org/#/c/21920/
If I understand wrong, I can reorganize :)
Comment 24 Yousuf Philips (jay) (retired) 2016-02-01 18:28:16 UTC
(In reply to aybuke from comment #23)
> Hi Yousuf,
> 
> I send this patch for you say: https://gerrit.libreoffice.org/#/c/21920/
> If I understand wrong, I can reorganize :)

Hi aybuke,

Yes my last comment was to be a further patch to be sent in to correct the current output, which is likely invalid CSV, as its using semi-colons rather than commas.

Regarding that patch, unfortunately it isnt giving me the output, as i dont think "OUString path(SvtPathOptions().GetConfigPath());" is extracting the correct path for the config directory in my parallel installation, where i've set 'UserInstallation=$SYSUSERCONFIG/libreofficedev/4_5_2' in bootstraprc.
Comment 25 jani 2016-02-01 18:45:14 UTC
(In reply to Yousuf (Jay) Philips from comment #24)
> (In reply to aybuke from comment #23)
> > Hi Yousuf,
> > 
> > I send this patch for you say: https://gerrit.libreoffice.org/#/c/21920/
> > If I understand wrong, I can reorganize :)
> 
> Hi aybuke,
> 
> Yes my last comment was to be a further patch to be sent in to correct the
> current output, which is likely invalid CSV, as its using semi-colons rather
> than commas.
> 
> Regarding that patch, unfortunately it isnt giving me the output, as i dont
> think "OUString path(SvtPathOptions().GetConfigPath());" is extracting the
> correct path for the config directory in my parallel installation, where
> i've set 'UserInstallation=$SYSUSERCONFIG/libreofficedev/4_5_2' in
> bootstraprc.

Actually the definition of csv is not comma (for one reason it is used in europe as decimal separator. The old csv standard used ";", and later it was defined that you can use any character as long as it does not appear in the values (except if enclosed by ""). You will see that most import filters allow you to specify which delimiter to use.

I too have a parallel installation (debug and production), and GetConfigPath() delivers the right directories to me. Test if your configs are separated, modify the configuration in one installation and make sure it is not changed in the other.

if you look at the code, you will see several places where GetConfigPath() is used, so I sure hope it works.
Comment 26 aybuke 2016-02-02 10:19:00 UTC
Hi everyone,

> > Yes my last comment was to be a further patch to be sent in to correct the
> > current output, which is likely invalid CSV, as its using semi-colons rather
> > than commas.

> 
> Actually the definition of csv is not comma (for one reason it is used in
> europe as decimal separator. The old csv standard used ";", and later it was
> defined that you can use any character as long as it does not appear in the
> values (except if enclosed by ""). You will see that most import filters
> allow you to specify which delimiter to use.

I try fix CSV format this patch:
https://gerrit.libreoffice.org/#/c/21920/5
2 objects and 3 elements in there:

output example:
Document Type;Command;Count
StartModule;.uno:OptionsTreeDialog;1
TextDocument;.uno:Bold(KeyModifier);2 TextDocument;.uno:CharBackgroundExt(BackColor);1 TextDocument;.uno:Save(KeyModifier);1 TextDocument;.uno:SelectAll;1

Shall I change elements as "uno command" and "count" or something else that you suggest?
 
> > Regarding that patch, unfortunately it isnt giving me the output, as i dont
> > think "OUString path(SvtPathOptions().GetConfigPath());" is extracting the
> > correct path for the config directory in my parallel installation, where
> > i've set 'UserInstallation=$SYSUSERCONFIG/libreofficedev/4_5_2' in
> > bootstraprc.

> 
> I too have a parallel installation (debug and production), and
> GetConfigPath() delivers the right directories to me. Test if your configs
> are separated, modify the configuration in one installation and make sure it
> is not changed in the other.
> 
> if you look at the code, you will see several places where GetConfigPath()
> is used, so I sure hope it works.

I'll do more research about GetConfigPath().
Comment 27 Yousuf Philips (jay) (retired) 2016-02-02 14:08:45 UTC
(In reply to jan iversen from comment #25)
> Actually the definition of csv is not comma (for one reason it is used in
> europe as decimal separator. The old csv standard used ";", and later it was
> defined that you can use any character as long as it does not appear in the
> values (except if enclosed by ""). You will see that most import filters
> allow you to specify which delimiter to use.

I've used and seen csv files for many years and know it to stand for "comma-separated values" and am aware that the format allows other field separators (see it in LO and mysql import/export dialogs), but still the standard format is to use a comma. Even LO defaults to export commas for CSVs.

https://en.wikipedia.org/wiki/Comma-separated_values

Ultimately i was bringing up the issue as aybuke was using commas on one line and then semi-colons on another in the csv file.

> I too have a parallel installation (debug and production), and
> GetConfigPath() delivers the right directories to me. Test if your configs
> are separated, modify the configuration in one installation and make sure it
> is not changed in the other.
>
> if you look at the code, you will see several places where GetConfigPath()
> is used, so I sure hope it works.

I have 11 parallel installations all running in their own separate config folder. Is there somewhere in the UI i can see where the path of GetConfigPath() appears? In Tools > Options > LibreOffice > Paths, all the paths there look fine. I searched ~/.config for a "usage" folder that was supposed to be created by

    path += "usage/";
    osl::Directory::createPath(path);

but couldnt find one.
Comment 28 Commit Notification 2016-02-03 07:53:16 UTC
aybuke committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=2db74425208033f848ddb5bac7277d30d72afcad

tdf#95505  Dump usage stats to text file.

It will be available in 5.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 29 Xisco Faulí 2016-09-15 21:17:47 UTC
Hi,
Is this bug fixed?
If so, could you please close it as RESOLVED FIXED?
Regards
Comment 30 Yousuf Philips (jay) (retired) 2017-05-04 10:57:18 UTC
(In reply to Xisco Faulí from comment #29)
> Is this bug fixed?
> If so, could you please close it as RESOLVED FIXED?

No not fully fixed. :(

@Baron: Maybe a weekend fun project when you get the time. :D