Bug 39593 - copy/paste code detector / fixing
Summary: copy/paste code detector / fixing
Status: NEW
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: LibreOffice (show other bugs)
Version:
(earliest affected)
unspecified
Hardware: Other All
: medium normal
Assignee: Not Assigned
URL:
Whiteboard: target:5.4.0 target:6.0.0 target:6.1....
Keywords: difficultyBeginner, easyHack, skillCpp, topicCleanup
Depends on:
Blocks: Clipboard
  Show dependency treegraph
 
Reported: 2011-07-27 03:09 UTC by Björn Michaelsen
Modified: 2019-03-10 06:12 UTC (History)
16 users (show)

See Also:
Crash report or crash signature:


Attachments
Removes duplicate code from sc/source/ui/view/output2.cxx (22.06 KB, patch)
2013-04-30 07:40 UTC, Minh Ngo
Details
Clones found with CCFinderX (124.60 KB, application/x-xz)
2014-06-09 10:31 UTC, Peter Senna Tschudin
Details
Code clones found with CCFinderX (20.33 KB, application/x-xz)
2014-06-09 13:07 UTC, Peter Senna Tschudin
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Björn Michaelsen 2011-07-27 03:09:26 UTC
=== copy/paste code detector / fixing ===

'''Background:''' There is a large amount of copy/paste/optional-rename coding going on in the LibreOffice code-base. This creates a maintenance disaster, bigger, slower code, etc. We should find a good cut/paste detector and run it over the code, and fix the output. There are a few known big chunks of code, perhaps more can be added here eg.

tools/source/fsys/tempfile.cxx vs. unotools/source/ucbhelper/tempfile.cxx - cf. CreateTemplName_Impl vs. filter/source/placeware/tempfile.cxx etc.

A good copy and paste detector is [http://www.harukizaemon.com/simian/ Simian]. The use is free for non-commercial projects. To get a space separated list of .cxx files in a directory, run:

find <directory> -name "*.cxx" | grep -v unxlng | tr '\n' ' ' > files

The Simian call to get code pieces with at least 30 equivalent lines is for example:

java -jar simian-2.3.31.jar -threshold=30 -language=c++ `cat files`

'''Skills:''' building, basic C++
Comment 1 Florian Reisinger 2012-05-18 09:04:14 UTC
Deteted "Easyhack" from summary
Comment 2 Minh Ngo 2013-04-30 07:40:02 UTC
Created attachment 78635 [details]
Removes duplicate code from sc/source/ui/view/output2.cxx

Hi,

Possibly removes copy/paste code from the output2.cxx. I have refactored
duplicated code into several methods.

It's my first patch for LO Project. So any criticism is welcome :)

Regards,
Minh
Comment 3 Michael Meeks 2013-04-30 08:18:27 UTC
Great to see your work Minh - I'll try to get to reviewing it, but Markus might get there faster (not sure :-).
Comment 4 Markus Mohrhard 2013-04-30 08:42:28 UTC
Will look into the patch later.
Comment 5 Michael Meeks 2013-05-02 10:54:20 UTC
Wow - this looks really nice Minh - good work there ! it's great to see duplicate code dying ... Just build testing before pushing to master; the code looks correct just reading it.
Comment 6 Commit Notification 2013-05-02 11:04:50 UTC
Minh Ngo committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=ca8b63523885c89c694799622c319260346b00f6

fdo#39593 - share copy/paste code, and add a few consts.



The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 7 Jelle van der Waa 2013-06-19 19:16:22 UTC
I have been bugging this bug, and pushed two patches to gerrit. But now I am stuck and need some guidance. I have found that these files are quiet similiar and I am uncertain how I can fix it.
 
Between lines 213 and 528 in /home/jelle/media/core/connectivity/source/drivers/mozab/MPreparedStatement.cxx
Between lines 212 and 527 in /home/jelle/media/core/connectivity/source/drivers/mork/MPreparedStatement.cxx
Found 278 duplicate lines in the following files:
Between lines 301 and 938 in /home/jelle/media/core/connectivity/source/drivers/mozab/MDatabaseMetaData.cxx
Between lines 296 and 933 in /home/jelle/media/core/connectivity/source/drivers/mork/MDatabaseMetaData.cxx
Found 305 duplicate lines in the following files:
Between lines 584 and 1092 in /home/jelle/media/core/connectivity/source/drivers/mork/MResultSet.cxx
Between lines 569 and 1077 in /home/jelle/media/core/connectivity/source/drivers/mozab/MResultSet.cxx
Comment 8 Jelle van der Waa 2013-06-19 19:59:42 UTC
Also the class in these files is identical. Not sure how to fix this either.
sd/source/ui/toolpanel/controls/MasterPageContainer.cxx 
sd/source/ui/sidebar/MasterPageContainer.cxx
Comment 9 Michael Meeks 2013-06-20 09:33:57 UTC
Hi Jelle, the sidebar and toolpanel duplication is expected, I guess eventually we will remove one or the other - but that will be under development for 4.2 so ... I'd skip that for now :-)

Similarly the mozab code is scheduled for destruction as/when we have some final pieces finished on windows we can drop it there - which we're looking forward to but - it still requires some work. So - good catches ! but I'd avoid those two areas :-)
Comment 10 Commit Notification 2013-06-20 13:01:33 UTC
Jelle van der Waa committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=07cd857a0e35c9f80f35e8c042ff0ae16aceaef7

fdo#39593 remove unused function 'checkBitmapImport' and clean up code



The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 11 Björn Michaelsen 2013-10-04 18:47:44 UTC
adding LibreOffice developer list as CC to unresolved EasyHacks for better visibility.

see e.g. http://nabble.documentfoundation.org/minutes-of-ESC-call-td4076214.html for details
Comment 12 Thomas Arnhold 2013-10-23 17:57:22 UTC
Those are nearly identical, too:

oox/source/export/ColorPropertySet.cxx
xmloff/source/chart/ColorPropertySet.cxx
oox/source/export/ColorPropertySet.hxx
xmloff/source/chart/ColorPropertySet.hxx
Comment 13 Michael Stahl (CIB) 2013-11-09 11:21:46 UTC
Comment on attachment 78635 [details]
Removes duplicate code from sc/source/ui/view/output2.cxx

patch found integrated as ca8b63523885c89c694799622c319260346b00f6
setting "obsolete" flag so it does not show up in bugzilla queries.
Comment 14 bfoman (inactive) 2014-03-07 21:02:59 UTC
Back to NEW as changing of the status is unclear and I don't see any request in comments for additional information.
@gandhishrey@gmail.com: you are the current Assignee - are you working on it? If not please reset to default Assignee.
Comment 15 Peter Senna Tschudin 2014-06-09 10:31:21 UTC
Created attachment 100719 [details]
Clones found with CCFinderX

I found code clones using CCFinderX(https://github.com/petersenna/ccfinderx-core). I was looking for big blocks of code, so I used the options:

$ ./ccfx/ccfx d cpp -k- -w f+g+ -b 500 -t 20 -d /var/tmp/libreoffice/core

What should I do next? Are you interested in clones that are smaller blocks of code? Are those clones the kind of issue this bug is about?
Comment 16 Thomas Arnhold 2014-06-09 10:47:35 UTC
Peter, this sounds promising. But all files inside workdir/ and instdir/ are just files which are used and generated at compile time. You can omit all of them.

The */qa/* dirs could also be omitted, those are testing code, which is generally a kind of copy and paste.

The results for sc/source/core/opencl/op_financial.cxx look good. Yeah, maybe you have to search for smaller chunks.

Is there any option to output the results as html report, like cppcheck does (sorry, I didn't look into CCFinderX)? If yes, we could easily upload the reports and fill an EasyHack, so new developers can solve those issues. You could fix them, too, if you like :)
Comment 17 Peter Senna Tschudin 2014-06-09 13:07:07 UTC
Created attachment 100734 [details]
Code clones found with CCFinderX

Code clones found with CCFinderX without compile time additional source code, and without */qa/* directories. This was generated for detecting smaller chunks. The command line:

$ ./ccfx/ccfx d cpp -k- -b 250 -t 20 -d /tmp/core/

@thomas-libo@arnhold.org I think it is not difficult to generate html from xml. I'm detecting the clones exactly to find something to fix. Which ones should I start with?
Comment 18 Thomas Arnhold 2014-06-14 11:44:16 UTC
Peter, here are some promising examples: clone ids 8831, 8589 and 8468 look like candidates to split them out inside a helper function. Looks like searching for clones which are inside different files is a good starting point.

Here are some examples how to fix copy and paste code:
288f0029c69ab0532165877637a146f774d5e740 and http://cgit.freedesktop.org/libreoffice/core/log/?qt=grep&q=copypasta
Comment 19 Alex Thurgood 2015-01-03 17:38:42 UTC Comment hidden (no-value)
Comment 20 Tor Lillqvist 2015-02-09 17:23:36 UTC
See bug #89267 for a specific, easy, already found case of copy-paste orgy.
Comment 21 Robinson Tryon (qubit) 2015-12-14 06:31:33 UTC Comment hidden (obsolete)
Comment 22 Robinson Tryon (qubit) 2016-02-18 14:51:52 UTC Comment hidden (obsolete)
Comment 23 Commit Notification 2017-02-01 16:31:56 UTC
qarkai committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=b12823aa81003e80372bd89db79bd6ba8e032a95

tdf#39593 move duplicate code to separate method

It will be available in 5.4.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 24 Mayank 2017-03-08 20:43:10 UTC
Is the bug settled ?
Comment 25 Michael Meeks 2017-03-09 09:07:48 UTC
I suspect there is a lot more to do =) It would be good to have more up-to-date output  of Simian / CCFinderX or whatever to locate duplicate code ... Thanks.
Comment 26 Buovjaga 2017-04-13 07:41:27 UTC
Noel pointed out this tool: https://pmd.github.io/pmd-5.5.5/usage/cpd-usage.html
Comment 27 Ekansh Jha 2017-08-03 12:45:16 UTC
I am new to LO. Just want to ask if this issue is still open.
Comment 28 Commit Notification 2017-11-22 04:54:19 UTC
tagezi committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=a0ebba3d8855fee0bcec04a10137ae3a4f9f0e77

tdf#39593 deduplicate code in ExplicitValueProvider

It will be available in 6.0.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 29 Commit Notification 2017-11-22 10:32:59 UTC
tagezi committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=2f34bec2a7b939e24c3c5e884fdfe2057b732248

tdf#39593 deduplicate code in ScOutputData

It will be available in 6.0.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 30 Commit Notification 2017-11-22 20:15:55 UTC
Eike Rathke committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=95da9d4c8f320d865939ef2cea268ad1aac20a8e

Follow our naming conventions, tdf#39593 follow-up

It will be available in 6.0.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 31 Commit Notification 2017-11-27 15:57:20 UTC
tagezi committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=ca2bc4b6d8fb10b2b9d7bf9646b1be22be88e2ef

tdf#39593 deduplicate code in ScOutputData

It will be available in 6.0.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 32 Commit Notification 2017-11-27 16:02:03 UTC
Eike Rathke committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=45d6b9aa1986c5e68e7bf03d995bc1f8cd1bc817

Address some nitpicks, tdf#39593 follow-up

It will be available in 6.0.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 33 Commit Notification 2017-12-04 19:42:16 UTC
tagezi committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=eafc42a2b46e70a0840a870b8c8cce85b32980c1

tdf#39593 deduplicate code in ScOutputData

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 34 Commit Notification 2017-12-22 21:23:07 UTC
tagezi committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=a5fe3117e1877fa0bfb55f1cf22cf9394b3f0795

tdf#39593 deduplicate code in ScCellShell

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 35 Commit Notification 2017-12-23 10:42:47 UTC
tagezi committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=e76bdaeb300e8420a8bc96e5e38b858821d22564

tdf#39593 deduplicate code in ScCellShell

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 36 Commit Notification 2018-01-20 08:48:27 UTC
tagezi committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=22dc2e5cd3e06e12f08dbb86df2df86860d301d6

tdf#39593 deduplicate code in ScCellShell

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 37 Commit Notification 2018-04-13 12:22:09 UTC
Hrishabh committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=f17a04df424dac553195c1922a4e4aa1f4ad5b9d

tdf#39593 fixed copy paste code in sw edlingu.cxx

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 38 Sanket Kumar Singh 2018-06-20 11:02:17 UTC
I wish to work on this bug.
Comment 39 Sanket Kumar Singh 2018-06-20 11:10:03 UTC
I will start to work on this bug.
Comment 40 Xisco Faulí 2018-07-21 02:37:36 UTC Comment hidden (obsolete)
Comment 41 Commit Notification 2018-08-28 07:02:13 UTC
Arkadiy Illarionov committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=eada4adee8a32be021ee3031d07e0ccda7217e0d

tdf#39593 deduplicate code using existing function

It will be available in 6.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 42 Commit Notification 2018-11-14 08:24:41 UTC
tagezi committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/+/ddcd36d9d6191bfba7dea179067767a9ee192608%5E%21

 tdf#39593 deduplicate code in TreeListBox

It will be available in 6.2.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 43 Commit Notification 2018-12-05 11:17:36 UTC
Dmitriy Shilin committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/+/6fa4c3e4778959d4591c053461329d396741ee64%5E%21

tdf#39593 vcl: extract SysColor list deletion

It will be available in 6.3.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 44 Commit Notification 2018-12-07 10:24:49 UTC
Dmitriy Shilin committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/+/af4fab260b4e5e6cd073d449081dd057b9715881%5E%21

tdf#39593 vcl: reduce copy-paste in ImplCalcOutSideRgn

It will be available in 6.3.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 45 rajprakash 2018-12-07 15:00:32 UTC
Hi,

I just started working on it

Regards,
rajprakash
Comment 46 Commit Notification 2018-12-09 19:06:09 UTC
Dmitriy Shilin committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/+/53a29871800c5b520767e18c1e3f10c4d5c4852e%5E%21

tdf#39593 vcl/win/gdi: extract pen resetting into method

It will be available in 6.3.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 47 Commit Notification 2018-12-10 04:47:22 UTC
Dmitriy Shilin committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/+/7311a88baa8c30eeb61d897f43ac3f5b481ed01f%5E%21

 tdf#39593 vcl/win/gdi: extract brush updating into method

It will be available in 6.3.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 48 Xisco Faulí 2018-12-10 15:54:34 UTC
I guess many people can work on this at the same time...
Comment 49 Commit Notification 2019-02-10 10:19:39 UTC
Arkadiy Illarionov committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/+/b95512e936b4eebb301c169e091bd9064c1eb2ac%5E%21

tdf#39593 use existing find() to reduce copypaste

It will be available in 6.3.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 50 rajprakash 2019-02-12 19:56:19 UTC
Hi,
Is this issue still available to take upon?

Regards,
raj
Comment 51 Commit Notification 2019-03-10 06:12:11 UTC
Arkadiy Illarionov committed a patch related to this issue.
It has been pushed to "master":

https://git.libreoffice.org/core/+/365ec7b14649b46c59e7dc3a1530b2b37bc908da%5E%21

tdf#39593 use existing clearConnectionPools() method

It will be available in 6.3.0.

The patch should be included in the daily builds available at
https://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
https://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.