Bug 42949 - accelerating compile times by removing unnecessary header #include
Summary: accelerating compile times by removing unnecessary header #include
Status: NEW
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: LibreOffice (show other bugs)
Version:
(earliest affected)
unspecified
Hardware: Other All
: medium normal
Assignee: Not Assigned
URL:
Whiteboard: target:5.2.0 target:5.3.0 target:5.4...
Keywords: difficultyInteresting, easyHack, skillCpp, topicCleanup
Depends on: 111081
Blocks:
  Show dependency treegraph
 
Reported: 2011-11-15 06:43 UTC by Michael Meeks
Modified: 2018-04-20 07:02 UTC (History)
13 users (show)

See Also:
Crash report or crash signature:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Meeks 2011-11-15 06:43:23 UTC
One of our problems is that compile times are too slow; part of the reason for this is header explosion - such that we parse and compile the same things again and again for each module. If we can identify headers that are included un-necessarily many times, then we should be able to accelerate the build for everyone.

After a build - we have full library dependencies in .d files, such that:

cat workdir/unxlngi6.pro/Dep/LinkTarget/Library/*.d | grep '^ ' | sed 's/\\.*//' | sort | uniq -c | sort -n > /tmp/headers.includes
for a in `cat /tmp/headers.includes`; do if test -f /opt/libreoffice/core/$a; then wc -c $a; else echo -n "$a        "; fi; done > /tmp/headers.csv

ONO. should give us a /tmp/headers.csv that has the size and include count of each header in the build [ presumably a faster, perl tool could be used to generate this ;-].

By multiplying these, we can get an estimate of the 'weight' that a header provides to the build; hence:

bytes to parse	include count size  filename
741439564	4454	166466	/opt/libreoffice/core/solver/unxlngi6.pro/inc/boost/preprocessor/seq/fold_left.hpp														
416181480	6754	61620	/opt/libreoffice/core/solver/unxlngi6.pro/inc/rtl/ustring.hxx															
Shows we parse 740Mb of the same file over 4000 includes - for the set of files I grokked. I up-loaded a spreadsheet here: http://users.freedesktop.org/~michael/headers.ods - but the include counts look suspiciously low here.

Anyhow - having found our problem headers; we then need to work out how we can reduce their use - preferably by finding out where such headers are included from, and how to reduce their use [ by adding class Foo; forward definitions eg. ].

Clearly that's not going to work for eg. the string classes which are used everywhere, but quite possibly we can reduce the boost/ related wasteage, as well as help to identify future areas for improvement (eg. aggregating OutputDevice from Window, rather than including it :-)
Comment 1 Björn Michaelsen 2011-11-18 03:31:20 UTC
Presumably there some headers also included without being used at all in the cxx (e.g. by splitting a cxx file or other quick refactor work). One could simply clear out one header (starting with the important/heavy ones as per description) and (at least in the new build system) do a "make -k". 

After that one should have a close look at the files that even compiled with the header being empty -- chances are, they do not need the include at all.
Comment 2 Michael Meeks 2011-11-18 07:40:00 UTC
IIRC, there is an 'Include what you need' tool somewhere that would be complementary (or perhaps better) than the suggestion below I think.
Comment 3 August Sodora 2011-11-19 18:34:25 UTC
The one that looks the most promising to me is http://code.google.com/p/include-what-you-use/ but I am having difficulty getting clang to build with the latest gcc.
Comment 4 Michael Meeks 2011-11-21 03:35:02 UTC
Hi August; it'd be cool to run include-what-you-use over LibO; the clang bootstrapping problem is amusing :-) I wonder which compiler is wrong.
Thanks for looking at this !
Comment 5 August Sodora 2011-11-21 11:51:46 UTC
I was able to successfully build LLVM+Clang by making sure that I had compat headers installed for libstdc++ and issuing the configure on llvm like this:

../llvm/configure --host=x86_64-redhat-linux --target=x86_64-redhat-linux --build=x86_64-redhat-linux

Apparently if the suffix on the includes directory for the libstdc++ stuff is not default, it has to be specified three times (host, target, build).
Comment 6 August Sodora 2011-11-21 13:13:39 UTC
Steps to build clang:
* Ensure that the appropriate compat c++ libraries are installed for your system. Typically this means gcc 3.*. For fedora the package is compat-gcc-34.
* Change directory to where you want the llvm directory placed.
* svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm
* cd llvm/tools
* svn co http://llvm.org/svn/llvm-project/cfe/trunk clang
* cd ../.. (back to where you started)
* mkdir build (for building without polluting the source dir)
* cd build
* ../llvm/configure --host=x86_64-redhat-linux --target=x86_64-redhat-linux --build=x86_64-redhat-linux (change the host, target, and build parameters to suit your system)
* make
* Remember what the value of pwd is here as it will be used when running include-what-you-use

To build include-what-you-use:
* cd llvm/tools/clang/tools
* svn co http://include-what-you-use.googlecode.com/svn/trunk/ include-what-you-use
* Edit Makefile so that include-what-you-use is one of the subdirs to build.
* make

To run:
* make -srkj3 CXX=<previous value of pwd>/Debug+Asserts/bin/include-what-you-use 2> iwyu.out

And you should see some lines in the output like the following:

/home/aasodora/Sources/libo/basic/source/app/process.cxx should add these lines:
#include <debug/map.h>                  // for map, map<>::const_iterator
#include <debug/safe_iterator.h>        // for _Safe_iterator, operator!=
#include <stddef.h>                     // for NULL
#include <map>                          // for _Rb_tree_const_iterator
#include <utility>                      // for pair
#include "basic/sbxdef.hxx"             // for SbxERR_NO_ACTIVE_OBJECT
#include "rtl/ustring.h"                // for rtl_uString, etc
#include "rtl/ustring.hxx"              // for OUString
#include "sal/types.h"                  // for sal_False, sal_Bool, etc
#include "tools/solar.h"                // for String, xub_StrLen

/home/aasodora/Sources/libo/basic/source/app/process.cxx should remove these lines:
- #include <tools/errcode.hxx>  // lines 33-33
- #include "precompiled_basic.hxx"  // lines 30-30

The full include-list for /home/aasodora/Sources/libo/basic/source/app/process.cxx:
#include <debug/map.h>                  // for map, map<>::const_iterator
#include <debug/safe_iterator.h>        // for _Safe_iterator, operator!=
#include <osl/process.h>                // for oslProcessInfo, etc
#include <stddef.h>                     // for NULL
#include <basic/process.hxx>            // for Process, Environment
#include <basic/sbxcore.hxx>            // for SbxBase
#include <basic/ttstrhlp.hxx>           // for CUniString
#include <map>                          // for _Rb_tree_const_iterator
#include <osl/file.hxx>                 // for FileBase
#include <tools/string.hxx>             // for String::operator OUString, etc
#include <utility>                      // for pair
#include "basic/sbxdef.hxx"             // for SbxERR_NO_ACTIVE_OBJECT
#include "rtl/ustring.h"                // for rtl_uString, etc
#include "rtl/ustring.hxx"              // for OUString
#include "sal/types.h"                  // for sal_False, sal_Bool, etc
#include "tools/solar.h"                // for String, xub_StrLen

include-what-you-use has a way to deal with some of these false positives, so that shouldn't be terribly difficult.

Unfortunately there are a couple of extraneous warnings in the output that I would like to do without. Repeat offenders are:

warning: argument unused during compilation: '-ggdb2'
warning: argument unused during compilation: '-fno-enforce-eh-specs'

I also seem to be getting warnings from boost, the stl, and our own code , often like:

In file included from /home/aasodora/Sources/libo/solver/unxlngx6/inc/boost/ptr_container/detail/static_move_ptr.hpp:23:
/home/aasodora/Sources/libo/solver/unxlngx6/inc/boost/ptr_container/detail/default_deleter.hpp:64:41: warning: unused parameter 'tt' [-Wunused-parameter]
    default_deleter(default_deleter<TT> tt) { }

The last thing that really bugs me is what I think is an include-what-you-use-bug:

include-what-you-use: /home/aasodora/Sources/llvm/tools/clang/tools/include-what-you-use/../../include/clang/AST/Type.h:478: const clang::ExtQualsTypeCommonBase* clang::QualType::getCommonPtr() const: Assertion `!isNull() && "Cannot retrieve a NULL type pointer"' failed.
Stack dump:
0.      <eof> parser at end of file
/bin/sh: line 1: 23068 Aborted                 $R/build/Debug+Asserts/bin/include-what-you-use -DCPPU_ENV=gcc3 -DDBG_UTIL -DENABLE_GRAPHITE -DENABLE_GTK -DGCC -DGXX_INCLUDE_PATH=/usr/include/c++/4.5.1 -DHAVE_GCC_VISIBILITY_FEATURE -DHAVE_THREADSAFE_STATICS -DLINUX -DOSL_DEBUG_LEVEL=1 -DSOLAR_JAVA -DSUPD=350 -DUNIX -DUNX -DVCL -DX86_64 -D_DEBUG -D_GLIBCXX_DEBUG -D_PTHREADS -D_REENTRANT -Wall -Wendif-labels -Wextra -fmessage-length=0 -fno-common -pipe -fPIC -Wshadow -Wsign-promo -Woverloaded-virtual -Wno-non-virtual-dtor -fvisibility=hidden -fvisibility-inlines-hidden -fno-strict-aliasing -std=c++0x -Wno-deprecated-declarations -ggdb2 -DEXCEPTIONS_ON -fexceptions -fno-enforce-eh-specs -O0 -c $S/basic/source/sample/object.cxx -o $W/CxxObject/basic/source/sample/object.o -MMD -MT $W/CxxObject/basic/source/sample/object.o -MP -MF $W/Dep/CxxObject/basic/source/sample/object.d -I$S/basic/source/sample/ -I$O/inc/stl -I$O/inc/external -I$O/inc -I$S/solenv/inc/unxlngx6 -I$S/solenv/inc -I$S/res -I/usr/lib/jvm/java/include -I/usr/lib/jvm/java/include/linux -I/usr/lib/jvm/java/include/native_threads/include -I$S/basic/inc/pch -I$S/basic/source/inc -I$O/inc/udkapi -I$O/inc/offapi



Ideally I would like to clean up the output a little bit before testing the auto-fix script.
Comment 7 Michael Meeks 2011-11-22 01:41:19 UTC
Wow - this is really good work August ! :-)

I suggest that before we push any of this, we should get some sample patch files for review onto the mailing list to discuss.

eg. removing the pch includes is prolly a bad idea :-)

Also (but it will take quite a while to test), it might be nice to compile before + after and time it to see if there is a real win ...

But exciting stuff :-)
Comment 8 August Sodora 2011-11-28 19:14:25 UTC
I haven't really had much luck cleaning up some of that output, but I figure it must be possible. In particular, I'm not even sure if I'm just redirecting too many output streams to the log file or if there is a way I can pass compiler flags to limit some warning output.

One thing that I have realized that is pretty disconcerting: the only files that are checked by the tool are hxx files with correspondingly named cxx files. I know that our code base definitely does not meet this requirement in most cases :(

Maybe if we can overcome these hurdles then we can start to play with the auto-fix tool.
Comment 9 August Sodora 2011-11-30 10:18:43 UTC
A better command to run is something like:

env IWYU_VERBOSE=1 make -srkj3 CXX="/home/aasodora/Sources/build/Debug+Asserts/bin/include-what-you-use -Qunused-arguments" CXXFLAGS="-Wno-unused-parameter" 2> iwyu.out

This should get rid of all warnings of the form:

warning: argument unused during compilation: '-ggdb2'
warning: argument unused during compilation: '-fno-enforce-eh-specs'

It also gets rid of the unused-parameter warning from boost. It seems like we are still getting some errors from boost and the stl during the build that I'm trying to figure out.
Comment 10 Florian Reisinger 2012-05-18 09:12:53 UTC
Deteted "Easyhack" from summary
Comment 11 Michael Stahl 2013-08-15 19:56:48 UTC
hi Marcos, since sadly August is no longer working on this maybe you are interested in playing with  http://code.google.com/p/include-what-you-use/ ?
Comment 12 Björn Michaelsen 2013-10-04 18:47:28 UTC
adding LibreOffice developer list as CC to unresolved EasyHacks for better visibility.

see e.g. http://nabble.documentfoundation.org/minutes-of-ESC-call-td4076214.html for details
Comment 13 Robinson Tryon (qubit) 2013-10-19 00:24:45 UTC Comment hidden (obsolete)
Comment 14 Riccardo Magliocchetti 2014-11-04 15:40:30 UTC
I've updated to gerrit a quick hackish patch to build with include-what-you-use: 
https://gerrit.libreoffice.org/#/c/12255/

Since it's very verbose i've added also support for partials compilation so you can do:

./configure --with-iwyu=/usr/bin/include-what-you-use
make sw 2> sw-warn-includes.log

And get only warnings for the single module.
Comment 15 Michael Stahl 2014-11-19 12:30:00 UTC
some issues that you should be aware of when doing cleanup:

1. we want each header file to be "complete" in the sense that it would build successfully as a compilation unit on its own; it should not depend on some other header being included before it into a source file to provide the types it uses, it should either include a header that declares the needed type, or contain a "forward declaration" of it.

2. removing an include from a source file because there is another include in the source file that transitively brings in the needed include is also undesirable: some time later somebody will refactor the code and reomve some include that is no longer directly needed and then it turns out that there are a lot of errors that need fixing because the source file was relying on indirectly included files.

3. it's also too easy to remove platform dependent or configuration dependent include this way; some of the code is behind some #ifdef. Ok, this is a more general problem and you have to be aware of it whatever approach you use. best to build with --enable-dbgutil to get all the debug code built.

include-what-you-use does indeed appear to be the best way to solve this because it avoids issues 1 and 2.



how to build it on Fedora 20:

yum install llvm-static llvm-devel clang-devel
tar -xzvf /tmp/include-what-you-use-3.4.src.tar.gz
cd include-what-you-use/
[ edit the CMakeLists.txt and replace "link_directories(${LLVM_PATH}/lib)" with "link_directories(${LLVM_PATH})" ]
cmake .
cmake  -DLLVM_PATH=/usr/lib/llvm-3.4 .
make

for LibreOffice use the --with-iwyu flag (see comment #14)



iwyu unfortunately only reports problems in the source file that is being compiled, and in a header file with the same basename as the source file.
so i've added a "make iwyudummy" target that can be used to build all include files: it will generate a iwyudummy/StaticLibrary_iwyudummy.mk that can be edited to those include files you want to clean up.

i've already used iwyu to clean up all the headers that are shipped in the sdk (i.e. listed in odk/Package_odk_headers.mk).

likely it would be a good approach to clean up include/ first in a bottom-up way following the module dependency graph http://ostrovsky.org/libo/lo.png

oh and there is no need to clean up the precompiled_*.hxx headers manually, since they are created automatically by some update_pch script anyway.
Comment 16 Michael Stahl 2015-07-10 21:44:16 UTC
added a little script "bin/includebloat.awk" to sort the included
headers by "badness" i.e. total bytes parsed.

picking some of the low hanging fruit there (primarily
excessive boost header inclusion) reduced the preprocessor input
from 36151616031 to 28560693617, that's a 7.5 GB (20%) saving.
Comment 17 Commit Notification 2015-10-01 09:36:15 UTC
Noel Grandin committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=aed0a639f4af8e630dbd6bd4f2e0368b1481eae3

tdf#42949 remove unnecessary includes using iwyu

It will be available in 5.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.
Comment 18 Robinson Tryon (qubit) 2015-12-14 07:00:21 UTC Comment hidden (obsolete)
Comment 19 Robinson Tryon (qubit) 2016-02-18 14:51:46 UTC Comment hidden (obsolete)
Comment 20 Commit Notification 2016-04-25 08:54:36 UTC
Jorenz Paragas committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=474eca1f9b42749665bbf69f6dc62c66ea4ad7fd

tdf#42949: clean up includes in include/oox/core using iwyu

It will be available in 5.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 21 Commit Notification 2016-04-26 11:56:38 UTC
Jorenz Paragas committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=6a4a15c87c03feffb90cc416ce22d2819e41442d

tdf#42949: clean up includes in include/oox/crypto with iwyu

It will be available in 5.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 22 Commit Notification 2016-04-28 10:02:05 UTC
Jorenz Paragas committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=29c4f7bd5863e34c449062aca6f8aee5ec7510a2

tdf#42949: prefer including Reference.hxx over Reference.h...

It will be available in 5.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 23 Commit Notification 2016-05-03 20:19:57 UTC
Jorenz Paragas committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=af1174b2dea72359e8eb0e15297c61cf82571250

tdf#42949: clean up includes in include/oox/drawingml with iwyu

It will be available in 5.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 24 Commit Notification 2016-05-04 10:35:53 UTC
Jorenz Paragas committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=24e56929445f0e3e22c7129bf18e912ae1ac6fc1

tdf#42949: clean up includes in include/oox/dump with iwyu

It will be available in 5.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 25 Commit Notification 2016-05-06 07:02:24 UTC
Jorenz Paragas committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=1aeca4b64aac45e64eaab32a0726850ff30dea31

tdf#42949: clean up includes in include/oox/export with iwyu

It will be available in 5.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 26 Commit Notification 2016-05-10 19:23:13 UTC
Jorenz Paragas committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=4bf7614503292fe69d3e5835621c1cdcaa4536cb

tdf#42949: clean up includes in include/oox/helper with iwyu

It will be available in 5.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 27 Commit Notification 2016-05-11 07:09:38 UTC
Jorenz Paragas committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=7b9ea3348ac08d775ae452132239853724c65a41

tdf#42949: clean up includes in include/oox/mathml with iwyu

It will be available in 5.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 28 Commit Notification 2016-05-13 10:08:31 UTC
Jorenz Paragas committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=96d5516429bc1846fc6cf7119590f7b05a65eef8

tdf#42949: clean up includes in include/oox/ole with iwyu

It will be available in 5.2.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 29 Commit Notification 2016-05-26 20:42:52 UTC
Jorenz Paragas committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=5c05f1b201be8c1b978dc8d409f3c4dbf0b4beb6

tdf#42949: clean up includes in include/oox/ppt with iwyu

It will be available in 5.3.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 30 Commit Notification 2016-06-10 08:35:34 UTC
Jorenz Paragas committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=4c8d656b897229d5a6f2126fe274d338ec997f76

tdf#42949: clean up includes in include/oox/token with iwyu

It will be available in 5.3.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 31 Commit Notification 2016-06-10 08:36:56 UTC
Jorenz Paragas committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=5b156b37d487b96ec19d65e01cd8cedd26a2150d

tdf#42949: clean up includes in include/oox/vml with iwyu

It will be available in 5.3.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 32 Commit Notification 2016-06-13 11:39:33 UTC
Jorenz Paragas committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=1812a6467defa244c2002dec6b055ad3983e5883

tdf#42949: clean up includes in include/vbahelper with iwyu

It will be available in 5.3.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 33 jani 2016-06-14 09:59:19 UTC
Seems solved
Comment 34 Michael Stahl 2016-06-14 10:25:58 UTC
uhh, do you know now many headers we have? with any luck, we are 5% done here :)
Comment 35 Commit Notification 2016-06-15 09:32:47 UTC
Jorenz Paragas committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=cef14b816aee25ce8a59a2200900989523372996

tdf#42949: clean up includes in include/filter with iwyu

It will be available in 5.3.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 36 jani 2016-06-16 06:12:19 UTC
(In reply to Michael Stahl from comment #34)
> uhh, do you know now many headers we have? with any luck, we are 5% done
> here :)

Got you point, was just trying to clean up in my part of the world :-)
Comment 37 Commit Notification 2016-06-16 12:24:27 UTC
Jorenz Paragas committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=26fa322583147f87511b2d77e8701228838f8c73

tdf#42949: clean up includes in include/dbaccess with iwyu

It will be available in 5.3.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 38 Commit Notification 2016-06-21 08:00:22 UTC
Jorenz Paragas committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=8a6bffd2fefcc81dc34951ba2e178d9938b59fb3

tdf#42949: clean up includes in include/formula with iwyu

It will be available in 5.3.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 39 Commit Notification 2016-11-21 17:07:20 UTC
Bartosz Kosiorek committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=069445f9cdf4d3f48cbfcf7cb4bf02d16f74956e

tdf#42949 Remove not needed uno headers from filters/xml

It will be available in 5.3.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 40 Commit Notification 2016-11-21 23:07:07 UTC
Bartosz Kosiorek committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=5a4e106be6778f7e96fa36d2076ef4e7a90f81d9

tdf#42949 Remove not needed uno headers from sc/source/core

It will be available in 5.3.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 41 Commit Notification 2016-11-23 07:31:13 UTC
Bartosz Kosiorek committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=c840c49b62b919ec4d0d947fe8cd530a785cf80d

tdf#42949 Remove unused uno headers from /sw/source/core

It will be available in 5.3.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 42 Commit Notification 2016-11-23 07:34:27 UTC
Bartosz Kosiorek committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=741b56fc09e8c9c4361baa7a876fe3c8b81710db

tdf#42949 Remove unused uno headers from /sd/source/core

It will be available in 5.3.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 43 Commit Notification 2016-11-23 07:38:46 UTC
Bartosz Kosiorek committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=9b2e69abfc6386578fe3313362f1bbfe74c36a14

tdf#42949 Remove unused uno headers from /sw/source/uibase

It will be available in 5.3.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 44 Commit Notification 2016-11-23 07:41:08 UTC
Bartosz Kosiorek committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=45c7ba7ad6befdc4c212c376ef9399976f6695e1

tdf#42949 Remove unnecessary headers from sw/source/filters

It will be available in 5.3.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 45 Commit Notification 2017-03-28 09:13:25 UTC
Jorenz Paragas committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=65aa28cc1c8a90865c3a9ea379722730c40ecb53

tdf#42949: clean up includes in include/svx/[a-c]* with iwyu

It will be available in 5.4.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 46 Commit Notification 2017-12-22 09:51:09 UTC
Miklos Vajna committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=f3e840541a117113be85d7733f20c5d5ab6552f1

tdf#42949 Fix some Include What You Use warnings

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 47 Commit Notification 2018-01-04 14:52:30 UTC
Miklos Vajna committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=67e1e2ee40dba196f706afb43d0379b29c3c0f42

tdf#42949 Fix some more Include What You Use warnings

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 48 Commit Notification 2018-02-01 14:20:00 UTC
Gabor Kelemen committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=4fae455fcdb0242b568b88f51726bd1596e100b0

tdf#42949 Remove unneeded helpids.h headers (1/3)

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 49 Commit Notification 2018-02-01 14:21:27 UTC
Gabor Kelemen committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=73bfe83a3985df447d4fd6b443277aac21cd9913

tdf#42949 Remove unneeded helpids.h headers (2/3)

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 50 Commit Notification 2018-02-01 14:21:38 UTC
Gabor Kelemen committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=f38c49cdc9df7acedcf08b895f4c99be85d6aa80

tdf#42949 Remove unneeded helpids.h headers (3/3)

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 51 Commit Notification 2018-04-03 07:42:00 UTC
Gabor Kelemen committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=14198f2191f216592f00b72221771704b3ce4636

tdf#42949 Remove unnecessary localization headers from sd

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 52 Commit Notification 2018-04-04 09:24:52 UTC
Gabor Kelemen committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=28b1443eb3a48e4ca0651cf3cd560f5da08de472

tdf#42949 Remove unnecessary localization headers from accessibility

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 53 Commit Notification 2018-04-04 09:25:01 UTC
Gabor Kelemen committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=89d26a4018626f43af9d3e15d3b9c49abdfde889

tdf#42949 Remove unnecessary localization headers from avmedia

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 54 Commit Notification 2018-04-05 14:10:44 UTC
Gabor Kelemen committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=158254119b7b20f5e4f825dc3c1e80c1efb91c95

tdf#42949 Remove unnecessary localization headers from basctl

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 55 Commit Notification 2018-04-05 20:01:53 UTC
Gabor Kelemen committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=e0aabd9c88897d64d1ce0eb294fb8352ce7f25e9

tdf#42949 Remove unnecessary localization headers from chart2

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 56 Commit Notification 2018-04-07 07:27:23 UTC
Gabor Kelemen committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=7b185bef65472d10f9fb65d24e0757b90172796c

tdf#42949 Remove unnecessary localization headers from cui

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 57 Miklos Vajna 2018-04-07 21:21:17 UTC
(In reply to Michael Stahl from comment #15)
> include-what-you-use does indeed appear to be the best way to solve this
> because it avoids issues 1 and 2.

I recently tried to make usage of the tool in the context of LO a bit easier:

1) If your Linux system is too old to build IWYU, I've uploaded a static binary here: https://dev-www.libreoffice.org/bin/

2) Also now there is a bin/find-unneeded-includes script in core.git that you can just run even on header files in e.g. sc/inc/*.hxx, for other modules where we have no idea yet what flags to use, see sc/IwyuFilter_sc.yaml for inspiration.

I hope that makes even more easier to fix IWYU warnings in our codebase.
Comment 58 Commit Notification 2018-04-10 07:36:49 UTC
Gabor Kelemen committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=342a5d13dfcca37a68cb996650e1ada868c5ce3e

tdf#42949 Remove unnecessary localization headers from desktop

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 59 Commit Notification 2018-04-11 08:28:06 UTC
Gabor Kelemen committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=e1a969ca643d41e034100aac588994f2d6c3ed05

tdf#42949 Remove unnecessary localization headers from framework

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 60 Commit Notification 2018-04-12 08:56:33 UTC
Gabor Kelemen committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=84e2614a75a615d6c8584b13a69b3368d2a12a3d

tdf#42949 Remove unnecessary localization headers from fpicker

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 61 Commit Notification 2018-04-13 14:39:40 UTC
Gabor Kelemen committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=83d247b9917ecf4e23d5a457252a998f233b5c30

tdf#42949 Remove unnecessary localization headers from dbaccess

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 62 Commit Notification 2018-04-16 08:12:41 UTC
Gabor Kelemen committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=104b26b246c94c8c66864b20d00e419d96b15961

tdf#42949 Remove unnecessary localization headers from sc

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 63 Commit Notification 2018-04-17 07:08:34 UTC
Gabor Kelemen committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=b3f094c2573de12bf3d386f19b924b536cb89af0

tdf#42949 Remove unnecessary localization headers from sw

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 64 Commit Notification 2018-04-18 07:38:21 UTC
Gabor Kelemen committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=40eb7f2d7ca5e4a60c9a5a62ddd202d7807cdb9f

tdf#42949 Remove unnecessary localization headers of editeng

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 65 Commit Notification 2018-04-18 07:39:41 UTC
Gabor Kelemen committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=00ff4d198313265ba736a6e234b4278166d1c3e4

tdf#42949 Remove unnecessary localization headers of sfx2

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 66 Commit Notification 2018-04-18 19:32:59 UTC
Gabor Kelemen committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=40c3b5dee84b419efcbc1efc1020d1c07332d670

tdf#42949 Remove unnecessary helpids.h from sc.hrc

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 67 Commit Notification 2018-04-19 09:16:33 UTC
Gabor Kelemen committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=e24a059906c9b7f399f0afcbfdcd5dd1aa663973

tdf#42949 Remove unnecessary localization headers from reportdesign

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.
Comment 68 Commit Notification 2018-04-20 07:02:02 UTC
Gabor Kelemen committed a patch related to this issue.
It has been pushed to "master":

http://cgit.freedesktop.org/libreoffice/core/commit/?id=663fd3d6e1f93ec989dc289e688d5dbfe434cbca

tdf#42949 Remove unnecessary localization headers of svtools

It will be available in 6.1.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.