Bug 147161 - [LODE] Consider cloning core/ repo with less history
Summary: [LODE] Consider cloning core/ repo with less history
Status: NEW
Alias: None
Product: LibreOffice
Classification: Unclassified
Component: LibreOffice (show other bugs)
Version:
(earliest affected)
unspecified
Hardware: All All
: medium normal
Assignee: Not Assigned
URL:
Whiteboard: reviewed:2022
Keywords: difficultyMedium, easyHack, skillScript
Depends on:
Blocks:
 
Reported: 2022-02-03 17:16 UTC by Olivier Hallot
Modified: 2023-01-10 21:21 UTC (History)
5 users (show)

See Also:
Crash report or crash signature:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Olivier Hallot 2022-02-03 17:16:46 UTC
Git clone on core/ brings full core/ history and is several GBytes large. Takes time to download.

New developers, especially developers that wants to try LibreOffice development/build are often discouraged in front of very long download times.

Adding "--depth=NN" or "--shallow-since=<date>" to LODE script can reduce the download times for the newcomer.

--- ESC minutes of 2022-02-02 ----
   + LODE improvements (ohallot)
       + 'git clone' of core/ brings full core/ history and is >Gbytes
       + same (and worse) for translations/
         + 12 hours for Olivier
       + can we amend with '--depth=NN' to shorten download times? (OH)
           + easier for dev newbies
           + NN=100 allows reducing the core.git size to <1GB
       + need advise from expert
       + certainly makes sense for submodules (Cloph)
         + less so for core.git if you want to switch branches
      + https://dev-www.libreoffice.org/bundles/ solves the "intrerrupted at 99%" problem (Miklos)
      + translations.git is the real problem: 8GB (Eike)
        + core.git 2.7GB
        + translation updates download took 2 hours the last time tried
      + think it's a mix of problems (Thorsten)
        + for non-EU, github is much faster to clone from
        + in EU, grabbing the tarballs is perhaps the quickest
          + maybe mirror that?
        + maybe lode.git should default to cloning translations.git in a shallow way?
    + would solve this by building in the cloud (Hossein)
      + e.g. github codespace, with visual studio code
      + build is done remotely, but vs code runs locally
      + Good for people with slow connection and slow computers
    + if you stick to the default options, you don't get translations.git (Stephan)
    + would like to make it easy to do a shallow clone of submodules (Olivier)
AI: file a bug / easy hack on this (Olivier)
      + maybe we could make it the default for submodules (Cloph)
        + can we still opt in to get the full history? (Stephan)
          + think so (Miklos)
AI: enable shallow clone for submodules by default (Cloph)
Comment 1 Miklos Vajna 2022-02-04 07:49:31 UTC
CC Cloph who may want to look into this. The safest subset seems to be doing this only for translations.git, where the cost/benefit ratio is the highest for the history.
Comment 2 Xisco Faulí 2022-02-04 14:21:12 UTC
Moving to NEW
Comment 3 Hossein 2022-02-07 00:00:13 UTC
Without sub-modules, the cloned size of cloned core tar.xz is: (for me)
core          -> 2.1G core.tar.xz
core, depth=1 -> 573M core1.tar.xz
Comment 4 Olivier Hallot 2022-02-07 11:35:53 UTC
On a fresh cygwin build, /translations/ went from 7.0 GB to 1.6 GB after "git gc --aggressive".

Note: /helcontents2/ and /dictionaries/ sub-modules are frequently patched part of the development.
Comment 5 Hossein 2022-08-04 14:05:36 UTC
Re-evaluating the EasyHack in 2022

This enhancement is still relevant. This is specially helpful for the newcomers who may want to work with the latest dev master with less download time.