Hello,

I would like to help out with this.
Please provide me with details, where I can install and configure it. 

Regards
David

Norbert, Robert: Could you bring David up to speed with where we are with Gerrit/Jenkins?

Hello,

still waiting for details about the status with this.

The questions to clarify are:

1. is jenkins already installed on master?
2. are slaves machines already identified?
3. how master connect to slave? SSL is the easiest way to go (that assumed that some user (jenkins?) is set up and public key authentication is in place)
4. the Admin-User Group who should be able to manage the slaves and the jobs and which authentication mechanism should jenkins use. LDAP? 
5. initial jobs/branches to build
6. how often should we build it and how long should we preserve the artifacts.

Regards
David

(In reply to comment #3)
> Hello,
> 
> still waiting for details about the status with this.
> 
> The questions to clarify are:
> 
> 1. is jenkins already installed on master?
> 2. are slaves machines already identified?
No,and they won't be 'identified as such'. think about SETI or Folding@Home

> 3. how master connect to slave? SSL is the easiest way to go (that assumed that
> some user (jenkins?) is set up and public key authentication is in place)
no, it is more a client-server model, where the 'salve' are the client, that contact the master when they are ready to do some work....
That is the specificity of the situation that is apparently not very Jenkins like
(at least I have not seen any plugin so far that seems to be working in that direction

> 4. the Admin-User Group who should be able to manage the slaves and the jobs
> and which authentication mechanism should jenkins use. LDAP? 
No, each slave is under the control of whichever volunteer make that machine available.
I suspect we could ask them to register... it migh be convinient to use ssh and interface with gerrit (which already has a registration and collect ssh-keys for users)
 
> 5. initial jobs/branches to build
> 6. how often should we build it and how long should we preserve the artifacts.
In a perfect world we would test-build every patch pushed in gerrit

Norbert

(In reply to comment #4)
> (In reply to comment #3)
> > Hello,
> > 
> > still waiting for details about the status with this.
> > 
> > The questions to clarify are:
> > 
> > 1. is jenkins already installed on master?
> > 2. are slaves machines already identified?
> No,and they won't be 'identified as such'. think about SETI or Folding@Home

Well, what would be the best way to start here? Obviously we should install it somewhere.
Would you install jenkins on some machine and provide me a credentials or should I install jenkins myself? I guess a normal user account with a couple of GB disk place should be sufficient to start with.
As jenkins per default store the whole configuration in ~/.jenkins a new user would be fine.

> > 3. how master connect to slave? SSL is the easiest way to go (that assumed that
> > some user (jenkins?) is set up and public key authentication is in place)
> no, it is more a client-server model, where the 'salve' are the client, that
> contact the master when they are ready to do some work....
> That is the specificity of the situation that is apparently not very Jenkins
> like
> (at least I have not seen any plugin so far that seems to be working in that
> direction
> 
> > 4. the Admin-User Group who should be able to manage the slaves and the jobs
> > and which authentication mechanism should jenkins use. LDAP? 
> No, each slave is under the control of whichever volunteer make that machine
> available.
> I suspect we could ask them to register... it migh be convinient to use ssh and
> interface with gerrit (which already has a registration and collect ssh-keys
> for users)
I don't understand this one. Why would you want to change the working system (jenkins)?
But may be I'm missing something here. I would prefer just to start and see how it goes.
The first step would be just install jenkins on master and we can establish the first job on the master.
Of course the limitation is the same OS.

> 
> > 5. initial jobs/branches to build
Well, I would just start with trunk.

> > 6. how often should we build it and how long should we preserve the artifacts.
> In a perfect world we would test-build every patch pushed in gerrit
Well I'm working also on gbuild conversion topic and we hope that after the whole LO is built with gbuild the significant time-gain can be achieved.  
> 
> Norbert

David

(In reply to comment #5)
> (In reply to comment #4)
> > (In reply to comment #3)
> > > Hello,
> > > 
> > > still waiting for details about the status with this.
> > > 
> > > The questions to clarify are:
> > > 
> > > 1. is jenkins already installed on master?
> > > 2. are slaves machines already identified?
> > No,and they won't be 'identified as such'. think about SETI or Folding@Home
> 
> Well, what would be the best way to start here? Obviously we should install it
> somewhere.
> Would you install jenkins on some machine and provide me a credentials or
> should I install jenkins myself? I guess a normal user account with a couple of
> GB disk place should be sufficient to start with.
> As jenkins per default store the whole configuration in ~/.jenkins a new user
> would be fine.

That is more for Robert. he is the admin guy... I'm just a guest on his machine :-)

but it maybe more convenient to do dev and testing on a local box of yours...
/me does not fancy doing dev on production box...

> > I suspect we could ask them to register... it migh be convinient to use ssh and
> > interface with gerrit (which already has a registration and collect ssh-keys
> > for users)
> I don't understand this one. Why would you want to change the working system
> (jenkins)?


The problem is:
WE are not in the scenario where we have a dedicated set of slave box under a centrlized control:
We can't afford that. what we want is a system where volonteer can 'donate' some CPU/Disk when they want.
So the 'master' ie jenkins cannot give order. it can only maintain a list of task to do and distribute them to vonlonteer machine that shows up.
That means:
1/ it is the 'slaves' that contact the master, not the other way around
2/ client will present a set of 'feature'. i.e which OS, what level, which options (autogen.sh) they support etc... this is tricky and will require an iterative approach, I think, to get it right eventually.
3/ the master need to keep track of who is doing what... and detect run-by to resubmit a task that was not completed in time to another slave
4/ practically we will have some dedicated box, which will reliably do the assigned task, the master need to be aware of them and use them to prioritize.

Gerrit already has an openid-based registration and collect ssh-keys, so it seems easier to re-use that rather than have yet another ssh-key collection point. but that is just a suggestion, with the caveat that I don't know jenkins

> > 
> > > 5. initial jobs/branches to build
> Well, I would just start with trunk.

We are not using cvs/svn... we don't have 'trunk'. But sure, building the 'master' branch for testing purpose is fine...
but eventually we want to interface with gerrit (there is a jenkins plugin for that IIRC) to get the event that trigger build... build of things that are not yet in git proper (that is the whole point of the exercise: build patch before they get committed to avoid master breakage as much as possible

Norbert

Hi,

LibreOffice jenkins master/slave build is up and running, you can see it here:

http://idaia.de:8080/jenkins


* master (ubuntu 10) on my root server (german based ISP).
  I'm using matrix based security, every user is assigned the privileges. 
  User cannot register, but have to be registered by admins.

* slave on my office machine (ubuntu 11) behind a firewall.
   as you mentioned slave donation model works fine with jenkins,
   no additional plugin are required for that. Out of the box, jenkins slave can connect via jnlp (http)        
   protocol, which my setup are using. If I disconnect master shows, the offline status of my slave.

* I'using only one jenkins Git Plugin.

TODOs:

Currently the follow commands are running unconditionally every time the build its started:
1. git fetch master
2. autogen.sh -- with hard coded set of configuration options
3. make clean
4. make

As the matter of fact we need some logic here: if build failed, we need a way to fix it and restart it from the point where it stopped. 

Integration with gerrit:

Our main purpose is to establish the automatic code review toolchain with the follow work flow:

Gerrit => Jenkins Gerrit Plugin pull on gerrit code review submit request and starts build => sucessfull build change review request as verified => Reviewer reviews the request and abandon it review and push to specific git branch.

I have an idea, how to do it.

http://www.infoq.com/articles/Gerrit-jenkins-hudson
https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+Trigger

Ciao
David

Hi,

I'm stuck with outdated helpcontent2 directory problem during the jenkins build.

The end point in the story is that jenkins build currently fails with the follow error message (sorry for this ugly german, I will switch the locale on my machine):

=============
(121/125) Building module helpcontent2
=============
Entering /home/david/wizball/workspace/LO-Ubuntu-Head/helpcontent2/source/auxiliary

gbuild module /home/david/wizball/workspace/LO-Ubuntu-Head/helpcontent2/source/auxiliary: make -f ../Makefile -j6  all slowcheck
make[1]: Betrete Verzeichnis '/home/david/wizball/workspace/LO-Ubuntu-Head/clone/help/helpcontent2/source/auxiliary'
make[1]: ../Makefile: Datei oder Verzeichnis nicht gefunden
make[1]: *** Keine Regel, um »../Makefile« zu erstellen.  Schluss.

I guess i should describe exactly what I'm doing (and in which order) to understand the problem ;-)

1. Git Jenkins Plugin is used to fetch the newest changes 
Repository URL: git://anongit.freedesktop.org/libreoffice/core
Refspec: +refs/heads/master:refs/remotes/origin/master
Branches to build: origin/master
2. ./autogen.sh
3. make clean
4. make

For the first time I did it, everything was up to date (the build failed with WaE anyway, that why I provided a couple of build fix patches, by the way ;-).

After 11 iterations though, the content of helpcontent2 directory is outdated ;-(

My current problem is, that the additional repository (help) seems to be outdated.
Build is failed, because last changes in helpcontent2/prj/ 
was not fetched with Jenkins Git Plugin - only core repo changes was fetched

(lately introduced empty dmake file is still missing in my local copy, thus the build.pl thinks, that help content module should be built with gbuild and looks for missing Makefile).

I guess that using ./g command instead of Jenkins Git Plugin would solve this issue?
The problem with it, that we lose jenkins SCM features like gerrit integrations and friends.

Another option would be to always clean the directories from foreign repos (like helpcontent2)?
Is there any command for it in some scripts?

Another option would be to unconditionally wipe out the whole workspace before build, fetching the dependencies during autogen process.
The problem with it: we can not rebuild/restart failed/broken build from the place where it stopped.

Last option would be to switch to using git submodule repository model and not to fetch the foreign repositories during the build (Git Plugin take care of submodule natively)?

Of course, one can specify more then one Git Repository in the same project, but they would be checked out in parallel directories - not an option.

Any ideas on this?
Another question: to investigate the jenkins/gerrit integration further, can I use http://gerrit.libreoffice.org test installation or should I install everything from scratch on my own infrastructure?
I guess to use it I would need credential of buildbot user on gerrit and admin access to gerrit too.

Ciao
David

(In reply to comment #8)
> Hi,
> 
> I'm stuck with outdated helpcontent2 directory problem during the jenkins
> build.
> 
> The end point in the story is that jenkins build currently fails with the
> follow error message (sorry for this ugly german, I will switch the locale on
> my machine):
> 
> =============
> (121/125) Building module helpcontent2
> =============
> Entering
> /home/david/wizball/workspace/LO-Ubuntu-Head/helpcontent2/source/auxiliary
> 
> gbuild module
> /home/david/wizball/workspace/LO-Ubuntu-Head/helpcontent2/source/auxiliary:
> make -f ../Makefile -j6  all slowcheck
> make[1]: Betrete Verzeichnis
> '/home/david/wizball/workspace/LO-Ubuntu-Head/clone/help/helpcontent2/source/auxiliary'
> make[1]: ../Makefile: Datei oder Verzeichnis nicht gefunden
> make[1]: *** Keine Regel, um »../Makefile« zu erstellen.  Schluss.
> 
> I guess i should describe exactly what I'm doing (and in which order) to
> understand the problem ;-)
> 
> 1. Git Jenkins Plugin is used to fetch the newest changes 
> Repository URL: git://anongit.freedesktop.org/libreoffice/core
> Refspec: +refs/heads/master:refs/remotes/origin/master
> Branches to build: origin/master
> 2. ./autogen.sh
> 3. make clean
> 4. make

Neat...

Ok, so I've been working ton tinbuild2 to allow for building of gerrit /for/*/* kind of things

I have not pushed these change yet... soon

The idea is:

the client 'build' should be a tindbuild2 process that do one-shot run to build a specific ref... and the config is such that the build only depend on core that way there is no issue of keeping the other repo in sync.
This is not perfect, but this will have to do until we migrate to git-submodules fo the 4 auxiliary repos...

iow you want to build with (on top of other platform specific things you need):

--with-help=no
--disable-dependency-tracking
--with-myspell-dicts=no
--with-help=no
--disable-binfilter

(In reply to comment #9)
> (In reply to comment #8)
> > Hi,
> > 
> > I'm stuck with outdated helpcontent2 directory problem during the jenkins
> > build.
> > 
> > The end point in the story is that jenkins build currently fails with the
> > follow error message (sorry for this ugly german, I will switch the locale on
> > my machine):
> > 
> > =============
> > (121/125) Building module helpcontent2
> > =============
> > Entering
> > /home/david/wizball/workspace/LO-Ubuntu-Head/helpcontent2/source/auxiliary
> > 
> > gbuild module
> > /home/david/wizball/workspace/LO-Ubuntu-Head/helpcontent2/source/auxiliary:
> > make -f ../Makefile -j6  all slowcheck
> > make[1]: Betrete Verzeichnis
> > '/home/david/wizball/workspace/LO-Ubuntu-Head/clone/help/helpcontent2/source/auxiliary'
> > make[1]: ../Makefile: Datei oder Verzeichnis nicht gefunden
> > make[1]: *** Keine Regel, um »../Makefile« zu erstellen.  Schluss.
> > 
> > I guess i should describe exactly what I'm doing (and in which order) to
> > understand the problem ;-)
> > 
> > 1. Git Jenkins Plugin is used to fetch the newest changes 
> > Repository URL: git://anongit.freedesktop.org/libreoffice/core
> > Refspec: +refs/heads/master:refs/remotes/origin/master
> > Branches to build: origin/master
> > 2. ./autogen.sh
> > 3. make clean
> > 4. make
> 
> Neat...
> 
> Ok, so I've been working ton tinbuild2 to allow for building of gerrit /for/*/*
> kind of things
> 
> I have not pushed these change yet... soon
> 
> The idea is:
> 
> the client 'build' should be a tindbuild2 process that do one-shot run to build
> a specific ref... and the config is such that the build only depend on core
> that way there is no issue of keeping the other repo in sync.
> This is not perfect, but this will have to do until we migrate to
> git-submodules fo the 4 auxiliary repos...
> 
> iow you want to build with (on top of other platform specific things you need):
> 
> --with-help=no
> --disable-dependency-tracking
> --with-myspell-dicts=no
> --with-help=no
> --disable-binfilter

Thank you for this suggestion.
I used it and now the jenkins build is fixed (but I did once time again ./g pull anyway.
But build exclusion is an ugly hack.
Another option would be to check the core and 4 aux. repos in one directory and then perform some linking magic from core to those directories.
On this way we would rely on pure git command (no ./g magic) and other SCM Git means (Jenkins Git Plugin).

Another question: I have very big interest to utilize gerrit.libreoffice.org
Why?
I'm involved in gbuild conversion with David and after some trivial modules, I'm going to convert non trivial ones: test tools and pyuno.

But even the simple module conversion  (like l10ntools) involve ca. 20 files and spread around the tree.
instead of bother you guys an IRC or send unfinished patches I would like to do it the right way: request an review in gerrit.

Yes, since yesterday I have write access to git repo and could (and would) create my own branch a lá gbuild_xyz,
but then the reviews have to switch and checkout my branch, and then still e cannot see it side by side,
and he cannot say on the line level: hey, dude, what are doing here is completely insane ;-)
 
So the question is, how can I push today my review request and ask David and may be you to take a look at it?

is simple git config remote.origin.pushurl to some specific value would be enough?
I'm on head, core repository (if that matters).
Yes, I created account on gerrit.libreoffice.org and I will import my public ssl key today.

Thank you in advance for your help!
David Ostrovsky

Hi,

I would like to ask a question about status of gerrit synchronization.
I would like to use gerrit already now and cannot because gerri is outdated.
Bjorn synced it once (I asked him to do this on IRC) and then I could push it once
with this command:

git push origin HEAD:refs/for/master

Bjorn is traveling now for the next 5 days (or so).
So who can help me with gerrit synchronization?

Cron job? 
Somebody from gerrit admins?
Or could somebody grant me the missing privileges?

What exactly do i need?
I would like to create gerrit review requests for master and feature banches, like this:
http://cgit.freedesktop.org/libreoffice/core/log/?h=feature/gbuild_testtools

That worked already once (and i was really happy: I created 8 Patch Sets in the same gerrit patch requests, thanks to Change Id git trigger):
https://gerrit.libreoffice.org/#/c/105/

;-)

Thank you in advance,
David

Hi,

i thought once again about aour discussion yesterday on IRC.
While i understand your position: not to change anything on the tb client, because there are 20 (or so) of them, we would change TB client in step 2 anyway, at least then when we would introduce task server (or how i call it BQM: build queue manager).

That why i recommend that we change it already now for the first task.

In addition i have another arguments:

1. Parsing the log outcome on the server is proved to be error prone process, as cloph_away confirmed yesterday. It is completely clear why: once you mess around with regex to identify something that look like an error, espesially in windows build, you will lose.

2. instead sending mails and letting the server side to be needlessly complex, KISS and just call one HHTP request, like i tried to convince you yesterday.
Why complex, because you need still to transform through mail the follow information:

job: id
log: content
result: 0 or 1
duration: how long

Jenkins already has such a built in facility, named "external job execution". I pointed you yetserday to java code for that. Today i want to point you to the documentation:

https://wiki.jenkins-ci.org/display/JENKINS/Monitoring+external+jobs#Monitoringexternaljobs-Submitarunprogramatically

as you can see there: all we need is to do something like that:

curl -X POST -d '<run><log encoding="hexBinary">4142430A</log><result>0</result><duration>2000</duration></run>' \
http://user:pass@myhost/jenkins/job/_jobName_/postBuildResult

where result is 0 or 1. No regex any more on server side and thus no guessing any more what the outcome of TB build is. TB knows the outcome exactly: it called the make and knows the process returns.

where _jobName_ is some unique job, for example on my jenkins machine (http://idaia.de/jenkins), you can find the jobs name here: http://idaia.de/jenkins/api/xml. But we still have to define the map from TB id to Jenkins Jobname.

My suggestion is to do the implementation in 2 steps:

1. to extend your tinderboxes with log mirroring feature (Mac OS X and/or Windows TB or both?) to do so i will write a small java client in first step.
2. after it is working i will migrate this java client part an whatever languag you want ... if you insist on z/OS Assembler, i can ask my IBM friends too ;-)

Question: what our master jenkins be?
Something like: ci.libreoffice.org? or builds.libreoffice.org?

Thorsten said me, that you donated machine that is currently idle in Nurenberg. Can we use it? it ios surely oversized, because master don't do anything,
but of course we could define some native jenkins jobs on the jenkins master machine too.

Or we can set up some VM, like gerrit one, but this would involve admin guys right?
The simplest way for now would to set up on my site.

Regards
David

(In reply to comment #12)
> Hi,
> 
> i thought once again about aour discussion yesterday on IRC.
> While i understand your position: not to change anything on the tb client,
> because there are 20 (or so) of them, we would change TB client in step 2
> anyway, at least then when we would introduce task server (or how i call it
> BQM: build queue manager).

No, you are confusing the existing tinderboxes which are controlling the git branches, and the 'gerrit extensions' of the concept that will control individual patch-set prior to them being merged.

The later does not mean we are going to abandon the formers.
There will still be builbot that build master and pre-release branches.
so maybe there will be some current tinderbox that will be re-purpose, the current setup of tinderbox + tinderbox server will not disappear, nor does it need to change.

So the development of a 'tinderbox for gerrit' system does not impact the current tinderbox setup.

(In reply to comment #12)
> 
> 1. Parsing the log outcome on the server is proved to be error prone process,
> as cloph_away confirmed yesterday. It is completely clear why: once you mess
> around with regex to identify something that look like an error, espesially in
> windows build, you will lose.
> 
[...]
> 
> Jenkins already has such a built in facility, named "external job execution". I
> pointed you yetserday to java code for that. Today i want to point you to the
> documentation:
> 
> https://wiki.jenkins-ci.org/display/JENKINS/Monitoring+external+jobs#Monitoringexternaljobs-Submitarunprogramatically
> 
> as you can see there: all we need is to do something like that:
> 
> curl -X POST -d '<run><log
> encoding="hexBinary">4142430A</log><result>0</result><duration>2000</duration></run>'
> \
> http://user:pass@myhost/jenkins/job/_jobName_/postBuildResult
> 
> where result is 0 or 1. No regex any more on server side and thus no guessing
> any more what the outcome of TB build is. TB knows the outcome exactly: it
> called the make and knows the process returns.

You are mis-intepretting the function of the regex.
They are _not_ used to determine if the build was sucessfull or not, that is done by the client and is a heder field in the email

they are used to present a 'summary log', to try to narrow down the log down to the pertinent part so that one does not have to download a multi-meg page with the browser to look at the result.

This is a vital usability feature that need to remain.

I would like to delegate the log file visualization for our gerrit-buildbot-plugin to jenkins (and not to mess with it in gerrit).

There are many ways. There is even shell script out there:
https://github.com/joemiller/hudson_wrapper/blob/master/hudson_wrapper.sh

@Norbert you don't even need JVM for that ;-) so we could consider to use it later in addition to master-tinderbox html-based log publication (master- & release-builds). There are many advantages why: nice UI, RSS, IRC ... we have access to over 2000 jenkins plugins.

Question: Robert uninstalled jenkins on VM2. If we would go with it, would we consider to have jenkins on the same VM where gerrit itself installed (VM2)? I would strong sugest to go with own VM-Continous-Integration, to isolate gerrit from jenkins, and to have in the long term the follow options:

* log file publication from gerrit patch verification
* log file publication direct from tinderboxes
* build some small platform undependent projects on jenkins instance itself (like java based buildbot-gerrit-plugin)

The domain name could be:

builds.libreoffice.org or ci.libreoffice.org, or, ...

If you do agree, could you please set up a new (test) VM?
Or should i start with jenkins installations on VM4?
@Robert could you please install jenkins on VM4 then?

Thanks
David

DavidO -- Hi! This bug sounds rather interesting.

Could you please give me a quick update on where we are with continuous integration?

Thanks!

(In reply to comment #16)
> DavidO -- Hi! This bug sounds rather interesting.
> 
> Could you please give me a quick update on where we are with continuous
> integration?

It is up and running and as always, with complex environments we still have a lot to do ;-)

In a perfect world we would lock push to master and route all changes through Gerrit, with patch set verification on all mandatory and optional platforms. Prerequisite for it is the availability of resources to be able to handle say 200-500 commits per day. In Gerrit change series can be used (dependent changes), so that the whole series is checked once.

For more information of current tinderbox => buildbot-gerrit-plugin => jenkins bridge see: 
[1], [2] and [3].


Ongoing work Gerrit-Buildbot plugin:

Gerrit master has UiAction feature: Plugin can contribute buttons and popup-windows. In that way we would like to put "Quick build" and "Expert Schedule" buttons on patch set action panel (change screen) in Gerrit.

"Quick build" would schedule a default build on 3 mandatory platforms.

"Expert Schedule" button would open a popup dialog and ask for user input: which platform should be build, which configure options, etc, p.p.


Ongoing work Tinderbox: add support for custom builds (configure options).

Ongoing work Jenkins:

Add parsing of logs, so that the error are shown and not the whole log [4].


[1] https://gerrit-test2.libreoffice.org/plugins/buildbot/Documentation/index.html

[2] https://github.com/davido/gerrit-buildbot-plugin

[3] https://review.idaia.de/#/q/status:open,n,z

[4] https://ci-test.libreoffice.org/job/buildbot/867/

adding LibreOffice developer list as CC to unresolved EasyHacks for better visibility.

see e.g. http://nabble.documentfoundation.org/minutes-of-ESC-call-td4076214.html for details

Since we have quite a complex setup and environment by now, not a trivial task. => Removing EasyHack

@Norbert: I assume this one can be closed, unless you still have use for this bug for tracking thing. Assigning to you for the honor of closing, since you (and David) did most of the work.

Migrating Whiteboard tags to Keywords: (DifficultyBeginner SkillScript TopicQA)
[NinjaEdit]