Which version do you use?

That should already be solved. Look into the file soffice2xmlhelp.xsl in folder "filter" in the *.oxt container. If it has a line
<xsl:when test="not(starts-with($masterstyle,'hlp_'))"><emph><xsl:apply-templates/></emph></xsl:when>
instead of 
<xsl:when test="not(starts-with($masterstyle,'hlp_'))"><xsl:apply-templates/></xsl:when>

then it is an old version.

I'm running my modified fork of 3.1.0 and it has

<xsl:when test="not(starts-with($masterstyle,'hlp_'))"><xsl:apply-templates/></xsl:when>

Can you please attach this special helpauthoring.oxt here. I know you have mentioned it on the mailing-list, but I do not find it.

And please say on which file you have tested it.

Do you see the same problem with the version 3.1.1

Created attachment 118648 [details]
forked version 0.5

The file tested - /source/text/shared/01/01020000.xhp

It shows up in 3.1.1 release, so it will be there also in the 3.1.0 release you are using.

I do not get the same as you. I get

Original XML
<emph><switchinline select="sys"><caseinline select="MAC">%PRODUCTNAME - Preferences</caseinline><defaultinline>Tools - Options</defaultinline></switchinline> - <link href="text/shared/optionen/01010300.xhp" name="%PRODUCTNAME - Paths">%PRODUCTNAME - Paths</link></emph>

Resaved XML
<switchinline select="sys"><caseinline select="MAC"><emph>%PRODUCTNAME - Preferences</emph>
</caseinline><defaultinline><emph>Tools - Options</emph></defaultinline></switchinline><emph> - </emph><link href="text/shared/optionen/01010300.xhp" name="%PRODUCTNAME - Paths"><emph>%PRODUCTNAME - Paths</emph></link>

Notice, that in the original XML the emph-element is the outer one and has the children 'switchinline' and 'link'. But both are not allowed as child of an emph-element in xmlhelp.dtd. Therefore the extension correctly moves the emph-element to the content of 'switchinline' and 'link'. There the emph-element is an allowed child and the file becomes valid against the xmlhelp.dtd.

Does your output really has a <emph>-tag before the <switchinline> tag and an </emph> element after the </link> element?

Created attachment 118658 [details]
screencast

(In reply to Regina Henschel from comment #5)
> Does your output really has a <emph>-tag before the <switchinline> tag and
> an </emph> element after the </link> element?

Yes. Check the screencast using oxt 3.1.1 and LO 5.0 daily. :D

I have used Version: 5.1.0.0.alpha1+
Build ID: fdafa5302df26df4f147b0d2efb6064389a1330c
TinderBox: Win-x86@62-merge-TDF, Branch:MASTER, Time: 2015-08-22_23:42:26
Locale: de-DE (de_DE)

I use master with my 3.1.0 fork.

Version: 5.1.0.0.alpha1+
Build ID: eb2e1ab4651350bffc53f618961a910bd3bbcfd9
TinderBox: Linux-rpm_deb-x86_64@70-TDF, Branch:master, Time: 2015-09-02_23:57:52
Locale: en-US (en_US.UTF-8)

Now I have download Version: 5.1.0.0.alpha1+
Build ID: 88c51cd55d1a9b29e62269c53b3923770253ab07
TinderBox: Win-x86@62-merge-TDF, Branch:MASTER, Time: 2015-09-11_13:09:15
Locale: de-DE (de_DE)

And now I see the error too.

Because I have used version 3.1.1 in both cases, I fear, that a change in the core interferes with the extension.

Created attachment 118755 [details]
Shortened version of /source/text/shared/01/01020000.xhp

Another way to reproduce:
Install the extension helpauthoring.oxt. You get the two filters xmlhelp2soffice.xsl and soffice2xmlhelp.xsl. They are registers, so that you can use them from Tools > XML Filter settings.

Select item 'XHP_Help' and then click on button 'Test XSLTs..'

I have done this with LO4.2 and LO5.1
The results from the import filter xmlhelp2soffice.xsl is identical.

But the export filter soffice2xmlhelp.xsl works in LO4.2 but not in LO5.1. Because here are really only the filters involved and not any Basic code of the extension, I think, the problem is not in the extension but in core.

The xml result of the export filter in LO5.2 is not even well-formed, although it uses the same input as in LO4.2.

Helpauthoring extension is deprecated. closing wontfix.