Currently a text box in a custom shape in Writer uses the frame style "Frame". You can e.g. change the writing direction in the style "Frame" to TopBottom-RightLeft and the text in the shape will change its writing mode.
It would be good to allow other frame styles. Currently you need to generate own frame styles for true frames in case you want to style them, because otherwise changes there will be automatically passed on to shapes with associated text box.
To reproduce: Insert a custom shape, add a text box and then write some text into it. Unselect shape.
Modify style "Frame". Goto tab "Option" and set text direction to "Right-to-left (vertical)".
Would be definitely a great enhancement. Though not much to add from UX here.