Change companion at Chapter 6

Hi guys.. I have a question for you.

I want to play a game with Yoshimo in the first part of the game and later change him with Jan.
I want to go to Spellhod with Yoshi (You know why..) but when I will save Imoen, I will had to wait until chapter 6 to leave here to her fate (I really don't like here..) for choose another companion for my group.

I want your opinion: it's ok? Or it's better take Jan from the begin?
I really like to play with Yoshimo in the first place and than change him with Jan, the only thing that stop me is change a companion at chapter 6 (and play the chapter 5 with one I don't like), near the end of SoA..

Thanks