CVar conversationMode

The action new Real ID Conversations take by default: "popout", "inline"

Default Setting

 * popout

Possible Settings

 * popout:New Real ID Conversations will be displayed in a new tab.
 * inline:Real ID Conversations will be displayed similarly to channels. They may be moved to a new window by right-clicking on the Conversation name in chat.

Changing this setting
In-game, type the following /console conversationMode setting

Alternatively, you can exit wow, open the Config-cache.wtf file, and add the following line: SET conversationMode setting