You probably have the audio channels mapped wrong - maybe upside down?
(click “configure” in HEX and make sure the channels maps correctly to the string tuning labels)
I just hooked up a GP-10 to a Win computer. It’s basically the same setup. The USB mode of the GP-10 has to be set to “Re-Guitar/Amp II” in order not to hear the processed GP-10 output signal but just the DAW return.
Create a “6.0 Music” input bus with connections as shown below. Create a “Music 6.0” audio track with the previous created bus as input. Insert the MG3 VST3 plugin (VST2 doesn’t work), configure the Hex channel inputs (1-6 / high e to low E string) and you’re good to go.