Everything depends on the program that you’re loading a plug-in into. A VST plug-in is generally created for, and tested on, Windows operating systems. An AU plug-in is generally created for, and tested on, Apple or Macintosh operating systems. ProTools has its own plug-in, AAX.
If both VST3 and VST2 variants of a plug-in are available, try the VST3 version first. VST3 is based on the newest VST specification so it will remain current and be supported longer than VST2. Steinberg released the VST3 specification in 2013 and quit supporting VST2 development in 2018. If you experience problems with a VST3 plug-in and a VST2 variant is available, you can always use the VST2 instead.
As mentioned earlier, AU plug-ins are designed and tested to work with Apple Macintosh computers. I know Logic can use them and I believe Garageband also can use them. Any other program that runs on a Mac should accept an AU plug-in.
Many programs designed to work in both operating systems, so called cross platform programs, accept both AU or VST plug-ins. That is helpful when a plug-in is available in one, but not the other, format. However, unless that is the case I think there is less potential for trouble by using the plug-in format on the operating system the plug-in is designed and tested in.