Steps taken:

1. Add non-steam game to steam: select the installer you downloaded from GOG as the file.

2. Change compatability setting to a defined version of proton. I used 8.0-5.

3. Run installer. This will install into your steam directory (for me this was /home/user/.steam/root/steamapps/compatdata/ - but run a search for "Freespace 2" if you are unsure). Note that the GOG version applies a patch directly to the proton files which is very important.

4. Launch game directly after installation when prompted. This should work out of the gate.

5. Now you just need to change the Steam shortcut of the installer you added to point at the game rather than the installer - use the directory setting you tried before, this time pointing to "Freespace 2.exe" and then run it (this ran a configuration program), and finally to FS2.exe (which now runs the game directly).

Done!