Just downloaded the SteamOS 3.4.6 update, which upgrades the RADeon Vulkan (RADV) driver version to 23.1.0_devel. I was able to get 20-30 FPS after tuning down the video settings. Game is playable with ray-tracing effects—my settings traded off visual fidelity to get the framerate floor up to 20 FPS.

Another observation was that, overall, Quake II RTX looked darker on my Steam Deck's built-in display when compared a UHD TV or a FHD monitor. I ended up adjusting the in-game settings to get the Steam Deck to look like my other displays.

Finally, IMHO you'll definitely want to create a custom controller profile or search for a community profile. At a minimum I'd recommend mapping: Ⓐ → space, Ⓑ → enter, Ⓧ → / [changes the position of the ray-traced "sun"], and the menu button (≡) → escape. There are many other things that need mapping—e.g., switching weapons and items—but should be enough to play the first level.

In-Game settings that were adjusted

Quake II RTX → Video

• resolution scaling options … → fixed resolution scaling: 60% (renders at 768x480)
• exposure bias(brightness): -0.2 EV
• contrast: 80%
• AMD FSR 1.0: yes
• global illumination: low
• ray tracing API: prefer KHR_ray_query