Here are some insights about the tech, per good (yes, they still exist!) journalists at medium.com:
- It's a 2D AI Filter. Input is only color buffer & motion vectors. The model doesn't see geometry, lights, PBR properties, normals, anything
- Extremely minimal "artistic control". You can turn things on/off or mix with alpha blending and control color grading and that's it NVIDIA says they're listening to devs about extra controls but IMO high-level, subtle knobs (like prompts to a diffusion app) are very unlikely.
- What the AI does is *inference*. It looks at the pixels of a jacket and recognizes leather. It looks at the pixels of shadows and reflections and guesses where the lights are, their intensity, color, etc. It figures out the 3D shape of objects by their contours, shadows, etc.
- Then it proceeds using those guesses to re-render the scene. If the guesses are good, the result can be a lot better than the original (at least for some definitions of better i.e. more realistic and detailed).
- Every scene/game had VERY simple illumination; often just ambient light. Sun/outdoors. They didn't pick their usual Path Traced demos with tons of light sources and emissives. I bet the model isn't even close to do all the guesswork for this. Especially without SSR effects.
- It can't even handle any fast moving object wells, but to be fair I don't know if those catastrophic frames in the FIFA demo should be blamed on the upscaler, FG, or the new AI filter. All of them have trouble with big disocclusions and very fast things. AI is not magic.
Thread on Twitter.
So basically, it's bad.