determining why glsl shaders fail on ati

category: code [glöplog]
I'm working on the same project as shuffle2. I'm primarily working on the Direct3D side of things, so I hope you'll understand what I'm talking about, even if my terminology doesn't quite fit to OpenGL.

In case is matters, the example shaders in the first post are supposed to copy the source texture (which is a render target) to the currently bound render target (which may be used as a texture lateron). During the copy, the source color texture is transformed by some input parameters which are passed via uniforms. We're using some of them a) as a (per-component) mask b) some of them as a (per-component) scaling factor c) some of them as a (per-component) offset. Those shaders are used to achieve render-to-texture like effects (we REALLY have to do it that way though, believe me). The GLSL code became kinda messy since they were ported from ARB assembly code though.

Anyway, here are some screenshots which show how things are going wrong on ATI GPUs:
- http://i1104.photobucket.com/albums/h325/no_cluez/glsl-fail1.png
The shaders above are used to generate the textures for the background hills and the main characters (which are above the copyright text)
- http://i1104.photobucket.com/albums/h325/no_cluez/glsl-fail2.png
The shaders above are used to render shadows in this game. However, stuff goes wrong and we just get a grey-ish rect above everything.
- http://i1104.photobucket.com/albums/h325/no_cluez/glsl-fail.png
- Stuff totally goes wrong here. In case you know this game you'll notice that the floor is NOT supposed to be red. The background is also heavily glitched.

Note that the "normal" geometry rendering works fine, it's just those shaders which fail. The "normal" rendering is done via a shader generator which uses the same "ugly glsl programming", so I doubt it's
that (especially since those example shaders are really trivially simple compared to what the shader generator is outputting).

Further investigation shows that turning off our usage of UBOs fixes the problems with the shaders mentioned above. However, it causes other kinds of issues (apparently in the generated shaders for rendering), e.g. http://i1104.photobucket.com/albums/h325/no_cluez/glsl-fail2-1.png . These indeed could be caused by "ugly glsl programming". They're not that important anyway, our main goal is fixing the problems with the shaders above for now, since they're a major showstopper for dropping Cg.

For reference, we are already checking for shader compilation / OpenGL errors (not warnings though).
added on the 2012-01-07 01:13:51 by neobrain neobrain
Are you running with a debug context? Those give pretty extensive error-messages.
added on the 2012-01-07 09:31:44 by hornet hornet
oh... d3d... on wii... no idea, good luck, have fun! :)
added on the 2012-01-07 09:33:11 by hornet hornet
d3d on pc emulating wii ;)
added on the 2012-01-07 10:11:05 by shuffle2 shuffle2
So does it fails when you start using multiple textures for instance? Are you sure you properly set your uniforms too?
added on the 2012-01-07 13:13:00 by nystep nystep
Sorry to hijack this tread, but it's ATI-shader related: I've made an ATI-fixed partyversion of our latest demo, but I'm not quite sure about one fix. It works on my machine (TM), but I'd appreciate it if somebody else with an ATI card could see if it works. It's been tested on a HD5770 card already. Here is the link: http://www.fulcrum-demo.org/filemanager/active?fid=122 . If I get confirmation that it works OK, I'll change the prod download link to that version, until we get a final done.

The main point I want to know is if the fractal at the title screen has one central tower and three identical smaller ones, like this: http://imageshack.us/photo/my-images/843/nvidiaf.jpg/. There seems to be a (driver?) bug that causes this: http://imageshack.us/photo/my-images/267/atic.png/, with one tower squashed and turned 45 degrees. (The details: I send an array with 6 post-transform matrices to the shader, and the 3th one gets corrupted. I've fixed it by hardcoding those values, but as I don't know the cause, I'm not confident the corruption happens to the same matrix on all cards. Nvidia works fine).
added on the 2012-01-16 20:32:20 by Seven Seven
Seven: Issues like those are often traced back to the application code not properly using glGetUniformLocation, but instead depending on NVIDIA-specific uniform packing rules. I'm not saying that IS the reason here, but it might be worth checking out.
added on the 2012-01-16 20:54:44 by kusma kusma
Kusma: I've never heard of any other way to set values then by using GlGetUniformLocation and glUniform. The variable in the shader is defined like this:

"uniform mat3 thePostMats[XNrTransX+1];"

(with the XNrTransX being substituted for the nr of flame transformations before compiling the shader) and in the main program:

float thePostMats[eMaxTrans][9];
... code that fills thePostMats, mostly with identity matrices ...
glUniformMatrix3fv( glGetUniformLocation(mGenProg->mProgram, "thePostMats"), mFlame->GetNrTrans(), false, thePostMats[0]);

In a debugger, I can see that the main thePostMats[] have the correct values. But the result look wrong unless I do something like this:

if (i != 3)
theResult = thePostMat * Bla;
theResult = Bla;

I'd love to hear any ideas about what can cause this and how to fix it, it makes me profoundly uneasy to have such obviously fragile workarounds in my code.
added on the 2012-01-16 21:30:46 by Seven Seven
Seven: The code looks correct in itself. But I was actually thinking that there might be some OTHER uniform that gets set on an invalid location (which could overlap with thePostMats[3]). You could try to use gDEBugger to inspect the uniform state when drawing to see that no entries have been corrupted, I guess...
added on the 2012-01-16 23:14:15 by kusma kusma
hey, just wanted to say that I finally got around to working on this - I've squashed compiler errors and warnings.
Now my NV hardware displays the same artifacts as ATI hardware. I guess that's a good thing, right? o_O
added on the 2012-01-19 19:48:01 by shuffle2 shuffle2
To correct that statement: the artifacts on ATI are somewhat improved, but now NV appears the same. So, at least there is some progress :) I think the rest deals with correcting interface between ogl and glsl.
added on the 2012-01-19 20:15:27 by shuffle2 shuffle2
go shuffle2 :)
added on the 2012-01-19 21:25:40 by ector ector
Kusma: corruption by later writes was something I already suspected, so I moved the thePostMats initialisation to be initialised the last. That doesn't make a difference. Barring some weird "caching the values and writing them in a different order" driver mechanism, I guess that rules out that possibility.

BTW, can anyone try if that ATI-fix works, and maybe report what model of ATI card you have? Here's the link again: http://www.fulcrum-demo.org/filemanager/active?fid=122

added on the 2012-01-20 22:57:04 by Seven Seven
Seven: Works fine on the AMD Radeon HD 6850 here.
added on the 2012-01-21 10:57:29 by Tolle Tolle