I just had a bit of an epiphany! Last night I was doing tests and swapped a dim led and a bright led around in the grid. Indeed the dim one was still dim after the swap and vice versa. That eliminates an earth leak in a row/column which I thought might be possible. But it was still bugging me because when I had the 2 LEDs removed and powered them up side by side out of the grid with separate power, they looked the same, I couldn't really tell the difference at all. I did put a tiny black dot on the underside of the weak one to make sure I didn't mix them up..
So after putting them back in I also noticed that the less power I applied, the bigger the "apparent" difference between the bright and dim LEDs became. So just now it occurred to me, one thing that would explain exactly this kind of issue is if the internal resistance of some of the LEDs was a little bit higher then average! Because I'm shorting the columns together and powering them with constant current drivers, it effectively makes 8 LEDs in parallel. If a column of 8 is getting a fixed amount of current, any LEDs with a higher internal resistance will get less current then their even share and therefore appear dimmer then the others! However, when powered alone with its own current, no other "less resistant" LEDs are out competing it, therefore it appears just as bright as any others.
IF that is correct, the brilliant thing is that my design will NOT run the LEDs in parallel! When it is complete, the "layer select" electronics will power only one layer at a time. The only reason I have the layers shorted together like I do now, is because I don't have the layer select electronics done yet and this was the easy and nasty way to get all the LEDs lit up to test the construction so far. So this might not be even an issue after all! That would be great, I'm not completely absolved yet, I'll have to do some test tonight to confirm this, but I have my fingers crossed.