So I started to work my way from the contrived example to the broken version. What I found was that adding the filters dynamically is whats causing the problem.
Here is a Twiddle that has the component layers. Things are working as expected.
Here is the broken version. Where I dynamically add the filters.
The difference between the 2 is that in the latter I add the filters dynamically. The filter object reference is the same. I dont understand why is breaks?