On 3/8 - 3/9 we saw two more runs with QT corruption.    These were both short, low rate, runs with no beam:

18067041
18068036

I analyzed 18067041:

* akio_fms17 configuration   w most FMS triggers in
* 1000 events
* ~100hz data rate
* no beam

On the face of it, the 100hz data rate would make it seem impossible for data corruption to occur.   However here is the sequence:

1.   No beam, so the scaler rates of the triggers are low --->  PS=1
2.   The FMS led firing at 2hz (and rings the FMS for 40 bunch crossings)
3.   The FMS led events have occupancy near 1.
4.   When FMS led event fires, no events are in the trigger, and we get an event EVERY bunch crossing for the next 30 bunches
5.   These events have high occupancy so they take a long time to clear
6.   Once the led has cleared, noise events keep coming in with low occupancy...

The trigger->readout time behaves as you would then expect, leading to wrap around corruption during the tail end of the LED event storm.

Easily fixable by including BTOW or other detector, or by using a prescale or by defining preceeded logic.

There is another point here, which is that we can easily see the effect of the "wrap-around" corruption:    The corruption occurs any time the trgger->readout is above 64k (with potentially a slight delay because the measurement is AFTER the readout).    The bunch crossing number stays flat while the even is being issued every event, however.   So you see the region between ~evt #27 and evt #33 are events that are probably still saturated with data, in reality, however the readout is fast because the new event (7ms later) is actually being read out...