FPS corruption

 I've added checking on every event for FPS 65536 bunch crossing wrap around in the event builders:

Currently, any errors go to /RTS/log/evb.log

The command line:

(daqman:/RTS/log)> egrep "Run|l2Threads" evb.log.* | egrep "Run|evts" | egrep -B 1 "BX"

will show all the runs with corruption (from the appropriate evb.log file).   (This is only part of last nights results):

--

[evb05    02:14:57 082] (evbx2): JEFF: evbx2.C [line 1346]: configureRun(#18082007)
[evb02    02:55:54 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption:  417087 evts of 677612:         fpre=4 evts, fpost=417085 evts
[evb09    02:55:54 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption:  693425 evts of 1125608:         fpre=5 evts, fpost=693425 evts
[evb08    02:55:54 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption:  804920 evts of 1306929:         fpre=5 evts, fpost=804920 evts
[evb04    02:55:54 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption:  417347 evts of 677731:         fpre=1 evts, fpost=417347 evts
[evb05    02:55:54 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption:  691186 evts of 1123255:         fpre=1 evts, fpost=691186 evts
[evb07    02:55:54 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption:  693161 evts of 1125602:         fpre=1 evts, fpost=693161 evts
[evb10    02:55:54 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption:  692416 evts of 1125610:         fpre=4 evts, fpost=692416 evts
[evb03    02:55:54 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption:  418228 evts of 678190:         fpre=0 evts, fpost=418228 evts
[evb06    02:55:54 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption:  692278 evts of 1123253:         fpre=1 evts, fpost=692278 evts
[evb01    02:55:55 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption:  417676 evts of 678197:         fpre=2 evts, fpost=417676 evts
--
[evb05    02:56:27 082] (evbx2): JEFF: evbx2.C [line 1346]: configureRun(#18082008)
[evb02    03:25:44 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption:  2 evts of 485585:         fpre=0 evts, fpost=2 evts
[evb08    03:25:44 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption:  13 evts of 936776:         fpre=2 evts, fpost=12 evts
[evb04    03:25:44 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption:  4 evts of 485443:         fpre=1 evts, fpost=4 evts
[evb09    03:25:44 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption:  11 evts of 806807:         fpre=3 evts, fpost=10 evts
[evb05    03:25:44 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption:  6 evts of 805126:         fpre=3 evts, fpost=3 evts
[evb03    03:25:44 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption:  5 evts of 486112:         fpre=1 evts, fpost=4 evts
[evb10    03:25:44 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption:  4 evts of 806812:         fpre=1 evts, fpost=3 evts
[evb07    03:25:44 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption:  7 evts of 806806:         fpre=3 evts, fpost=5 evts
[evb06    03:25:44 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption:  8 evts of 805123:         fpre=1 evts, fpost=7 evts
[evb01    03:25:45 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption:  4 evts of 486113:         fpre=0 evts, fpost=4 evts

There are 3 different sorts of corruption I saw last night.    Nearly all of the runs show the corruption seen in run #18082008 above.    This seems to be a very low level ~4e-4% of valid errors.  You can see the spectrum of the trg->readout RCC counters in the JEVP plots now:

However the statistics are way too low.   Here you see the times for every event, and there are several corrupt events.   There doesn't seem to be any pathological reason for the delays, however.  They are simply the outliers...

The second issue seem to be that one of the RCC boards can reset its BX counter to 0 at some point in the run.   This occured in run #18082007:
Here The values plotted are the bare 32 bit bunch crossings for the FPRE, FPOST & L0 trigger.   The triangle shape of all the plots is due to 32 bit wrap around of the bunch crossing every 4.5 minutes.    Here, after 4 minutes or so you see the FPOST mysteriously reset, but continue to count for the rest of the run.

In one of the previous blogs I showed this sometimes happens in the QT boards as well...