I've added checking on every event for FPS 65536 bunch crossing wrap around in the event builders:
(daqman:/RTS/log)> egrep "Run|l2Threads" evb.log.* | egrep "Run|evts" | egrep -B 1 "BX"
will show all the runs with corruption (from the appropriate evb.log file). (This is only part of last nights results):
--
[evb05 02:14:57 082] (evbx2): JEFF: evbx2.C [line 1346]: configureRun(#18082007)
[evb02 02:55:54 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption: 417087 evts of 677612: fpre=4 evts, fpost=417085 evts
[evb09 02:55:54 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption: 693425 evts of 1125608: fpre=5 evts, fpost=693425 evts
[evb08 02:55:54 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption: 804920 evts of 1306929: fpre=5 evts, fpost=804920 evts
[evb04 02:55:54 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption: 417347 evts of 677731: fpre=1 evts, fpost=417347 evts
[evb05 02:55:54 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption: 691186 evts of 1123255: fpre=1 evts, fpost=691186 evts
[evb07 02:55:54 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption: 693161 evts of 1125602: fpre=1 evts, fpost=693161 evts
[evb10 02:55:54 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption: 692416 evts of 1125610: fpre=4 evts, fpost=692416 evts
[evb03 02:55:54 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption: 418228 evts of 678190: fpre=0 evts, fpost=418228 evts
[evb06 02:55:54 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption: 692278 evts of 1123253: fpre=1 evts, fpost=692278 evts
[evb01 02:55:55 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption: 417676 evts of 678197: fpre=2 evts, fpost=417676 evts
--
[evb05 02:56:27 082] (evbx2): JEFF: evbx2.C [line 1346]: configureRun(#18082008)
[evb02 03:25:44 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption: 2 evts of 485585: fpre=0 evts, fpost=2 evts
[evb08 03:25:44 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption: 13 evts of 936776: fpre=2 evts, fpost=12 evts
[evb04 03:25:44 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption: 4 evts of 485443: fpre=1 evts, fpost=4 evts
[evb09 03:25:44 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption: 11 evts of 806807: fpre=3 evts, fpost=10 evts
[evb05 03:25:44 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption: 6 evts of 805126: fpre=3 evts, fpost=3 evts
[evb03 03:25:44 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption: 5 evts of 486112: fpre=1 evts, fpost=4 evts
[evb10 03:25:44 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption: 4 evts of 806812: fpre=1 evts, fpost=3 evts
[evb07 03:25:44 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption: 7 evts of 806806: fpre=3 evts, fpost=5 evts
[evb06 03:25:44 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption: 8 evts of 805123: fpre=1 evts, fpost=7 evts
[evb01 03:25:45 082] (evbx2): JEFF: l2Threads.C [line 322]: BX Corruption: 4 evts of 486113: fpre=0 evts, fpost=4 evts
There are 3 different sorts of corruption I saw last night. Nearly all of the runs show the corruption seen in run #18082008 above. This seems to be a very low level ~4e-4% of valid errors. You can see the spectrum of the trg->readout RCC counters in the JEVP plots now:
However the statistics are way too low. Here you see the times for every event, and there are several corrupt events. There doesn't seem to be any pathological reason for the delays, however. They are simply the outliers...
The second issue seem to be that one of the RCC boards can reset its BX counter to 0 at some point in the run. This occured in run #18082007:
Here The values plotted are the bare 32 bit bunch crossings for the FPRE, FPOST & L0 trigger. The triangle shape of all the plots is due to 32 bit wrap around of the bunch crossing every 4.5 minutes. Here, after 4 minutes or so you see the FPOST mysteriously reset, but continue to count for the rest of the run.
In one of the previous blogs I showed this sometimes happens in the QT boards as well...