HyperCache integration, phase I report

First part of the StHyperCache addon integration into to StDbLib

 

1. Core classes added to StDbLib : HyperCache interface, Cache factory, Local File Cache implementation, WebService cache implementation, utilities. Memcache support classes would be added when libmemcache becomes available at STAR external libs location;

2. Basic integration is done, tests under way. For now, only Local File cache could be enabled, due to missing WebService server;

3. Future steps: (a) extend existing implementation to Memcache, (b) implement fine-grained config options (only core options implemented right now);

TESTS:

0. Code level:
  a) memory leaks examined by Valgrind, none found;
  b) comments are sparse right now, but I'm working on that :)
  c) class diagram and workflow would be (supposedly) presented at some STAR S&C meeting;

1. Performance, Basic Level:  typical user script to fetch TPC voltages for specific date from db (no chain, just St_db_Maker and single db call);
Three variants compared:
  a) no caching,
  b) caching exists but disabled,
  c) caching enabled and cache is empty,
  d) caching enabled and cache is full;

All variants produce identical results (workflow is presumed). There is no sense to study speed-up at this moment, due to very basic and tiny request to db.

2. Performance, BFC Level:
a) test BFC chain was run on just 2 events - no problems found by looking at log files briefly (tracks produced, hits available etc). For detailed study see (b).
b) test BFC chain is being run on 100 events. Please check back this page sometime soon to see the test results. Expected summary will include timings for various modes of operation, correctness of the output etc..
с) I rely on nighly tests infrastructure for extended tests, since it has multple years, chains and datasets in config. Will ask Lidia for help.

test chain (from Lidia): bfc.C(500,"DbV20100301,pp2009c,ITTF,VFPPVnoCTB,beamLine,BEmcChkStat, btofDat,Corr4,OSpaceZ2,OGridLeak3D", "/star/rcf/test/daq/2009/115/st_physics_10115020_raw_5020001.daq")

100 evts test

BFC percentages for QA :INFO  - QAInfo:Maker St_db_Maker::db
Cache not integrated Ast =  2.97( 0.4%) Cpu =  1.38( 0.2%)  baseline measurement
Cache integrated, disabled Ast =  3.02( 0.4%) Cpu =  1.30( 0.2%)  as it is right now in DEV
Cache integrated, enabled, empty Ast =  3.00( 0.4%) Cpu =  1.57( 0.2%)  max resource usage expected
Cache integrated, enabled, full Ast =  2.37( 0.3%) Cpu =  1.46( 0.2%)  min resource usage expected

 

Or, as a plain list with extra info reported by St_db_Maker:

1. NO CACHE INTEGRATED:
QA :INFO  - QAInfo:Maker          St_db_Maker::db                   Ast =  2.97( 0.4%) Cpu =  1.38( 0.2%)
St_db_Maker:INFO  - St_db_Maker::dbStat : Evts = 100 dbEvts=1 Evts/dbEvts = 100.0
St_db_Maker:INFO  - St_db_Maker::dbStat : dbData =      68.5 dbTime=20.3204 dbData/dbTime=3.36923
St_db_Maker:INFO  - St_db_Maker::dbStat : dbTime =      20.3 dbEvts=1 dbTime/dbEvts =20.3204
St_db_Maker:INFO  - St_db_Maker::dbStat : dbCpu  =       6.0 dbEvts=1  dbCpu/dbEvts =6.05
St_db_Maker:INFO  - St_db_Maker::dbStat : dbTime/tot  =      2.36 dbCpu/tot=      0.70

2. CACHE DISABLED:
QA :INFO  - QAInfo:Maker          St_db_Maker::db                   Ast =  3.02( 0.4%) Cpu =  1.30( 0.2%)
St_db_Maker:INFO  - St_db_Maker::dbStat : Evts = 100 dbEvts=1 Evts/dbEvts = 100.0
St_db_Maker:INFO  - St_db_Maker::dbStat : dbData =      68.5 dbTime=200.204 dbData/dbTime=0.341973
St_db_Maker:INFO  - St_db_Maker::dbStat : dbTime =     200.2 dbEvts=1 dbTime/dbEvts =200.204
St_db_Maker:INFO  - St_db_Maker::dbStat : dbCpu  =       6.1 dbEvts=1  dbCpu/dbEvts =6.13
St_db_Maker:INFO  - St_db_Maker::dbStat : dbTime/tot  =     23.42 dbCpu/tot=      0.72

3. CACHE ENABLED, EMPTY:
QA :INFO  - QAInfo:Maker          St_db_Maker::db                   Ast =  3.00( 0.4%) Cpu =  1.57( 0.2%)
St_db_Maker:INFO  - St_db_Maker::dbStat : Evts = 100 dbEvts=1 Evts/dbEvts = 100.0
St_db_Maker:INFO  - St_db_Maker::dbStat : dbData =      68.5 dbTime=149.012 dbData/dbTime=0.459456
St_db_Maker:INFO  - St_db_Maker::dbStat : dbTime =     149.0 dbEvts=1 dbTime/dbEvts =149.012
St_db_Maker:INFO  - St_db_Maker::dbStat : dbCpu  =      10.8 dbEvts=1  dbCpu/dbEvts =10.76
St_db_Maker:INFO  - St_db_Maker::dbStat : dbTime/tot  =     17.25 dbCpu/tot=      1.25

4. CACHE ENABLED, FULL
QA :INFO  - QAInfo:Maker          St_db_Maker::db                   Ast =  2.37( 0.3%) Cpu =  1.46( 0.2%)
St_db_Maker:INFO  - St_db_Maker::dbStat : Evts = 100 dbEvts=1 Evts/dbEvts = 100.0
St_db_Maker:INFO  - St_db_Maker::dbStat : dbData =      68.5 dbTime=48.3724 dbData/dbTime=1.41536
St_db_Maker:INFO  - St_db_Maker::dbStat : dbTime =      48.4 dbEvts=1 dbTime/dbEvts =48.3724
St_db_Maker:INFO  - St_db_Maker::dbStat : dbCpu  =      10.5 dbEvts=1  dbCpu/dbEvts =10.5
St_db_Maker:INFO  - St_db_Maker::dbStat : dbTime/tot  =      5.59 dbCpu/tot=      1.21