Grid Production

Summary of Reconstruction Production on GRID
 



Dataset name

Description

 Events Submit Date

Finish Date

Number of jobs submitted

Output size

Efficiency

Cluster or Site

CPU in hours

rcf1304

pp200/pythia6_410/55_65gev/cdf_a/y2006c/gheisha_on

 118K 2007-06-11

2007-06-12

60

35GB

98%

pdsf.nersc.gov

14hours

rcf1302

pp200/pythia6_410/45_55gev/cdf_a/y2006c/gheisha_on

 118K 2007-06-01

2007-06-02

60

29.4GB

100%

pdsf.nersc.gov

14hours

rcf1303

pp200/pythia6_410/35_45gev/cdf_a/y2006c/gheisha_on

 119K 2007-06-02

2007-06-02

120

36.2GB

97%

pdsf.nersc.gov

11hours

rcf1306

pp200/pythia6_410/25_35gev/cdf_a/y2006c/gheisha_on

 393K 2007-06-04

2007-06-06

200

119GB

98%

pdsf.nersc.gov

41hours

rcf1307

pp200/pythia6_410/15_25gev/cdf_a/y2006c/gheisha_on

 391K 2007-06-06

2007-06-07

200

114GB

98%

pdsf.nersc.gov

34hours

rcf1308

pp200/pythia6_410/11_15gev/cdf_a/y2006c/gheisha_on

 416K 2007-06-08

2007-06-10

210

115GB

98%

pdsf.nersc.gov

39hours

rcf1309

pp200/pythia6_410/9_11gev/cdf_a/y2006c/gheisha_on

 409K 2007-06-10

2007-06-12

210

109GB

98%

pdsf.nersc.gov

47hours

rcf1310

pp200/pythia6_410/7_9gev/cdf_a/y2006c/gheisha_on

 420K 2007-06-13

2007-06-14

210

107GB

100%

pdsf.nersc.gov

31hours

rcf1311

pp200/pythia6_410/5_7gev/cdf_a/y2006c/gheisha_on

 394K 2007-06-14

2007-06-16

199

96GB

98%

pdsf.nersc.gov

48hours

rcf1317

pp200/pythia6_410/4_5gev/cdf_a/y2006c/gheisha_on

 683K 2007-06-16

2007-06-19

343

158GB

99%

pdsf.nersc.gov

69hours

rcf1318

pp200/pythia6_410/3_4gev/cdf_a/y2006c/gheisha_on

 688K 2007-06-19

2007-06-22

345

152GB

100%

pdsf.nersc.gov

78hours

rcf1319

pp200/pythia6_410/minbias/cdf_a/y2006c/gheisha_on

 201K 2007-06-22

2007-06-23

120

21GB

99%

pdsf.nersc.gov

13hours

rcf1321

pp62/pythia6_410/3_4gev/cdf_a/y2006c/gheisha_on

 250K 2007-06-25

2007-06-26

125

41GB

100%

pdsf.nersc.gov

20hours

rcf1320

pp62/pythia6_410/4_5gev/cdf_a/y2006c/gheisha_on

 400K 2007-06-26

2007-06-27

200

67GB

100%

pdsf.nersc.gov

28hours

rcf1322

pp62/pythia6_410/5_7gev/cdf_a/y2006c/gheisha_on

 218K 2007-06-24

2007-06-25

110

38GB

100%

pdsf.nersc.gov

17hours

rcf1323

pp62/pythia6_410/7_9gev/cdf_a/y2006c/gheisha_on

 220K 2007-06-29

2007-06-30

110

39GB

100%

pdsf.nersc.gov

18hours

rcf1324

pp62/pythia6_410/9_11gev/cdf_a/y2006c/gheisha_on

 220K 2007-06-30

2007-06-30

110

41GB

100%

pdsf.nersc.gov

14hours

rcf1325

pp62/pythia6_410/11_15gev/cdf_a/y2006c/gheisha_on

 220K 2007-07-01

2007-07-02

110

41GB

100%

pdsf.nersc.gov

19hours

rcf1326

pp62/pythia6_410/15_25gev/cdf_a/y2006c/gheisha_on

 220K 2007-07-03

2007-07-04

110

40GB

100%

pdsf.nersc.gov

21hours

rcf1327

pp62/pythia6_410/25_35gev/cdf_a/y2006c/gheisha_on

 220K 2007-07-04

2007-07-05

110

38GB

100%

pdsf.nersc.gov

18hours

rcf1312

pp200/pythia6_410/7_9gev/bin1/y2004y/gheisha_on

 539K 2007-07-13

2007-07-18

272

143GB

99.6%

pdsf.nersc.gov

53hours

rcf1313

pp200/pythia6_410/9_11gev/bin2/y2004y/gheisha_on

 758K 2007-07-19

2007-07-22

380

203GB

100%

pdsf.nersc.gov

72hours

rcf1314

pp200/pythia6_410/11_15gev/bin3/y2004y/gheisha_on

 116K 2007-07-31

2007-08-01

58

32GB

100%

pdsf.nersc.gov

182hours

rcf1315

pp200/pythia6_410/11_15gev/bin4/y2004y/gheisha_on

 420K 2007-08-04

2007-08-05

210

119GB

100%

pdsf.nersc.gov

527hours

rcf1316

pp200/pythia6_410/11_15gev/bin5/y2004y/gheisha_on

 158K 2007-08-08

2007-08-09

79

45GB

100%

pdsf.nersc.gov

183hours
rcf1317 pp200/pythia6_410/4_5gev/cdf_a/y2006c/gheisha_on 683K              
rcf1318 pp200/pythia6_410/3_4gev/cdf_a/y2006c/gheisha_on 688K 2007-06-04 2007-06-04 360 83.4GB 95.8% fnal.gov 619hours
rcf1319 pp200/pythia6_410/minbias/cdf_a/y2006c/gheisha_on 201K 2007-06-04 2007-06-04 120 11.7GB 100.0% fnal.gov 105hours
rcf1320 pp62/pythia6_410/4_5gev/cdf_a/y2006c/gheisha_on 400K 2007-06-06 2007-06-06 200 35.7GB 100.0% fnal.gov 241hours
rcf1321 pp62/pythia6_410/3_4gev/cdf_a/y2006c/gheisha_on 250K 2007-06-06 2007-06-06 125 21.6GB 100.0% fnal.gov 139hours
rcf1322 pp62/pythia6_410/5_7gev/cdf_a/y2006c/gheisha_on 218K 2007-06-07 2007-06-07 110 20.1GB 100.0% fnal.gov 114hours
rcf1323 pp62/pythia6_410/7_9gev/cdf_a/y2006c/gheisha_on 220K 2007-06-07 2007-06-07 110 20.6GB 100.0% fnal.gov 112hours
rcf1324 pp62/pythia6_410/9_11gev/cdf_a/y2006c/gheisha_on 220K 2007-06-07 2007-06-07 110 20.6GB 99.0% fnal.gov 124hours
rcf1325 pp62/pythia6_410/11_15gev/cdf_a/y2006c/gheisha_on 220K 2007-06-07 2007-06-07 110 20.7GB 100.0% fnal.gov 91hours
rcf1326 pp62/pythia6_410/15_25gev/cdf_a/y2006c/gheisha_on 220K 2007-06-08 2007-06-08 110 20.2GB 100.0% fnal.gov 132hours
rcf1327 pp62/pythia6_410/25_35gev/cdf_a/y2006c/gheisha_on 220K 2007-06-08 2007-06-09 110 18.3GB 100.0% fnal.gov 133hours
rcf1501 pp200/pythia6_410/minbias/cdf_a/y2006g/gheisha_on 1.99M 2008-09-30 2008-11-12 1991 412GB 99.6% nersc.gov 1,026hours
rcf1504 1DplusOnly/gkine/pt10/eta1_5/y2005g/gheisha_on 1.097M 2009-01-16 2009-02-10 1102 80.7GB 99.9% nersc.gov 1,216hours
rcf9003 pp200/pythia6_410/5_7gev/cdf_a/y2007g/gheisha_on 389K part grid and part local, because of urgency (high priority) ec2.internal  
rcf9004 pp200/pythia6_410/7_9gev/cdf_a/y2007g/gheisha_on 408K part grid and part local, because of urgency (high priority) ec2.internal  
rcf9005 pp200/pythia6_410/9_11gev/cdf_a/y2007g/gheisha_on 401K 200-03-07 2009-03-17 782 333.7GB 99.10% ec2.internal 13,022hours
rcf9010 pp200/pythia6_410/45_55gev/cdf_a/y2007g/gheisha_on 118K part grid and part local, because of urgency (high priority) ec2.internal  
rcf9011 pp200/pythia6_410/55_65gev/cdf_a/y2007g/gheisha_on 119K 2009-03-07 2009-03-11 295 108.4GB 100% ec2.internal 8,060hours
rcf10020 pp200/pythia6_422/2_3gev/tune100/y2005h/gheisha_on 115K 2010-4-07 2010-4-07 115 406.5GB 99.1% pdsf.nersc.gov 946hours
pdsf10021 pp200/pythia6_422/3_4gev/tune100/y2005h/gheisha_on 114K 2010-4-07 2010-4-09 115 438.4GB 99.1% pdsf.nersc.gov 1,728hours
pdsf10022 pp200/pythia6_422/4_5gev/tune100/y2005h/gheisha_on 114K 2010-4-08 2010-4-09 115 458.6GB 99.1% pdsf.nersc.gov 1,926hours
pdsf10023 pp200/pythia6_422/5_7gev/tune100/y2005h/gheisha_on 116K 2010-4-09 2010-4-12 115 983.1GB 96.6% pdsf.nersc.gov 1,293hours
pdsf10024 p200/pythia6_422/7_9gev/tune100/y2005h/gheisha_on 1.19M 2010-4-08 2010-4-17 1200 9615.4GB 95.8% pdsf.nersc.gov 18,261hours
pdsf10025 pp200/pythia6_422/9_11gev/tune100/y2005h/gheisha_on 115K 2010-4-10 2010-4-12 115 1018.4GB 98.2% pdsf.nersc.gov 951hours
pdsf10026 pp200/pythia6_422/11_15gev/tune100/y2005h/gheisha_on 115K 2010-4-12 2010-4-13 115 509.5GB 94.6% pdsf.nersc.gov 965hours
pdsf10027 pp200/pythia6_422/15_25gev/tune100/y2005h/gheisha_on 112K 2010-4-13 2010-4-14 115 466.0GB 83.5% pdsf.nersc.gov 822hours
pdsf10028 pp200/pythia6_422/25_35gev/tune100/y2005h/gheisha_on 114K 2010-4-13 2010-4-13 115 525.7GB 89.5% pdsf.nersc.gov 999hours
pdsf10029 pp200/pythia6_422/35_infgev/tune100/y2005h/gheisha_on 104K 2010-4-13 2010-4-14 115 521.9GB 90.4% pdsf.nersc.gov 442hours
pdsf10030
AuAu7.7/hijing_382/B0_20/minbias/y2010a/gheisha_on
 1.02M  part grid and part local, becuase of urgency (high priority) pdsf.nersc.gov  129,939hours
pdsf10031
AuAu11.5/hijing_382/B0_20/minbias/y2010a/gheisha_on
 400K  2010-08-06 2010-08-14 2000 14,598GB 94.5% pdsf.nersc.gov 95,938hours
pdsf10033 AuAu7.7/hijing_382/B0_20/minbias/y2010a/gheisha_on  3.0M 2010-12-06 2011-01-30  15,300 10TB 89.4%  pdsf.nersc.gov 465,000hours
pdsf11010 pp200/pythia6_423/minbias/highptfilt/y2005i/tune_pro_pt0 3.4M 2011-02-14 2011-02-20 1,700 1.024TB 97.17% pdsf.nersc.gov 22,022hours
pdsf11000 pp200/pythia6_220/fmspi0filt/default/y2008e/gheisha_on 1.2M 2011-05-23 2011-06-01 600 403GB 20% pdsf.nersc.gov 9,800hours
pdsf11001 pp200/pythia6_220/minbias/default/y2008e/gheisha_on 300K 2011-05-21 2011-05-22 150 84GB 100% pdsf.nersc.gov 600hours
pdsf11002  dAu200/herwig_382/fmspi0filt/shadowing_on/y2008e/gheisha_on 200K 2011-06-02 2011-06-03 250 207GB 88% pdsf.nersc.gov 2500hours
pdsf11003 dAu200/herwig_382/fmspi0filt/shadowing_off/y2008e/gheisha_on 200K 2011-06-03 2011-06-04 250 233GB 100% pdsf.nersc.gov 2500hours
pdsf11011
pp200/pythia6_423/highptfilt/jp2filt/y2005i/tune_pro_pt0

45M (100k filtered)

 2011-06-24 2011-07-14 4,500 653G 88.8% pdsf.nersc.gov 67,500hours
pdsf11010

pp200/pythia6_423/minbias/highptfilt/y2005i/tune_pro_pt0

(Expanding statistics for preexisting dataset pdsf11010)

 30.3M

part grid (2,940jobs) and part local, becuase of urgency (high priority)

2011-08-05 2011-08-26 5,500 8.50T(inc. .FZD) ??%

 pdsf.nersc.gov 82,500hours
pdsf11020 tracker review 2012 1K 2011-08-29 2011-09-05 10 484M 100% pdsf.nersc.gov 7.8hours
pdsf11021 tracker review 2012 10K 2011-08-29 2011-09-05 100 27G 100% pdsf.nersc.gov 600hours
pdsf11022 tracker review 2012 10K 2011-08-29 2011-09-06 250 102G 98.00% pdsf.nersc.gov 2,500hours
pdsf11023 tracker review 2012 10K 2011-08-29 2011-09-08 700 332G 98.14% pdsf.nersc.gov 3,500hours
pdsf11024 tracker review 2012 10K 2011-08-29 2011-09-05 100 28G 100% pdsf.nersc.gov 900hours
pdsf11025 tracker review 2012 10K 2011-08-29 2011-09-05 100 5.1G 100% pdsf.nersc.gov 200hours
pdsf11026 tracker review 2012 10K 2011-08-14 2011-08-16 200 94GB 100% pdsf.nersc.gov 2,000hours
pdsf11027 pending pending pending pending pending pending pending pending pending

Notes:

Notes for getting file size from catalog:

* This method is an approximation because the .fzd files are not cataloged, however there size is about the same as the geant.root files so an approximation is done as:

[rcas6010] ~/> get_file_list.pl -keys 'sum(size)' -cond 'path~pp200/pythia6_422/2_3gev/tune100/y2005h/gheisha_on,storage=HPSS'
27758788009
[rcas6010] ~/> get_file_list.pl -keys 'sum(size)' -cond 'path~pp200/pythia6_422/2_3gev/tune100/y2005h/gheisha_on,filetype=MC_reco_geant,storage=HPSS'
14106791434
[rcas6010] ~/> echo `echo "(27758788009+14106791434)/100000000" | bc`" GB"
418 GB

The true dataset value is 406.5GB so there is a +2.75% error.

 

The dataset description can be found at:

http://www.star.bnl.gov/public/comp/prod/MCProdList.html

 

#Example of getting the size
SELECT CONCAT(SUM(size_workerNode) / 1000000000 , 'GB')
FROM MasterIO f
WHERE f.`jobID_MD5` = 'C88562E422AF05783ACF43F6172DC95A';

#Example of fining Start Time
SELECT j.`jobID_MD5`, j.`submitTime`, f.`name_requester`
FROM MasterIO f, MasterJobEfficiency j
WHERE f.`jobID_MD5` = 'C88562E422AF05783ACF43F6172DC95A'
AND f.`jobID_MD5`= j.`jobID_MD5`
AND f.`name_requester` IS NOT NULL
ORDER BY `submitTime` ASC
LIMIT 3;


#Example of fining end time
SELECT j.`jobID_MD5`, j.`endTime`, f.`name_requester`
FROM MasterIO f, MasterJobEfficiency j
WHERE f.`jobID_MD5` = 'C88562E422AF05783ACF43F6172DC95A'
AND f.`jobID_MD5`= j.`jobID_MD5`
AND f.`name_requester` IS NOT NULL
ORDER BY `endTime` DESC
LIMIT 3;

 

Notes finding the number if events from catalog:

[rcas6010] ~/> get_file_list.pl -keys 'sum(events)' -cond 'path~pp200/pythia6_422/2_3gev/tune100/y2005h/gheisha_on,filetype=MC_reco_geant,storage=HPSS'
115000

*Note select only one type of file ( filetype=MC_reco_geant ), else you will be double counting.

#Example of getting the production Efficiency:


SELECT concat(
((SELECT count(*) AS jobsCount FROM MasterJobEfficiency j
WHERE submitAttempt = 1
AND overAllState = 'success'
AND j.`jobID_MD5` = 'C88562E422AF05783ACF43F6172DC95A'
) * 100 ) /
(SELECT count(*) AS jobsCount FROM MasterJobEfficiency j
WHERE submitAttempt = 1
AND j.`jobID_MD5` = 'C88562E422AF05783ACF43F6172DC95A'
),'%');

 

#Example of getting the run time. Note there is a filter for run-way jobs:

SELECT AVG((`endTime` - `startTime`) / 60 / 60) FROM MasterJobEfficiency f
WHERE endTime > 0
AND startTime > 0
AND ((`endTime` - `startTime`) / 60 / 60) < 200
AND f.`jobID_MD5` = 'C88562E422AF05783ACF43F6172DC95A';