Home » Software & Computing » Grid and Cloud

Grid Production

Updated on Fri, 2011-09-16 11:26. Originally created by lbhajdu on 2007-06-08 19:18. Under:
Summary of Reconstruction Production on GRID
 



Dataset name

Description

 Events Submit Date

Finish Date

Number of jobs submitted

Output size

Efficiency

Cluster or Site

CPU in hours

rcf1304

pp200/pythia6_410/55_65gev/cdf_a/y2006c/gheisha_on

 118K 2007-06-11

2007-06-12

60

35GB

98%

pdsf.nersc.gov

14hours

rcf1302

pp200/pythia6_410/45_55gev/cdf_a/y2006c/gheisha_on

 118K 2007-06-01

2007-06-02

60

29.4GB

100%

pdsf.nersc.gov

14hours

rcf1303

pp200/pythia6_410/35_45gev/cdf_a/y2006c/gheisha_on

 119K 2007-06-02

2007-06-02

120

36.2GB

97%

pdsf.nersc.gov

11hours

rcf1306

pp200/pythia6_410/25_35gev/cdf_a/y2006c/gheisha_on

 393K 2007-06-04

2007-06-06

200

119GB

98%

pdsf.nersc.gov

41hours

rcf1307

pp200/pythia6_410/15_25gev/cdf_a/y2006c/gheisha_on

 391K 2007-06-06

2007-06-07

200

114GB

98%

pdsf.nersc.gov

34hours

rcf1308

pp200/pythia6_410/11_15gev/cdf_a/y2006c/gheisha_on

 416K 2007-06-08

2007-06-10

210

115GB

98%

pdsf.nersc.gov

39hours

rcf1309

pp200/pythia6_410/9_11gev/cdf_a/y2006c/gheisha_on

 409K 2007-06-10

2007-06-12

210

109GB

98%

pdsf.nersc.gov

47hours

rcf1310

pp200/pythia6_410/7_9gev/cdf_a/y2006c/gheisha_on

 420K 2007-06-13

2007-06-14

210

107GB

100%

pdsf.nersc.gov

31hours

rcf1311

pp200/pythia6_410/5_7gev/cdf_a/y2006c/gheisha_on

 394K 2007-06-14

2007-06-16

199

96GB

98%

pdsf.nersc.gov

48hours

rcf1317

pp200/pythia6_410/4_5gev/cdf_a/y2006c/gheisha_on

 683K 2007-06-16

2007-06-19

343

158GB

99%

pdsf.nersc.gov

69hours

rcf1318

pp200/pythia6_410/3_4gev/cdf_a/y2006c/gheisha_on

 688K 2007-06-19

2007-06-22

345

152GB

100%

pdsf.nersc.gov

78hours

rcf1319

pp200/pythia6_410/minbias/cdf_a/y2006c/gheisha_on

 201K 2007-06-22

2007-06-23

120

21GB

99%

pdsf.nersc.gov

13hours

rcf1321

pp62/pythia6_410/3_4gev/cdf_a/y2006c/gheisha_on

 250K 2007-06-25

2007-06-26

125

41GB

100%

pdsf.nersc.gov

20hours

rcf1320

pp62/pythia6_410/4_5gev/cdf_a/y2006c/gheisha_on

 400K 2007-06-26

2007-06-27

200

67GB

100%

pdsf.nersc.gov

28hours

rcf1322

pp62/pythia6_410/5_7gev/cdf_a/y2006c/gheisha_on

 218K 2007-06-24

2007-06-25

110

38GB

100%

pdsf.nersc.gov

17hours

rcf1323

pp62/pythia6_410/7_9gev/cdf_a/y2006c/gheisha_on

 220K 2007-06-29

2007-06-30

110

39GB

100%

pdsf.nersc.gov

18hours

rcf1324

pp62/pythia6_410/9_11gev/cdf_a/y2006c/gheisha_on

 220K 2007-06-30

2007-06-30

110

41GB

100%

pdsf.nersc.gov

14hours

rcf1325

pp62/pythia6_410/11_15gev/cdf_a/y2006c/gheisha_on

 220K 2007-07-01

2007-07-02

110

41GB

100%

pdsf.nersc.gov

19hours

rcf1326

pp62/pythia6_410/15_25gev/cdf_a/y2006c/gheisha_on

 220K 2007-07-03

2007-07-04

110

40GB

100%

pdsf.nersc.gov

21hours

rcf1327

pp62/pythia6_410/25_35gev/cdf_a/y2006c/gheisha_on

 220K 2007-07-04

2007-07-05

110

38GB

100%

pdsf.nersc.gov

18hours

rcf1312

pp200/pythia6_410/7_9gev/bin1/y2004y/gheisha_on

 539K 2007-07-13

2007-07-18

272

143GB

99.6%

pdsf.nersc.gov

53hours

rcf1313

pp200/pythia6_410/9_11gev/bin2/y2004y/gheisha_on

 758K 2007-07-19

2007-07-22

380

203GB

100%

pdsf.nersc.gov

72hours

rcf1314

pp200/pythia6_410/11_15gev/bin3/y2004y/gheisha_on

 116K 2007-07-31

2007-08-01

58

32GB

100%

pdsf.nersc.gov

182hours

rcf1315

pp200/pythia6_410/11_15gev/bin4/y2004y/gheisha_on

 420K 2007-08-04

2007-08-05

210

119GB

100%

pdsf.nersc.gov

527hours

rcf1316

pp200/pythia6_410/11_15gev/bin5/y2004y/gheisha_on

 158K 2007-08-08

2007-08-09

79

45GB

100%

pdsf.nersc.gov

183hours
rcf1317 pp200/pythia6_410/4_5gev/cdf_a/y2006c/gheisha_on 683K              
rcf1318 pp200/pythia6_410/3_4gev/cdf_a/y2006c/gheisha_on 688K 2007-06-04 2007-06-04 360 83.4GB 95.8% fnal.gov 619hours
rcf1319 pp200/pythia6_410/minbias/cdf_a/y2006c/gheisha_on 201K 2007-06-04 2007-06-04 120 11.7GB 100.0% fnal.gov 105hours
rcf1320 pp62/pythia6_410/4_5gev/cdf_a/y2006c/gheisha_on 400K 2007-06-06 2007-06-06 200 35.7GB 100.0% fnal.gov 241hours
rcf1321 pp62/pythia6_410/3_4gev/cdf_a/y2006c/gheisha_on 250K 2007-06-06 2007-06-06 125 21.6GB 100.0% fnal.gov 139hours
rcf1322 pp62/pythia6_410/5_7gev/cdf_a/y2006c/gheisha_on 218K 2007-06-07 2007-06-07 110 20.1GB 100.0% fnal.gov 114hours
rcf1323 pp62/pythia6_410/7_9gev/cdf_a/y2006c/gheisha_on 220K 2007-06-07 2007-06-07 110 20.6GB 100.0% fnal.gov 112hours
rcf1324 pp62/pythia6_410/9_11gev/cdf_a/y2006c/gheisha_on 220K 2007-06-07 2007-06-07 110 20.6GB 99.0% fnal.gov 124hours
rcf1325 pp62/pythia6_410/11_15gev/cdf_a/y2006c/gheisha_on 220K 2007-06-07 2007-06-07 110 20.7GB 100.0% fnal.gov 91hours
rcf1326 pp62/pythia6_410/15_25gev/cdf_a/y2006c/gheisha_on 220K 2007-06-08 2007-06-08 110 20.2GB 100.0% fnal.gov 132hours
rcf1327 pp62/pythia6_410/25_35gev/cdf_a/y2006c/gheisha_on 220K 2007-06-08 2007-06-09 110 18.3GB 100.0% fnal.gov 133hours
rcf1501 pp200/pythia6_410/minbias/cdf_a/y2006g/gheisha_on 1.99M 2008-09-30 2008-11-12 1991 412GB 99.6% nersc.gov 1,026hours
rcf1504 1DplusOnly/gkine/pt10/eta1_5/y2005g/gheisha_on 1.097M 2009-01-16 2009-02-10 1102 80.7GB 99.9% nersc.gov 1,216hours
rcf9003 pp200/pythia6_410/5_7gev/cdf_a/y2007g/gheisha_on 389K part grid and part local, because of urgency (high priority) ec2.internal  
rcf9004 pp200/pythia6_410/7_9gev/cdf_a/y2007g/gheisha_on 408K part grid and part local, because of urgency (high priority) ec2.internal  
rcf9005 pp200/pythia6_410/9_11gev/cdf_a/y2007g/gheisha_on 401K 200-03-07 2009-03-17 782 333.7GB 99.10% ec2.internal 13,022hours
rcf9010 pp200/pythia6_410/45_55gev/cdf_a/y2007g/gheisha_on 118K part grid and part local, because of urgency (high priority) ec2.internal  
rcf9011 pp200/pythia6_410/55_65gev/cdf_a/y2007g/gheisha_on 119K 2009-03-07 2009-03-11 295 108.4GB 100% ec2.internal 8,060hours
rcf10020 pp200/pythia6_422/2_3gev/tune100/y2005h/gheisha_on 115K 2010-4-07 2010-4-07 115 406.5GB 99.1% pdsf.nersc.gov 946hours
pdsf10021 pp200/pythia6_422/3_4gev/tune100/y2005h/gheisha_on 114K 2010-4-07 2010-4-09 115 438.4GB 99.1% pdsf.nersc.gov 1,728hours
pdsf10022 pp200/pythia6_422/4_5gev/tune100/y2005h/gheisha_on 114K 2010-4-08 2010-4-09 115 458.6GB 99.1% pdsf.nersc.gov 1,926hours
pdsf10023 pp200/pythia6_422/5_7gev/tune100/y2005h/gheisha_on 116K 2010-4-09 2010-4-12 115 983.1GB 96.6% pdsf.nersc.gov 1,293hours
pdsf10024 p200/pythia6_422/7_9gev/tune100/y2005h/gheisha_on 1.19M 2010-4-08 2010-4-17 1200 9615.4GB 95.8% pdsf.nersc.gov 18,261hours
pdsf10025 pp200/pythia6_422/9_11gev/tune100/y2005h/gheisha_on 115K 2010-4-10 2010-4-12 115 1018.4GB 98.2% pdsf.nersc.gov 951hours
pdsf10026 pp200/pythia6_422/11_15gev/tune100/y2005h/gheisha_on 115K 2010-4-12 2010-4-13 115 509.5GB 94.6% pdsf.nersc.gov 965hours
pdsf10027 pp200/pythia6_422/15_25gev/tune100/y2005h/gheisha_on 112K 2010-4-13 2010-4-14 115 466.0GB 83.5% pdsf.nersc.gov 822hours
pdsf10028 pp200/pythia6_422/25_35gev/tune100/y2005h/gheisha_on 114K 2010-4-13 2010-4-13 115 525.7GB 89.5% pdsf.nersc.gov 999hours
pdsf10029 pp200/pythia6_422/35_infgev/tune100/y2005h/gheisha_on 104K 2010-4-13 2010-4-14 115 521.9GB 90.4% pdsf.nersc.gov 442hours
pdsf10030
AuAu7.7/hijing_382/B0_20/minbias/y2010a/gheisha_on
 1.02M  part grid and part local, becuase of urgency (high priority) pdsf.nersc.gov  129,939hours
pdsf10031
AuAu11.5/hijing_382/B0_20/minbias/y2010a/gheisha_on
 400K  2010-08-06 2010-08-14 2000 14,598GB 94.5% pdsf.nersc.gov 95,938hours
pdsf10033 AuAu7.7/hijing_382/B0_20/minbias/y2010a/gheisha_on  3.0M 2010-12-06 2011-01-30  15,300 10TB 89.4%  pdsf.nersc.gov 465,000hours
pdsf11010 pp200/pythia6_423/minbias/highptfilt/y2005i/tune_pro_pt0 3.4M 2011-02-14 2011-02-20 1,700 1.024TB 97.17% pdsf.nersc.gov 22,022hours
pdsf11000 pp200/pythia6_220/fmspi0filt/default/y2008e/gheisha_on 1.2M 2011-05-23 2011-06-01 600 403GB 20% pdsf.nersc.gov 9,800hours
pdsf11001 pp200/pythia6_220/minbias/default/y2008e/gheisha_on 300K 2011-05-21 2011-05-22 150 84GB 100% pdsf.nersc.gov 600hours
pdsf11002  dAu200/herwig_382/fmspi0filt/shadowing_on/y2008e/gheisha_on 200K 2011-06-02 2011-06-03 250 207GB 88% pdsf.nersc.gov 2500hours
pdsf11003 dAu200/herwig_382/fmspi0filt/shadowing_off/y2008e/gheisha_on 200K 2011-06-03 2011-06-04 250 233GB 100% pdsf.nersc.gov 2500hours
pdsf11011
pp200/pythia6_423/highptfilt/jp2filt/y2005i/tune_pro_pt0

45M (100k filtered)

 2011-06-24 2011-07-14 4,500 653G 88.8% pdsf.nersc.gov 67,500hours
pdsf11010

pp200/pythia6_423/minbias/highptfilt/y2005i/tune_pro_pt0

(Expanding statistics for preexisting dataset pdsf11010)

 30.3M

part grid (2,940jobs) and part local, becuase of urgency (high priority)

2011-08-05 2011-08-26 5,500 8.50T(inc. .FZD) ??%

 pdsf.nersc.gov 82,500hours
pdsf11020 tracker review 2012 1K 2011-08-29 2011-09-05 10 484M 100% pdsf.nersc.gov 7.8hours
pdsf11021 tracker review 2012 10K 2011-08-29 2011-09-05 100 27G 100% pdsf.nersc.gov 600hours
pdsf11022 tracker review 2012 10K 2011-08-29 2011-09-06 250 102G 98.00% pdsf.nersc.gov 2,500hours
pdsf11023 tracker review 2012 10K 2011-08-29 2011-09-08 700 332G 98.14% pdsf.nersc.gov 3,500hours
pdsf11024 tracker review 2012 10K 2011-08-29 2011-09-05 100 28G 100% pdsf.nersc.gov 900hours
pdsf11025 tracker review 2012 10K 2011-08-29 2011-09-05 100 5.1G 100% pdsf.nersc.gov 200hours
pdsf11026 tracker review 2012 10K 2011-08-14 2011-08-16 200 94GB 100% pdsf.nersc.gov 2,000hours
pdsf11027 pending pending pending pending pending pending pending pending pending

Notes:

Notes for getting file size from catalog:

* This method is an approximation because the .fzd files are not cataloged, however there size is about the same as the geant.root files so an approximation is done as:

[rcas6010] ~/> get_file_list.pl -keys 'sum(size)' -cond 'path~pp200/pythia6_422/2_3gev/tune100/y2005h/gheisha_on,storage=HPSS'
27758788009
[rcas6010] ~/> get_file_list.pl -keys 'sum(size)' -cond 'path~pp200/pythia6_422/2_3gev/tune100/y2005h/gheisha_on,filetype=MC_reco_geant,storage=HPSS'
14106791434
[rcas6010] ~/> echo `echo "(27758788009+14106791434)/100000000" | bc`" GB"
418 GB

The true dataset value is 406.5GB so there is a +2.75% error.

 

The dataset description can be found at:

http://www.star.bnl.gov/public/comp/prod/MCProdList.html

 

#Example of getting the size
SELECT CONCAT(SUM(size_workerNode) / 1000000000 , 'GB')
FROM MasterIO f
WHERE f.`jobID_MD5` = 'C88562E422AF05783ACF43F6172DC95A';

#Example of fining Start Time
SELECT j.`jobID_MD5`, j.`submitTime`, f.`name_requester`
FROM MasterIO f, MasterJobEfficiency j
WHERE f.`jobID_MD5` = 'C88562E422AF05783ACF43F6172DC95A'
AND f.`jobID_MD5`= j.`jobID_MD5`
AND f.`name_requester` IS NOT NULL
ORDER BY `submitTime` ASC
LIMIT 3;


#Example of fining end time
SELECT j.`jobID_MD5`, j.`endTime`, f.`name_requester`
FROM MasterIO f, MasterJobEfficiency j
WHERE f.`jobID_MD5` = 'C88562E422AF05783ACF43F6172DC95A'
AND f.`jobID_MD5`= j.`jobID_MD5`
AND f.`name_requester` IS NOT NULL
ORDER BY `endTime` DESC
LIMIT 3;

 

Notes finding the number if events from catalog:

[rcas6010] ~/> get_file_list.pl -keys 'sum(events)' -cond 'path~pp200/pythia6_422/2_3gev/tune100/y2005h/gheisha_on,filetype=MC_reco_geant,storage=HPSS'
115000

*Note select only one type of file ( filetype=MC_reco_geant ), else you will be double counting.

#Example of getting the production Efficiency:


SELECT concat(
((SELECT count(*) AS jobsCount FROM MasterJobEfficiency j
WHERE submitAttempt = 1
AND overAllState = 'success'
AND j.`jobID_MD5` = 'C88562E422AF05783ACF43F6172DC95A'
) * 100 ) /
(SELECT count(*) AS jobsCount FROM MasterJobEfficiency j
WHERE submitAttempt = 1
AND j.`jobID_MD5` = 'C88562E422AF05783ACF43F6172DC95A'
),'%');

 

#Example of getting the run time. Note there is a filter for run-way jobs:

SELECT AVG((`endTime` - `startTime`) / 60 / 60) FROM MasterJobEfficiency f
WHERE endTime > 0
AND startTime > 0
AND ((`endTime` - `startTime`) / 60 / 60) < 200
AND f.`jobID_MD5` = 'C88562E422AF05783ACF43F6172DC95A';

 

»