HPSS disk cache and Water marking

During Run 14, I looked into the HPSS cache usage and its water marking behavior.
Since a picture is worth 1,000 words, let us look at a large period of time

In short, the upper WaterMark is kicking too early for the 52 TB cache. In average over a large period, the 70% dropped cache occupancy at an average of ~ 15% (could be even lower but the HWM is of interrests). Even at the worst moment, the HWM never went beyond 83% or a 14% dispersion to average at worst.

I would tend to conlude we could increase the HWM from 70% safely to 80%. An absolute dispersion of 14% would still be at a safe 94% while a relative dispersion of 14% would be reaching 93%. Being too conservative (by leaving where it is) instead of the proposed HWM @ 42 TB cleans the cache too often and decrease the opportunity to recover files not yet on tape (6 TB worth).

However, the RCF want to keep the margin in case they do have a maintenance of the back-end tape system requiring hours of downtime (then the extra space would fill up). 6 TB worth at 1.2 Gb/sec is = 9 GB/mnts or 540 GB/hour so a ~ 11.4 hours downtime margin.