CMS Guide on how to calculate luminosity
What is this?
This document provides instruction for how to calculate the luminosity information
for CMS in the open data environment. This is done using brilcalc, which is a command line tool from the CMS Beam Radiation Instrumentation and Luminosity (BRIL) group.
Before you continue, consider that some of the information that you wish to calculate is already available for 2010, 2011, 2012, and 2015 data. The information available includes:
- the integrated luminosity for validated runs and luminosity sections
- the trigger prescales by run and luminosity section
- the luminosity by luminosity section for the list of validated runs and luminosity sections
Prerequisites
There are three options for using brilcalc: using a docker container with the brilcalc software installed, or installing brilcalc either in the CMS open data Virtual Machine (VM) or in the CMS open data docker container.
Brilcalc container
If you have docker installed, donwload and run the container with
docker run -it --name brilws gitlab-registry.cern.ch/cms-cloud/brilws-docker
The software is preinstalled in the container and you can directly test brilcalc.
Virtual machine
Follow the instructions here to install the CMS VM and then the instructions for getting started.
Click on the CMS Shell icon on the desktop to open the terminal. At the terminal prompt, run the command
cmsrel CMSSW_5_3_32
and then change directory
cd CMSSW_5_3_32/src
and then finally intialize the CMS environment
cmsenv
Now we are ready to install the brilcalc package.
Docker container
Following the instructions here we download and install an image and run a Docker container using the following command:
docker run --name my_od -P -p 5901:5901 -it cmsopendata/cmssw_5_3_32-slc6_amd64_gcc472 /bin/bash
When completed successfully we should be at the command prompt:
cmsusr@a17b2a79d067 ~/CMSSW_5_3_32/src $
Now we are ready to install the brilcalc package.
Install brilcalc
cvmfs
If you are using the VM, or if are using the docker container and have the cvmfs file system installed locally, you have access to cvmfs.
In these cases, follow these instructions:
-
Set your
PATH:export PATH=$HOME/.local/bin:/cvmfs/cms-bril.cern.ch/brilconda/bin:$PATH -
Install using
pipwith the command:pip install --user brilwsAt the end of a successful installation you should see the messages:
Successfully built brilws Installing collected packages: brilws Successfully installed brilws -
Check the installation by running the command:
brilcalc --versionwhich should output
3.6.2
Now you are ready to test brilcalc.
conda
Alternatively you can install a conda environment:
-
In the command prompt go to the home directory in your container with the command
cd -
Fetch the installer for the
brilcalcconda environment:wget https://cern.ch/cmslumisw/installers/linux-64/Brilconda-1.1.7-Linux-x86_64.sh(you may need to use the
--no-check-certificateoption withwget) -
Run the installer:
bash Brilconda-1.1.7-Linux-x86_64.sh -b -p <localbrilcondabase>where we substitute
<localbrilcondabase>withbrilconda. This is the directory in which thebrilcalctools will be installed. -
Once installed successfully add the tools to your
PATHwith the following command:export PATH=$HOME/.local/bin:$HOME/brilconda/bin:$PATH -
Then install
brilcalcwith this command:pip install brilws -
Finally check by running the following:
brilcalc --versionwhich should output
3.53
Now you are ready to test brilcalc.
Test brilcalc
As a test, let's see what the integrated luminosity was for run 160431 by running the command
brilcalc lumi -c web -r 160431
which should output
#Data tag : 19v3 , Norm tag: onlineresult
+-------------+-------------------+-----+------+----------------+----------------+
| run:fill | time | nls | ncms | delivered(/ub) | recorded(/ub) |
+-------------+-------------------+-----+------+----------------+----------------+
| 160431:1615 | 03/14/11 03:08:07 | 243 | 218 | 8781.334977241 | 7879.289611261 |
+-------------+-------------------+-----+------+----------------+----------------+
#Summary:
+-------+------+-----+------+-------------------+------------------+
| nfill | nrun | nls | ncms | totdelivered(/ub) | totrecorded(/ub) |
+-------+------+-----+------+-------------------+------------------+
| 1 | 1 | 243 | 218 | 8781.334977241 | 7879.289611261 |
+-------+------+-----+------+-------------------+------------------+
Note: It is important that you use the -c web option when running brilcalc. This specifies that you use indirect access to BRIL servers via web cache. For users of CMS open data outside CERN and CMS this is the only option that will work.
Note: There is a useful help option for brilcalc and its commands:
brilcalc --help
usage:
brilcalc (-h|--help|--version)
brilcalc [--debug|--warn] <command> [<args>...]
commands:
lumi Luminosity
beam Beam
trg Trigger configurations
See 'brilcalc <command> --help' for more information on a specific command.
Common queries
Integrated luminosity for a run
See test above
Integrated luminosity for validated runs and luminosity sections
First obtain the file with the list of validated runs and luminosity sections. Here we use the list for 2011 data.
wget http://opendata.cern.ch/record/1001/files/Cert_160404-180252_7TeV_ReRecoNov08_Collisions11_JSON.txt
It is recommended to steer the output to a file (e.g. called 2011lumi.txt) using this command:
brilcalc lumi -c web -i Cert_160404-180252_7TeV_ReRecoNov08_Collisions11_JSON.txt > 2011lumi.txt
The output will appear as follows:
#Data tag : v1 , Norm tag: onlineresult
+-------------+-------------------+------+------+----------------+---------------+
| run:fill | time | nls | ncms | delivered(/ub) | recorded(/ub) |
+-------------+-------------------+------+------+----------------+---------------+
| 160431:1615 | 03/14/11 03:14:43 | 200 | 200 | 7366.884 | 7366.875 |
| 160577:1622 | 03/16/11 01:44:38 | 53 | 53 | 1719.888 | 1631.386 |
| 160578:1622 | 03/16/11 02:20:03 | 175 | 175 | 5201.105 | 4952.155 |
| 160871:1636 | 03/19/11 02:43:57 | 141 | 141 | 109739.586 | 106354.581 |
| 160872:1636 | 03/19/11 03:50:05 | 38 | 38 | 28346.738 | 23977.109 |
| 160873:1636 | 03/19/11 04:20:20 | 147 | 147 | 104914.534 | 103031.004 |
The units are inverse microbarns as default and can be changed to inverse femtobarns with the option -u /fb. Note that the lists of validated runs are usually given for the full data-taking period, whereas the open data are a part of it. See the intructions below on how to get the values over a range of runs.
Select luminometer
CMS measures the luminosity with different luminometers (luminosity detectors) and algorithms. You can choose which to use with the --type option as below. Here we use the list of validated runs from 2012.
brilcalc lumi --type pxl -c web -i Cert_190456-208686_8TeV_22Jan2013ReReco_Collisions12_JSON.txt -u /fb > 2012lumi.txt
In this example the pixel detectors are used. This is the preferred option. For 2010 data a calculation using the hadronic forward calorimeters is used and is given by the option --type hfoc.
For Run-2 data, the luminometer giving the best value for each luminosity section is recorded in a 'normtag' file, provided in the 2015 luminosity information record ('normtag_PHYSICS_2015.json'). Use it with the corresponding list of validated runs from 2015 as in
brilcalc lumi -c web -i Cert_13TeV_16Dec2015ReReco_Collisions15_25ns_JSON_v2.txt -u /fb --normtag normtag_PHYSICS_2015.json > 2015lumi.txt
Note You may notice at the end of the output luminosity sections that are listed in the json quality file but do not have any luminosity values corresponding to them. These correspond to sections that are left-overs at the end of a run, which where still tagged as STABLE RUN, but actually did not provide any luminosity. These are safe to ignore as they do not contain any events.
Integrated luminosity for validated runs and luminosity sections over a range of runs
First obtain the file with the list of validated runs and luminosity sections. Here we use the list for 2011 data.
wget http://opendata.cern.ch/record/1001/files/Cert_160404-180252_7TeV_ReRecoNov08_Collisions11_JSON.txt
RunA of 2011 proton-proton data comprises runs 160431 to 173692 (inclusive) so to calculate the integrated luminosity for this era run the command:
brilcalc lumi -c web --begin 160431 --end 173692 -i Cert_160404-180252_7TeV_ReRecoNov08_Collisions11_JSON.txt > RunA2011lumi.txt
For 2015 data, obtain the corresponding list of validated runs:
wget https://opendata.cern.ch/record/14210/files/Cert_13TeV_16Dec2015ReReco_Collisions15_25ns_JSON_v2.txt
and the normtage file
wget https://opendata.cern.ch/record/1055/files/normtag_PHYSICS_2015.json
RunD of 2015 proton-proton data comprises runs 256630 to 260627 (inclusive) so to calculate the integrated luminosity for this era using the best luminometer estimates defined in normtag_PHYSICS_2015.json run the command:
brilcalc lumi -c web --begin 256630 --end 260627 -i Cert_13TeV_16Dec2015ReReco_Collisions15_25ns_JSON_v2.txt --normtag normtag_PHYSICS_2015.json > RunD2015lumi.txt
Integrated luminosity for validated runs and luminosity sections, separated by luminosity sections
If you want to fetch the integrated luminosity by luminosity section and output the results to a csv file (which is recommended), first obtain the file with the list of validated runs and luminosity sections. Here we use the list for 2011 data.
wget http://opendata.cern.ch/record/1001/files/Cert_160404-180252_7TeV_ReRecoNov08_Collisions11_JSON.txt
Then run the command:
brilcalc lumi --byls --output-style csv -c web -i Cert_160404-180252_7TeV_ReRecoNov08_Collisions11_JSON.txt > my2011lumibyls.csv
The contents of the csv file will appear as below:
#Data tag : v1 , Norm tag: None
#run:fill,ls,time,beamstatus,E(GeV),delivered(/ub),recorded(/ub),avgpu,source
160431:1615,19:19,03/14/11 03:14:43,STABLE BEAMS,3500,39.312,39.312,3.4,hfoc
160431:1615,20:20,03/14/11 03:15:07,STABLE BEAMS,3500,39.368,39.368,3.4,hfoc
160431:1615,21:21,03/14/11 03:15:30,STABLE BEAMS,3500,39.305,39.305,3.4,hfoc
160431:1615,22:22,03/14/11 03:15:53,STABLE BEAMS,3500,35.843,35.843,3.1,hfoc
160431:1615,23:23,03/14/11 03:16:17,STABLE BEAMS,3500,37.093,37.093,3.2,hfoc
160431:1615,24:24,03/14/11 03:16:40,STABLE BEAMS,3500,39.620,39.620,3.4,hfoc
160431:1615,25:25,03/14/11 03:17:03,STABLE BEAMS,3500,34.396,34.396,3.0,hfoc
160431:1615,26:26,03/14/11 03:17:27,STABLE BEAMS,3500,39.874,39.874,3.4,hfoc
Integrated luminosity for a HLT trigger path
Here as an example we calculate the total integrated luminosity for a particular for high-level triggers whose name matches the
pattern HLT_DoubleMu and output the results to a csv file:
brilcalc lumi -c web -r 173692 --hltpath="HLT_DoubleMu*" --output-style csv > run173692_DoubleMu.csv
Note: It is strongly recommended to make your HLT query as specific as possible and to output to csv.
Calculate trigger prescales for a run and trigger path
brilcalc trg --prescale -c web -r 148002 --hltpath "HLT_Jet*"
+--------+-------+----------+-------------+---------------------------------+-------+--------------------+
| run | cmsls | prescidx | totprescval | hltpath/prescval | logic | l1bit/prescval |
+--------+-------+----------+-------------+---------------------------------+-------+--------------------+
| 148002 | 1 | 1 | 12000 | HLT_Jet15U/20 | ONE | L1_SingleJet6U/600 |
| 148002 | 1 | 1 | 12000 | HLT_Jet15U_HcalNoiseFiltered/20 | ONE | L1_SingleJet6U/600 |
| 148002 | 1 | 1 | 1200 | HLT_Jet30U/1200 | ONE | L1_SingleJet20U/1 |
| 148002 | 1 | 1 | 170 | HLT_Jet50U/170 | ONE | L1_SingleJet30U/1 |
| 148002 | 1 | 1 | 30 | HLT_Jet70U_v2/30 | ONE | L1_SingleJet40U/1 |
| 148002 | 1 | 1 | 1 | HLT_Jet100U_v2/1 | ONE | L1_SingleJet60U/1 |
| 148002 | 1 | 1 | 1 | HLT_Jet140U_v1/1 | ONE | L1_SingleJet60U/1 |
+--------+-------+----------+-------------+---------------------------------+-------+--------------------+
Change of prescales over a run
brilcalc trg --prescale -c web -r 173243
+--------+-------+----------+
| run | cmsls | prescidx |
+--------+-------+----------+
| 173243 | 1 | 5 |
| 173243 | 14 | 4 |
+--------+-------+----------+
