CTADIRAC issueshttps://gitlab.cta-observatory.org/cta-computing/dpps/workload/CTADIRAC/-/issues2024-03-18T11:27:18+01:00https://gitlab.cta-observatory.org/cta-computing/dpps/workload/CTADIRAC/-/issues/104doc: Add note on Opensearch configuration2024-03-18T11:27:18+01:00Natthan.Pigouxdoc: Add note on Opensearch configuration* Minimal permissions on the certificates/index prefix:
* indices:data/write/update
* indices:admin/get
* indices:admin/aliases/get
* indices:admin/mappings/get
* cluster:monitor/main
* DIRAC Configuration:
* in dirac.cfg, conf...* Minimal permissions on the certificates/index prefix:
* indices:data/write/update
* indices:admin/get
* indices:admin/aliases/get
* indices:admin/mappings/get
* cluster:monitor/main
* DIRAC Configuration:
* in dirac.cfg, configure the Opensearch/Elastic client:
```
Services
{
Monitoring
{
CTADIRAC-alma
{
Databases
{
MonitoringDB
{
Host = opensearch-api
Port = 9200
SSL = False
IndexPrefix = index-prefix
CRT = True
ca_certs = /path/to/ca
client_cert = /path/to/cert
client_key =/path/to/cert-key
}
}
}
}
WorkloadManagement
{
CTADIRAC-alma
{
Databases
{
ElasticJobParametersDB
{
Host = opensearch-api
Port = 9200
SSL = False
IndexPrefix = index-prefix
CRT = True
ca_certs = /path/to/ca
client_cert = /path/to/cert
client_key =/path/to/cert-key
}
}
}
}
}
```
* Operations/\<Setup\>
* MonitoringBackends
* WMSHistory = Monitoring
* DataOperation = Monitoring
* PilotsHistory = Monitoring
* PilotSubmissionMonitoring = Monitoring
* AgentMonitoring = Monitoring
* ServiceMonitoring = Monitoring
* RMSMonitoring = Monitoring
* Services/JobMonitoring/useESForJobParametersFlag = True
* Restart the services JobStateJobMonitoring, WMSAdministrator, StatesAccountingAgent, etc
* See: index prefix fix in DIRAC for ElasticJobParametersDBNatthan.PigouxNatthan.Pigouxhttps://gitlab.cta-observatory.org/cta-computing/dpps/workload/CTADIRAC/-/issues/103Improve cta-job-get-parameters2024-03-08T10:10:50+01:00Natthan.PigouxImprove cta-job-get-parameters* [ ] : Show allowed `FIELDS`
* [ ] : --stat should be set by default* [ ] : Show allowed `FIELDS`
* [ ] : --stat should be set by defaultNatthan.PigouxNatthan.Pigouxhttps://gitlab.cta-observatory.org/cta-computing/dpps/workload/CTADIRAC/-/issues/102Mismatched versions in pyproject.toml and on pypi2024-02-28T12:12:53+01:00Maximilian LinhoffMismatched versions in pyproject.toml and on pypiThe pyproject.toml in the main branch here has version 2.2.37 yet there is a version 2.2.39 on Pypi: https://pypi.org/project/CTADIRAC/
How can this be?The pyproject.toml in the main branch here has version 2.2.37 yet there is a version 2.2.39 on Pypi: https://pypi.org/project/CTADIRAC/
How can this be?https://gitlab.cta-observatory.org/cta-computing/dpps/workload/CTADIRAC/-/issues/101Test integration tests with docker compose2024-02-14T17:18:30+01:00Luisa ArrabitoTest integration tests with docker composeTest the script available in DIRAC to run integration tests either locally in a development environment either in the CI:
https://github.com/DIRACGrid/DIRAC/blob/integration/integration_tests.py
The goal is to develop some integration t...Test the script available in DIRAC to run integration tests either locally in a development environment either in the CI:
https://github.com/DIRACGrid/DIRAC/blob/integration/integration_tests.py
The goal is to develop some integration tests also for the CTADIRAC extension and include them in the CI.https://gitlab.cta-observatory.org/cta-computing/dpps/workload/CTADIRAC/-/issues/99Develop scripts to submit calibpipe CWL workflows2024-03-22T15:03:14+01:00Luisa ArrabitoDevelop scripts to submit calibpipe CWL workflowshttps://gitlab.cta-observatory.org/cta-computing/dpps/workload/CTADIRAC/-/issues/98Develop generic scripts to submit CWL workflows2024-01-25T16:06:02+01:00Luisa ArrabitoDevelop generic scripts to submit CWL workflowshttps://gitlab.cta-observatory.org/cta-computing/dpps/workload/CTADIRAC/-/issues/91__init__.py in base directory of repository2023-12-08T13:33:51+01:00Maximilian Linhoff__init__.py in base directory of repositoryThis file is either at the wrong place or obsolete. It certainly is not part of the installed python package (in src/CTADIRAC)This file is either at the wrong place or obsolete. It certainly is not part of the installed python package (in src/CTADIRAC)https://gitlab.cta-observatory.org/cta-computing/dpps/workload/CTADIRAC/-/issues/89Check the way LHCb does the system test in the CI2024-01-09T14:56:21+01:00Natthan.PigouxCheck the way LHCb does the system test in the CIhttps://gitlab.cern.ch/lhcb-dirac/LHCbDIRAC/-/blob/master/.gitlab-ci.yml#L304-324https://gitlab.cern.ch/lhcb-dirac/LHCbDIRAC/-/blob/master/.gitlab-ci.yml#L304-324Natthan.PigouxNatthan.Pigouxhttps://gitlab.cta-observatory.org/cta-computing/dpps/workload/CTADIRAC/-/issues/86Test to create a report using ES API2023-10-09T11:04:42+02:00Natthan.PigouxTest to create a report using ES APIhttps://gitlab.cta-observatory.org/cta-computing/dpps/workload/CTADIRAC/-/issues/85Add quality check in CtaApplyModelsJob2023-10-06T13:34:05+02:00Luisa ArrabitoAdd quality check in CtaApplyModelsJobWe should add a quality check in the CtaApplyModelsJob to make sure that only 1 model file is downloaded as a result of cta-prod-get-file-by-query.We should add a quality check in the CtaApplyModelsJob to make sure that only 1 model file is downloaded as a result of cta-prod-get-file-by-query.https://gitlab.cta-observatory.org/cta-computing/dpps/workload/CTADIRAC/-/issues/84How to Update the TransformationDB if not ? 2024-03-22T10:52:20+01:00Natthan.PigouxHow to Update the TransformationDB if not ? Natthan.PigouxNatthan.Pigouxhttps://gitlab.cta-observatory.org/cta-computing/dpps/workload/CTADIRAC/-/issues/83Let time to transformation before flushing2023-10-09T15:31:49+02:00Natthan.PigouxLet time to transformation before flushingNatthan.PigouxNatthan.Pigouxhttps://gitlab.cta-observatory.org/cta-computing/dpps/workload/CTADIRAC/-/issues/68Bug fix in cta-prod-create-dataset-description2023-09-07T08:36:31+02:00Luisa ArrabitoBug fix in cta-prod-create-dataset-descriptionFix `cta-prod-create-dataset-description` (see below). Not sure why it happens:
```
$ cta-prod-create-dataset-description --MCCampaign=PROD5 --site=Paranal --array_layout=AlphaNectarcam --az=180 --zen=20.0 --ctapipe_ver=v0.19.2 --nsb=1
...Fix `cta-prod-create-dataset-description` (see below). Not sure why it happens:
```
$ cta-prod-create-dataset-description --MCCampaign=PROD5 --site=Paranal --array_layout=AlphaNectarcam --az=180 --zen=20.0 --ctapipe_ver=v0.19.2 --nsb=1
Traceback (most recent call last):
File "/Users/arrabito/ctadirac_client/diracos/bin/cta-prod-create-dataset-description", line 8, in <module>
sys.exit(main())
File "/Users/arrabito/ctadirac_client/diracos/lib/python3.9/site-packages/DIRAC/Core/Base/Script.py", line 55, in __call__
raise NotImplementedError("Something is very wrong")
NotImplementedError: Something is very wrong
```https://gitlab.cta-observatory.org/cta-computing/dpps/workload/CTADIRAC/-/issues/67Fix the output of cta-prod-show-dataset2023-09-06T10:40:35+02:00Luisa ArrabitoFix the output of cta-prod-show-datasetFix the `cta-prod-show-dataset` output for the case where the dataset does not exist.
Currently we get:
```
$ cta-prod-show-dataset Prod6_Paranal_Prod6-Hyperarray_NSB1x_gamma-diffuse_North_20deg_0.008div_R1_test
Available datasets: 1...Fix the `cta-prod-show-dataset` output for the case where the dataset does not exist.
Currently we get:
```
$ cta-prod-show-dataset Prod6_Paranal_Prod6-Hyperarray_NSB1x_gamma-diffuse_North_20deg_0.008div_R1_test
Available datasets: 1
Prod6_Paranal_Prod6-Hyperarray_NSB1x_gamma-diffuse_North_20deg_0.008div_R1_test
Storage usage for all datasets
|_. Name |_. N files |_. Size(TB) |
Traceback (most recent call last):
File "/Users/arrabito/ctadirac_client/diracos/bin/cta-prod-show-dataset", line 8, in <module>
sys.exit(main())
File "/Users/arrabito/ctadirac_client/diracos/lib/python3.9/site-packages/DIRAC/Core/Base/Script.py", line 74, in __call__
return entrypointFunc._func()
File "/Users/arrabito/PycharmProjects/CTADIRAC/src/CTADIRAC/Core/scripts/cta_prod_show_dataset.py", line 226, in main
name, n_files, size, mq = get_dataset_info(dataset_name)
File "/Users/arrabito/PycharmProjects/CTADIRAC/src/CTADIRAC/Core/scripts/cta_prod_show_dataset.py", line 49, in get_dataset_info
res = dataset_dict['Successful'][dataset_name][dataset_name]
KeyError: 'Prod6_Paranal_Prod6-Hyperarray_NSB1x_gamma-diffuse_North_20deg_0.008div_R1_test'
```https://gitlab.cta-observatory.org/cta-computing/dpps/workload/CTADIRAC/-/issues/66Correct the output of cta-transformation-get-file2023-09-06T10:38:14+02:00Luisa ArrabitoCorrect the output of cta-transformation-get-fileThe output of the cta-transformation-get-files is sligthly wrong, e.g.:
```
$ cta-transformation-get-files transID.list --FileStatus=Assigned --dumpFull
Reading transformation IDs from input file: transID.list
Found 7 transformations
...The output of the cta-transformation-get-files is sligthly wrong, e.g.:
```
$ cta-transformation-get-files transID.list --FileStatus=Assigned --dumpFull
Reading transformation IDs from input file: transID.list
Found 7 transformations
100 files selected for transformation 4243 with status Assigned
No files selected for transformation 4241 with status Assigned
0 files dumped in transAssignedFiles_NoneTasks_4243-4241.transID_lfns
```
We should correct the last line of the output to give the total nb of files found for all the transformations and not just the nb for the last one.https://gitlab.cta-observatory.org/cta-computing/dpps/workload/CTADIRAC/-/issues/65Allow to run on multiple cores for CTA training jobs2023-09-06T10:30:12+02:00Luisa ArrabitoAllow to run on multiple cores for CTA training jobsIn 2.20.21 we have added :
`self.setNumberOfProcessors(minNumberOfProcessors=32)`
for the 2 Cta Training jobs.
This is needed for these special jobs which take great benefit by running on multiple cores.
We could eventually allow the...In 2.20.21 we have added :
`self.setNumberOfProcessors(minNumberOfProcessors=32)`
for the 2 Cta Training jobs.
This is needed for these special jobs which take great benefit by running on multiple cores.
We could eventually allow the possibility to specify the nb of cores in the yml interface of cta-prod-submit.
Then, we should also check how to configure DIRAC so that a given site can both run jobs with 1 core or multiple cores requirements.
Some tests should be done to find the correct configuration.
For now I've only checked that setting:
`NumberOfProcessors = 32`
in the CE configuration makes sure that jobs with:
`self.setNumberOfProcessors(minNumberOfProcessors=32)`
match and run on multiple cores.https://gitlab.cta-observatory.org/cta-computing/dpps/workload/CTADIRAC/-/issues/64Add a check in CtaApplyModel jobs2023-09-06T10:21:03+02:00Luisa ArrabitoAdd a check in CtaApplyModel jobsAdd a check after the download energy step of the CtaApplyModel jobs to make sure that only 1 file will be be downloaded.Add a check after the download energy step of the CtaApplyModel jobs to make sure that only 1 file will be be downloaded.https://gitlab.cta-observatory.org/cta-computing/dpps/workload/CTADIRAC/-/issues/63Simplify output of dirac-prod-get-all2023-09-06T10:17:49+02:00Luisa ArrabitoSimplify output of dirac-prod-get-allThe output of dirac-prod-get-all is a little bit long, e.g.:
```
399 Prod6_Paranal_div0.01135_rest_merge Active 627 2023-08-11 17:19:33 2023-08-11 17:20:53 /O=GRID-FR/C=FR/O=CNRS/OU=LUPM/CN=Lui...The output of dirac-prod-get-all is a little bit long, e.g.:
```
399 Prod6_Paranal_div0.01135_rest_merge Active 627 2023-08-11 17:19:33 2023-08-11 17:20:53 /O=GRID-FR/C=FR/O=CNRS/OU=LUPM/CN=Luisa Arrabito cta_prod
400 Prod6_Paranal_div0.01453_proton_merge Active 628 2023-08-11 17:22:43 2023-08-11 17:23:44 /O=GRID-FR/C=FR/O=CNRS/OU=LUPM/CN=Luisa Arrabito cta_prod
```
Probably a shorter output would be more readable, e.g.:
```
Prod6_Paranal_div0.01135_rest_merge Active 627 2023-08-11 17:19:33
Prod6_Paranal_div0.01453_proton_merge Active 628 2023-08-11 17:22:43
```https://gitlab.cta-observatory.org/cta-computing/dpps/workload/CTADIRAC/-/issues/62Send the report by mail: Make it works on majestix-vm2023-09-14T11:07:50+02:00Natthan.PigouxSend the report by mail: Make it works on majestix-vmNatthan.PigouxNatthan.Pigouxhttps://gitlab.cta-observatory.org/cta-computing/dpps/workload/CTADIRAC/-/issues/57Improve Transformations failure management2024-03-22T10:52:20+01:00Natthan.PigouxImprove Transformations failure managementImprove Transformations failure management.Improve Transformations failure management.Natthan.PigouxNatthan.Pigoux