Test fefs performances on worker nodes prepared for kubernetes

Context:

The possibility to use kubernetes with the current north site cluster of Tokyo University is been evaluated. It can not be easily installed because the worker nodes need to have access to the shared file system, and there are no drivers for fefs on alma9. As a result, there are 2 possibilities that are investigated by the DPPS team (also requiring k8s to be available onsite):

• use an outdated, unsupported kubernetes version on centos7 nodes, test on cp50
  • give access to DPPS team https://gitlab.cta-observatory.org/cta-computing/on-site-ict/on-site-ict-datacentre/onsite-ict-dpps/on-site-ict-dpps-documentation/-/work_items/1
  • install old kubernetes https://gitlab.cta-observatory.org/cta-computing/on-site-ict/on-site-ict-datacentre/onsite-ict-dpps/on-site-ict-dpps-documentation/-/issues/2
    • troubles with installing kubernetes with regular ansible configuration
    • settled for KinD which is not a suitable solution for SAG (handling multiple machines on multiple clusters would add a lot of complexity)
  • nothing to test, regular fefs speed are expected since usual setup with fefs CENTOS 7 drivers.
• upgrade the nodes to alma9 and use NFS v3 mounting to write/read data to /fefs/, test on cp46
  • setting up the alma9 node https://gitlab.cta-observatory.org/cta-computing/on-site-ict/on-site-ict-datacentre/onsite-ict-dpps/on-site-ict-dpps-documentation/-/issues/3
  • set-up kubernetes on the node
  • test fefs performances
    • LST tests
    • SAG tests: tracked in this issue
    • DPPS tests

SAG tests are tracked in this issue

• Test complete RTA software with streamers running on the same node (localhost)
• Test complete RTA with streamers running on nodes, infiniband network
• Test complete RTA while DPPS BDMS is running on the same nodes

FYI @sami.caroff