Convert string-like-arrays to arrays

Store arrays in model parameters as arrays and not as strings (see simtools issue).

Conversion

Use this script to convert all string-type arrays to really arrays:

import json
import pathlib
import numpy as np

from simtools.utils.general import convert_string_to_list

def find_json_files(directory):
    """Find all JSON files in a directory and subdirectories."""
    return list(pathlib.Path(directory).rglob("*.json"))

def read_json(file_path):
    """Read a JSON file into a dictionary."""
    with open(file_path, "r", encoding="utf-8") as f:
        return json.load(f)

def check_value_type(data):
    """Check if the 'value' entry is a string."""
    return isinstance(data.get("value"), str)

def check_conversion(value):
    """Check if general.convert_string_to_list() returns a list."""
    result = convert_string_to_list(value)
    return isinstance(result, (list, np.ndarray))

def main(directory):
    json_files = find_json_files(directory)
    
    for file in json_files:
        data = read_json(file)
        
        if check_value_type(data):
            
            if check_conversion(data["value"]):
                print(f"{file}: CONVERT {data['value']} -> {convert_string_to_list(data['value'])}")

                data["value"] = convert_string_to_list(data["value"])
                with open(file, "w", encoding="utf-8") as f:
                    json.dump(data, f, indent=4,  sort_keys=False)
            else:
                print(f"{file}: NO_CONVERT {data['value']} -> {convert_string_to_list(data['value'])}")

if __name__ == "__main__":
    main("../simulation-models/simulation-models/model_parameters") 

Tests

Uploaded this branch to a local model data base and run all unit and integration tests successfully against them.

If the reviewer wants to test this:

use the following .env settings in your dev container:

# Environmental variables
SIMTOOLS_DB_API_PORT=27017 #Port on the MongoDB server
SIMTOOLS_DB_SERVER='simtools-mongodb'
SIMTOOLS_DB_API_USER='api' # username for MongoDB
SIMTOOLS_DB_API_PW='password' # Password for MongoDB
SIMTOOLS_DB_API_AUTHENTICATION_DATABASE='admin'
SIMTOOLS_DB_SIMULATION_MODEL='CTAO-Simulation-ModelParameters-v0-6-0'
SIMTOOLS_SIMTEL_PATH='/workdir/sim_telarray'

Start the container with (assuming you have setup the local DB with the scripts in the database_scripts directory):

docker  run --rm -it -v "$(pwd)/:/workdir/external" --network simtools-mongo-network ghcr.io/gammasim/simtools-dev:latest bash -c "source /workdir/env/bin/activate && cd /workdir/external/simtools && pip install -e . && bash"

go to simtools/database_scripts/

./upload_from_model_repository_to_db.sh CTAO-Simulation-ModelParameters-v0-6-0

After that, just run unit and integration tests as usual.

assigned to @GernotMaier

marked this merge request as ready

changed the description

Attach additional the printout from converting the entries.convert.log

This all looks good. Should I approve already or are you still checking things?

This is ready to be reviewed.

requested review from @ogueta

approved this merge request

merged

mentioned in commit e5d4a0c5