adding to the solution suggested by @itsamemarkus, I also set the :ref/default-open-blocks-level to 1 in my config.edn. This way the child blocks of the linked references are expanded to only 1 level (reference).
Below are the 2 settings that I have changed:
;; This limits the number of linked refrences visible in the linked reference
ref/linked-references-collapsed-threshold 5
;; This limits the expansion level of the child blocks of the linked references
:ref/default-open-blocks-level 1