For each of these cases, we use the dierent LLM-based threat elicitation tools and collect their outputs. We furthermore explore the eect of using dierent LLMs, a mix of local and hosted models, and in total process 56 distinct threat models generated by automated tools.

We compare these outputs against a set of ground truth baselines, created by human threat modelers. In these baselines, we further distinguish between threats identied by novices and experts, and we further take into account the amount of agreement among the human threat modelers about the identied threats as a weight or penalty factor. This allows for in-depth evaluation of the generated results against human threat analysis.

We use NLP technology –sentence transformers– to perform threat mapping, i.e. to decide whether a generated threat indeed refers to the same issue as one of the threats documented by the human threat modelers. More specically, the attack-BET sentence transformer is used, which was trained specically on security terminology, and not surprisingly proved most capable for this purpose. We calculate semantic similarity, which expresses the extent to which two threat descriptions refer to the same core issue, regardless of phrasing. Based on that, we quantify performance, expressed in precision, recall, F1-score.