Exploring over a decade of systems engineering research center: A community detection and text analytics approach - Zavala - Systems Engineering - Wiley Online Library

incose.onlinelibrary.wiley.com

Araceli Zavala

Summary

This document is a research article that analyzes over a decade of systems engineering research conducted by the Systems Engineering Research Center (SERC). The key points from the article are:

The authors used text mining and network analysis techniques on 248 technical reports published by SERC between 2009-2023 to identify research streams, topics, evolution of systems engineering research, and key authors.
Four main research streams were identified across the years, focusing on areas like

enterprises and systems of systems,
trusted systems,
systems engineering and management transformation, and
human capital development.

Seven key research topics emerged:

technology collaboration,
systems engineering hub,
innovative design,
synergy models,
integrated systems,
applications, and
cognitive technology.

The evolution of systems engineering research showed the emergence of concepts like

system of systems,
complex systems,
cost and lifecycle considerations,
digital engineering,
AI, machine learning etc. over the years.

The prominent SERC authors with the most publications and citations were identified, along with visualizations of the co-authorship networks.

Overall, the article provides insights into the diverse and evolving nature of systems engineering research within SERC over the past decade, the key research themes pursued, and the important researchers driving the field forward. The methodology showcases how text analytics can be used to map the landscape of a research field.

Exemplars of SE Practice

The article mentions several specific systems engineering research examples to illustrate the key topics and trends identified. Some of these include:

A study by DeLaurentis [11] in 2011 that quantified the impact of attacks on communications in a system-of-systems (SoS) context, examining the behavior and interactions of constituent systems under cyberattack.
Research by Agarwal et al. [29] on the dynamic planning of incorporating new systems and upgrading existing systems over time under threats, limited budget, and uncertainty - exemplifying work on SoS.
A methodology proposed in 2014-2015 for a structured approach to recognizing and assessing ship capability portfolios during product design stages, touching on cost, schedule and lifecycle considerations [30, 31].
The Helix project (2017-2021), a multi-year research effort focused on understanding what makes systems engineers effective [32, 33], resulting in the development of Atlas 1.1, a theory of effective systems engineers [17].
Research supporting the Department of Defense's digital engineering transformation, creating a roadmap for integrating traditional and learning systems to enable automation of systems engineering practices [34, 35].
An analysis by Cody and Beling (2023) [37] presenting a transfer learning framework from a systems theoretic perspective, covering aspects like homogeneous and heterogeneous transfer, domain adaptation, and negative transfer.
Work by Carter et al. (2019) [38] presenting methodologies for addressing cybersecurity early in a system's lifecycle.
Research identifying factors in the evolution of Green Supply Chain frameworks inside and outside the aviation industry [39].
Tregubov and Lane's [40] research about Kanban-based scheduling system workflow in multilevel systems, illustrating the importance of resilient systems and constituent systems in 2013.

These examples span a range of systems engineering applications and methodological developments, from defense and aerospace to supply chain and digital engineering, illustrating the diversity of SERC's research portfolio over the years.

MBSE

The article discusses Model-Based Systems Engineering (MBSE) as one of the key research areas within the Systems Engineering and Systems Management Transformation (SEMT) theme of SERC. Specifically, it mentions:

"Regarding SEMT, it manages the decision-making capabilities and determines how and when to apply different strategies for an agile response to complex systems. SERC has focused on advancing the practice of Model-Based Systems Engineering (MBSE), which involves using models to represent complex systems throughout their life cycle. Their research has developed methodologies, frameworks, and tools to support effective MBSE implementation [13]. This includes work on model integration, model verification and validation, and model-driven decision-making [14]."

Two specific examples of MBSE-related research are cited:

1. Dunbar et al. [15] illustrate the SEMT area by presenting a digital engineering framework for integrating semantic web technologies into engineering design and analysis tasks.

2. Another example of SEMT is the proposed framework for selecting metrics for implementing MBSE followed by digital transformation [16].

These examples highlight SERC's efforts in advancing MBSE practices, particularly in developing frameworks and methodologies for model integration, verification, validation, and decision-making. The research aims to support the effective implementation of MBSE in complex systems engineering projects across various domains.

However, the article does not provide a comprehensive review of all MBSE-related research within SERC. The examples mentioned serve to illustrate the broader SEMT research theme and its relevance to MBSE.

Future Directions - Addressing Complexity, AI & LLM

The article suggests that systems engineering research and practice are evolving to address the increasing complexity of systems and the need for more agile, data-driven approaches. It highlights several emerging trends and future directions:

1. Incorporation of AI and machine learning: The article notes the introduction of terms like "AI," "machine learning," and "deep learning" in recent years (2014-2018), indicating a growing interest in applying these techniques to systems engineering problems. The authors suggest that future research should explore how AI can be more effectively involved in complex systems.
2. Digital engineering transformation: The article discusses research supporting the DoD's digital engineering transformation, aiming to integrate traditional and learning systems to enable automation of systems engineering practices [34, 35]. This suggests a trend towards more digitalized and automated approaches in SE.
3. Resilience and adaptability: The article highlights research on resilient systems, from supply chains to constituent systems, as a way to cope with increasing complexity. This indicates a focus on designing systems that can withstand disruptions and adapt to changing conditions.
4. Human factors and cognitive technology: The emergence of the "Cognitive Technology" community in the topic analysis suggests a growing interest in technologies that mimic human cognitive processes and can interpret complex, unstructured data. This points to a future where SE increasingly incorporates human factors and cognitive considerations.

Regarding the role of Large Language Models (LLMs) and AI, the article does not explicitly discuss this. However, given the increasing importance of AI and machine learning in SE research, it is plausible that LLMs could play a significant role in future SE practice, such as:

Automating documentation and knowledge management: LLMs could be used to generate and maintain systems engineering documentation, capture and share knowledge, and facilitate collaboration among distributed teams.
Assisting in requirements analysis and design synthesis: LLMs could help process and analyze large amounts of unstructured data (e.g., stakeholder interviews, legacy documents) to identify requirements and generate design concepts.
Supporting model-based systems engineering: LLMs could be integrated with MBSE tools to enable natural language interactions with system models, automate model documentation, and facilitate model-based collaboration.

However, the specific roles and impact of LLMs in future SE practice would require further targeted research and experimentation, beyond the scope of this particular article.

Authors

Araceli Zavala orcid.org/0000-0002-8011-7561 Professor School of Engineering and Sciences, Tecnologico de Monterrey, Guadalajara, Mexico [Google Scholar]
Dinesh Verma orcid.org/0009-0009-3531-0280 Professor School of Systems and Enterprises, Stevens Institute of Technology Hoboken, Hoboken, New Jersey, USA, Executive Director of SERC.
Jose E. Ramirez Marquez jmarquez@stevens.edu orcid.org/0000-0002-0965-1446 Professor School of Systems and Enterprises, Stevens Institute of Technology Hoboken, Hoboken, New Jersey, USA [IEEE Explore]

Correspondence

Jose E. Ramirez Marquez, Professor School of Systems and Enterprises, Stevens Institute of Technology Hoboken, Hoboken, NJ 07030, USA.

SE Metrics

The article uses several bibliometric indicators to identify key systems engineering researchers within the SERC community:

1. Number of technical reports (TRs): The authors rank researchers based on the number of SERC technical reports they have authored or co-authored. Table 1 lists the researchers with more than 9 TRs, with Dr. Dinesh Verma, Dr. Gary Witus, Dr. Jon Wade, and Dr. Nicole Hutchison topping the list with 25 or more TRs each.
2. h-index: The h-index is a widely used metric that quantifies a researcher's productivity and citation impact. It is defined as the maximum value of h such that the researcher has published h papers that have each been cited at least h times. The authors report the h-index for each of the top contributors, with Dr. Barry Boehm having the highest h-index of 51.
3. Number of citing documents: This metric represents the total number of publications that have cited a given researcher's work. It indicates the overall impact and influence of a researcher's output. Dr. Barry Boehm again tops this list with 12,823 citing documents.
4. Co-authorship network centrality: While not explicitly quantified, the authors use network analysis to visualize the co-authorship patterns among SERC researchers. The size of a researcher's node in the network corresponds to their number of TRs, while the thickness of the edges represents the frequency of collaboration between researchers. Central nodes in this network are considered important contributors and influencers within the SERC community.

The authors use these metrics in combination to identify the most productive, impactful, and collaborative researchers within the SERC systems engineering community. However, they acknowledge that these metrics are based solely on SERC technical reports and do not capture the researchers' full body of work or impact beyond SERC.

It's worth noting that the article does not delve into the limitations or potential biases of these metrics, such as the exclusion of other types of publications (e.g., journal articles, conference papers) or the skewed distribution of citations across fields and over time. A more comprehensive evaluation of researcher impact might require additional metrics and contextual information.

1 INTRODUCTION

Over the last decade, more than 20 universities have nurtured Systems Engineering (SE) research within the Systems Engineering Research Center (SERC). SERC is a University-Affiliated Research Center (UARC) sponsored by the U.S. Department of Defense (DoD) that has brought faculty, researchers, and students together to work on SE research across different domains of functional interest.¹ SERC conducts long-term research that makes significant progress in the SE research areas by focusing on collaboration and educating future engineers and system engineers.² Its research is broadly implemented in DoD and industry bases with a high impact on large-scale acquisitions of complex systems and solutions. Based on this, our research aims to understand the evolution of SE based on SERC-related Technical Reports (TR). By analyzing these reports, we gain valuable insights into the developments, trends, or discoveries within the SE domain. For the valuable information that we can gain from TRs, SERC was selected for the study. Also, while SERC researchers developed the concept of SE Body of Knowledge and implemented it via the engagement of the global systems research and education community, it is today one of three organizations that provide governance and sponsoring support. The other two organizations are the International Council on Systems Engineering (INCOSE) and the IEEE Systems Council (IEEE-SYSC). This is further described in ref. [3]. The SERC plays a crucial role in forging collaborations among academia, government, and industry to address SE challenges research. This involves fostering collaboration among SE researchers, end-users, and SE research organizations. Additionally, the SERC transfers its research findings to educators and practitioners. After 15 years of running projects and research with its collaborators and maturing its positioning within the national security ecosystem, SERC now has annual research expenditures in excess of $25 million.⁴

In the last 15 years (roughly 2009 to early 2023), SERC has developed 660 documents related to different projects in the SE domain. For example, among the documents, there are 300 presentations, 248 TR, 89 conference papers, 31 videos, 10 journal articles, one book, and other documents. SERC findings have transitioned into courses across the SERC universities and beyond.² Due to the importance of how SE is involved throughout the life cycle of a system, from its conceptual design, manufacturing, deployment, use, and disposal, we conducted a meta-analysis of the research conducted within the SERC using a text analytics approach. By developing a Natural Language Process (NLP) Analysis framework, this article aims to understand the evolution of the research areas and applications in SE research thus far and perhaps unveil directions of new techniques and new topics. The input to this meta-analysis framework constitutes all the TRs that researchers have generated while conducting research within the SERC. These TRs contain information regarding applications, methods, and techniques to address specific SE problems.

The main reason for selecting TRs is that every project under SERC support delivers a report of the findings in a formal document that provides detailed information and analysis of the conducted research. TRs might have written records of experiments or the results of a scientific project. We know that the research is usually published in journals or conference proceedings; however, it is difficult to track all of the research related to a specific project, and not all are on the SERC web page.

Using text mining and network analysis techniques, our methodology proposes to analyze the information obtained from TRs to infer research streams, topics, and evolution in SE research. We define a research stream as a series of related papers on one topic, each progressing to dig deeper.⁵ The main contributions of this article are as follows: (1) identifying research streams over the course of a decade, (2) classifying research topics, (3) unveiling insights on how SE research has evolved, and (4) identifying authors with the most collaborations and the most important authors for SE within SERC over this timeframe.

The remainder of this article is organized as follows: Section 2 briefly describes the SE discipline. Section 3 presents the research questions. Section 4 explains the text mining and network analysis methodology to find research orientations, topics, the evolution of SE research, and paramount authors. Section 5 presents the proposed methodology's empirical results and visualizations. We discuss conclusions and future work in Section 6.

2 BACKGROUND

This section summarizes the four thematic areas of SERC presented in the 2020 annual report and gives examples of existing literature. Starting with the SE definition and its key aspects. SE is a multidisciplinary field created to deal with the intricacies of complex systems. According to INCOSE, SE can be defined as “a transdisciplinary and integrative approach to enable the successful realization, use, and retirement of engineered systems, using systems principles and concepts, and scientific, technological, and management methods”.⁶ SERC has investigated the challenges and opportunities associated with systems thinking and the management of complex systems-of-systems (SoS). Its research has examined system integration, interoperability, scalability, and adaptability issues. By developing frameworks and methodologies, SERC aims to enable effective management and optimization of large-scale complex systems and SoS. Hossain and Jaradat⁷ studied the multiple definitions for SE that can be represented as: Holistic Approach, Interdisciplinary Approach, Requirement Driven, Integration & Design, Life Cycle, Iterative Process, and Management Oriented.

The key aspects of SE can be listed as follows according to INCOSE⁶:

Integrating stakeholders' goals and needs,
Establishing life cycle, process, and governance structures,
Generating concepts and architectures,
Base lining, modeling, and evaluating the architecture,
Identifying roles and relationships between system parts, and
Verifying and validating the system parts.

There are a few highlights of the extensive research conducted by the SERC over a decade. Their efforts have significantly contributed to advancing the field of SE and have practical implications for the design, development, and operation of complex systems across various domains. SERC considers a research strategy that identifies four research thematic areas: Enterprises and Systems of Systems (ESOS), Trusted Systems (TS), SE and Systems Management Transformation (SEMT), and Human Capital Development (HCD).¹ Having defined the importance of SERC's research to SE, we will briefly describe each thematic area and present some related research.

ESOS focuses on understanding the relationship of large-scale, complex systems composed of diverse components such as people, processes, technologies, and resources (smaller systems). However, all these components work together to create a more complex and capable system. An example is the study conducted by Dahmann and DeLaurentis,⁸ which addressed questions about SoS and their problems for analysis, architecting, and engineering. Resilience has been another area of study to make informed decisions about SoS, regarding system design Watson et al.,⁹ recovery strategies Uday and Marais,¹⁰ and more.

As far as TS is concerned, it provides ways to develop sustainable systems that are safe, secure, adaptable, and survivable. SERC has been at the forefront in the area of cybersecurity and resilience. They have explored approaches to design secure and resilient systems, incorporating considerations for system architecture, threat modeling, risk assessment, and mitigation strategies. This research area aims to enhance the ability of systems to withstand cyber threats and recover from disruptions.¹¹ This is exemplified by the tool cybersecurity body of knowledge (CYBOK), which can find associated attack vectors given a system model.¹²

Regarding SEMT, it manages the decision-making capabilities and determines how and when to apply different strategies for an agile response to complex systems. SERC has focused on advancing the practice of Model-Based Systems Engineering (MBSE), which involves using models to represent complex systems throughout their life cycle. Their research has developed methodologies, frameworks, and tools to support effective MBSE implementation.¹³ This includes work on model integration, model verification and validation, and model-driven decision-making.¹⁴ For example, Dunbar et al.¹⁵ illustrates this area clearly by presenting a digital engineering framework for integrating semantic web technologies into engineering design and analysis tasks. Another example of what SEMT means is the proposed framework for selecting metrics for implementing MBSE followed by digital transformation.¹⁶

Finally, HCD centers on ensuring not only the quality and quantity of systems engineers but also the critical knowledge and skills of the workforce and how to pass on SE knowledge. This is evident in the case of the Helix project,¹⁷ in which they investigate what makes systems engineers effective and why. In addition, they found out what makes organizations effective at SE.

Awareness of SE trends is not recent; an example is the study of Broniatowski,¹⁸ in which he focused his study around the concepts of the “ilities”. “Ilities”, also known as “non-functional requirements” are properties of the system's architecture rather than characteristics (e.g., flexibility, reliability, sustainability, availability, maintainability). The study covered the years from 1990 to 2017 of SE made by members of the Council on Engineering Systems Universities (CESUN). Since the Council's foundation in 2004, it has provided a mechanism for the member universities to work together in engineering systems as a new field of study. Significant analysis and discussion on SE over the years 1998–2016 was presented by Bhatia and Mesmer,¹⁹ in which they analyzed four SE and design engineering journals with the most frequencies of occurrences of SE related topics. However, the dataset used in Broniatowski¹⁸ are only abstracts from the “Web of Science”, and then, they used Latent Dirichlet Allocation (LDA) for topic analysis. In contrast, the dataset used in ref. [19] considered the complete text and used different methods. However, they considered two journals related to engineering design and 5 random years out of the 19 years of the articles' availability. Our study considers all the TRs generated by SERC since its foundation (2009–2022).

Combining different methodologies, SERC has explored the application of agile and lean thinking to SE to reduce waste and increase system flexibility. Moreover, SERC has contributed to advancing decision analysis and optimization techniques in SE by developing quantitative models, algorithms, and decision-support tools to aid in complex decision-making processes. This section has attempted to briefly describe the work relating to SE within SERC.

3 QUESTIONS TO BE ANSWERED

The objective is to identify SE research's relevant topics and evolution over a decade, providing insights on research streams, essential topics, and leading authors within the SERC. Our modeling approach addresses the following questions from a text analytic approach:

1. Which research streams can be identified over a decade of research?
2. What are the relevant topics in SE research?
3. How has SE research evolved over a decade?
4. Who are the key authors of SE research at SERC?

4 MODELING APPROACH

The modeling approach aims to capture the research topics and evolution of SE within and via the SERC. This approach covers data pre-processing, text, and network analytic techniques for knowledge discovery.

Figure 1 presents the framework for the data analysis. The proposed framework has four stages. Stage 1 corresponds to the Data Collection phase, which compiles TRs from 2009 to early 2023. Stage 2 consists of Data Pre-Processing, corresponding to data cleaning by removing unnecessary information and transforming the inputs into an adequate representation for the analysis. Stage 3 is Text Analytics, which aims to identify the different research streams through document similarity. Finally, Stage 4 involves Network Analytics, where we model the text into semantic networks, first by using the Louvain Community, then, based on term proximity, we used the co-occurrence to find SE's research topics and their evolution. In addition, an author network representation is needed to find out the authors with more collaborations and their research network.

Details are in the caption following the image — Framework for text and Network analytics for TRs within SERC. SERC, systems engineering research center; TRs, technical reports.

4.1 Stage 1: Data collection

The Data collection is all the documents that researchers from SERC have developed over a decade. These documents are categorized as conference papers, journal articles, presentations, posters, white papers, annual, and TRs, videos, workshops, dissertations, and so forth. For this analysis, we only consider the TRs from 2009 to early 2023, representing reports for every research project developed by SERC. There are 248 TRs out of the 660 total documents. This represents 37.6% of the SERC documents that can be fully downloaded. An example of a TR raw text is shown in Figure 2. We used the MongoDB library to retrieve the documents from SERC.¹ This library offers solutions for handling, storing, and retrieving unstructured or semi-structured data.

4.2 Stage 2: Data pre-processing

Data pre-processing corresponds to transforming the raw text into an adequate representation for the analysis. This stage has four steps, as follows:

Tokenization: Breaking the text into segments, such as words, sentences, and paragraphs, before data pre-processing. In this step, we used the Natural Language Toolkit (NLTK).²⁰ Tokenization is a fundamental pre-processing step in NLP tasks because it enables further analysis and understanding of the text. In addition, it helps standardize text representation, extract meaningful information, and facilitate subsequent tasks such as part-of-speech (POS) tagging, named entity recognition, and language modeling. Tokenization starts the NLP process.

Entities removal: Unnecessary information (data entities) related to the names of researchers and organizations and other terms related to languages, dates, time, different types of numbers representation, and locations were removed. We used SpaCy Entity Recognizer for this step for the raw TRs.²¹ This step provides flexibility and control over which named entities are considered during text processing or analysis, allowing analysts to tailor the pipeline according to their needs and requirements. The entities considered are PERSON, LANGUAGE, DATE, TIME, PERCENT, MONEY, QUANTITY, ORDINAL, CARDINAL, ORG, and GPE. For example, the entity GPE considers countries, cities, and states. Another example is the entity MONEY which covers monetary values, including units.

Special characters and stop words removal: The objective of removing stop words in text analytics is to eliminate common words that do not carry significant meaning or contribute to the understanding of the text. Removing these stop words and special characters can shift the focus to more relevant and informative terms, improving the accuracy and efficiency of text analysis. For example, words such as report number, copyright, and confidentiality.²² A TR might have URLs or dates that are not relevant to our study, all of them were removed. Several examples of stop words that were eliminated are ‘which’, ‘do’, ‘a’, or ‘every’.

Bigrams: Bigrams are the concatenation of two consecutive words, which helps gain insights into contextual relationships and improve language understanding. SE or best-practices are good illustrations of bigrams within this research. Subsequently, the most frequent bigrams were used as the base for the clustering algorithm.

4.3 Stage 3: Text analytics

The text analytics stage intends to identify the different research streams from a semantic perspective by comparing documents' word similarities through time. For example, a group of documents is considered “similar” if these share akin words measured by their surrounding words. Thus, it can be inferred shifts in research from one period to another by identifying the similarities.

First, group the documents' clean unigrams at a year level; this will help to understand how research has evolved through time. Second, use a pre-trained sentence-transformers model based on Bidirectional Encoder Representations from Transformers (BERT) to map the text into an effective numerical representation. BERT learns contextual representations of words by considering their left and right contexts, allowing it to capture more accurate semantic understanding. An option is the model “all-distilroberta-v1”, which has an average performance of 59.84%.²³ This model maps sentences and paragraphs to a 768-dimensional dense vector space. Third, calculate the cosine similarity between the group of documents at a year level to every other year. The output is a score similarity from 0 to 1. A value closer to one represents a higher similarity than a value closer to zero. Finally, visualize the similarity scores into a square heatmap to identify the different research streams from a semantic approach at a document level and then at a year level.

Also, this stage represents the baseline for the research streams because of the heatmap results at a year level. At this point, the analyst intervention is crucial to identify the SERC research areas.

4.4 Stage 4: Network analytics

In the network analytics stage, text is modeled into a graph where connectivity is based on word proximity within the document. To identify clusters, we implemented Louvain community detection. The algorithm works by iteratively optimizing a modularity score, which measures the quality of community structure. For each iteration, nodes are merged into communities to maximize this modularity score, resulting in cohesive groups of nodes with similar connectivity patterns.^{24, 25} These clusters depict bigrams that usually appear together in the documents. For example, a common bigram is “SE”, which has a frequency of more than 1000 times. Then, an expert analyzes these clusters to determine the topic and uses Artificial Intelligence (ChatGPT) to assign a cluster title.²⁶ Followed by visuals to clearly illustrate the different topics. The steps to model the semantic network (Louvain communities) are as follows:

1. Nodes: Filter the most frequent and relevant bigrams.
2. Links: Find the pairs of bigrams proximate to each other within a size window to the right. We set a window size of 10 due to the slight difference between left, right, and both.²⁷
3. Links weight: Calculate the ratio between the frequency of the pair with the overall bigrams frequency.
4. Implement the Louvain community detection for all documents and each research stream of the previous stage.

The next step is to use the co-occurrence network to understand how the term “SE” has evolved. Even though the steps are similar to the previous step. Here, the co-occurrence network is per year. Also, it is necessary to identify the bigrams (nodes) directly linked to the “SE” node. Once we have all these bigrams, we compare the networks to determine terms particular to one period, the core terms constant through time, or new concepts introduced in a specific year. Creating a heatmap is the best way to visualize the core and new concepts per year.

Finally, for the authors' network, it is necessary to create a matrix representing the authors with more participation in TRs and their relationships with other authors. For this analysis, it is important to consider authors with at least two TRs. Each node illustrates an author, while the node size is the total publications of that author. In other words, the bigger the node, the more TRs the author has. The arc represents a connection with other researchers, and the arc thickness the number of collaborations between them.

5 EMPIRICAL RESULTS

Over the course of a decade, the 25 collaborating universities and faculty within the SERC have developed 660 different types of research documents, of this number 248 are final research project TRs. These involved 479 researchers (industry members, faculty, postdocs, and doctoral students). The data used for this analysis was focused on the 248 TRs. The purpose of a TR is to convey information, findings, analysis, and recommendations related to a specific project. Also, it provides detailed and organized documentation of research, experiments, designs, analyzes, and other technical activities of the project. Figure 3 illustrates the distribution of all documents and TRs per year from 2009 to early 2023. TRs represent 37.57% of the total work research output relative to the documents analyzed. The remaining part of the empirical results will address each research question from a text analytics approach.

5.1 Research streams

To answer our research question 1: Which research streams can be identified over a decade of research? We performed a document similarity, analyzing the similarity among the reports. Sorting the documents by publication date, Figures 4 and 5 illustrate the results by comparing at a document and year level, respectively. Figure 4 shows the similarity among documents. This visual presents two darker areas. The first is in the top-left, where we observed a group of documents with high similarity values, followed by a small darker area in the middle. However, within the rest of the graph, several darker areas are scattered. The similarity values range between 0 and 1. Values closer to one represent high similarity, whereas values closer to zero are orthogonal vectors or perpendicular with null similarity.²⁸ Thus, a value of 0.6 represents high similarity. It is important to note that the similarity rate starts at 0.064, given the granularity of the analysis.

Figure 5 shows a summary per year with the research stream areas and the number of TRs involved. This graph distinguishes three periods: The first research stream corresponds from 2009 to 2011 at the top-left. This area contains 24 documents. The second research stream is from 2014 to 2015 in the middle of the heatmap, represented by 55 documents. These two areas are similar to the ones spotted in Figure 4. Another darker area at the bottom-right has high similarity values between 2017 and 2021, representing the third research stream with 85 documents. In addition, research stream four considers eight scatter years with a total of 135 documents. Even though the latter is disjointed by the years 2016 and 2019, they share alike similarity values (0.67–0.86). For this reason, we analyzed it as well. The other research streams are considered to have a lower priority than the rest due to their similarity values. However, it is worth its analysis. Research streams five and six (highlighted in green) have 84 and 116 documents, respectively. Lastly, research stream seven considers 2014, 2015, and 2017 with 69 documents (highlighted in yellow). As we aggregate documents by year, the similarity rate starts at 0.39, representing a significant similarity value among documents.

The identified research streams are based on analyzing the words used and their context. This insight will later be discussed in Section 5.3 to understand what is particular about each stream and what changes from one another.

5.2 SE topics

To answer our research question 2: What are the relevant topics to sponsors and faculty when analyzing SE research conducted over the course of a decade? By implementing Louvain communities, there are seven main communities. The algorithm uses the most relevant and frequent bigrams as nodes, and the links are related to the proximity between these bigrams in the document.

The semantic network includes information on all TRs (248) from 2009 to early 2023 and has 133 nodes with 2304 links. It has a density of 0.2625 and an average clustering of 0.4615. The node degree measures the number of links connected to a node. The top 15 terms with the highest degree are as follows: SE, decision-making, engineering research, systems engineers, life-cycle, research team, task order, complex systems, modeling simulation, SoS, research task, methods and processes, methods and tools, model-based, engineering systems. These terms reflect the common language within SERC research.

Figure 6 illustrates the results of the Louvain communities, identifying seven groups. The title for each community was found using Artificial Intelligence (AI) according to the bigrams of each community. Below are the communities' titles, a brief explanation of each one, and the relationship with the four SERC thematic areas.

1. Technology Collaboration: refers to the interaction of different technologies, tools, and methodologies to work collectively on the design, development, and implementation of complex systems. In general, the main objective is to achieve a common goal. This community is related to SEMT and HCD.
2. SE Hub: concentrates terms related to the activities done in SE research and the roles that a system engineer performs at any process. This community has a clear connection with the ESOS area.
3. Innovative Design: relates to how systems engineers perform or innovate their tasks, for example, using new tools, methods, technology, or techniques. SEMT is the SERC area that fits better to this community.
4. Synergy Models: brings up specific methods and techniques used in the field, such as systems approach, modeling, simulation, agent-based models, AI, game theory, optimization, and machine learning. These methods and techniques contribute to SE in gaining a holistic view of how different elements contribute to a complex system's overall behavior and performance. SEMT and ESOS describe the linkage with this community well.
5. Integrated Systems: denotes the terms used to measure performance, processes, methods, and assessment. Integrated systems are designed to work together seamlessly, ensuring the overall system functions efficiently and effectively. There is a clear association with the TS area within this community.
6. Applications: aggregate use cases, industry, and government transition and application. They refer more to the implementation of a project or process. Undoubtedly, all the SERC areas might fit under this community; however, we believe SEMT is more accurate.
7. Cognitive Technology: alludes to technologies that mimic human cognitive processes such as perception, reasoning, learning, and problem-solving. Moreover, these technologies are designed to understand, interpret, and interact with complex and unstructured data. Cognitive technologies often incorporate elements of AI, machine learning, NLP, and data analytics to perform tasks that traditionally require human intelligence. Both SEMT and TS areas bear with this community.

The results demonstrate the richness of SE research, from applications and methods to the numerous research activities the field demands. Moreover, the context reflects the uniqueness of SE, such as complex systems, SoS, complexity, and multiple stakeholders. As explained in the following section, SE research has expanded to new methods like AI, machine learning, and deep learning.

5.3 Evolution of SE

To answer research question 3: How has SE research evolved over a decade? We analyze how the term “SE” relates to other concepts. Therefore, for each research stream of Figure 5, the Louvain community is applied to discover new topics in that timeframe. In the top two rows of Figure 7 are the four main research streams, while in the bottom rows are other research stream relationships worth analyzing. For example, Research Stream 1 covers 3 years of research focusing on the topics related to the Development Insights Network and SE Hub. Also, this research stream has more communities (15 communities), meaning that researchers covered the vast methods that SE considers. However, according to the visualization, the most significant community is the SE Hub, which relates more to the ESOS area. In contrast, Research Stream 2 has 2 years, and the center of attention is on Innovative and Engineering Solutions. Notably, the community related to Engineering Solutions was introduced during this timeframe. This means the SEMT area started getting more attention within SERC projects. Then, Research Stream 3 involves 5 years with more emphasis on Engineering Methods and Operational Development. Followed by Research Stream 4, which has disjoint years but similar values. This Research Stream introduced the term AI in SE research.

The other Research Streams represent other meaningful clusters with less similarity values. For instance, Research Stream 5 relates more to Systems Innovation, while Research Stream 6 focuses on the community named Systemic Mastery Hub and introduces a new community named Systems Decision. The Systemic Mastery Hub community considers application projects and how to train systems engineers inside organizations. Yet Research Streams 5 and 6 share the community of Integrated Systems. Finally, Research Stream 7 contemplates 3 years. Here, the emphasis is on Integrated Engineering Solutions and Systems Insights.

To identify which concepts were introduced in the leading research streams (1, 2, 3, and 4), a co-occurrence network directly captures the terms linked to “SE.” The top terms with a higher degree besides “SE” are presented for each co-occurrence network. The results are as follows for the three Research Streams. Research Stream 4 is not considered here because its years are covered in the first three. The network is not presented, only the leading concepts.

Research Stream 1 (2009–2011): systems engineers, life cycle, SoS, methods processes, SoS, complex systems, software development, software engineering, security requirements, product line.
Research Stream 2 (2014-2015): systems engineers, theory effective, decision-making, effective systems, engineering research, life-cycle, task order, cost schedule, modeling simulation.
Research Stream 3 (2017–2021): systems engineers, mission engineering, systems engineer, methods tools, helix team, research team, processes tools.

These results show that between 2009–2011, terms such as SoS and complex systems became more relevant. This is exemplified in the work undertaken by DeLaurentis,¹¹ which quantified attacks' impact on communications. They study the behavior of SoS and the interactions between the constituent systems when there is a cyberattack. Another example of what is meant by SoS is the research done by Agarwal et al.,²⁹ in which they cover the challenging endeavor of dynamic planning of incorporating new systems and upgrading existing systems over some time under threats, limited budget, and uncertainty. This previous research could be considered part of the TS SERC area.

In 2014 and 2015, the terms cost schedule and life-cycle became relevant. This can be seen in the methodology proposed for a more structured approach to recognizing and assessing ship capability portfolios in the stages of product design.³⁰ Another example is the work to improve the methods and techniques used to design and acquire complex military systems.³¹

By 2017–2021, words related to the research team and helix team emerged. Evidence of this can be seen in the Helix project, a multi-year research focusing on understanding what makes systems engineers effective.^{32, 33} As a result, The Helix team developed Atlas 1.1: An Update to the Theory of Effective Systems Engineers.¹⁷ This research might be considered part of the HCD area of SERC. An additional example is the research on supporting the DoD digital Engineering transition. This research created a roadmap with fundamental aspects of Digital Engineering Transformation and how the interaction between traditional and learning systems should enable SE practice automation.^{34, 35} More emphasis on methods, processes, tools, and new concepts, such as mission engineering, emerged during this timeframe. A notable example of all these terms is the analysis of work-at-home policies on systems engineers during the COVID-19 pandemic.³⁶ Also, Cody and Beling 2023³⁷ clearly illustrate this trend within their transfer learning framework from a systems theoretic perspective. Their framework included homogeneous and heterogeneous transfer, domain adaptation, inductive and transductive transfer, negative transfer, and other variables.

Figure 8 reflects the introduction of the relevant concepts in the study of SE per year. For example, the term security requirements was introduced in 2010, while systems thinking and systems architecture were introduced in 2011. Several terms appear together due to the type of projects. Big data and visual analytics are good illustrations of this in 2014. Turning now to words related to informatics, in 2012, words such as systems software, cloud computing, cyber-security, cyber-attacks, and open source appeared. An example is the work by Carter et al. 2019,³⁸ which presented two methodologies for addressing cyber-security at the earliest stages of a new system's life cycle.

Other vital areas introduced in 2012 are supply chain and human factors; however, these terms were more relevant the following year. A helpful example of supply chain research is identifying factors to the evolution of the Green Supply Chain framework within and outside the aviation industry.³⁹ In defense, we found terms like defense systems and air force in 2013 and mission engineering in 2018. In 2013, concepts such as resilient systems and constituent systems directly affected the type of projects. Tregubov and Lane⁴⁰ illustrate this point clearly with their research about Kanban-based scheduling system workflow in multilevel systems.

For data-driven investigation, words such as visual analytics, AI, and big data came into view in 2014, while machine learning popped up in 2018. Research that shows these terms' relationship is the work done by McDermott et al.,⁴¹ in which they stated that implementing AI, Machine Learning, and Autonomation to complex and critical systems need holistic and system-oriented approaches.

To conclude, in 2020 and 2021, the terms mission engineering and surrogate pilot were introduced due to the project of using the Open Model Based Engineering Environment developed by NASA/JPL⁴² as the Authoritative Source of Truth (AST) of the surrogate pilot. The surrogate pilot is a team from academia, government, and industry that assesses the Naval Air Systems Command SE Transformation framework by simulating a real-time collaboration within an AST instead of standard paper-based documents.⁴³

This section has reviewed the relevant terms introduced in each year of the SERC's life and gave several examples of projects developed by its researchers for several terms.

5.4 Authors

To answer research question 4: Who are the key authors for SE research at SERC? The strength of SERC derives from its research faculty network across the 25 collaborators. Table 1 shows the authors' names who have contributed to SERC with more than nine TRs. In addition, for each category, the author with more citations is shown in bold font. The table shows the h-index and the number of publications that have cited each researcher's work. We retrieved this information from SCOPUS. The h-index is calculated using the number of publications for which an author has been cited by other authors at least that same number of times.

TABLE 1. Authors with more than nine TRs from 2009–2022 h-index and number of documents that cited the researcher's work.

Authors	No. of TRs	h-index	Documents that cited their work
Dr. Dinesh Verma	26	15	758
Dr.Gary Witus	25	8	312
Dr. Jon Wade	25	10	842
Dr. Nicole Hutchison	25	3	31
Dr. Douglas Bodner	22	12	310
Dr. Mark Blackburn	21	11	397
Mr. Ralph Griffin	21	No info	No info
Dr. Barry Boehm	20	51	12,823
Dr. Cihan Dagli	20	26	2425
Barry Horowitz		15	840
Dr. Louis Pape	19	7	94
Dr. Nil Kilicay-Ergin	19	10	230
Dr. Abhijit Gosavi	18	17	1481
Dr. David Enke	18	22	1645
Dr. Dincer Konur	18	16	886
Dr. Richard Turner	17	12	1044
Dr. Ruwen Qin	17	18	1026
Dr. Siddhartha Agarwal	17	6	86
Dr. William Rouse	17	33	4079
Dr. Donna Rhodes	16	19	918
Ms. Megan Clifford	16	1	6
Dr. Jo Ann Lane	15	12	493
Dr. Renzhong Wang	15	6	137
Dr. Russell Peak	15	11	469
Ram Deepak Gottapu	15	5	106
Dr. Adam Ross	14	21	948
Dr. Peter Beling	14	22	1429
Mr. Thomas McDermott	14	5	85
Dr. Cesare Guariniello	13	8	146
Dr. Daniel DeLaurentis	13	28	2161
Dr. Walter Bryzik	13	24	1425
Dr. Peter Dominick	12	9	349
Dr. Robert Cloutier	12	11	545
Peizhu Zhang	12	6	2
Dr. Abhijit Deshmukh	11	15	1086
Dr. Devanandham Henry	11	8	891
Dr. Kevin Sullivan	11	30	2306
Dr. Michael Pennotti	11	5	102
Dr. Stan Rifkin	11	4	298
Dr. Mary Bone	10	5	273
Dr. Navindran Davendralingam	10	9	197
Dr. Valerie Sitterle	10	7	267
Dr. Forrest Shull	9	38	4,221
Dr. Jose Ramirez-Marquez	9	41	4752
Dr. Karen Marais	9	21	1381
Dr. Michael Pennock	9	9	259
Dr. Paul Grogan	9	10	385
Dr. Tommer Ender	9	5	151
John Colombi	9	11	581

Abbreviation: TRs, technical reports.

However, to visualize researchers' importance within the SERC and the relationship among researchers, Figure 9 presents the network of researchers whose names appeared in more than two TRs. We decided to exclude researchers who appeared only once because, in most cases, they were Ph.D. students who, for several reasons, worked at SERC at that time. Each node of the network represents a researcher with at least two collaborations, while the arc shows connections with other researchers. The thicker the arc is, the more TRs have been written within their network. It can be visualized that several authors commonly have worked together due to the thickness of the arc; for example, at the top of the network, a group of researchers has worked together on the Flexible and Intelligent Learning architecture of SoS. Below this group, it is easy to spot other researchers' networks whose work has been focused on transforming SE through Model-Centric engineering. The rest of the network has many connections, representing the interaction between researchers and the involvement in diverse projects. The other two groups of researchers have focused on cyber security. For a better visualization, several segments of researchers were framed with their respective thematic SERC areas. However, the selected thematic area does not exclude other research areas for those researchers.

Finally, the graph shows the names of researchers who have relationships with other authors. Furthermore, they have worked on several SERC projects and are considered leading researchers. However, all researchers from Table 1 should be regarded as preeminent due to their contribution to the SERC on different projects.

6 CONCLUSION AND FUTURE RESEARCH

The SERC is a prominent research organization specializing in SE. Over a decade, it has conducted extensive research in various domains, including defense, aerospace, and technology, as we found with our analysis. SERC's primary mission is to advance the field of SE through collaborative efforts between academia, industry, and government agencies. Diverse projects have been involved in the evolution of SERC. They all related to the importance of SE in more than one field. In addition, the management of complex systems nowadays has been vital to cope with the complexity of such systems. After our analysis, we answered the research questions related to research orientations, topics, the evolution of SE research, and preeminent authors. Our findings illustrate the diversity and richness of SE research and the context of complexity by which it operates. Concepts like game theory, AI, simulation, systems thinking, systems architecture, and visual analytics reflect the diverse methods and tools used to model complex systems. Moreover, SERC introduces relevant aspects such as human factors and resilience to cope with the interaction of humans and cyber-physical systems. SERC has been adapting to new threats; thus, studying resilient systems from supply chain to constituent systems has allowed them to cope with their complexity. Their work often leads to the development of innovative methodologies, frameworks, and tools that enhance complex systems' efficiency, effectiveness, and resilience. Despite the four research areas SERC considers, the communities presented in this work have more granularity regarding SE research. More questions arise from this research, such as “What topics should SERC pursue in the future?”, How would AI affect SE? and How is knowledge being transferred besides the SE handbook?

Evidently, with the increase of new technology in any system, there will be more research in the SE field. Considering this evidence, future research seems to continue exploring critical terms in SE, such as complex systems or SoS that may reveal new methods, tools, and applications. Likewise, terms such as AI can show different ways to solve complex problems. SERC has significantly impacted advancing the practice and theory of SE, benefiting a broad range of industries and sectors. Advances in AI will have a positive impact on the field of SE. Researchers should consider future investigations into how AI can be involved in complex systems. Also, our methodology can be implemented in datasets outside SERC to visualize other types of research in other countries or research centers.

Aerospace Electronic and Defense Systems

Wednesday, May 22, 2024

Decades of Progress at SERC: How Research Grew and Propagated