Writing about aerospace and electronic systems, particularly with defense applications. Areas of interest include radar, sonar, space, satellites, unmanned plaforms, hypersonic platforms, and artificial intelligence.
We did a lot of research on this one because there's not a
lot of public information on this lesser-known Russian submarine
Our Story begins really in 1987. The cold war is really at
its peak. This is right before the five-year or so decline to the end of the
Soviet Union and they're just going all out with their spending coming up with
new ideas. A lot of the money at the end of the Soviet Union was really put
into rocket technology and missiles and things like that, but they also put a
lot of money into the Navy and one of the new ideas was to get away from
nuclear propulsion for our submarines because conventional is cheaper for one
and we can get a lot of performance out of these diesel conventionally powered
submarines and that spurred on this idea part of which was from the kilo.
The kilo class conventionally powered submarine was a major success
for the Soviet navy. They built a ton of those boats and they improved to
modernize them over and over again to where they were going to be very well. The
kilo is absolutely famous for being an outstanding sub and this submarine the
lot of submarine that nobody knows about is the replacement for the kilo
This is the new kilo. This is the new super stealth sub that
is entering Russian Service as of this year and nobody knows about it, so
that's what we're going to fix today. We're going to tell you about this new
sub and what a hunk of junk it is. The reason why it's so secret is they're
embarrassed to tell anybody about it. This is a complete train wreck from
design to construction to sea trials that nearly killed people on board. It's a
mess This submarine was designed by the very respected Ruben Central Design
Bureau. Now they have a ton of successes under their belt. They've built every
major nuclear submarine for the Russian Navy and a couple submarines that were
not nuclear, as well designed them including the kilo class so they can design Subs.
But something went terribly wrong with this one.
Designed around a unique implementation of air-independent propulsion (AIP) technology, the new Project 677 Lada
submarines were meant to provide the Russian navy with a modernized,
cost-efficient complement to nuclear-powered submarines. But the Lada
project stalled amid technical difficulties, leading the manufacturer to
abandon AIP propulsion altogether in favor of a traditional
diesel-electric system. Project 677’s place on the looser list is a reflection
of the fact that the Russian shipbuilding industry has failed to
implement the submarines’ core defining feature, dooming what was a
potentially innovative class to long-term technical irrelevance.
Russia's AIP-powered Lada-class submarines: A significant advancement in non-nuclear underwater warfare
Russia's Lada-class submarines, also known as Project 677, utilize
air-independent propulsion (AIP) technology, setting them apart as the
first class of AIP-powered submarines in the Russian Navy. This
technological leap allows for underwater operations without the need for
atmospheric oxygen, enhancing their stealth capabilities and
effectiveness in anti-submarine warfare.
Improvements over the
Kilo-class submarines include better acoustic signatures, advanced
combat systems, a mono-hull design, increased speed, and reduced surface
displacement for maneuverability. Lada-class submarines are equipped
with modern sonar systems, automated combat control, and countermeasure
electronic support.
Despite initial plans to replace older Kilo-class
submarines, construction of additional Improved Kilo-class boats has
been ordered due to setbacks with the Lada-class program. Overall, the
development and implementation of the Lada-class submarines represent a
significant advancement in non-nuclear underwater warfare technologies.
Developer: Stealth Capabilities of 677 Lada Subs Outshine Its Predecessors
The
677 Lada-class diesel-electric submarines are designed for
reconnaissance, surveillance, anti-submarine warfare, anti-shipping, and
mine-laying missions. The subs are equipped with advanced technology,
including improved stealth capabilities, enhanced communication systems,
and advanced torpedoes.
The
stealth capabilities of the Project 677 Lada non-nuclear submarines are
significantly superior to those of their predecessors, said Igor
Vilnit, general director of the submarine's developer, the Rubin Central Design Bureau of Marine Engineering.
The
first serial-produced Project 677 Lada submarine, the Kronstadt, built
at the Admiralty shipyards in St. Petersburg, is being readied for
delivery to the Russian Navy.
"In
terms of stealth parameters, this submarine is several times superior
to its predecessors. The boat maintains an extremely low noise level
thanks to its specially designed equipment. Additionally, it extensively
incorporates modern acoustic protection equipment, an external
anti-hydrolocation coating, and carefully designed hull contours to
ensure low visibility," Vilnit said in an interview.
Rubin's general director highlighted that Project 677 submarines
have a "very powerful" hydroacoustic system. What's more, he
spotlighted the impressive missile-torpedo arsenal, and radio-electronic
weaponry found in these subs.
"I
would like to emphasize the hydroacoustics of the Lada: it is not only a
wider range of acoustic waves, it is a significantly longer range of
target detection. It took a lot of effort to achieve this result," Vilnit specified.
A
"high-speed" hydroacoustic data processing system is required to
distinguish a useful signal from interference. In addition, it is
necessary to classify a target, determine its speed, depth, and
direction of travel.
Russia’s New Lada-Class Submarine Fails to Meet Expectations
Here’s What You Need to Remember: The second Lada submarine—Kronstadt, the
one being referenced by TASS’ source—is now reportedly on the cusp of
entering service, following a tortured construction process involving
several sweeping revisions. But Kronstadt, too, is a standard
diesel-electric submarine.
Russia’s first Lada-class submarine will finally be handed over to the Navy in 2022, according to recent reports.
“The first serial Project 677 Lada submarine is planned to be
received in 2022,” a defense industry insider told Russia’s TASS state
news. The TASS report noted that this information has not been publicly
corroborated by Russian officials.
As first conceived, the Lada-class was designed around a
unique air-independent propulsion (AIP) system. This technology comes
in different variants, but the core concept is the same: AIP is a
propulsion technique that allows a submarine to function without direct
access to outside air. Depending on its exact implementation, AIP
technology can bring a wide range of performance benefits: notably,
reductions in displacement and crew size, significantly reduced noise
generation when compared to standard diesel-electric systems, and
greater endurance. These enhancements can make AIP submarines an
attractive choice for littoral, local, or low-intensity missions that
don’t require the virtually unlimited endurance, range, and sustained
speeds of nuclear-powered submarines. The Lada boats’ arsenal will include six 533-millimeter torpedo tubes for a total capacity of eighteen torpedoes, as well ten Kalibr cruise missiles.
Several other navies are dabbling in AIP technology—some, with
seemingly greater success than Russia. The Chinese People’s Liberation
Army Navy’s (PLAN) Type 039AYuan-class submarines are powered by the proven, widely-established Stirling AIP method—as
many as fifteen such submarines are currently in service, with several
more under construction. But Russia’s shipbuilding industry has
struggled to integrate the fuel cell AIP systems intended for the Lada boats. The first Lada model, Sankt Peterburg, was
laid down in 2004 and commissioned six years later with a standard
diesel-electric propulsion system; that submarine has since been
repurposed into a testbed vessel. The second Lada submarine—Kronstadt, the
one being referenced by TASS’ source—is now reportedly on the cusp of
entering service, following a tortured construction process involving
several sweeping revisions. But Kronstadt, too, is a standard
diesel-electric submarine.
It was later revealed that the third Lada boat, Velikiye Luki,
will likewise be launched with a traditional diesel-electric propulsion
system. Аlexander Buzakov, head of the Admiralty Shipyard in Saint
Petersburg, said in late 2019 that there are no plans to equip Project
677 submarines with AIP systems. It is not fully clear if Buzakov was
referring only to the next two Lada submarines, or to the entire range. The news puts the future of the Lada-class, which was mainly distinguished from its Kilo-class predecessor by its innovative AIP system, into doubt. At least two more Lada submarines are slated to be laid down in 2022 and commissioned as part of Russia’s 2027 state armament cycle.
Mark Episkopos is a national security reporter for The National Interest.
Underwater Coffins: 5 Worst Submarines from Russia
by Mark Episkopos
~4 minutes
For decades, the Soviet Union has been one of the world’s leading
producers of cutting-edge submarine technology. The Russian Federation
has since picked up its predecessor’s mantle, challenging its NATO
rivals with two formidable new submarine
classes. But submarine design and construction is a notoriously tricky
business—these projects often don’t pan out for a wide range of
technical, logistical, and even political reasons. Here are the five worst Russian and Soviet submarines.
November
TheNovember-class was the USSR’s first line of nuclear-powered attack submarines,
inaugurated in 1958. The class suffered from serious safety and
reliability issues stemming from its experimental reactor
implementation, resulting in a series of catastrophic accidents that
earned the November boats the reputation of bonafide underwater coffins
among the Soviet sailors unfortunate enough to be assigned to them. The
1970 sinking of the K-8 November-class submarine during Naval
exercises is remembered as one of history’s greatest submarine
disasters, costing the lives of fifty-two servicemen.
Victor II
TheVictor I-class nuclear-powered attack submarines brought a revolutionary leap in postwar Soviet submarine design, but the Victor II revision did little to build on these advancements. The Victor II
update sought to reduce the class’ noise generation, but the changes
were too little, too late. The Soviets soon discovered, in part through
information collected by the Walker spy ring, that the Victor II vessels still lagged far behind their American counterparts in acoustics. Seven Victor II submarines were built from 1972 through 1978—the line was then cut short and replaced by the markedly more successfulVictor III-class.
Lada
Designed around a unique implementation of air-independent propulsion (AIP) technology, the new Project 677 Lada
submarines were meant to provide the Russian navy with a modernized,
cost-efficient complement to nuclear-powered submarines. But the Lada
project stalled amid technical difficulties, leading the manufacturer to
abandon AIP propulsion altogether in favor of a traditional
diesel-electric system. Project 677’s place on this list is a reflection
of the fact that the Russian shipbuilding industry has failed to
implement the submarines’ core defining feature, dooming what was a
potentially innovative class to long-term technical irrelevance.
Project-685 Plavnik
The only submarine of her class, the K-278 Komsomoletswas intended as a testbed for new naval technologies. A fire broke out aboard the Komsomolets in
1989, setting off a series of events that caused the submarine to sink
and led to the deaths of forty-two crew members. The damage wrought by Komsomolets has
managed to outlive the submarine itself: further investigations
discovered plutonium leakage from the wrecked, sunken submarine’s
nuclear torpedoes, prompting ongoing concerns of environmental
contamination.
Pravda
Introduced in 1935, the three Pravda-class submarines were
used by the USSR during the Second World War as transport boats. But the
Soviet military quickly realized that the Pravda class was woefully
underpowered for its role. The Pravda-class was crippled by its
poor maneuverability, unacceptably long diving time, and small crush
depth, making it one of the least successful submarine lines ever to
serve in the Soviet navy.
Mark Episkopos is a national security reporter for the National Interest.
According to information published by Tass on January 31, 2024, the
Russian Navy has officially commissioned the Project 677 Lada
diesel-electric submarine Kronshtadt. Follow Navy Recognition on Google News at this link
Commissioning ceremony of the Project 677 Lada submarine Kronshtadt. (Picture source: Tvspb)
The induction ceremony, marked by the hoisting of the naval flag,
took place at the Admiralty Shipyards in St. Petersburg, a key facility
within the United Shipbuilding Corporation.
During the event,
Admiral Nikolay Yevmenov, Commander-in-Chief of the Russian Navy,
announced the integration of the "Kronstadt" into the fleet, assigning
it to the Kola Flotilla within the Northern Fleet.
It possesses a surface speed of 10 knots and an underwater speed of
21 knots. The submarine operates at a working depth of 250 meters, with a
maximum submersion depth of 300 meters. With an autonomy of 45 days at
sea, it accommodates a crew of 35.
Regarding dimensions, the submarine has a surface displacement of
1,765 tons and a submerged displacement of 2,650 tons. It measures 66.8
meters in length overall, with a maximum hull width of 7.1 meters and an
average draft of 6.6 meters.
The submarine's power plant features a diesel-electric engine with
full electric motion, including two diesel generators each producing
1250 kW, an all-mode electric motor of 4050 - 5500 horsepower, two
backup electric motors of 102 horsepower each, a single low-noise
propeller, and two battery banks each containing 120 elements.
Its
armament comprises a mine-torpedo system with six bow torpedo tubes of
533 mm, capable of holding 18 torpedoes or mines. The submarine is also
equipped with the "Kalibr" missile system. For air defense, it carries
"Igla-1M", "Strela-3M", and "Verba" anti-aircraft missile systems, with 8
missiles in transport and launch containers.
About the class
They are designed for anti-submarine warfare, anti-surface warfare,
mine-laying missions, as well as reconnaissance and surveillance. This
versatility makes them suitable for protecting naval bases, seashores,
and sea lanes.
Their advanced capabilities enable them to operate in various
environments, from the Arctic to more temperate climates, extending
Russia's strategic reach and influence. The Lada-class submarines are
well-suited for A2/AD strategies, where the goal is to deny an adversary
access to certain critical regions.
Lada-Class: Russia's Failed Diesel Submarine (No One Will Buy It)
In July, the Project 677 Lada-class (NATO reporting name "St.
Petersburg") diesel-electric submarine Kronstadt completed a deep-sea
immersion as part of its continuing sea trials, within the maritime
ranges of the Baltic Fleet.
The submarine crew verified the functioning of all onboard mechanisms
and systems, practicing control algorithms at great depths and under
various surfacing conditions, Naval Recognition reported at the time.
During the sea trials, the Kronstadt reached a depth of 180 meters at
one of the Baltic Fleet's ranges, while the dive was facilitated by
fleet forces and resources, including the rescue ship SS-750. The
results of the submarine's dive were reported to the Commander-in-Chief
of the Russian Navy, Nikolay Yevmenov.
Lada-Class: The Slow Boat
Construction of the Kronstadt – the first serial submarine of Project
677, following the Sankt Peterburg – began in July 2005 but was
suspended by the Ministry of Defense of the Russian Federation in 2009
until 2013.
The submarine was only launched in 2018, and it took until the end of
2021 for the boat to commence its sea trials, which were prolonged due
to ongoing modernization.
The submarine had been scheduled to join the Russian fleet this year,
yet, there have been no reports that has occurred. When the Kronstadt
finally enters service, it is expected to serve in the Northern Fleet,
where the lead submarine of the project, Sankt Peterburg, is already in
service. That decision had been announced at a conference call meeting
at the Ministry of Defense in early March this year.
It was during that call that Russian Minister of Defense Sergey
Shoigu confirmed that the submarine would carry Kalibr cruise missiles.
"The first issue on the agenda is the construction of the large
submarine Kronshtadt that will enter service with the Northern Fleet.
The ship is set to feature Kalibr cruise missiles, the latest radar,
sonar and navigational systems. This will considerably boost its combat
efficiency," Shoigu was quoted as saying by state media outlet Tass.
Lada-Class Key Details
Project 677 Lada-class submarines are often referred to as the fourth generation of diesel-electric submarines, developed by a Russian Rubin design bureau.
It is essentially an improved version of the Kilo-class and was
designed to be fitted with an air-independent propulsion (AIP) along
with new combat systems. The AIP system was meant to increase submerged
endurance to 45 days, while its submerged cruising range was 500
nautical miles (900 km) at three knots.
The boats have a surface displacement of about 1,750 tonnes and can
develop an underwater speed of up to 21 knots, and an endurance of 45
days. The Lada-class subs are armed with Kalibr cruise missile systems
along with six 533 mm torpedo tubes for a mix of 18 torpedoes or
tube-launched missiles. These may include the Alfa (NATO reporting name
SS-N-27 or Sizzler) multi-role cruise missiles, or Oniks (SS-N-26)
anti-ship cruise missiles.
The boats reportedly have a crew of 35 including officers and sailors.
The boats were initially developed and designed to protect naval
bases, coastal installations, and sea lanes from hostile submarines and
ships, while these boats can also perform patrol and surveillance tasks,
including anti-submarine warfare (ASW) and anti-surface warfare (AsuW)
operations.
Problem-Plagued Platform
Sankt Petersburg (B-585), the lead submarine of the project, was
launched in late 2004 and commissioned in 2010. However, she was not
accepted by the Russian Navy as it was discovered there were issues with the boat's propulsion and that its sonar systems did not meet Russian specifications.
Construction on the remaining boats of Project 677 was thus frozen.
The issues with the lead submarine were eventually addressed, but
only after several years of serving as a "test platform," she was
formally accepted into service with the Russian Navy last year. Sankt
Petersburg officially joined the Northern Fleet in September 2021.
Russia was unable to resolve the issues with the fuel cells for the
AIP, and as a result the second boat of the class, the Kronstadt, was
fitted with an ordinary diesel-electric propulsion system without the
AIP system.
Currently, the Admiralty Shipyard is also building one more
Lada-class submarine, the future Velikiye Luki, while the first steel
was cut for the next two boats with a third also on order.
Originally a full dozen of the diesel-electric boats were ordered, but
given the issues with the program, it is unclear if that order has been
pulled.
Export Model – With No Buyers
Russia had further developed an export variant of the Lada-class: the
Project 1650 Amur-class (named for the Amur River), which was designed
for markets including India and China, while Morocco has also been
offered one. The export submarine could be offered in various
configurations with a displacement of 550 to 1,850 tonnes and be
equipped with a variety of weapon systems.
To date, there have been no buyers for the submarines, and given the
problems with the boats as well as the sanctions placed on Russia,
following its unprovoked invasion of Ukraine, the Amur-class could be
dead in the water.
Author Experience and Expertise
Peter Suciu is a Michigan-based writer. He has contributed
to more than four dozen magazines, newspapers, and websites with over
3,200 published pieces over a twenty-year career in journalism. He
regularly writes about military hardware, firearms history,
cybersecurity, politics, and international affairs. Peter is also a Contributing Writer for Forbes and Clearance Jobs. You can follow him on Twitter: @PeterSuciu.
6 x 533 mm torpedo tubes, for 18 torpedoes,
anti-submarine or anti-ship missiles
Other
mines in place of missiles and torpedoes
The Project
677 or Lada class is a diesel-electric patrol submarine, developed
by a Russian Rubin design bureau. It is an improved version of the
Kilo class, fitted with an
air-independent propulsion and new combat systems. The previous Kilo
class was a basic submarine, simple in design and technology. Also
it achieved respectable export sales. Its major operators are China,
India and Iran. Development of the Lada class commenced in the early
1980s. It was rather protracted. The goal was to develop a submarine
that would be much quieter than its predecessor. The lead boat was
commissioned only in 2010. It turned out that the new boat has
fallen far short of requirement.
The lead
boat was laid down at the Admiralty Shipyard in St. Petersburg in
1997 and launched in 2004. It was commissioned in 2010 and is in
service with the Baltic Fleet. The Admiralty Shipyard laid down
another three
submarines of this class. The lead boat, Sankt Peterburg, was
extensively tested by the Russian Navy,
before entering service. Though it turned out that this submarine has
fallen far short of requirements. The main problem was its propulsion
system. Also there were a number of other major issues. Russia
invested a lot of time and resources in development of the Lada
class boats, however this project turned out to be a failure. One of
the reasons was that after collapse of the Soviet Union a number of
companies that produced various components for Soviet submarines
simply closed down or stopped production of military equipment. Some
of the companies ended up in independent Ukraine. So at that time
Russians lacked all necessary equipment and expertise to build these
new advanced boats. Between 2009-2011 construction of the follow-on
boats was suspended due to multiple major issues with the lead boat.
Rubin design bureau was ordered to make changes to the project.
The Sankt Peterburg was used only as a test boat for testing various
equipment, rather than for active duty. Incomplete boats were heavily
redesigned and
were built to an improved
project. In 2013 construction of the second boat resumed. In 2015
construction of the third boat resumed - it was re-launched due to
the redesign. However construction of the 4th boat remains suspended
and there are no plans to complete it. Production of the Lada class
boats was stopped in favor of more traditional Project 636
Varshavyanka (Improved Kilo) class boats.
However in 2019 a contract was signed for construction of two more
Lada class boats. Most likely that it included the 4th boat
Petrozavodsk. In 2020 a 6th boat of the class was ordered.
Interestingly the lead boat Sankt Peterburg began its
active duty only in 2021. Originally it was planned
that the Lada class boats will have a service life of 30 years.
The Lada
class submarines are designed to protect naval bases, costal
installations and sea lanes from hostile submarines and ships. These
boats can also perform patrol and
surveillance tasks.
The Lada
class boats had a number of new and unusual design features.
Designers abandoned a number of proven features in order to achieve
better performance. These include new anti-sonar coating of the hull,
which reduces acoustic signature of the boats. Submarines are fitted with sophisticated
sonar equipment with bow and flank arrays, as well as towed array.
The Lada
class has six 533 mm torpedo tubes for a mix of 18 torpedoes or
tube-launched missiles. These include Alfa (Western reporting name SS-N-27
or Sizzler) multi-role cruise missiles, or
Oniks (SS-N-26) anti-ship
cruise missiles.
This
submarine class is fitted with a fuel cell plant, which gives air
independent propulsion with oxygen/hydrogen fuel cells and
electric/chemical generators. The Air Independent Propulsion (AIP) system increases the Lada
class submerged endurance to 45 days. The submerged cruising range
is 500 nautical miles (900 km) at 3 knots. However it appeared that
design of the Russian AIP was rather raw and had numerous problems. Notably
the fuel cells were poor. Russia could not develop more advanced
fuel cells due to funding problems and lack of expertise. As a result the second
boat of the class, the Kronstadt, was fitted with an ordinary
diesel-electric propulsion system without the AIP system. In 2022
the second boat of the class, Kronstadt, completed factory sea
trials
Variants
Amur class,
or Project 1650, a less capable version, intended for export. It is
named after the Amur river. Design work has been completed for a
whole family of submarines with a displacement ranging from 550 to 1 850 tons
and various weapon systems.
Knowledge sharing about emerging threats is crucial in the
rapidly advancing field of cybersecurity and forms the foundation of
Cyber Threat Intelligence. In this context, Large Language Models are
becoming increasingly significant in the field of cybersecurity,
presenting a wide range of opportunities.
This study explores the
capability of chatbots such as ChatGPT, GPT4all, Dolly,Stanford Alpaca,
Alpaca-LoRA, and Falcon to identify cybersecurity-related text within
Open Source Intelligence.
We assess the capabilities of existing chatbot
models for Natural Language Processing tasks. We consider binary
classification and Named Entity Recognition as tasks.
This study
analyzes well-established data collected from Twitter, derived from
previous research efforts. Regarding cybersecurity binary
classification, Chatbot GPT-4 as a commercial model achieved an
acceptable F1-score of 0.94, and the open-source GPT4all model achieved
an F1-score of 0.90. However, concerning cybersecurity entity
recognition, chatbot models have limitations and are less effective.
This study demonstrates the capability of these chatbots only for
specific tasks, such as cybersecurity binary classification, while
highlighting the need for further refinement in other tasks, such as
Named Entity Recognition tasks.
Subjects:
Cryptography and Security (cs.CR); Machine Learning (cs.LG)
From: Samaneh Shafee [view email] [v1]
Fri, 26 Jan 2024 13:15:24 UTC (683 KB)
Summary
Here is a summary of the key points from the documents:
The documents present an
evaluation of the capabilities of chatbots and large language models
(LLMs) for natural language processing tasks in cyberthreat detection.
Specifically, the research focuses on binary text classification and
named entity recognition using Twitter data.
Several chatbot models are
examined, including open source options like GPT4all, Dolly, Alpaca, and
Falcon, as well as commercial versions of ChatGPT. Their performance is
compared on classifying tweets as cybersecurity-related or not, and on
identifying organization and product version entities.
For binary classification,
ChatGPT and GPT4all achieve the best results, with F1 scores above 0.9.
The other models range from 0.64 to 0.86 F1. ChatGPT is also most
accurate for entity recognition. Overall, commercial models outperform
open source, but GPT4all comes closest.
Challenges identified include
inconsistencies in chatbot responses, requiring manual validation, and
limitations in providing precise named entities without additional
fine-tuning. Prompt engineering is noted as an important factor in
optimizing chatbot performance.
The potential of LLMs and
chatbots for cyberthreat detection is demonstrated, but refinement is
still needed for optimal real-world application, particularly on
specialized tasks like entity extraction. The study provides insights
into strengths and weaknesses of different models.
F1 Scores
The F1 score is a common evaluation metric used to measure the accuracy of models for classification and information retrieval tasks. It is the harmonic mean of precision and recall:
Precision is the fraction of
predicted positive cases that are correctly real positives. It measures
how many selected items are relevant.
Recall is the fraction of real
positive cases that are correctly predicted positive. It measures how
many relevant items are selected.
The F1 score balances both
precision and recall into a single metric. It ranges from 0 to 1, with 1
being perfect precision and recall, and 0 being the worst.
The F1 score is commonly used instead of raw accuracy when there is an uneven class distribution. It gives a more realistic measure of a model's performance by accounting for false positives and false negatives. Models with high precision but low recall, or vice versa, are penalized compared to models where both are high.
Based on the documents, here are the key F1 scores reported for the cybersecurity binary classification task:
ChatGPT-3.5-turbo (16k context): 0.9431
ChatGPT-4 (8k context): 0.9410
GPT4all: 0.9049
Dolly 2.0 (12B parameters): 0.8612
Falcon (40B parameters): 0.8511
Alpaca-LoRA (65B parameters): 0.8477
Stanford Alpaca (30B parameters): 0.6415
The commercial ChatGPT models achieved the highest F1 scores on this task, with GPT4all being the top performing open source model. The scores ranged from 0.6415 to 0.9431 across the different chatbot models examined. The multi-task LSTM model from previous work scored 0.9470.
For the named entity recognition task, the F1 scores were lower overall. ChatGPT-4 achieved 0.41 for organization extraction and 0.54 for product version extraction when tested on a subset of the data. On the full dataset, its F1 score was only 0.10 for extracting all entity types.
So in summary, the F1 scores indicate ChatGPT and GPT4all performed best for classification, while all models struggled more with precise entity extraction, highlighting a limitation in applying chatbots directly for this specialized NLP task.
- LASIGE, Faculdade de Ciências, Universidade de Lisboa, Portugal
Previous relevant publications:
Ferreira et al. (2020) analyzed threat data on Twitter
Dionisio et al. (2019, 2020) worked on cyberthreat detection from Twitter using neural networks
Alves et al. (2021) presented a system to process tweets for threat awareness
Institutions:
LASIGE - Laboratory of Software Engineering, Faculty of Sciences, University of Lisbon
Artifacts:
The authors used a dataset of
31281 tweets collected and labeled by Alves et al. (2020). This dataset
could potentially be requested for independent verification.
Code and models do not seem to be directly shared, but the approaches are described in enough detail to reimplement if needed.
Results are presented
comprehensively including F1 scores, execution times, prompt examples,
and classification outputs. This allows independent assessment.
In summary, the key artifacts are the labeled Twitter dataset and the detailed experimental results. The datasets and implementation details provide potential avenues for reproducing or extending the study if access was granted to the Twitter data.
Some other References:
Daniel Iwugo, "Large Language Models and Cybersecurity – What You Should Know," freecodecamp.org
cybersecurity.devAI in Cybersecurity: The
Role of Large Language Models | cybersecurity.dev
Identity-based attacks are on the rise, with phishing remaining the most common
and second-most expensive attack vector. Some attackers are using AI to
craft more convincing phishing messages and deploying bots to get
around automated defenses designed to spot suspicious behavior.
At the same time, a continued increase in enterprise applications
introduces challenges for IT teams who must support, secure, and manage
these applications, often with no increase in staffing.
The number of connected devices continues to grow, introducing
security risks due to an increase in the attack surface. This is
compounded by potential vulnerabilities associated with each device.
While there are many security tools and applications available to
help enterprises defend against attacks, integrating and managing a
large number of tools introduces more cost, complexity, and risk.
Cybersecurity is among the top three challenges
for CEOs, second to environmental sustainability and just ahead of tech
modernization. Generative AI can be transformational for cybersecurity.
It can help security analysts find the information they need to do
their jobs faster, generate synthetic data to train AI models to
identify risks accurately, and run what-if scenarios to better prepare
for potential threats.
Using AI to keep pace with an expanding threat landscape
Cybersecurity is a data problem, and the vast amount of data
available is too large for manual screening and threat detection. This
means human analysts can no longer effectively defend against the most
sophisticated attacks because the speed and complexity of attacks and
defenses exceed human capacity. With AI, organizations can achieve 100
percent visibility of their data and quickly discover anomalies,
enabling them to detect threats faster.
Although the exponentially increasing quantity of data poses a
challenge for threat detection, AI-based approaches to cyber defense
require access to training data. In some cases, this isn’t readily
available, because organizations don’t typically share sensitive data.
With generative AI, synthetic data can help address the data gap and
improve cybersecurity AI defense.
One of the most effective ways of synthesizing and contextualizing
data is through natural language. The advancements of large language
models (LLMs) are expanding threat detection and data generation
techniques that improve cybersecurity.
This post explores three use cases showing how generative AI and LLMs
improve cybersecurity and provides three examples of how AI foundation
models for cybersecurity can be applied.
Copilots boost the efficiency and capabilities of security teams
Staffing shortages for cybersecurity professionals persist. Security copilots with retrieval-augmented generation (RAG)
enable organizations to tap into existing knowledge bases and extend
the capabilities of human analysts, making them more efficient and
effective.
Copilots learn from the behaviors of security analysts, adapt to
their needs, and provide relevant insights that guide them in their
daily work, all in a natural interface. Organizations are quickly
discovering the value of RAG chatbots.
By 2025, two-thirds of businesses will leverage a combination of
generative AI and RAG to power domain-specific, self-service knowledge
discovery, improving decision efficacy by 50%1.
In addition to not having enough cybersecurity personnel,
organizations are challenged in training new and existing employees.
With copilots, cybersecurity professionals can get near real-time
responses and guidance on complex deployment scenarios without the need
for additional training or research.
While security copilots can bring transformational benefits to an
organization, they’re only useful when they can provide fast, accurate,
and up-to-date information. The NVIDIA AI Chatbot with Retrieval-Augmented Generation workflow
provides a great starting point. It demonstrates how to build agents
and chatbots that can retrieve the most up-to-date information in
real-time and provide accurate responses in natural language.
Generative AI can dramatically improve common vulnerability defense
Patching software security issues are becoming increasingly challenging as the number of reported security flaws in the common vulnerabilities and exposures (CVEs) database
hit a record high in 2022. With over 200,000 cumulative vulnerabilities
reported as of the third quarter of 2023, it’s clear that a traditional
approach to scanning and patching has become unmanageable.
Using the NVIDIA Morpheus
LLM engine integration, NVIDIA built a pipeline to address CVE risk
analysis with RAG. Security analysts can determine whether a software
container includes vulnerable and exploitable components using LLMs and
RAG.
This method enabled analysts to investigate individual CVEs 4X
faster, on average, and identify vulnerabilities with high accuracy so
patches could be prioritized and addressed accordingly.
Foundation models for cybersecurity
While pretrained models are useful for many applications, there are
times when it’s beneficial to train a custom model from scratch. This is
helpful when there’s a specific domain with a unique vocabulary or the
content has properties that do not conform to traditional language
paradigms and structures.
In cybersecurity, this is observed with certain types of raw logs.
Think about a book and how words form sentences, sentences form
paragraphs, and paragraphs form chapters. There’s an inherent structure
that is part of the language model. Contrast that to data contained in a
format like JSON-lines or CEF. Proximity of the data keys and values
doesn’t have the same meaning.
Using custom foundation models presents multiple opportunities.
Addressing the data gap: while making better use of
the influx of data can lead to improved cybersecurity, the quality of
the data matters. When there is a lack of available training data, the
accuracy of detecting threats is compromised. Generative AI can help
address the data gap with synthetic data generation, or by using large
models to generate data to train smaller models.
Performing “what if” scenarios: novel threats are
challenging to defend against without data sets to build the defenses.
Generative AI can be used for attack simulations and to perform “what
if” scenarios—to test against attack patterns that haven’t yet been
experienced. This dynamic model training, based on evolving threats and
changing patterns in data can help to improve overall security.
Feed downstream anomaly detectors: use large models
to generate data that train downstream, lightweight models used for
threat detection, which can reduce infrastructure costs while keeping
the same level of accuracy.
NVIDIA performed many experiments and trained several
cybersecurity-specific foundation models, including one based on GPT-2
style models referenced as CyberGPT. One of those is a model that is
trained on identity data (including application logs like Azure AD).
With this model, one can generate highly realistic synthetic data that
addresses a data gap and can perform “what if” scenarios.
Figure 2 shows the Rogue2 F1 scores for CyberGPT models of various
sizes, with each instance achieving around 80% accuracy. This means that
8 out of 10 logs generated are virtually indistinguishable from logs
generated by real network users.
As for training times, a supercomputer isn’t necessary to realize
quality results. In testing, training times were as low as 12 GPU hours
for a GPT-2-small model with character-level tokenization. This model is
trained on 2.3M rows of over 100 user logs with 1,000 iterations. This
model was trained on multiple types of data, including Azure,
SharePoint, Confluence, and Jira.
Experiments were also run with tokenizers–primarily character-level
tokenizers, off-the-shelf byte pair encoding (BPE) tokenizers, and
custom-trained tokenizers. While there are benefits and drawbacks to
each, the best performance comes as a result of training custom
tokenizers. This not only enables more efficient use of resources due to
the custom vocabulary, but it results in reduced tokenization errors
and can handle log-specific syntax.
While these results reflect experiments with language models, the same tests with LLMs achieve similar results.
Synthetic data generation provides 100% detection of spear phishing e-mails
Spear phishing e-mails are highly targeted, and therefore, very
convincing. The only real difference between a spear phishing (and, in
general, any effective phishing campaign) and a benign e-mail is the
intent of the sender. This makes spear phishing challenging to defend
against with AI because there is a lack of available training data.
To explore the potential of synthetic data generation in enhancing
spear phishing e-mail detection, a pipeline was constructed using NVIDIA
Morpheus.
With off-the-shelf models, the spear phishing detection pipeline
missed 16% (about 600) of malicious e-mails. The uncaught malicious
e-mails were then used to create a new synthetic dataset. A new intent
model was learned from the synthetically generated e-mails, and
integrated into our spear phishing detection pipeline. The addition of
this new intent model feature in the detection pipeline resulted in 100%
detection of spear phishing e-mails trained solely on synthetic
e-mails.
The NVIDIA AI platform
is uniquely positioned to help address these challenges–building in
security at multiple levels. At the hardware infrastructure level, and
beyond the data center perimeter to the edge of every server, while also
providing tools that help to secure your data with AI.
Learn more
Watch the session from Bartley Richardson, head of cybersecurity
engineering at NVIDIA, to see demonstrations of the use cases
illustrated in this post. Learn about integrating language models and
cybersecurity featured at NVIDIA LLM Developer Day.
Check out the November 2023 release of NVIDIA Morpheus to access the new LLM engine integration feature, and get started with accelerated AI for cybersecurity.
Find out how NVIDIA NeMo provides an easy way to get started with building, customizing, and deploying generative AI models.
NVIDIA Morpheus and NeMo are included with NVIDIA AI Enterprise, the enterprise-grade software that powers the NVIDIA AI platform.
IDC FutureScape: Worldwide Artificial Intelligence and Automation 2024 Predictions, #AP50341323, October 2023 ↩︎