Generative AI a new headache for cybersecurity

Author Moonshot.news

Published on: 13/01/2025

Generative AI is augmenting cybercriminal capabilities, contributing to an uptick in attacks manipulating people to share information they shouldn’t share, World Economic Forum says in a cybersecurity outlook. Microsoft’s Digital Crimes Unit is initiating a US court case to stop cybercriminals developing tools specifically designed to bypass the safety guardrails of generative AI services to create offensive and harmful content. 

“Ultimately, overcoming today’s challenges requires not just technological innovation but a shift in perspective. Cyber resilience must be recognised as a collective responsibility, with organisations of all sizes working together to fortify the interconnected networks that underpin the digital economy.” 

“Further, there is a need for decisive leadership action to prioritise cybersecurity among and between organisations; beyond technical indicators, robust criteria rooted in the economic implications of cyber insecurity will be required. A united leadership team, in which business and cyber leaders see eye to eye on the cyber risks facing the organisation, is critical to navigating growing cyber complexity.”

“While 66% of organisations expect AI to have the most significant impact on cybersecurity in the year to come, only 37% report having processes in place to assess the security of AI tools before deployment”, the WEF outlook reports. 

“This reveals the paradox of the gap between the recognition of AI-driven cybersecur

ity risks and the rapid implementation of AI without the necessary security safeguards to ensure cyber resilience.”

WEF reports that 72% of respondents have an increase in organisational cyber risks, with ransomware remaining a top concern. 

“Nearly 47% of organisations cite adversarial advances powered by generative AI (GenAI) as their primary concern, enabling more sophisticated and scalable attacks.” 

“In 2024 there was a sharp increase in phishing and social engineering attacks, with 42% of organizations reporting such incidents.” 

WEF says that regulations are increasingly seen as an important factor for improving baseline cybersecurity and building trust. 

“However, their proliferation and disharmony are creating significant challenges for organisations, with more than 76% of chief information security officers (CISOs) at the World Economic Forum’s Annual Meeting on Cybersecurity in 2024 reporting that fragmentation of regulations across jurisdictions greatly affects their organisations’ ability to maintain compliance.”

“Since 2024, the cyber skills gap has increased by 8%, with two out of three organisations reporting moderate-to-critical skills gaps, including a lack of essential talent and skills to meet their security requirements. Furthermore, only 14% of organisations are confident that they have the people and skills they need today.”

The 2025 report finds that a series of compounding factors are driving an escalating complexity in the cyber landscape: 

– Geopolitical tensions are contributing to a more uncertain environment. 

– Increased integration and dependence on more complex supply chains are leading to a more opaque and unpredictable risk landscape. 

– The rapid adoption of emerging technologies is contributing to new vulnerabilities and new threats. 

“Ransomware remains the top organisational cyber risk year on year, with 45% of respondents ranking it as a top concern in this year’s survey.”

WEF says that significant innovations in ransomware attacks can be expected. 

“This is compounded by the continued adoption of Ransomware-as-a-Service (RaaS), entrenching the commoditisation of the ransomware model. 

Cyber-enabled fraud ranks as the second-highest organisational cyber risk for 2025, viewed by CEOs as a significant threat alongside ransomware and supply chain disruptions. 

“At the same time, identity theft climbs to the top of the agenda, emerging as the primary personal cyber risk for both CISOs and CEOs.”

Nearly 60% of organisations state that geopolitical tensions have affected their cybersecurity strategy. 

Geopolitical turmoil has also affected the perception of risks, with one in three CEOs citing cyber espionage and loss of sensitive information/ intellectual property (IP) theft as their top concern, while 45% of cyber leaders are concerned about disruption of operations and business processes. Supply chain vulnerabilities are emerging as the top ecosystem cyber risk 

The Microsoft case is initiated at the Eastern District of Virginia. The company says that cybercriminals remain persistent and relentlessly innovate their tools and techniques to bypass even the most robust security measures. 

“With this action, we are sending a clear message: the weaponisation of our AI technology by online actors will not be tolerated.”  

As alleged in our court filings, Microsoft has observed a foreign-based threat–actor group developing sophisticated software that exploited exposed customer credentials scraped from public websites. 

“In doing so, they sought to identify and unlawfully access accounts with certain generative AI services and purposely alter the capabilities of those services.” 

“Cybercriminals then used these services and resold access to other malicious actors with detailed instructions on how to use these custom tools to generate harmful and illicit content.” 

The company says it has revoked cybercriminal access, put in place countermeasures, and enhanced its safeguards to further block such malicious activity in the future.                   

“ Every day, individuals leverage generative AI tools to enhance their creative expression and productivity. Unfortunately, and as we have seen with the emergence of other technologies, the benefits of these tools attract bad actors who seek to exploit and abuse technology and innovation for malicious purposes.”