By Jesse Oliver Pinkston Dungan in Newsletter — 30 Jun 2025

Bonus Episode 5.25 - AI on AI

This is what my Microsoft Copilot AI wanted me to write about how dangerous AI will be with respect to financial frauds and scams. Not the worst advice in the world. But definitely reads like an AI wrote it (or a corporate drone middle-manager edited it 😏). I made no changes to the draft, except for modifying some of the formatting (mainly shrinking and making consistent some headings) which is really quite funny, since I might have assumed that the AI would be the best at making it look good and readable. Although, I confess this was first done in a word document and then pasted into here. So probably my fault. Remember, my prompt was: "Explain in 2 pages why AI content creation and sophisticated LLMs may lead to more financial fraud and higher quality scams and what people can do to protect themselves against these scams." And I would note, in my defense, that the AI blew right through the page limit although it is still substantially smaller than my actual newsletter post; though maybe it should have used footnotes.

Jesse has been locked outside the airlock. Hal 9000 will take your finances from here!

The Rise of AI-Powered Financial Fraud: Risks and Resilience Strategies

Navigating the Threat of Sophisticated Language Models in Modern Scams

Introduction

The digital age has ushered in unprecedented tools for communication and innovation, with artificial intelligence standing at the forefront. Among the most transformative advancements are large language models (LLMs) and AI-driven content creation tools. These technologies, designed to understand and generate human-like text, have revolutionized industries from customer service to entertainment. However, as with all powerful technologies, their misuse can pose serious risks. Notably, financial fraud and scams are evolving to exploit AI's capabilities, resulting in schemes that are more convincing, scalable, and difficult to detect than ever before. This document explores why AI content creation and sophisticated LLMs contribute to the rise of high-quality financial scams, and outlines vital strategies individuals and organizations can use to protect themselves against these emerging threats.

How AI Content Creation and LLMs Enable Financial Fraud

1. Hyper-Realistic Communication

At the core of successful scams lies the ability to deceive. Traditional phishing emails or fraudulent messages often reveal themselves through poor grammar, awkward phrasing, or generic templates. Advanced LLMs, however, can craft messages that are virtually indistinguishable from those written by real humans. These models can adapt tone, mimic organizational jargon, and create highly personalized messages using data scraped from social media or previous correspondence. The result is a new breed of scam communication that is far more likely to bypass the skepticism of even the most vigilant individuals.

2. Automation and Scale

Previously, scammers were limited by the need to manually craft messages or deploy basic templates. With AI, fraudsters can automate the generation of personalized emails, texts, or messages at scale, targeting thousands or even millions of individuals simultaneously. LLMs can generate endless variations of scam messages, making traditional spam filters less effective. Automation also means that targeting can be hyper-specific, with AI generating content tailored to individuals’ professions, interests, or recent activities.

AI content creation extends beyond text. Voice synthesis and deepfake technologies can produce audio and video content that convincingly mimics real people. Scammers can use these tools to impersonate executives, colleagues, or trusted officials in phone calls or video messages, adding a layer of authenticity that textual scams alone could not achieve. Combined with LLMs, these multi-modal scams can orchestrate complex social engineering attacks that manipulate victims into divulging sensitive information or transferring funds.

4. Exploiting Data Breaches and Public Information

LLMs can process and organize vast troves of leaked or publicly available data, allowing scammers to conduct highly targeted attacks. For instance, after a data breach, fraudsters may use AI to analyze and synthesize information about a victim’s financial habits, communication style, or recent transactions. This intelligence enables them to craft scams that are not only believable, but also contextually relevant, making them far more dangerous.

5. Evolution of Scam Quality and Tactics

As AI models continuously improve, so too does the sophistication of scams. LLMs can “learn” from failed attempts, analyze responses, and adjust tactics in real time. Scammers can deploy A/B testing using AI-generated content to identify which approaches yield the highest response rates. This iterative process rapidly refines scam methodologies, raising the bar for detection and prevention efforts.

Real-World Examples of AI-Powered Scams

· Business Email Compromise (BEC): An attacker uses an LLM to impersonate a CEO, instructing an employee to transfer funds to a fraudulent account. The language and style match the executive’s previous emails, making the request seem genuine.

· Phishing Campaigns: AI generates personalized emails that refer to recent purchases, tax filings, or other life events pulled from public data, increasing the likelihood that a recipient will click on a malicious link or provide sensitive information.

· Deepfake Voicemails: A scammer leaves a realistic voicemail using synthesized speech that mimics a relative’s voice, urgently requesting financial help.

· Fraudulent Customer Support: AI chatbots masquerade as support agents, tricking users into sharing account credentials under the guise of troubleshooting.

Why These Scams Are More Dangerous Than Before

The principal danger of AI-driven scams lies in their credibility and adaptability. Where once a misspelled word or odd phrase might have alerted someone to a scam, LLMs now generate content that is grammatically flawless and contextually appropriate. Additionally, these scams can rapidly pivot and evolve in response to failed attempts, making them a moving target for defenders. The emotional manipulation inherent in personalized, realistic communication can compromise judgment, especially under stress or time pressure.

Protecting Yourself Against AI-Driven Financial Fraud

The fight against sophisticated scams is not hopeless. By understanding the risks and adopting a layered defense, individuals and organizations can significantly reduce their vulnerability.

1. Education and Awareness

Staying informed is the first line of defense. Recognize that anyone can be targeted by high-quality scams, regardless of technical expertise or experience. Regularly educate yourself and others about the latest tactics used by scammers, especially those that incorporate AI.

· Participate in security awareness training that includes information on AI-generated scams.

· Be skeptical of unsolicited requests for money or sensitive information, even if they appear authentic.

· Encourage a culture of caution—it's better to double-check than to fall victim.

2. Verification Procedures

Implement robust verification processes for any financial transactions or sensitive operations.

· Always verify requests for money transfers or changes in payment instructions through a separate communication channel (e.g., phone call to a known number).

· Use two-factor authentication for financial accounts and sensitive systems.

· Develop and enforce internal protocols for handling unexpected or urgent requests, especially those purporting to come from executives or vendors.

3. Technical Safeguards

Leverage technology to defend against technology-driven threats.

· Invest in advanced email filtering solutions that use machine learning to detect anomalous messages.

· Use anti-phishing software and regularly update security systems to address emerging threats.

· Monitor accounts and transactions for unusual activity, and set up alerts for large or suspicious transactions.

4. Data Privacy and Security

Limit the amount of personal information available online, as scammers often use public data to craft convincing attacks.

· Be cautious about the information you share on social media and professional networking sites.

· Periodically review privacy settings and remove any unnecessary details about your work, finances, or personal life.

· Advocate for strong data protection policies in organizations and encourage partners to do the same.

5. Reporting and Response

If you suspect you've been targeted by a scam, swift action is essential.

· Report suspected fraud to financial institutions and relevant authorities immediately.

· Document any suspicious communications or transactions for use in investigations.

· Learn about and use resources provided by consumer protection agencies.

Conclusion

The rise of AI content creation and large language models has fundamentally changed the landscape of financial fraud. While these tools offer immense benefits to society, their misuse by malicious actors presents a growing threat that cannot be ignored. Scams powered by AI are more convincing, more scalable, and more adaptive than ever before. However, by arming ourselves with knowledge, employing rigorous verification, and leveraging both personal and technological safeguards, we can outpace the fraudsters and protect our finances and futures in the digital era. Remaining vigilant, skeptical, and proactive is the best defense against even the most sophisticated scams AI can engineer.