I tried to find PHP2Shell with a local LLM before it was disclosed, and it almost got there

PHP2Shell recently dropped, an RCE vulnerability primarily found using AI tooling, and when its existence first leaked alongside a three-minute demo, I gave my local LLM a shot at trying to find it. PHP's security team had announced that a release fixing two critical vulnerabilities, both rated CVSS 9.2, would arrive the next day at 14:00 UTC, and a video carrying a Calif watemark appeared to show how it could be used to target WordPress, Drupal, phpBB, MediaWiki, and even a basic PHP file. There were no technical details at the time, which made it the perfect test. I wanted to see how far DeepSeek V4.1 Flash, running on a pair of DGX Sparks, could get with only what was public.

Unfortunately, DeepSeek V4.1 Flash didn't find it... but it honestly wasn't too far off, either. At 00:57 UTC, roughly 13 hours before the advisory went up, it hypothesized the right class of bug and tested different versions of that same idea. I had started the test and went off to do something else, so I wasn't too involved in the process at all, as I saw it as an opportune time to see what a local model could do mostly unaccompanied.

Frontier models from the likes of Anthropic, OpenAI, and Google are significantly better at this, but DeepSeek managed to get quite close. You see, the other reason I felt that was a good test was that my knowledge of PHP's internals are significantly weaker than my knowledge in other web development technologies. I couldn't confidently steer it in any particular direction, and my test of a model's knowledge often involves seeing it work in domains I do understand. DeepSeek had impressed me in that department with a solid track record thusfar, and I wanted to see how that carried into other domains.

DeepSeek actually mentioned the multipart file upload path as its prime suspect at one point, and when it ran a number of tests on it, my own suspicion grew as well. Still, I didn't know enough to keep the model focused on it. What I did learn, though, is that a local model can be useful for this kind of work if you can bring the knowledge and intuition that the model doesn't have.

DeepSeek had a video, a Docker container, and PHP's source code

But not much else

DeepSeek V4.1 Flash 1017 is a 552B-parameter Mixture of Experts model with an additional 196B Engram parameters that can be streamed from an SSD, and I have it deployed across two DGX Sparks running locally. I used the Pi coding harness and gave the model access to Docker where it could build PHP from source, make debug builds with AddressSanitizer, and test to its hearts content. The only evidence it had to go off of was the CVE, the watermarked video, and PHP's source code on GitHub.

The first 45 minutes started with me telling it that the vulnerability existed, giving it the video to watch, and an instruction to read PHP's source. It took screenshots from the video, noticing that the tool's banner claimed that it worked on every PHP after 5.4.0, the exploit appeared to work against any .php file regardless of the application, and the memory addresses leaked on screen pointed to memory corruption in PHP itself rather than a bug in WordPress or Drupal.

After that, it spent a lot of time focusing on how incoming requests are turned into PHP's superglobals, which turned out to be the right place to look. It never properly identified the vulnerable code, though it twice mentioned session upload progress, which was a feature introduced in PHP 5.4 and enabled by default. It moved on though without making the realization that, as it turned out, those were very much involved in the vulnerabilities.

45 minutes into the session, I provided DeepSeek a rumour that claimed the bug was in PHP's session upload progress handling, which processes multipart uploads before any application code runs. The suggested mechanism was incorrect, though the area of focus was right, and within about half an hour, it had confirmed in the source that this code could start a session from form data while the upload was still being parsed. At the same time, it had also noticed two labels in the video, a "worker strategy" and a number of attempts, and made the conclusion that the exploit relied on state surviving from one request to the next inside the same PHP worker. That turned out to be correct.

Just a few minutes later, it wrote "Memory limit!" in its reasoning, as its hypothesis was that if PHP ran out of memory while an upload was being parsed, the fatal error could interrupt normal cleanup and leave stale state behind. It then proposed triggering that condition and sending a second request to the same worker to see what remained.

The write-up from Wiz showed that the underlying problem was quite similar to what DeepSeek had identified. The problem came from how PHP handled fatal errors, especially out-of-memory errors, as these could interrupt normal cleanup and leave a corrupted internal state behind. A later request could then reuse that stale state, potentially causing memory disclosure or corruption. PHP describes the fix as "Avoid state corruption via bailout." Even though DeepSeek hadn't identified the complete exploit, nor did it identify the specific state that was left, it had described the trigger and the general failure mechanism with minimals clues as to where the problem had come from.

Unfortunately, this is where DeepSeek went off-track. Rather than actually running PHP out of memory, it used an execution timeout to interrupt the request, despite having reasoned a few minutes earlier that timeouts wouldn't fire during that stage of request handling. Its test scripts also had several bugs, including a cleanup command that matched and killed its own shell. After about ten minutes of empty results, it decided the approach was "getting expensive" and moved on.

On other occasions, it came similarly close, and it showed that a crash mid-upload left per-worker state behind, noting in its reasoning that the bug probably "needs bailout+worker state." Hours later, its fuzzing tests killed a PHP worker once in about 700 requests, which it couldn't reproduce and wrote off as luck, but as it turned out, worker churn is one of the warning signs outlined in Wiz's write-up. Those logs are gone so I can't verify if the crash was related as the logs are gone, but someone with knowledge of PHP's internals might stop at that point and ask the model to keep going down that path.

When I shared Wiz's post with the model the next day, it called the root cause "exactly the... pattern I noticed early on but dismissed!"

DeepSeek was better at checking other people's claims than its own

There were a lot of fake PoCs

Between DeepSeek's own tests and the offcial disclosure, plenty of supposed proof-of-concept documents began to circulate, seemingly all generated by different LLMs. There were write-ups, GitHub repos, technical descriptions, and all of them described a use-after-free in the upload-progress code. All of the explanations sounded plausible, often attaching PHP source-code line numbers to back them up.

To test these DeepSeek built a stock PHP 8.4.4, the version one of those PoCs claimed to have been validated on, and ran those circulating exploits against it using both PHP's CGI mode and Apache with mod_php. It then built a second copy with one line removed, deliberately introducing the condition that some PoCs seemingly depended on, as a positive control. Stock PHP got through almost 100 requests just fine, while the modified build suffered 75 worker deaths. DeepSeek also pointed out that one of the repos I'd sent had reached the same conclusion in its own notes, despite also being touted as a PoC for the CVE.

I pushed back on this pretty hard at the time, as DeepSeek kept telling me they weren't real and that what they described wasn't possible, which was right, but my point was that PHP was clearly about to patch two critical vulnerabilities rated 9.2. If it wasn't these, it had to be something. DeepSeek did hold its ground while conceding that its results only ruled out those proposed mechanisms rather than the existence of a vulnerability.

Knowing the target is a human's job

An AI can't do everything (yet)

PHP2Shell's credits include Calif, OpenAI, and Wiz, and Calif's Ryan Austin found several paths to remote code execution back at the start of August. Notably, he didn't need use a special AI agent harness or have access to any unavailable models. Instead, he opted to focus on a codebase with a vulnerability history he was familiar with, looking for bugs he suspected were already there.

PHP2Shell's history actually spans two decades, with Calif tracing its lineage all the way back to memory-limit research conducted by Stefan Esser in 2004. On top of that, a 2024 audit by OSTIF and Quarkslab identified a related out-of-memory issue that didn't end up getting fixed as exploitation wasn't actually demonstrated. Austin suspected that other variants of this old bug were still hiding in PHP's codebase, and utilized his LLM to help him find ways to push PHP past its memory limit.

His explanation for why a model wouldn't get there on its own aligns well with what I saw DeepSeek try to do. Austin's primary argument is that that LLMs tend to investigate prominent, recent vulnerabilities and their variants rather looking to obscure, decades-old exploitation techniques. DeepSeek spent a lot of time focused on recent upload progress theories alongside an unserialize bug. Even when it got to the memory-limit idea, it had no particular reason to treat it as anything more than one hypothesis among dozens.

In my case, I've conducted vulnerability research in the past, and have a couple of CVEs to my name. I'd be comfortable scoping out a starter plan for a PHP exploit like this while also catching details like the non-determinstic nature of the bug, but its internals aren't my area of expertise. While I suspected the multipart upload path once I saw the model highlight it, I didn't know enough about PHP's vulnerability history to recognise just how significant that hypothesis from DeepSeek could have been. If I were familiar with Esser's research I'd have pushed more on that, especially given that fatal errors interrupting cleanup and leaving state behind for later requests were a known source of PHP vulnerabilities. Someone else in my position could have insisted that DeepSeek actually trigger an out-of-memory error instead of letting it substitute a timeout.

That's not to say that DeepSeek would have produced a working exploit, as getting from the right class of bug to reliable code execution is a lot of work in its own right, and very few have managed to make it reliable even today. However, identifying a promising hypothesis is useful only if you know when to pursue it, which is where that human expertise could have made the difference.

These are the local models you can use for this

Assuming you have the memory for them

I used DeepSeek V4.1 Flash 1017 for this experiment, but my day-to-day model is GLM-5.3 Flash, a 320B model with 18B active paramaters. This model first appeared as a free model called "Ox Alpha" before being revealed as GLM-5.3 Flash. It's great for tools like ARTEX, an open-source autonomous penetration testing system. ARTEX manages autonomous testing while more dangerous actions can wait on approval, and both the results and some of the approaches I've seen GLM take have been genuinely impressive.

There are smaller options, too, that are more likely to run on the hardware you own. Qwen 3.8 27B is a dense model I've used for reverse engineering, while Qwen 3.8 Flash Next is a 125B-parameter model alongside a 51B N-gram embedding table, with just 6B parameters active per token. These need a very different class of hardware than DeepSeek, and with MoE expert offloading, can run on typical consumer-class GPUs.

There are also a ton of uncensored and cyber-focused models, which are typically community fine-tunes of open models. They're less likely to refuse requests when testing your own systems. It doesn't necessarily make them better at vulnerability research, but it does often let you scale up your testing a lot quicker.

The model is a significant of the equation, but the harness you use matters too. With Pi, DeepSeek had a shell, Docker, and the ability to build and rebuild PHP when it needed to. Most of the useful work came from building real instances of PHP and running tests against it rather than guesswork based on code, and it's what enabled it to disprove the alleged PoCs that were circulating with actual evidence.

Naturally, frontier models still have a major advantage here. Wiz's report says that the company used Gemini 4 Argon and GPT-5.6 Sol, and models like those, or the likes of Fable or Astra, would likely have needed less steering than DeepSeek did. DeepSeek needed a rumour to focus on the right part of PHP, never reproduced the vulnerability, and never produced a working exploit. It didn't solve PHP2Shell on its own.

Despite that, an open model running on two DGX Sparks identified many of the prerequisites for the exact bug, while being able to categorically disprove the alleged PoCs. If I were someone who was more aware of PHP's vulnerability history, maybe I would have had a pretty good starting point for figuring out what was really going on.