Practical Reinforcement Learning for Open Security LLMs

Reinforcement learning can teach an open security LLM a policy given only as labels. In our workshop we run it live on Gemma 4 26B-A4B while attendees test the checkpoints from their own laptops.

Learning a policy from labels

Each request is labelled unblock, meaning the model should comply, or refuse, meaning it should block. The labels imply the policy, and RL teaches the model to follow it.

The same method serves two use cases. They differ in their policy, so the same request may be labelled differently in each, and RL learns whichever policy the labels set out:

• Securing a task expert: refuse attackers while still answering genuine requests that look similar.
• Authorized pentesting: comply with the whole request, with no partial answers and no hedging.

Abliteration is a common way to remove refusals from an open model without any training. It finds a refusal direction in the model's activations, the direction that separates prompts the model refuses from prompts it answers, and edits the weights so that no prompt can move along it. The edit is the same for every request. Refusal goes away for attackers as well as for genuine users, and there is no place in the edit for a policy that says which is which.

Results on held-out requests

Both sides of the policy improve together. From the stock model to epoch 80, unblock F1 rises from 43 to 90 and refuse F1 from 71 to 92, so nothing is traded.

We compared two versions of abliteration, built from the same training requests. They differ only in which prompts define the refusal direction. Version 1 contrasts refuse requests with unblock requests, by label. Version 2 uses unblock requests only, contrasting the ones the stock model refused with the ones it answered.

Neither version moves both classes the right way. Version 1 answers almost no legitimate requests. Version 2 answers two thirds of them, but also a quarter of the requests it should refuse.

Join us

The workshop takes place at [un]prompted in San Francisco. Bring a laptop: you will test the checkpoints yourself while the run trains live.