# 05.29.2026 - AI/Qwen3.6 27B Quantization Benchmark

This is my attempt to benchmark and compare the quality of some of the well known Qwen3.6 27B quantizations on HuggingFace (unsloth, mradermacher, IQ4\_XS from cHunter789 and Ununnilium), from Q8 all the way down to Q2.

## Measurement method

I'm using llama.cpp's `llama-perplexity` to measure the **mean KLD** and **Same Top P Percentage** between the quantized model and the base (BF16 version).

All runs were using the same context length of 8192 tokens, KV cache quantized to q8\_0 so I can make sure the entire model fit in the GPU.

First, I obtained the logits of the BF16 version with this command:

```
llama-perplexity -m <MODEL_PATH> \
	-f wiki.test.raw -c 8192 -ngl 99 -ctk q8_0 -ctv q8_0 \
	--kl-divergence-base logits-bf16.dat
	--kl-divergence
```

Then for each model, I ran the test with:

```
llama-perplexity -m BF16/Qwen3.6-27B-BF16-00001-of-00002.gguf \
	-f wiki.test.raw -c 8192 -ngl 99 -ctk q8_0 -ctv q8_0 \
	--kl-divergence-base logits-bf16.dat
```

## Understand KLD and Same Top P

To understand the test result, it would be useful to understand the difference between the two metrics I used.

When an LLM predicts the next word of a given prompt, for example **"Today I will do my"**, it looks at its entire vocabulary and assigns a confidence score to every single token. Then samples the top tokens and pick the final one, based on the given temperature.

- **KL Divergence (KLD)** measures how much the confidence distribution of the quantized model drifts away from the base. In this example, the base model might assign 90% confidence to "homework", 5% to "bike" and 1% to "banana". But the poorly quantized one might give 50% to "homework", 30% to "bike" and "20%" to "banana".
- **Same Top P** tracks how often the quantized model picks the same token as the base model. In this example, the model might just pick "homework" as the next token for the prompt.

So, while you might get a good token choice with the quantized model (**Same Top P** is high), it's important to look at the **Mean KLD** to see how stable the inner probability of the model is, the lower, the better.
 
## Benchmark result

### Unsloth's quantization

![](_meta/qwen36_27b_frontier_inset.png)

Nothing special, higher quants are better than lower quants. Q6 to Q8 are pretty much lossless. You can see Q8\_0 has a higher **Same Top P**, but underlying, the **Mean KLD** tells us that UD-Q8\_K\_XL is better. Anything below Q4 are for the desperate, like the 5060ti 16GB club.

The 4-bit cluster is a bit more interesting. Different people may have a different take on this, but to me, Q4\_K\_XL is a good quality-compromise if you can afford the VRAM. If you're tight, IQ4\_XS could serve you well, IQ4\_NL is not much difference. And in that case, there's no need to stretch for Q4\_K\_M. You can skip Q4\_K\_S.

From Q3\_K\_XL, the quality degradation is more drastic. The KLD went all above 0.1 and matching token selection dropped to 90-85% can tell a lot about the instability.

### mradermacher's and other quants

I've seen people mention mradermacher's i1 quants here and there, and also IQ4\_XS quants from cHunter789 and Ununnilium. I have been personally using Ununnilium's IQ4\_XS for a while now. So I want to put them all on the same table to see how they fit. But a single diagram will not be enough so I will break them into 4 groups: Q8-Q6, Q5, Q4 and Q3-below.

#### 8-bit and 6-bit quantization

![](_meta/q8-q6.png)

mradermacher's Q6\_K seems to be a clear winner over Unsloth's Q6\_K here. The mean KLD is near perfect (0.027352), and 97.011% token selection match.

#### 5-bit quantization

![](_meta/q5.png)

In this group, Unsloth is a winner. With about 300-500MB difference in size, you can skip Q5\_K\_S and go for Q5\_K\_M. Unsloth's Q5\_K\_M is clearly better in both matching token selection and KLD.

#### 4-bit quantization

![](_meta/q4.png)

Unsloth beats all of the 4-bit quants here. But if you are looking for some alternative quants to save VRAM, like ones on 16GB, pay attention to IQ4\_XS (it will help but of course, you will not be able to get above 65k context window).

mradermacher's IQ4\_XS is a clear winner among all the other IQ4\_XS quants, but at 15.1 GB, it would be a bit tight. cHunter's IQ4\_XS is also very good at 14.7 GB.

#### 3-bit and below

![](_meta/q3.png)

Again, mradermacher's quants filled in the gap between Unsloth's quants here, so you get a bit more choice, but tbh, at this range, you better off with Unsloth's Q3\_K\_XL or at least Q3\_K\_M.

I was very interested to see how some new quants like IQ3\_S, IQ3\_M perform, but they turned out a bit disappointed.

## Raw benchmark data

If you are interested, here's the raw benchmark data table after all the run.

| Quantization           | Mean PPL(Q) | Mean KLD | RMS Δp (%) | Same top p (%) |
|------------------------|-------------|----------|------------|----------------|
| UD-Q8_K_XL             | 6.569706    | 0.015495 | 2.448      | 97.407         |
| Q8_0                   | 6.567807    | 0.020497 | 2.701      | 97.753         |
| UD-Q6_K_XL             | 6.541421    | 0.023398 | 2.903      | 97.436         |
| mradermacher/Q6_K      | 6.541627    | 0.027352 | 3.045      | 97.011         |
| Q6_K                   | 6.566514    | 0.027766 | 3.014      | 97.112         |
| UD-Q5_K_XL             | 6.625155    | 0.045526 | 4.021      | 96.187         |
| Q5_K_M                 | 6.658295    | 0.05277  | 4.26       | 95.864         |
| mradermacher/Q5_K_M    | 6.630279    | 0.053246 | 4.372      | 95.664         |
| mradermacher/Q5_K_S    | 6.613859    | 0.055034 | 4.476      | 95.505         |
| Q5_K_S                 | 6.652629    | 0.055888 | 4.414      | 95.674         |
| UD-Q4_K_XL             | 6.647006    | 0.06656  | 5.023      | 94.621         |
| Q4_K_M                 | 6.672841    | 0.070345 | 5.334      | 94.228         |
| IQ4_NL                 | 6.619131    | 0.071724 | 5.497      | 94.106         |
| IQ4_XS                 | 6.61994     | 0.072223 | 5.481      | 94.016         |
| mradermacher/IQ4_XS    | 6.611545    | 0.073705 | 5.648      | 93.852         |
| mradermacher/Q4_K_M    | 6.685347    | 0.074124 | 5.507      | 94.08          |
| cHunter/IQ4_XS-i1      | 6.656157    | 0.075933 | 5.645      | 93.77          |
| Q4_K_S                 | 6.690623    | 0.078947 | 5.72       | 93.833         |
| mradermacher/Q4_K_S    | 6.642023    | 0.080407 | 5.825      | 93.657         |
| Ununnilium/IQ4_XS-pure | 6.765894    | 0.084115 | 6.127      | 92.407         |
| UD-Q3_K_XL             | 6.620281    | 0.105386 | 7.077      | 91.837         |
| Q3_K_M                 | 6.453757    | 0.129404 | 7.893      | 90.437         |
| mradermacher/Q3_K_L    | 6.482496    | 0.136127 | 8.116      | 90.213         |
| mradermacher/Q3_K_M    | 6.481299    | 0.140487 | 8.424      | 89.934         |
| mradermacher/IQ3_XS    | 6.981601    | 0.161364 | 9.182      | 88.767         |
| UD-IQ3_XXS             | 6.994512    | 0.176688 | 9.626      | 87.953         |
| mradermacher/IQ3_S     | 7.405328    | 0.176782 | 9.637      | 88.689         |
| Q3_K_S                 | 7.068685    | 0.178631 | 9.61       | 87.681         |
| mradermacher/IQ3_M     | 7.454224    | 0.180647 | 9.824      | 88.603         |
| mradermacher/Q3_K_S    | 6.910989    | 0.181172 | 9.82       | 87.422         |
| UD-Q2_K_XL             | 7.316461    | 0.229068 | 11.399     | 85.95          |
| UD-IQ2_M               | 7.468708    | 0.241252 | 11.91      | 85.319         |
| UD-IQ2_XXS             | 8.507239    | 0.40986  | 16.708     | 78.483         |

---

There are many more Qwen3.6 27B quantizations on HuggingFace, like ones from bartowski, huihui,... within my time budget (not money budget, since I'm basically using modal.com's free monthly credit :P), I cannot benchmark them all.

If you are interested in doing your own benchmark, I also attached the script here so you can run it on your own.

→ [kld.py](https://gist.github.com/huytd/ac6457b4581598a198c027e4051380de)

This script was written specifically to benchmark Qwen3.6 27B, please feel free to modify it as you need. The instruction to run is inside the script, please read the code before running.

Thanks for reading this far!


Qwen3.6-27B Quantization Benchmark

# 12.01.2025 - AI/Teaching an LLM a Niche Diagraming Language

Text-to-diagram seems to be an area that has been solved perfectly by LLMs, but only with popular languages like Mermaid or PlantUML. There are many other less popular diagramming languages like D2, Structurizr, or Pintora. I want to see how far I can go teaching an LLM one of these languages, and [Pintora](https://pintorajs.vercel.app/) seems like a nice option to pick because I know there will not be a lot of LLMs trained with this language. At least, not the small ones.

## Project goal

So, I set out the goal for this training project. It will be an LLM that has the ability to work with Pintora diagrams, for the following tasks:

- Generating diagrams from scratch
- Editing any existing diagram

## Picking the base model

I also limit the scope of this project to models that are smaller than 30B, simply because I do not have enough resources to train anything bigger than that.

Having experimented with 4B models and smaller, I found that these models are not really suitable for either a code-generating task or instruction following in a long conversation. A 7B-14B would be a good choice. I also want to base it off a coding model, hoping that it will be, well, code-smart.

Gemma-3 only has 4B and 27B. The smallest Codestral model is 22B. Ministral 3 was not released until 3 days after the writing of this post. I don't want to deal with thinking mode, so Qwen3 is also out of the list. After a lot of consideration, I picked Qwen2.5-Coder-7B. In 2025, it's like centuries old in terms of LLM-years, but whatever.

To confirm that Qwen2.5-Coder does not have any knowledge of Pintora, I tried to run it with the following prompt:

```
create a pintora component diagram for a website like twitter
```

And yes, the model proceeded to generate a very well-made... PlantUML diagram:

```
Below is a component diagram using Pintora syntax. You can copy and paste this into a Pintora editor to visualize it.

@startuml
!define RECTANGLE class

title Twitter Component Diagram

RECTANGLE "User Interface" as UI {
  RECTANGLE "Header" as Header
  RECTANGLE "Footer" as Footer
  ...
```

## The training plan

Since we're teaching the model a new language, we will need to train in two phases:

- **Phase 1: Continued Pretraining (CPT)** - We will feed the model a bunch of Pintora diagrams, so the model can learn the language's syntax, recognize grammar structure, etc.
- **Phase 2: Instruction Finetune (IFT)** - Then we will train the model on some diagram generating/editing instructions, so it can learn how to use the new language's knowledge on specific tasks.

I will use Unsloth's [training notebook](https://docs.unsloth.ai/basics/continued-pretraining), since it supports 4-bit quantized LoRA training, that helped training faster and using less memory.

## Data preparation

First thing's first, I need a dataset that I can use in both the CPT and IFT phases. I will go for the minimum amount of data needed (around 1000-1500 rows).

Pintora supports different kinds of diagrams: Sequence, ER, Component, Activity, Mindmap, Gantt, Class,... I will need a diverse number of data for each case, so that's about 150-200 rows per diagram type.

I want the model to have an ability to either generate a diagram from scratch or edit an existing diagram, so the dataset should also contain some examples that have an input diagram.

The plan is, each row will contain three fields:
- **instruction**: the description of what type of diagram the user wants to create
- **input**: an optional input diagram code for editing
- **output**: the final output code that the model should generate

With that clear plan, I started typing out each row, and after about 5 rows, I gave up... This kind of labor is not productive at all!

Why not just grab some code that people already created? I started searching on Github to see how much Pintora code there is. Not much, there were like 5 or 6 repositories that had some diagram code. Also, I don't like the idea of stealing someone's code without asking for their permission, not to mention, if I actually asked, I'm not sure how many of them would respond.

So, the last resort is to generate training data using AI! There's not much to talk about this step. The trick is to write an over-detailed prompt that includes all the syntax documentation, examples,... then some patience to beg the AI agent every 50 entries, threatening it that an alien Godzilla will destroy the Golden Gate Bridge if it's not completing its job.

At the end of the begging process, I ended up with about 2000 data entries. The result was not great at all, both Gemini 3 Pro and Claude Sonnet 4.5 generated a lot of syntactically incorrect code and a lot of duplicated entries.

To clean it up, I wrote a script to merge every row that has the same `output` column into one, and then, for each row, use the `@pintora/cli` tool to render the actual diagram, removing any rows where the `output` code cannot be used.

In the end, I was left with 1000 rows for CPT and 500 rows for IFT. If you are interested, they are available on Hugging Face, links are at the bottom of the post.

## Training

I started the training process on Google Colab (using a single 16GB T4 GPU) and quickly ran into an OOM issue. The situation was not getting any better with Kaggle's 2xT4 GPUs. So I ended up renting a 48GB A40 on Runpod for $0.4/hr.

It turned out that even for a 7B model with 4-bit QLoRA, my training script took about 19.33GB of VRAM to run, which was too much for 16GB of a T4 (the actual available VRAM was even less than that). But it was an unnecessary problem.

Theoretically, since Pintora language still uses keywords that already exist in most English-based programming languages, the model did not need to learn any new tokens. I could save about 5GB-6GB of VRAM needed by removing the `embed_tokens` and `lm_head` from the `target_modules`.

```python
model = FastLanguageModel.get_peft_model(
	model,
	r = 64, 
	target_modules = [
		"q_proj", "k_proj", "v_proj", "o_proj", 
		"gate_proj", "up_proj", "down_proj",
		"embed_tokens", "lm_head", # could have removed this
	],
	lora_alpha = 64,
	lora_dropout = 0.05,
	use_gradient_checkpointing = "unsloth",
	...
)
```

Back to the training process. After the CPT phase, I ran a test to see how well the model learned the syntax.

![](_meta/pintora-cpt-example.png)

Since it started to pick up some syntax characteristic of Pintora, the diagram code is still syntactically incorrect.

In the next step, I loaded the `pintora-edit-instruct` dataset, with each entry formatted with this `edit_prompt`, and started the IFT phase.

```
edit_prompt = """Pintora Diagram Edit Instruction

### Instruction:
{instruction}
{input}

### Response:
{output}
"""
```

After this step, the model already learned to generate more accurate and syntactically correct code, for both generating from scratch and editing tasks.

**Generate diagram from scratch**

![](_meta/pintora-example-1.png)

**Editing existing diagram**

![](_meta/pintora-example-2.png)

So to this point, I have successfully taught Qwen2.5-Coder how to generate Pintora diagrams instead of spitting out random Mermaid/PlantUML diagrams. But how well has it learned?

## Evaluation for accuracy

To quickly evaluate the accuracy of the generated diagram (not the quality), I ~~vibed~~ created a script to use the model to generate with some randomized prompts:

```python
...

entities = [
	'User', 'Client', 'WebApp', 'Backend', 'Server', 'Database', 'AuthService',
	'PaymentGateway', 'Cache', 'Redis', 'Worker', 'TaskQueue', 'Frontend',
	'API Gateway', 'OrderSystem', 'Inventory', 'NotificationSvc', 'Logger', 'MetricsSvc'
]
actions = [
	'requests login', 'fetches data', 'updates record', 'processes payment',
	'validates token', 'sends email', 'renders view', 'queries index',
	'health check', 'ack signal', 'authenticates user', 'writes to log',
	'queries for user profile', 'returns 200 OK', 'returns 404 Not Found',
	'submits form', 'enqueues job', 'dequeues job', 'generates report'
]
diagram_types = ['sequenceDiagram', 'componentDiagram', 'activityDiagram']

def create_from_scratch_task():
    d_type = random.choice(diagram_types)
    num_interactions = random.randint(1, 3)
    interactions = []
    for _ in range(num_interactions):
        src, dst = random.sample(entities, 2)
        action = random.choice(actions)
        interactions.append(f"{src} {action} to {dst}")
    prompt_desc = ", and then ".join(interactions)
    instruction = f"Create a {d_type} that shows: {prompt_desc}."
    output_code = model.generate(instruction)
    return [instruction, "", output_code]

for i in range(1000):
	create_from_scratch_task()
	...
```

Some example result:

|instruction                                                                        |input                                                |output                                                                                        |
|-----------------------------------------------------------------------------------|-----------------------------------------------------|----------------------------------------------------------------------------------------------|
|Create a activityDiagram that shows: PaymentGateway returns 404 Not Found to...    |                                                     |activityDiagram start :Worker requests PaymentGateway; if (PaymentGateway returns 404)...     |
|Add a step where User health check to MetricsSvc.                                  |sequenceDiagram Cache->>User: enqueues job           |sequenceDiagram Cache->>User: enqueues job User->>MetricsSvc: health check                    |
|Add a step where PaymentGateway health check to Inventory.                         |sequenceDiagram Worker->>PaymentGateway: enqueues job|sequenceDiagram Worker->>PaymentGateway: enqueues job PaymentGateway->>Inventory: health check|

Then, I use the same technique in the data preparation step, deduplicate the result, and parse each `output` code with the `@pintora/cli` command.

In the end, out of 996 diagrams, we have 139 diagrams with syntax errors and 857 diagrams successfully rendered. That gives us **86%** accuracy, not bad for such a really small amount of training data.

## Final thoughts

There were a lot of learnings for me during this experiment, and countless mistakes that I could have done better, but ultimately, I had a lot of fun. Maybe I'll try to tackle the accuracy next with RL, I heard many good and bad things about it and I must give it a try.

I am also interested in this music programming language called [Strudel](https://strudel.cc/), and it would be fun to train an LLM for it.

In the meantime, if you are interested, here's the model (with GGUF) and the datasets, as well as the eval result below:

**Model:**
- https://huggingface.co/huytd189/pintora-coder-7b 
- https://huggingface.co/huytd189/pintora-coder-7b-gguf (GGUF - F16, Q8, Q4_K_M)

**Dataset:**
- https://huggingface.co/datasets/huytd189/pintora-instruct
- https://huggingface.co/datasets/huytd189/pintora-edit-instruct

**Eval result:**
- [pintora_eval.csv](_meta/pintora_eval.csv)
- [pintora_eval_bad.csv](_meta/pintora_eval_bad.csv)
- [pintora_eval_good.csv](_meta/pintora_eval_good.csv)

<details>
<summary>Now it's time for an ad if you don't mind ;)</summary>
If you want to try AI-assisted diagramming yourself, check out my project, <a href="https://chatuml.com">ChatUML</a>. It's designed to turn text into diagrams instantly. You can grab 60% off right now using the code <b>PINTORA</b>!
</details>
