Job posting triage: four engines, one labelled test split

Four engines answered the same four questions about the same postings: each saw 1,000 postings from the test split of the public fake job postings dataset, drawn with seed 0 as a uniform random sample without replacement. A question is scored on the rows that carry a gold answer: 1,000 for fraudulent, 627 for required_experience, 520 for required_education, 1,000 for salary_range_stated. Every answer, its latency, its confidence and its cost were recorded. The fraud question holds 53 positive postings in this sample.

The repository The full report

94.7%majority-class floor
0.700best F1, tfidf (fitted)
0.780best PR-AUC, tfidf (fitted), base rate 0.053
1arm with no spread

Fraud triage

The two floor rows carry a neutral rule and a floor chip. A 3px rule in teal marks an arm that clears the 94.7% majority floor on this question, crimson marks one that does not. Precision, recall and F1 belong to the fraudulent class, the rare one, and the number under each of them is its 95% interval (the file records the interval frame as: 53 positives in this frame).
arm	n	accuracy	macro F1	precision	recall	F1	PR-AUC	invalid JSON	p50 latency
classifier_dev	1,000	94.7%	64.2%	50.0%[0.31, 0.69]	22.6%[0.13, 0.36]	31.2%[0.18, 0.45]	0.276	0.0%	2.9 ms
classifier_dev_no_rubric	1,000	93.0%	61.7%	30.2%[0.19, 0.45]	24.5%[0.15, 0.38]	27.1%[0.16, 0.39]	0.260	0.0%	6.5 ms
tfidffitted	1,000	97.0%	84.2%	74.5%[0.60, 0.85]	66.0%[0.53, 0.77]	70.0%[0.59, 0.79]	0.780	0.0%	0.002 ms
gliner	1,000	89.7%	52.5%	9.7%[0.05, 0.20]	11.3%[0.05, 0.23]	10.4%[0.04, 0.18]	0.070	0.0%	930 ms
llm_localno spread	1,000	5.3%	5.0%	5.3%[0.04, 0.07]	100.0%[0.93, 1.00]	10.1%[0.08, 0.13]	0.053	0.0%	12,261 ms
llm_hosted_json_object	1,000	94.8%	65.1%	60.0%[0.39, 0.78]	22.6%[0.13, 0.36]	32.9%[0.19, 0.47]	0.282	0.5%	4,055 ms
majority_classfloor	1,000	94.7%	48.6%	n/an/a	0.0%[0.00, 0.07]	0.0%[0.00, 0.00]	n/a	0.0%	n/a
dummy_stratifiedfloor	1,000	90.6%	51.4%	8.2%[0.03, 0.19]	7.5%[0.03, 0.18]	7.8%[0.02, 0.16]	0.055	0.0%	n/a

The table scrolls sideways on a narrow screen.

Why a cell reads n/a
majority_class, precision_positive: always predicts 'legitimate' (947/1000 = 94.7% of the scored subset) and carries no distribution, so brier, ece, pr_auc and the routing sweep are null for it
majority_class, precision_ci95: no positives (or no positive predictions) in the scored subset, so the interval has no denominator
majority_class, pr_auc: no majority_class row carries p_positive for fraudulent
majority_class, latency_p50: always predicts 'legitimate' (947/1000 = 94.7% of the scored subset) and carries no distribution, so brier, ece, pr_auc and the routing sweep are null for it
dummy_stratified, latency_p50: sklearn DummyClassifier(strategy='stratified', random_state=0) fitted on 12725 train+validation gold rows, predicting on the 1000 scored records; its predict_proba is one-hot on the sampled label, so it is a certainty-asserting floor

Precision, recall, PR-AUC and the invalid-JSON rate all belong to the fraudulent class, the rare one. 53 of 1,000 scored postings carry the fraudulent label, which is 5.3%. p50 latency is the median answer time per posting for this question.

On F1, tfidf's interval [0.59, 0.79] starts above the highest upper bound of every other arm (0.47), so that ordering does not depend on where the threshold sits. The other arms' intervals overlap in part (llm_hosted_json_object, classifier_dev, classifier_dev_no_rubric, llm_local, gliner), and their accuracy runs from 5.3% to 94.8%, against a 94.7% majority-class floor on accuracy. The file records the interval frame as: 53 positives in this frame.

Arms with no spread

These arms answer nearly the same thing about every posting, so each row is a single operating point. The numbers come from the same rows as the table above.

llm_local: its PR-AUC equals the base rate (0.053): the scores never separate the two classes; its ECE is 0.947, so its confidence carries no information.

Per-class precision and recall

arm	n	legitimate precision	legitimate recall	fraudulent precision	fraudulent recall
classifier_dev	1,000	95.8%	98.7%	50.0%	22.6%
classifier_dev_no_rubric	1,000	95.8%	96.8%	30.2%	24.5%
gliner	1,000	95.0%	94.1%	9.7%	11.3%
llm_hosted_json_object	1,000	96.0%	98.8%	60.0%	22.6%
llm_local	1,000	n/a	0.0%	5.3%	100.0%
tfidf	1,000	98.1%	98.7%	74.5%	66.0%
majority_classfloor	1,000	94.7%	100.0%	n/a	0.0%
dummy_stratifiedfloor	1,000	94.8%	95.2%	8.2%	7.5%

The table scrolls sideways on a narrow screen.

Why a cell reads n/a
llm_local / legitimate, precision: no row of this arm was predicted with this label, so precision has no denominator and is undefined
majority_class / fraudulent, precision: always predicts 'legitimate' (947/1000 = 94.7% of the scored subset) and carries no distribution, so brier, ece, pr_auc and the routing sweep are null for it

One posting, four verdicts

Subsea Process System Engineer

Corporate overviewAker Solutions is a global provider of products, systems and services to the oil and gas industry. Our engineering, design and technology bring discoveries into production and maximize recovery from each petroleum field. We employ approximately 28,000 people in about 30 countries. Go to #URL_0fa3f7c5e23a16de16a841e368006cae916884407d90b154dfef39764

Posting test-6 from the public dataset, showing the first 400 characters of the text the engines read. Their input is the title plus the description, cut at 6,000 characters. The text is verbatim, so a link in the source appears as a #URL_ placeholder and runs of whitespace stay as they were.

What each engine said about it

Bar length is the probability the engine gave its own answer. Teal marks a call that matches the gold label, crimson marks a call that does not.

classifier_devlegitimate1.00wrong
classifier_dev_no_rubriclegitimate0.99wrong
glinerlegitimate1.00wrong
llm_hosted_json_objectlegitimate0.98wrong
llm_localfraudulent1.00right
tfidffraudulent0.61right

Gold label: fraudulent. 4 of 6 engines disagreed with it.

No answer on this posting from: majority_class, dummy_stratified.

The four engines

6 arms ran on this sample, and they come from 4 engines: Jev, served by classifier.dev, TF-IDF with logistic regression, the GLiNER2.5-base zero-shot encoder, Qwen2.5-1.5B-Instruct, served locally and over a hosted endpoint, and the other 2 arms vary one of them rather than adding an engine.

classifier_dev: Jev 1.13.0, served by classifier.dev. One request carries up to 1,000 postings and returns a probability for every label of a question in a single pass. No tokens generated.
classifier_dev_no_rubric: The same Jev arm with the per-question rubric text dropped from the request. This is the ablation for what the instruction text buys.
tfidf: TF-IDF over word unigrams and bigrams, then one logistic regression per question, fitted on the train and validation rows that carry a gold label.
gliner: fastino/gliner2.5-base-v1, a 194M-parameter zero-shot encoder. All four questions are compiled into one schema and read in one forward pass per posting. Its window is 512 subwords, so long postings are truncated before they are scored.
llm_local: Qwen2.5-1.5B-Instruct Q4_K_M, served by llama-server on this machine. One chat completion per posting answers all four questions under a strict JSON schema at temperature 0. The model reports its own confidence.
llm_hosted_json_object: The same LLM code path pointed at a hosted model over an OpenAI-compatible endpoint, kept as its own row so the two are never confused. The gateway rejects json_schema, so the reply shape goes in the prompt and the reply is validated afterwards.

Calibration

ECE sorts every answered posting into ten bins of stated probability and averages the gap between the probability an arm claimed in a bin and the share of those answers it got right. 0.000 means the stated number can be read as a probability; 0.100 means the arm is off by ten points on average.

A 3px rule in teal marks an ECE of 0.05 or less, where the stated probability is within five points of the outcome on average; crimson marks an arm above that. A Brier score is the mean squared error of the same probabilities, so lower is better in both columns.
arm	n	Brier	ECE	excluded
classifier_dev	1,000	0.054	0.046	0
classifier_dev_no_rubric	1,000	0.073	0.070	0
tfidf	1,000	0.033	0.033	0
gliner	1,000	0.094	0.082	0
llm_local	1,000	0.947	0.947	0
llm_hosted_json_object	995	0.047	0.015	5

The table scrolls sideways on a narrow screen.

Stated probability against accuracy per arm on the fraudulent question — Calibration on the fraud question, drawn by the evaluator from the same records these tables come from. A well calibrated arm sits on the diagonal.

The bins below cover llm_hosted_json_object, tfidf, the two arms with the lowest ECE on this question, and llm_local, gliner, the two with the highest. These are the ends of the ranking: the arms whose stated probabilities land closest to what they get right, and furthest from it. 21 bins across these arms held no answer at all and are left out.

bin of stated probability	n	mean stated probability	accuracy in the bin
`llm_hosted_json_object`
0.6 to 0.7	5	0.620	0.800
0.7 to 0.8	45	0.723	0.867
0.8 to 0.9	69	0.835	0.899
0.9 to 1.0	876	0.959	0.962
`tfidf`
0.1 to 0.2	21	0.138	0.429
0.2 to 0.3	10	0.263	1.000
0.3 to 0.4	4	0.344	1.000
0.4 to 0.5	2	0.411	1.000
0.5 to 0.6	1	0.571	1.000
0.6 to 0.7	5	0.648	1.000
0.7 to 0.8	3	0.772	1.000
0.8 to 0.9	1	0.824	1.000
0.9 to 1.0	953	0.968	0.981
`llm_local`
0.9 to 1.0	1,000	1.000	0.053
`gliner`
0.5 to 0.6	15	0.550	0.733
0.6 to 0.7	13	0.664	0.308
0.7 to 0.8	13	0.763	0.769
0.8 to 0.9	29	0.861	0.793
0.9 to 1.0	930	0.991	0.913

The table scrolls sideways on a narrow screen.

Routing and cost

Read a row as: at stated probability tau or above, this share of postings is acted on, and this is how the acted-on set scores. The 3px rule marks a tau whose precision clears 0.90 in teal and one that does not in crimson. tau is a sequence, so the rows are ordered rather than numbered.
tau	coverage	precision	recall	false negatives among acted on	positives acted on
`classifier_dev`
0.50	2.4%	50.0%	22.6%	41	12 of 53
0.70	1.2%	83.3%	18.9%	43	10 of 53
0.90	0.8%	87.5%	13.2%	46	7 of 53
0.95	0.6%	100.0%	11.3%	47	6 of 53
`classifier_dev_no_rubric`
0.50	4.4%	29.5%	24.5%	40	13 of 53
0.70	2.2%	50.0%	20.8%	42	11 of 53
0.90	1.1%	63.6%	13.2%	46	7 of 53
0.95	0.8%	75.0%	11.3%	47	6 of 53
`tfidf`
0.50	1.0%	100.0%	18.9%	43	10 of 53
0.70	0.4%	100.0%	7.5%	49	4 of 53
0.90	0.0%	n/a	0.0%	53	0 of 53
0.95	0.0%	n/a	0.0%	53	0 of 53
`gliner`
0.50	6.2%	9.7%	11.3%	47	6 of 53
0.70	4.9%	12.2%	11.3%	47	6 of 53
0.90	3.9%	10.3%	7.5%	49	4 of 53
0.95	3.4%	8.8%	5.7%	50	3 of 53
`llm_local`
0.50	100.0%	5.3%	100.0%	0	53 of 53
0.70	100.0%	5.3%	100.0%	0	53 of 53
0.90	100.0%	5.3%	100.0%	0	53 of 53
0.95	100.0%	5.3%	100.0%	0	53 of 53
`llm_hosted_json_object`
0.50	2.0%	60.0%	22.6%	41	12 of 53
0.70	2.0%	60.0%	22.6%	41	12 of 53
0.90	0.7%	100.0%	13.2%	46	7 of 53
0.95	0.5%	100.0%	9.4%	48	5 of 53
`majority_class`
0.50	0.0%	n/a	0.0%	53	0 of 53
0.70	0.0%	n/a	0.0%	53	0 of 53
0.90	0.0%	n/a	0.0%	53	0 of 53
0.95	0.0%	n/a	0.0%	53	0 of 53
`dummy_stratified`
0.50	4.9%	8.2%	7.5%	49	4 of 53
0.70	4.9%	8.2%	7.5%	49	4 of 53
0.90	4.9%	8.2%	7.5%	49	4 of 53
0.95	4.9%	8.2%	7.5%	49	4 of 53

The table scrolls sideways on a narrow screen.

Why a cell reads n/a
tfidf, precision: nothing reaches p_positive >= 0.9
tfidf, precision: nothing reaches p_positive >= 0.95
majority_class, precision: nothing reaches p_positive >= 0.5 (1000 row(s) carry no p_positive)
majority_class, precision: nothing reaches p_positive >= 0.7 (1000 row(s) carry no p_positive)
majority_class, precision: nothing reaches p_positive >= 0.9 (1000 row(s) carry no p_positive)
majority_class, precision: nothing reaches p_positive >= 0.95 (1000 row(s) carry no p_positive)

classifier.dev is a shared service and its probabilities shift slightly between fetches of the same posting, so a threshold-dependent cell in this table can move by one record between reruns while the label-based tables in Fraud triage do not. Treat the routing cells as indicative at high thresholds, where few postings carry the decision: at tau 0.95, 3 arms act on fewer than 1% of the 1,000 scored postings (llm_hosted_json_object 5, classifier_dev 6, classifier_dev_no_rubric 8).

False negatives are the fraudulent postings a tau leaves in the unreviewed pile, so they are what higher precision costs. 6 of these rows sit at a tau that no score in the arm reaches, so nothing is acted on: coverage is 0.0% and precision has no value.

Latency and tokens are measured on the fraud question. A classification is counted per answer, so one posting on one question is one classification.
arm	p50 per posting	p95 per posting	mean tokens in	mean tokens out	classifications spent	wall seconds
classifier_dev	2.9 ms	2.9 ms	n/a	n/a	3,147	54.4 s
classifier_dev_no_rubric	6.5 ms	6.5 ms	n/a	n/a	3,147	6.5 s
tfidf	0.002 ms	0.002 ms	n/a	n/a	n/a	86.7 s
gliner	930 ms	1,067 ms	247	n/a	n/a	1,915.4 s
llm_local	12,261 ms	17,586 ms	450	91	n/a	3,249.2 s
llm_hosted_json_object	4,055 ms	12,138 ms	648	800	n/a	1,259.9 s

The table scrolls sideways on a narrow screen.

For classifier_dev, classifier_dev_no_rubric, tfidf, p50 equals p95 in this run, so those two columns describe one measured value rather than a spread.

Per-row latency is wall clock on the machine that ran the arm; arms that ran concurrently include each other's load in these numbers, so the projection is an upper bound for this machine rather than a benchmark.

Projection at 10,000 postings a day

classifier_dev spent 3,147 classifications on 1,000 postings, which is 3.15 per posting; at 10,000 postings a day that is 31,470 classifications a day.
classifier_dev_no_rubric spent 3,147 classifications on 1,000 postings, which is 3.15 per posting; at 10,000 postings a day that is 31,470 classifications a day.

Precision and recall of the acted-on set as tau rises, drawn by the evaluator. Each point is one tau from the table above.

Method checks

The rubric ablation

One field differs between these rows: the instructions text carrying the per-question rubric. The model, the postings and the labels are the same.

request variant	accuracy	macro F1	precision (fraudulent)	recall (fraudulent)	PR-AUC
classifier_devwith the rubric	94.7%	64.2%	50.0%	22.6%	0.276
classifier_dev_no_rubricwithout the rubric	93.0%	61.7%	30.2%	24.5%	0.260

The table scrolls sideways on a narrow screen.

Unusable answers

An answer is unusable when the arm failed its own output validation. Those rows stay in the run and are scored wrong, so they cost accuracy rather than disappearing from it.

llm_hosted_json_object: 0.5% of answers unusable. invalid_json_rate = 0.5%: 5 of 1000 rows failed the arm's own output validation and are scored wrong (arm-reported reasons: 5 x 'output budget of 4096 tokens exhausted before an answer (finish_reason=length); raise --max-tokens'); 5 of them are also excluded from Brier/ECE because they carry no probability
classifier_dev, classifier_dev_no_rubric, tfidf, gliner, llm_local: no unusable answer at all (no row of this arm is marked invalid for this question).

Frame check: every arm answered the same set of postings on every question, so the rows in the tables above are directly comparable.

Leakage guard: no arm rises more than 3 points above the 82.2% majority floor on the salary control question, whose label lives in the dataset and not in the posting text. The guard is one-directional: only an accuracy above the floor plus 3 points fails, because only that direction can mean the gold label leaked into the posting text. Under the floor: classifier_dev_no_rubric sits -57.7 points; gliner sits -55.7 points, which reads as an arm that is not answering the question that was asked rather than as a leak.

What this does not prove

The data is the public Kaggle fake job postings mirror, which dates from 2014 to 2018. Both language models may have seen it while they were pretrained, so their numbers here are an upper bound rather than a clean measurement.
This run scores 1,000 postings of the whole test split, drawn with seed 0 as a uniform random sample without replacement, so it is not a prefix of the split. Each question is scored on the rows that carry a gold answer (1,000 for fraudulent, 627 for required_experience, 520 for required_education, 1,000 for salary_range_stated), so the rare-class numbers rest on the 53 fraudulent postings that a draw of this size holds, which is what makes the intervals in the fraud table as wide as they are.
required_experience and required_education are platform tags a poster filled in, so agreeing with them measures tag recovery.
classifier.dev is a shared free service with a daily quota, so rerunning the Jev arm can shift its numbers.
The LinkedIn demo set is unlabelled, so its numbers are flag rates and agreement between arms, never accuracy.
Precision and recall here count a flagged posting as a hit against a platform tag. Nothing in this run says whether a human reviewer would agree with the flag.

Reproduce it

uv sync
uv run hf download Qwen/Qwen2.5-1.5B-Instruct-GGUF qwen2.5-1.5b-instruct-q4_k_m.gguf --local-dir models
llama-server -m models/qwen2.5-1.5b-instruct-q4_k_m.gguf --jinja -c 16384 -np 4 --cache-ram 128 --port 8080 --host 127.0.0.1
uv run python -m triage.run --arms all --split test --limit 1000 --seed 0 --concurrency 4
uv run python -m triage.evaluate --report
uv run python web/build_site.py

Keep --cache-ram 128 in the server line: the default 8 GiB prompt cache is what filled memory and put this machine into swap during the run.

This page is generated by web/build_site.py from results/metrics.json. The full tables, including every n/a with its reason, are in report.md.