Loading...
Polish NLP had no benchmark you could trust to compare models fairly. Results were reported on inconsistent dataset splits, with no shared evaluation protocol and no way to reproduce a published score.
12 task families · 17 dataset views · 10 evaluated model setups · still taking commits
Twelve task families across seventeen dataset views, drawn from legal language, reviews, news, Wikipedia, social media and general-domain corpora — the breadth is the point, because a benchmark narrow enough to overfit tells you nothing.
The eleven datasets shipped in the repository:
| Task | Dataset |
|---|---|
| Named entity recognition | KPWr-NER |
| Part-of-speech tagging | NKJP-POS |
| Sentiment classification | PolEmo2 |
| Aspect-based sentiment | AspectEmo |
| Abusive clause detection | abusive-clauses-pl |
| Political advertising detection | political-advertising-pl |
| Punctuation restoration | 2021-punctuation-restoration |
| Question answering | DYK |
| Summarisation | PSC |
| Textual entailment | CDSC-E |
| Extractive QA | Wikipedia · NKJP · KPWr |
Two of the eleven were built for this benchmark rather than borrowed for it: political-advertising-pl, the first publicly open dataset for detecting political advertising in Polish, and abusive-clauses-pl, covering abusive clauses in Polish consumer contracts. Both are released under my own name — the same loop the roadmap below describes, already running.
Every dataset version is pinned with DVC and every training run is recorded in Weights & Biases. A number on the leaderboard is therefore reproducible rather than merely reported — you can trace it back to the exact data and the exact run that produced it. That is the difference between a benchmark you can build on and a benchmark you have to take on faith.
LEPISZCZE was published in 2022, but it was never finished — the repository still takes commits, and extractive question answering landed after the paper. Two directions are open.
More datasets. Broader task coverage beyond the original eleven, with the same requirement that made the benchmark worth building: every result reproducible, every dataset version pinned.
Legal Polish. Legal language is where general-purpose Polish benchmarks go quiet — the vocabulary, the sentence length, the entity structure all diverge from the news and social-media text most Polish datasets are drawn from. Work on real legal corpora, through AI-Tax and JuDDGES, is the natural source of evaluation tasks nobody else is positioned to contribute.
Applied work feeding an open benchmark, rather than running beside it.
Have a Polish dataset worth benchmarking, or a model to submit? Get in touch.
@inproceedings{augustyniak2022lepiszcze,
author = {Augustyniak, Lukasz and Tagowski, Kamil and Sawczyn, Albert and
Janiak, Denis and Bartusiak, Roman and Szymczak, Adrian and others},
booktitle = {Advances in Neural Information Processing Systems},
title = {This is the way: designing and compiling LEPISZCZE,
a comprehensive NLP benchmark for Polish},
year = {2022}
}