Łukasz Augustyniak

Research & Publications

Peer-reviewed research contributions in AI, NLP, and machine learning, published in top-tier conferences and journals.

25
Published Papers
N/A
Citations
17
Venues
22
Collaborators

Featured Publications

Published
2023

Why Aren't We NER Yet? Artifacts of ASR Errors in Named Entity Recognition in Spontaneous Speech Transcripts

Piotr Szymanski and Lukasz Augustyniak and Mikolaj Morzy and Adrian Szymczak and Krzysztof Surdyk and Piotr Zelasko

ACL

πŸŽ™οΈπŸ’₯ Ever wondered why AI still messes up names in conversations? This paper reveals the spectacular failure of name detection in spontaneous speech, showing it's not just speech recognition errors - spoken language is inherently messy and breaks traditional AI models! Bonus: they prove everyone's been measuring success wrong this whole time. πŸ€–πŸ“Š

View full abstract

Transcripts of spontaneous human speech present a significant obstacle for traditional NER models. The lack of grammatical structure of spoken utterances and word errors introduced by the ASR make downstream NLP tasks challenging. In this paper, we examine in detail the complex relationship between ASR and NER errors which limit the ability of NER models to recover entity mentions from spontaneous speech transcripts. Using publicly available benchmark datasets (SWNE, Earnings-21, OntoNotes), we present the full taxonomy of ASR-NER errors and measure their true impact on entity recognition. We find that NER models fail spectacularly even if no word errors are introduced by the ASR. We also show why the F1 score is inadequate to evaluate NER models on conversational transcripts.

NERSpeech Processing
View Paper
Published
2023

Massively Multilingual Corpus of Sentiment Datasets and Multi-faceted Sentiment Classification Benchmark

Lukasz Augustyniak and Szymon Wozniak and Marcin Gruza and Piotr Gramacki and Krzysztof Rajda and Mikolaj Morzy and Tomasz Kajdanowicz

NeurIPS

🌍❀️😑 Detecting emotions in text is super hard when cultural nuances matter - a thumbs up might mean different things in Tokyo vs. Toronto! This mega-project collected 79 high-quality sentiment datasets across 27 languages and ran hundreds of experiments to figure out what actually works. πŸ—£οΈβœ¨

View full abstract

Despite impressive advancements in multilingual corpora collection and model training, developing large-scale deployments of multilingual models still presents a significant challenge. This is particularly true for language tasks that are culture-dependent. One such example is the area of multilingual sentiment analysis, where affective markers can be subtle and deeply ensconced in culture. This work presents the most extensive open massively multilingual corpus of datasets for training sentiment models. The corpus consists of 79 manually selected datasets from over 350 datasets reported in the scientific literature based on strict quality criteria. The corpus covers 27 languages representing 6 language families. Datasets can be queried using several linguistic and functional features. In addition, we present a multi-faceted sentiment classification benchmark summarizing hundreds of experiments conducted on different base models, training objectives, dataset collections, and fine-tuning strategies.

Sentiment AnalysisMultilingualBenchmarks & Datasets
View Paper
Published
2022

This is the way: designing and compiling LEPISZCZE, a comprehensive {NLP

Lukasz Augustyniak and Kamil Tagowski and Albert Sawczyn and Denis Janiak and Roman Bartusiak and Adrian Szymczak and Arkadiusz Janz and Piotr Szymanski and Marcin Watroba and Mikolaj Morzy and Tomasz Kajdanowicz and Maciej Piasecki

NeurIPS

πŸ‡΅πŸ‡±πŸ”¬ LEPISZCZE (Polish for 'glue') is the comprehensive AI benchmark that Polish NLP desperately needed! While English gets all the fancy testing suites, this team created a flexible, version-controlled benchmark to level the playing field and provide a blueprint for other under-resourced languages. πŸš€πŸ“ˆ

View full abstract

The availability of compute and data to train larger and larger language models increases the demand for robust methods of benchmarking the true progress of LM training. Recent years witnessed significant progress in standardized benchmarking for English. Benchmarks such as GLUE, SuperGLUE, or KILT have become de facto standard tools to compare large language models. Following the trend to replicate GLUE for other languages, the KLEJ benchmark has been released for Polish. In this paper, we evaluate the progress in benchmarking for low-resourced languages. We note that only a handful of languages have such comprehensive benchmarks. We also note the gap in the number of tasks being evaluated by benchmarks for resource-rich English/Chinese and the rest of the world. In this paper, we introduce LEPISZCZE (the Polish word for glew, the Middle English predecessor of glue), a new, comprehensive benchmark for Polish NLP with a large variety of tasks and high-quality operationalization of the benchmark. We design LEPISZCZE with flexibility in mind. Including new models, datasets, and tasks is as simple as possible while still offering data versioning and model tracking. In the first run of the benchmark, we test 13 experiments (task and dataset pairs) based on the five most recent LMs for Polish. We use five datasets from the Polish benchmark and add eight novel datasets. As the paper's main contribution, apart from LEPISZCZE, we provide insights and experiences learned while creating the benchmark for Polish as the blueprint to design similar benchmarks for other low-resourced languages.

NLP
View Paper
Published
2020

WER we are and WER we think we are

Piotr Szymanski and Piotr Zelasko and Mikolaj Morzy and Adrian Szymczak and Marzena Zyla{-

EMNLP

🎀🀨 This paper is basically calling BS on speech recognition companies claiming super-low error rates! When tested on real conversations instead of clean benchmark data, even the best commercial systems performed way worse than advertised. Reality check delivered! πŸ“‰πŸ’―

View full abstract

Natural language processing of conversational speech requires the availability of high-quality transcripts. In this paper, we express our skepticism towards the recent reports of very low Word Error Rates (WERs) achieved by modern Automatic Speech Recognition (ASR) systems on benchmark datasets. We outline several problems with popular benchmarks and compare three state-of-the-art commercial ASR systems on an internal dataset of real-life spontaneous human conversations and HUB'05 public benchmark. We show that WERs are significantly higher than the best reported results. We formulate a set of guidelines which may aid in the creation of real-life, multi-domain datasets with high quality annotations for training and testing of robust ASR systems.

EMNLP
View Paper

All Publications

PublishedConference Paper2023

Why Aren't We NER Yet? Artifacts of ASR Errors in Named Entity Recognition in Spontaneous Speech Transcripts

Piotr Szymanski and Lukasz Augustyniak and Mikolaj Morzy and Adrian Szymczak and Krzysztof Surdyk and Piotr Zelasko

ACL β€’ Pages 1746--1761

πŸŽ™οΈπŸ’₯ Ever wondered why AI still messes up names in conversations? This paper reveals the spectacular failure of name detection in spontaneous speech, showing it's not just speech recognition errors - spoken language is inherently messy and breaks traditional AI models! Bonus: they prove everyone's been measuring success wrong this whole time. πŸ€–πŸ“Š

NERSpeech Processing
View Paper
PublishedConference Paper2023

Massively Multilingual Corpus of Sentiment Datasets and Multi-faceted Sentiment Classification Benchmark

Lukasz Augustyniak and Szymon Wozniak and Marcin Gruza and Piotr Gramacki and Krzysztof Rajda and Mikolaj Morzy and Tomasz Kajdanowicz

NeurIPS

🌍❀️😑 Detecting emotions in text is super hard when cultural nuances matter - a thumbs up might mean different things in Tokyo vs. Toronto! This mega-project collected 79 high-quality sentiment datasets across 27 languages and ran hundreds of experiments to figure out what actually works. πŸ—£οΈβœ¨

Sentiment AnalysisMultilingualBenchmarks & Datasets
View Paper
PublishedJournal Article2023

Electoral Agitation Data Set: The Use Case of the Polish Election

Mateusz Baran and Mateusz WΓ³jcik and Piotr Kolebski and Michal Bernaczyk and Krzysztof Rajda and Lukasz Augustyniak and Tomasz Kajdanowicz

CoRR

Polish NLPPolitical Analysis
View Paper
PublishedConference Paper2022

Electoral Agitation Dataset: The Use Case of the Polish Election

Mateusz Baran and Mateusz WΓ³jcik and Piotr Kolebski and Michal Bernaczyk and Krzysztof Rajda and Lukasz Augustyniak and Tomasz Kajdanowicz

LREC β€’ Pages 32--36

πŸ—³οΈπŸ‡΅πŸ‡± Election watchdogs can't keep up with political spam on social media, so this team built an AI to detect illegal electioneering! They annotated 6,112 Polish tweets and trained a model (yes, named HerBERT) to automatically flag campaign violations during elections. Democracy meets machine learning! πŸ€–βš–οΈ

Benchmarks & DatasetsPolish NLPPolitical Analysis
View Paper
PublishedConference Paper2022

This is the way: designing and compiling LEPISZCZE, a comprehensive {NLP

Lukasz Augustyniak and Kamil Tagowski and Albert Sawczyn and Denis Janiak and Roman Bartusiak and Adrian Szymczak and Arkadiusz Janz and Piotr Szymanski and Marcin Watroba and Mikolaj Morzy and Tomasz Kajdanowicz and Maciej Piasecki

NeurIPS

πŸ‡΅πŸ‡±πŸ”¬ LEPISZCZE (Polish for 'glue') is the comprehensive AI benchmark that Polish NLP desperately needed! While English gets all the fancy testing suites, this team created a flexible, version-controlled benchmark to level the playing field and provide a blueprint for other under-resourced languages. πŸš€πŸ“ˆ

NLP
View Paper
PublishedConference Paper2022

Assessment of Massively Multilingual Sentiment Classifiers

Krzysztof Rajda and Lukasz Augustyniak and Piotr Gramacki and Marcin Gruza and Szymon Wozniak and Tomasz Kajdanowicz

the β€’ Pages 125--140

πŸ€”πŸ’ͺ Bigger isn't always better! This massive study tested 11 models across 80 datasets in 27 languages to show that a smaller, faster model that works well for ALL languages beats a giant model that's only 2% better for English. Practical wisdom for the real world! 🌐⚑

Sentiment AnalysisMultilingual
View Paper
PublishedJournal Article2022

Assessment of Massively Multilingual Sentiment Classifiers

Krzysztof Rajda and Lukasz Augustyniak and Piotr Gramacki and Marcin Gruza and Szymon Wozniak and Tomasz Kajdanowicz

CoRR

Sentiment AnalysisMultilingual
View Paper
PublishedConference Paper2021

Fact-checking: relevance assessment of references in the Polish political domain

Albert Sawczyn and Jakub Binkowski and Denis Janiak and Lukasz Augustyniak and Tomasz Kajdanowicz

Knowledge-Based and Intelligent Information {\& β€’ Pages 1285--1293

πŸ•΅οΈπŸ‡΅πŸ‡± Fake news spreads faster than fact-checkers can debunk it, especially in elections! This team built an AI system to automate the first crucial step of fact-checking for Polish - figuring out which sources are actually relevant to verify a claim. Fighting misinformation, one algorithm at a time! πŸ›‘οΈπŸ“°

Polish NLPPolitical Analysis
View Paper
PublishedJournal Article2021

Comprehensive analysis of aspect term extraction methods using various text embeddings

Lukasz Augustyniak and Tomasz Kajdanowicz and Przemyslaw Kazienko

Comput. Speech Lang. β€’ Pages 101217

πŸ”πŸ’¬ When you review a restaurant saying "the pasta was amazing but service was terrible," AI needs to extract those specific aspects (pasta, service). This deep dive tests every trick in the book to figure out what actually works best - spoiler: bidirectional networks + good word embeddings + CRF layers = winning combo! 🍝⭐

Aspect-Based AnalysisEmbeddings
View Paper
PublishedConference Paper2020

WER we are and WER we think we are

Piotr Szymanski and Piotr Zelasko and Mikolaj Morzy and Adrian Szymczak and Marzena Zyla{-

EMNLP β€’ Pages 3290--3295

🎀🀨 This paper is basically calling BS on speech recognition companies claiming super-low error rates! When tested on real conversations instead of clean benchmark data, even the best commercial systems performed way worse than advertised. Reality check delivered! πŸ“‰πŸ’―

EMNLP
View Paper
PublishedConference Paper2020

Punctuation Prediction in Spontaneous Conversations: Can We Mitigate {ASR

Lukasz Augustyniak and Piotr Szymanski and Mikolaj Morzy and Piotr Zelasko and Adrian Szymczak and Jan Mizgajski and Yishay Carmiel and Najim Dehak

st Annual Conference of the International Speech Communication Association β€’ Pages 4906--4910

πŸ“πŸŽ™οΈ Speech recognition mishears words (their/there/they're anyone?), which makes adding punctuation to transcripts super messy! This clever solution tweaks word embeddings to handle those confusing homonyms better, boosting punctuation accuracy by up to 9%. Your transcripts just got way more readable! ✨

Speech Processing
View Paper
PublishedJournal Article2020

Political Advertising Dataset: the use case of the Polish 2020 Presidential Elections

Lukasz Augustyniak and Krzysztof Rajda and Tomasz Kajdanowicz and Michal Bernaczyk

CoRR

Benchmarks & DatasetsPolish NLPPolitical Analysis
View Paper
PublishedConference Paper2019

Aspect Detection using Word and Char Embeddings with (Bi) {LSTM

Lukasz Augustyniak and Tomasz Kajdanowicz and Przemyslaw Kazienko

nd {IEEE β€’ Pages 43--50

Aspect-Based AnalysisEmbeddings
View Paper
PublishedConference Paper2019

Avaya Conversational Intelligence: {A

Jan Mizgajski and Adrian Szymczak and Robert Glowski and Piotr Szymanski and Piotr Zelasko and Lukasz Augustyniak and Mikolaj Morzy and Yishay Carmiel and Jeff Hodson and Lukasz WΓ³jciak and Daniel Smoczyk and Adam WrΓ³bel and Bartosz Borowik and Adam Artajew and Marcin Baran and Cezary Kwiatkowski and Marzena Zyla{-

th Annual Conference of the International Speech Communication Association β€’ Pages 3659--3660

th Annual Conference of the International Speech Communication Association
View Paper
PublishedJournal Article2019

Towards Better Understanding of Spontaneous Conversations: Overcoming Automatic Speech Recognition Errors With Intent Recognition

Piotr Zelasko and Jan Mizgajski and Mikolaj Morzy and Adrian Szymczak and Piotr Szymanski and Lukasz Augustyniak and Yishay Carmiel

CoRR

Speech Processing
View Paper
PublishedJournal Article2019

Aspect Detection using Word and Char Embeddings with (Bi)LSTM and {CRF

Lukasz Augustyniak and Tomasz Kajdanowicz and Przemyslaw Kazienko

CoRR

Aspect-Based AnalysisEmbeddings
View Paper
PublishedJournal Article2019

Extracting Aspects Hierarchies using Rhetorical Structure Theory

Lukasz Augustyniak and Tomasz Kajdanowicz and Przemyslaw Kazienko

CoRR

Aspect-Based Analysis
View Paper
PublishedJournal Article2019

WordNet2Vec: Corpora agnostic word vectorization method

Roman Bartusiak and Lukasz Augustyniak and Tomasz Kajdanowicz and Przemyslaw Kazienko and Maciej Piasecki

Neurocomputing β€’ Pages 141--150

πŸ•ΈοΈβž‘οΈπŸ“Š What if instead of learning word meanings from massive text collections, we could extract them from WordNet's language knowledge graph? WordNet2Vec does exactly that - transforming the linguistic network into word vectors that capture each word's role in the entire language! Works across languages and crushed it on Amazon reviews. πŸŒπŸ’‘

Neurocomputing
View Paper
PublishedConference Paper2017

Method for Aspect-Based Sentiment Annotation Using Rhetorical Analysis

Lukasz Augustyniak and Krzysztof Rajda and Tomasz Kajdanowicz

Intelligent Information and Database Systems - β€’ Pages 772--781

Sentiment AnalysisAspect-Based Analysis
View Paper
PublishedConference Paper2016

Fast and Accurate - Improving Lexicon-Based Sentiment Classification with an Ensemble Methods

Lukasz Augustyniak and Piotr Szymanski and Tomasz Kajdanowicz and Przemyslaw Kazienko

Intelligent Information and Database Systems - β€’ Pages 108--116

Sentiment Analysis
View Paper
PublishedJournal Article2016

Comprehensive Study on Lexicon-based Ensemble Classification Sentiment Analysis

Lukasz Augustyniak and Piotr Szymanski and Tomasz Kajdanowicz and Wlodzimierz Tuliglowicz

Entropy β€’ Pages 4

Sentiment Analysis
View Paper
PublishedConference Paper2015

Sentiment Analysis for Polish Using Transfer Learning Approach

Roman Bartusiak and Lukasz Augustyniak and Tomasz Kajdanowicz and Przemyslaw Kazienko

Second European Network Intelligence Conference β€’ Pages 53--59

Sentiment AnalysisPolish NLP
View Paper
PublishedConference Paper2014

Belief Propagation Method for Word Sentiment in WordNet 3.0

Andrzej Misiaszek and Przemyslaw Kazienko and Marcin Kulisiewicz and Lukasz Augustyniak and Wlodzimierz Tuliglowicz and Adrian Popiel and Tomasz Kajdanowicz

Intelligent Information and Database Systems - β€’ Pages 263--272

Sentiment Analysis
View Paper
PublishedConference Paper2014

Simpler is better? Lexicon-based ensemble sentiment classification beats supervised methods

Lukasz Augustyniak and Tomasz Kajdanowicz and Piotr Szymanski and Wlodzimierz Tuliglowicz and Przemyslaw Kazienko and Reda Alhajj and Boleslaw K. Szymanski

{IEEE/ACM β€’ Pages 924--929

Sentiment Analysis
View Paper
PublishedConference Paper2014

An Approach to Sentiment Analysis of Movie Reviews: Lexicon Based vs. Classification

Lukasz Augustyniak and Tomasz Kajdanowicz and Przemyslaw Kazienko and Marcin Kulisiewicz and Wlodzimierz Tuliglowicz

Hybrid Artificial Intelligence Systems - β€’ Pages 168--178

Sentiment Analysis
View Paper

Research Interests

Natural Language ProcessingLarge Language ModelsMultilingual AINamed Entity RecognitionSentiment AnalysisSpeech ProcessingPolish NLPBenchmarks & DatasetsAspect-Based AnalysisCross-lingual Understanding