|
Analyzing Transformers Representations. A Linguistic Perspective| old_uid | 19851 |
|---|
| title | Analyzing Transformers Representations. A Linguistic Perspective |
|---|
| start_date | 2021/12/17 |
|---|
| schedule | 11h UTC+1 |
|---|
| online | no |
|---|
| details | Location/connection: Due to current restrictions, you are invited to
attend online (Google Meet). The link will be shared on the day in this
google doc
https://docs.google.com/document/d/1bnuIxo9F2WLPsdI9VIn233fMVwybFSUequrUh4vlYkk/edit?usp=sharing. |
|---|
| summary | Transformers have become a key component in many NLP models,
arguably because of their capacity to uncover contextualized
distributed representation of tokens from raw texts. Many works have
striven to analyze these representations to find out whether they are
consistent with models derived from linguistic theories and how they
could explain their ability to solve an impressive number of NLP tasks.
In this talk, I will present two series of experiments falling within
this line of research and aiming at highlighting the information flows
within a Transformer network. The first series of experiments focuses
on the long distance agreement task (e.g. between a verb and its
subject), one of the most popular methods to assess neural networks’
ability to encode syntactic information. I will present several
experimental results showing that transformers are able to build an
abstract, high-level sentence representation rather than solely
capturing surface statistical regularities. In a second series of
experiments, I will use a controlled set of examples to investigate how
gender information circulates in an encoder-decoder architecture
considering both probing techniques as well as interventions on the
internal representations used in the MT system.
Joint work with Bingzhi Li, Benoit Crabbé, Lichao Zhu, Nicolas
Bailler and François Yvon |
|---|
| responsibles | Seddah |
|---|
| |
|