Part of Advances in Neural Information Processing Systems 32 (NeurIPS 2019)
Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, Marco Baroni
Despite renewed interest in emergent language simulations with
neural networks, little is known about the basic properties of the
induced code, and how they compare to human language. One
fundamental characteristic of the latter, known as Zipf's Law of
Abbreviation (ZLA), is that more frequent words are efficiently
associated to shorter strings. We study whether the same pattern
emerges when two neural networks, a speaker'' and alistener'',
are trained to play a signaling game. Surprisingly, we find that
networks develop an \emph{anti-efficient} encoding scheme,
in which the most frequent inputs are associated to the longest messages,
and messages in general are skewed towards the maximum length threshold.
This anti-efficient code appears easier to discriminate for the listener,
and, unlike in human communication, the speaker does not impose a
contrasting least-effort pressure towards brevity. Indeed, when the
cost function includes a penalty for longer messages, the resulting
message distribution starts respecting ZLA. Our analysis stresses
the importance of studying the basic features of emergent
communication in a highly controlled setup, to ensure the latter
will not strand too far from human language. Moreover, we present a
concrete illustration of how different functional pressures can lead
to successful communication codes that lack basic properties of
human language, thus highlighting the role such pressures play in
the latter.