{"title": "Disentangled behavioural representations", "book": "Advances in Neural Information Processing Systems", "page_first": 2254, "page_last": 2263, "abstract": "Individual characteristics in human decision-making are often\n  quantified by fitting a parametric cognitive model to subjects'\n  behavior and then studying differences between them in the associated\n  parameter space.  However, these models often fit behavior more poorly\n  than recurrent neural networks (RNNs), which are more flexible and\n  make fewer assumptions about the underlying decision-making processes.\n  Unfortunately, the parameter and latent activity spaces of RNNs are\n  generally high-dimensional and uninterpretable, making it hard to use\n  them to study individual differences.  Here, we\n  show how to benefit from the flexibility of RNNs while representing\n  individual differences in a low-dimensional and interpretable space.\n  To achieve this, we propose a novel end-to-end learning framework in\n  which an encoder is trained to map the behavior of subjects into a\n  low-dimensional latent space. These low-dimensional representations\n  are used to generate the parameters of individual RNNs corresponding\n  to the decision-making process of each subject.  We introduce terms\n  into the loss function that ensure that the latent dimensions are\n  informative and disentangled, i.e.,\nencouraged to have distinct effects on behavior. This allows them to\nalign with separate facets of\n  individual differences. We illustrate the performance\n   of our framework on synthetic data as well as a dataset including the behavior\n   of patients with psychiatric disorders.", "full_text": "Disentangled behavioral representations\n\nAmir Dezfouli1\u2217 Hassan Ashtiani2 Omar Ghattas13\nRichard Nock145 Peter Dayan6# Cheng Soon Ong14\n\n1Data61, CSIRO 2McMaster University 3University of Chicago\n\n4Australian National University 5University of Sydney 6Max Planck Institute\nCorresponding authors \u2217amir.dezfouli@data61.csiro.au #dayan@tue.mpg.de\n\nAbstract\n\nIndividual characteristics in human decision-making are often quanti\ufb01ed by \ufb01tting\na parametric cognitive model to subjects\u2019 behavior and then studying differences\nbetween them in the associated parameter space. However, these models often \ufb01t\nbehavior more poorly than recurrent neural networks (RNNs), which are more \ufb02ex-\nible and make fewer assumptions about the underlying decision-making processes.\nUnfortunately, the parameter and latent activity spaces of RNNs are generally high-\ndimensional and uninterpretable, making it hard to use them to study individual\ndifferences. Here, we show how to bene\ufb01t from the \ufb02exibility of RNNs while\nrepresenting individual differences in a low-dimensional and interpretable space.\nTo achieve this, we propose a novel end-to-end learning framework in which an\nencoder is trained to map the behavior of subjects into a low-dimensional latent\nspace. These low-dimensional representations are used to generate the parameters\nof individual RNNs corresponding to the decision-making process of each subject.\nWe introduce terms into the loss function that ensure that the latent dimensions are\ninformative and disentangled, i.e., encouraged to have distinct effects on behav-\nior. This allows them to align with separate facets of individual differences. We\nillustrate the performance of our framework on synthetic data as well as a dataset\nincluding the behavior of patients with psychiatric disorders.\n\n1\n\nIntroduction\n\nThere is substantial commonality among humans (and other animals) in the way that they learn from\nexperience in order to make decisions. However, there is often also considerable variability in the\nchoices of different subjects in the same task [Carroll and Maxwell, 1979]. Such variability is rooted\nin the structure of the underlying processes; for example, subjects can differ in their tendencies to\nexplore new actions [e.g., Frank et al., 2009] or in the weights they give to past experiences [e.g.,\nden Ouden et al., 2013]. If meaningfully disentangled, these factors would crisply characterise\nthe decision-making processes of the subjects, and would provide a low-dimensional latent space\nthat could be used for many other tasks including studying the behavioral heterogeneity of subjects\nendowed with the same psychiatric labels. However, extracting such representations from behavioral\ndata is challenging, as choices emerge from a complex set of interactions between latent variables\nand past experiences, making disentanglement dif\ufb01cult.\nOne promising approach proposed for learning low-dimensional representations of behavioral data is\nthrough the use of cognitive modelling [e.g., Navarro et al., 2006, Busemeyer and Stout, 2002]; for\nexample using a reinforcement learning framework [e.g., Daw, 2011]. In this approach, a parametrised\ncomputational model is assumed to underlie the decision-making process, and the parameters of this\nmodel \u2013 such as the tendency to explore and the learning rate \u2013 are found by \ufb01tting each subject\u2019s\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fFigure 1: The model comprises an encoder (shown in red) and decoder (blue). The encoder is the\ncomposition of an RNN and a series of fully-connected layers. The encoder maps each whole input\nsequence (in the rectangles on the left) into a point in the latent space (depicted by (z1, z2)-coordinates\nin the middle) based on the \ufb01nal state of the RNN. The latent representation for each input sequence is\nin turn fed into the decoder (shown in blue). The decoder generates the weights of an RNN (called the\nlearning network here) and is shown by the dotted lines on the right side. The learning network is the\nreconstruction of the closed-loop dynamical process of the subject which generated the corresponding\ninput sequence based on experience. This takes as inputs the previous reward, rn\nt\u22121, previous action,\nt is then multiplied by matrix\nt\u22121, and its previous state, xn\nan\nW n to generate unnormalised probabilities (logits) vn\nt for taking each action in the next trial, which\nare converted (through a softmax) to actual probabilities, \u03c0n\nt (.). The negative-log-likelihoods of the\ntrue actions are used to de\ufb01ne the reconstruction loss, LRE, which along with LDIS and LSEP are used\nto train the model. The LDIS term induces disentanglement at the group level, and LSEP separates the\neffects of each dimension of the latent space on the output of the learning network.\n\nt\u22121, and outputs its next state (xn\n\nt ). xn\n\nchoices. Their individual parameters are treated as the latent representations of each subject. This\napproach has been successful at identifying differences between subjects in various conditions [e.g.,\nYechiam et al., 2005]; however, it is constrained by its reliance on the availability of a suitable\nparametric model that can capture behavior and behavioral variability across subjects. In practice,\nthis often leads to manually designing and comparing a limited number of alternative models which\nmay not \ufb01t the behavioral data closely.\nAn alternative class of computational models of human decision-making involves Recurrent Neural\nNetworks (RNNs) [e.g., Dezfouli et al., 2019, 2018, Yang et al., 2019]. RNNs can model a wide range\nof dynamical systems [Siegelmann and Sontag, 1995] including human decision-making processes,\nand make fewer restrictive assumptions about the underlying dynamical processes. However, unlike\ncognitive models, RNNs typically: (i) have large numbers of parameters, which (ii) are hard to\ninterpret. This renders RNNs impractical for studying and modelling individual differences.\nHere we develop a novel approach which bene\ufb01ts from the \ufb02exibility of RNNs, while representing\nindividual differences in a low-dimensional and interpretable space. For the former, we use an\nautoencoder framework [Rumelhart et al., 1985, Tolstikhin et al., 2017] in which we take the behaviors\nof a set of subjects as input and automatically build a low-dimensional latent space which quanti\ufb01es\naspects of individual differences along different latent dimensions (Figure 1). As in an hyper-networks\n[Ha et al., 2016, Karaletsos et al., 2018], the coordinates of each subject within this latent space are\nthen used to generate the parameters of an RNN which models the decision-making processes of\nthat subject. To address interpretability, we introduce a novel contribution to the autoencoder\u2019s loss\nfunction which encourages the different dimensions of the latent space to have separate effects on\npredicted behavior. This allows them to be interpreted independently. We show that this model is\nable to learn and extract low-dimensional representations from synthetically-generated behavioral\ndata in which we know the ground truth, and we then apply it to experimental data.\n\n2 The model\n\nData. The data D \u2261 {Dn}N\nn=1 comprise N input sequences, in which each sequence n \u2208 {1 . . . N}\nconsists of the choices of a subject on a sequential decision-making task. In input sequence n \u2208\n\n2\n\nRNNz1<latexit sha1_base64=\"yl55mjaQ5FGhj99HcYtcX7mFKtw=\">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8eKpi20oWy2m3bpZhN2J0IN/QlePCji1V/kzX/jts1BWx8MPN6bYWZemEph0HW/ndLa+sbmVnm7srO7t39QPTxqmSTTjPsskYnuhNRwKRT3UaDknVRzGoeSt8PxzcxvP3JtRKIecJLyIKZDJSLBKFrp/qnv9as1t+7OQVaJV5AaFGj2q1+9QcKymCtkkhrT9dwUg5xqFEzyaaWXGZ5SNqZD3rVU0ZibIJ+fOiVnVhmQKNG2FJK5+nsip7Exkzi0nTHFkVn2ZuJ/XjfD6DrIhUoz5IotFkWZJJiQ2d9kIDRnKCeWUKaFvZWwEdWUoU2nYkPwll9eJa2LuufWvbvLWoMUcZThBE7hHDy4ggbcQhN8YDCEZ3iFN0c6L86787FoLTnFzDH8gfP5AwSCjYA=</latexit><latexit sha1_base64=\"yl55mjaQ5FGhj99HcYtcX7mFKtw=\">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8eKpi20oWy2m3bpZhN2J0IN/QlePCji1V/kzX/jts1BWx8MPN6bYWZemEph0HW/ndLa+sbmVnm7srO7t39QPTxqmSTTjPsskYnuhNRwKRT3UaDknVRzGoeSt8PxzcxvP3JtRKIecJLyIKZDJSLBKFrp/qnv9as1t+7OQVaJV5AaFGj2q1+9QcKymCtkkhrT9dwUg5xqFEzyaaWXGZ5SNqZD3rVU0ZibIJ+fOiVnVhmQKNG2FJK5+nsip7Exkzi0nTHFkVn2ZuJ/XjfD6DrIhUoz5IotFkWZJJiQ2d9kIDRnKCeWUKaFvZWwEdWUoU2nYkPwll9eJa2LuufWvbvLWoMUcZThBE7hHDy4ggbcQhN8YDCEZ3iFN0c6L86787FoLTnFzDH8gfP5AwSCjYA=</latexit><latexit sha1_base64=\"yl55mjaQ5FGhj99HcYtcX7mFKtw=\">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8eKpi20oWy2m3bpZhN2J0IN/QlePCji1V/kzX/jts1BWx8MPN6bYWZemEph0HW/ndLa+sbmVnm7srO7t39QPTxqmSTTjPsskYnuhNRwKRT3UaDknVRzGoeSt8PxzcxvP3JtRKIecJLyIKZDJSLBKFrp/qnv9as1t+7OQVaJV5AaFGj2q1+9QcKymCtkkhrT9dwUg5xqFEzyaaWXGZ5SNqZD3rVU0ZibIJ+fOiVnVhmQKNG2FJK5+nsip7Exkzi0nTHFkVn2ZuJ/XjfD6DrIhUoz5IotFkWZJJiQ2d9kIDRnKCeWUKaFvZWwEdWUoU2nYkPwll9eJa2LuufWvbvLWoMUcZThBE7hHDy4ggbcQhN8YDCEZ3iFN0c6L86787FoLTnFzDH8gfP5AwSCjYA=</latexit><latexit sha1_base64=\"yl55mjaQ5FGhj99HcYtcX7mFKtw=\">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8eKpi20oWy2m3bpZhN2J0IN/QlePCji1V/kzX/jts1BWx8MPN6bYWZemEph0HW/ndLa+sbmVnm7srO7t39QPTxqmSTTjPsskYnuhNRwKRT3UaDknVRzGoeSt8PxzcxvP3JtRKIecJLyIKZDJSLBKFrp/qnv9as1t+7OQVaJV5AaFGj2q1+9QcKymCtkkhrT9dwUg5xqFEzyaaWXGZ5SNqZD3rVU0ZibIJ+fOiVnVhmQKNG2FJK5+nsip7Exkzi0nTHFkVn2ZuJ/XjfD6DrIhUoz5IotFkWZJJiQ2d9kIDRnKCeWUKaFvZWwEdWUoU2nYkPwll9eJa2LuufWvbvLWoMUcZThBE7hHDy4ggbcQhN8YDCEZ3iFN0c6L86787FoLTnFzDH8gfP5AwSCjYA=</latexit>z2<latexit sha1_base64=\"R/3clTzm+nXZiDAT3HmS7xMZEDs=\">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKUI8FLx4r2g9oQ9lsJ+3SzSbsboQa+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJzdzvPKLSPJYPZpqgH9GR5CFn1Fjp/mlQG5QrbtVdgKwTLycVyNEclL/6w5ilEUrDBNW657mJ8TOqDGcCZ6V+qjGhbEJH2LNU0gi1ny1OnZELqwxJGCtb0pCF+nsio5HW0yiwnRE1Y73qzcX/vF5qwms/4zJJDUq2XBSmgpiYzP8mQ66QGTG1hDLF7a2EjamizNh0SjYEb/XlddKuVT236t1dVRokj6MIZ3AOl+BBHRpwC01oAYMRPMMrvDnCeXHenY9la8HJZ07hD5zPHwYGjYE=</latexit><latexit sha1_base64=\"R/3clTzm+nXZiDAT3HmS7xMZEDs=\">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKUI8FLx4r2g9oQ9lsJ+3SzSbsboQa+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJzdzvPKLSPJYPZpqgH9GR5CFn1Fjp/mlQG5QrbtVdgKwTLycVyNEclL/6w5ilEUrDBNW657mJ8TOqDGcCZ6V+qjGhbEJH2LNU0gi1ny1OnZELqwxJGCtb0pCF+nsio5HW0yiwnRE1Y73qzcX/vF5qwms/4zJJDUq2XBSmgpiYzP8mQ66QGTG1hDLF7a2EjamizNh0SjYEb/XlddKuVT236t1dVRokj6MIZ3AOl+BBHRpwC01oAYMRPMMrvDnCeXHenY9la8HJZ07hD5zPHwYGjYE=</latexit><latexit sha1_base64=\"R/3clTzm+nXZiDAT3HmS7xMZEDs=\">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKUI8FLx4r2g9oQ9lsJ+3SzSbsboQa+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJzdzvPKLSPJYPZpqgH9GR5CFn1Fjp/mlQG5QrbtVdgKwTLycVyNEclL/6w5ilEUrDBNW657mJ8TOqDGcCZ6V+qjGhbEJH2LNU0gi1ny1OnZELqwxJGCtb0pCF+nsio5HW0yiwnRE1Y73qzcX/vF5qwms/4zJJDUq2XBSmgpiYzP8mQ66QGTG1hDLF7a2EjamizNh0SjYEb/XlddKuVT236t1dVRokj6MIZ3AOl+BBHRpwC01oAYMRPMMrvDnCeXHenY9la8HJZ07hD5zPHwYGjYE=</latexit><latexit sha1_base64=\"R/3clTzm+nXZiDAT3HmS7xMZEDs=\">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKUI8FLx4r2g9oQ9lsJ+3SzSbsboQa+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJzdzvPKLSPJYPZpqgH9GR5CFn1Fjp/mlQG5QrbtVdgKwTLycVyNEclL/6w5ilEUrDBNW657mJ8TOqDGcCZ6V+qjGhbEJH2LNU0gi1ny1OnZELqwxJGCtb0pCF+nsio5HW0yiwnRE1Y73qzcX/vF5qwms/4zJJDUq2XBSmgpiYzP8mQ66QGTG1hDLF7a2EjamizNh0SjYEb/XlddKuVT236t1dVRokj6MIZ3AOl+BBHRpwC01oAYMRPMMrvDnCeXHenY9la8HJZ07hD5zPHwYGjYE=</latexit>LDIS<latexit sha1_base64=\"FoCyHyV2A4z+04iCucXqZxMxXQg=\">AAAB/3icbVDLSsNAFJ3UV62vqODGzWARXJVEBF0WdKHgoqJ9QBPCZDpph04ezNyIJWbhr7hxoYhbf8Odf+Ok7UJbDwwczrmXe+b4ieAKLOvbKC0sLi2vlFcra+sbm1vm9k5LxamkrEljEcuOTxQTPGJN4CBYJ5GMhL5gbX94XvjteyYVj6M7GCXMDUk/4gGnBLTkmXtOSGBAiciuc88B9gDZxdVt7plVq2aNgeeJPSVVNEXDM7+cXkzTkEVABVGqa1sJuBmRwKlgecVJFUsIHZI+62oakZApNxvnz/GhVno4iKV+EeCx+nsjI6FSo9DXk0VaNesV4n9eN4XgzM14lKTAIjo5FKQCQ4yLMnCPS0ZBjDQhVHKdFdMBkYSCrqyiS7BnvzxPWsc126rZNyfVOp7WUUb76AAdIRudojq6RA3URBQ9omf0it6MJ+PFeDc+JqMlY7qzi/7A+PwBY5qWNw==</latexit><latexit sha1_base64=\"FoCyHyV2A4z+04iCucXqZxMxXQg=\">AAAB/3icbVDLSsNAFJ3UV62vqODGzWARXJVEBF0WdKHgoqJ9QBPCZDpph04ezNyIJWbhr7hxoYhbf8Odf+Ok7UJbDwwczrmXe+b4ieAKLOvbKC0sLi2vlFcra+sbm1vm9k5LxamkrEljEcuOTxQTPGJN4CBYJ5GMhL5gbX94XvjteyYVj6M7GCXMDUk/4gGnBLTkmXtOSGBAiciuc88B9gDZxdVt7plVq2aNgeeJPSVVNEXDM7+cXkzTkEVABVGqa1sJuBmRwKlgecVJFUsIHZI+62oakZApNxvnz/GhVno4iKV+EeCx+nsjI6FSo9DXk0VaNesV4n9eN4XgzM14lKTAIjo5FKQCQ4yLMnCPS0ZBjDQhVHKdFdMBkYSCrqyiS7BnvzxPWsc126rZNyfVOp7WUUb76AAdIRudojq6RA3URBQ9omf0it6MJ+PFeDc+JqMlY7qzi/7A+PwBY5qWNw==</latexit><latexit sha1_base64=\"FoCyHyV2A4z+04iCucXqZxMxXQg=\">AAAB/3icbVDLSsNAFJ3UV62vqODGzWARXJVEBF0WdKHgoqJ9QBPCZDpph04ezNyIJWbhr7hxoYhbf8Odf+Ok7UJbDwwczrmXe+b4ieAKLOvbKC0sLi2vlFcra+sbm1vm9k5LxamkrEljEcuOTxQTPGJN4CBYJ5GMhL5gbX94XvjteyYVj6M7GCXMDUk/4gGnBLTkmXtOSGBAiciuc88B9gDZxdVt7plVq2aNgeeJPSVVNEXDM7+cXkzTkEVABVGqa1sJuBmRwKlgecVJFUsIHZI+62oakZApNxvnz/GhVno4iKV+EeCx+nsjI6FSo9DXk0VaNesV4n9eN4XgzM14lKTAIjo5FKQCQ4yLMnCPS0ZBjDQhVHKdFdMBkYSCrqyiS7BnvzxPWsc126rZNyfVOp7WUUb76AAdIRudojq6RA3URBQ9omf0it6MJ+PFeDc+JqMlY7qzi/7A+PwBY5qWNw==</latexit><latexit sha1_base64=\"FoCyHyV2A4z+04iCucXqZxMxXQg=\">AAAB/3icbVDLSsNAFJ3UV62vqODGzWARXJVEBF0WdKHgoqJ9QBPCZDpph04ezNyIJWbhr7hxoYhbf8Odf+Ok7UJbDwwczrmXe+b4ieAKLOvbKC0sLi2vlFcra+sbm1vm9k5LxamkrEljEcuOTxQTPGJN4CBYJ5GMhL5gbX94XvjteyYVj6M7GCXMDUk/4gGnBLTkmXtOSGBAiciuc88B9gDZxdVt7plVq2aNgeeJPSVVNEXDM7+cXkzTkEVABVGqa1sJuBmRwKlgecVJFUsIHZI+62oakZApNxvnz/GhVno4iKV+EeCx+nsjI6FSo9DXk0VaNesV4n9eN4XgzM14lKTAIjo5FKQCQ4yLMnCPS0ZBjDQhVHKdFdMBkYSCrqyiS7BnvzxPWsc126rZNyfVOp7WUUb76AAdIRudojq6RA3URBQ9omf0it6MJ+PFeDc+JqMlY7qzi/7A+PwBY5qWNw==</latexit>xnt1<latexit sha1_base64=\"BN+VMrBMKvoavnEgTZ6+fbdhQXA=\">AAAB/XicbVDLSsNAFJ3UV62v+Ni5GSyCG0sigi4LblxWsA9oY5hMJ+3QySTM3Ig1BH/FjQtF3Pof7vwbJ20X2npg4HDOvdwzJ0gE1+A431ZpaXllda28XtnY3NresXf3WjpOFWVNGotYdQKimeCSNYGDYJ1EMRIFgrWD0VXht++Z0jyWtzBOmBeRgeQhpwSM5NsHvYjAMAizh9zP4NTN7zKZ+3bVqTkT4EXizkgVzdDw7a9eP6ZpxCRQQbTuuk4CXkYUcCpYXumlmiWEjsiAdQ2VJGLayybpc3xslD4OY2WeBDxRf29kJNJ6HAVmssiq571C/M/rphBeehmXSQpM0umhMBUYYlxUgftcMQpibAihipusmA6JIhRMYRVTgjv/5UXSOqu5Ts29Oa/W8ayOMjpER+gEuegC1dE1aqAmougRPaNX9GY9WS/Wu/UxHS1Zs5199AfW5w8MPJV8</latexit><latexit sha1_base64=\"BN+VMrBMKvoavnEgTZ6+fbdhQXA=\">AAAB/XicbVDLSsNAFJ3UV62v+Ni5GSyCG0sigi4LblxWsA9oY5hMJ+3QySTM3Ig1BH/FjQtF3Pof7vwbJ20X2npg4HDOvdwzJ0gE1+A431ZpaXllda28XtnY3NresXf3WjpOFWVNGotYdQKimeCSNYGDYJ1EMRIFgrWD0VXht++Z0jyWtzBOmBeRgeQhpwSM5NsHvYjAMAizh9zP4NTN7zKZ+3bVqTkT4EXizkgVzdDw7a9eP6ZpxCRQQbTuuk4CXkYUcCpYXumlmiWEjsiAdQ2VJGLayybpc3xslD4OY2WeBDxRf29kJNJ6HAVmssiq571C/M/rphBeehmXSQpM0umhMBUYYlxUgftcMQpibAihipusmA6JIhRMYRVTgjv/5UXSOqu5Ts29Oa/W8ayOMjpER+gEuegC1dE1aqAmougRPaNX9GY9WS/Wu/UxHS1Zs5199AfW5w8MPJV8</latexit><latexit sha1_base64=\"BN+VMrBMKvoavnEgTZ6+fbdhQXA=\">AAAB/XicbVDLSsNAFJ3UV62v+Ni5GSyCG0sigi4LblxWsA9oY5hMJ+3QySTM3Ig1BH/FjQtF3Pof7vwbJ20X2npg4HDOvdwzJ0gE1+A431ZpaXllda28XtnY3NresXf3WjpOFWVNGotYdQKimeCSNYGDYJ1EMRIFgrWD0VXht++Z0jyWtzBOmBeRgeQhpwSM5NsHvYjAMAizh9zP4NTN7zKZ+3bVqTkT4EXizkgVzdDw7a9eP6ZpxCRQQbTuuk4CXkYUcCpYXumlmiWEjsiAdQ2VJGLayybpc3xslD4OY2WeBDxRf29kJNJ6HAVmssiq571C/M/rphBeehmXSQpM0umhMBUYYlxUgftcMQpibAihipusmA6JIhRMYRVTgjv/5UXSOqu5Ts29Oa/W8ayOMjpER+gEuegC1dE1aqAmougRPaNX9GY9WS/Wu/UxHS1Zs5199AfW5w8MPJV8</latexit><latexit sha1_base64=\"BN+VMrBMKvoavnEgTZ6+fbdhQXA=\">AAAB/XicbVDLSsNAFJ3UV62v+Ni5GSyCG0sigi4LblxWsA9oY5hMJ+3QySTM3Ig1BH/FjQtF3Pof7vwbJ20X2npg4HDOvdwzJ0gE1+A431ZpaXllda28XtnY3NresXf3WjpOFWVNGotYdQKimeCSNYGDYJ1EMRIFgrWD0VXht++Z0jyWtzBOmBeRgeQhpwSM5NsHvYjAMAizh9zP4NTN7zKZ+3bVqTkT4EXizkgVzdDw7a9eP6ZpxCRQQbTuuk4CXkYUcCpYXumlmiWEjsiAdQ2VJGLayybpc3xslD4OY2WeBDxRf29kJNJ6HAVmssiq571C/M/rphBeehmXSQpM0umhMBUYYlxUgftcMQpibAihipusmA6JIhRMYRVTgjv/5UXSOqu5Ts29Oa/W8ayOMjpER+gEuegC1dE1aqAmougRPaNX9GY9WS/Wu/UxHS1Zs5199AfW5w8MPJV8</latexit>ant1<latexit sha1_base64=\"xMovVCBOML7Z1VLF3K8Y9jsU4M4=\">AAAB8HicbVDLSgNBEOz1GeMr6tHLYBC8GHZF0GPAi8cI5iHJGmYnk2TIzOwy0yuEJV/hxYMiXv0cb/6Nk2QPmljQUFR1090VJVJY9P1vb2V1bX1js7BV3N7Z3dsvHRw2bJwaxusslrFpRdRyKTSvo0DJW4nhVEWSN6PRzdRvPnFjRazvcZzwUNGBFn3BKDrpgT7qbobnwaRbKvsVfwayTIKclCFHrVv66vRiliqukUlqbTvwEwwzalAwySfFTmp5QtmIDnjbUU0Vt2E2O3hCTp3SI/3YuNJIZurviYwqa8cqcp2K4tAuelPxP6+dYv86zIROUuSazRf1U0kwJtPvSU8YzlCOHaHMCHcrYUNqKEOXUdGFECy+vEwaF5XArwR3l+UqyeMowDGcwBkEcAVVuIUa1IGBgmd4hTfPeC/eu/cxb13x8pkj+APv8wdqBZAI</latexit><latexit sha1_base64=\"xMovVCBOML7Z1VLF3K8Y9jsU4M4=\">AAAB8HicbVDLSgNBEOz1GeMr6tHLYBC8GHZF0GPAi8cI5iHJGmYnk2TIzOwy0yuEJV/hxYMiXv0cb/6Nk2QPmljQUFR1090VJVJY9P1vb2V1bX1js7BV3N7Z3dsvHRw2bJwaxusslrFpRdRyKTSvo0DJW4nhVEWSN6PRzdRvPnFjRazvcZzwUNGBFn3BKDrpgT7qbobnwaRbKvsVfwayTIKclCFHrVv66vRiliqukUlqbTvwEwwzalAwySfFTmp5QtmIDnjbUU0Vt2E2O3hCTp3SI/3YuNJIZurviYwqa8cqcp2K4tAuelPxP6+dYv86zIROUuSazRf1U0kwJtPvSU8YzlCOHaHMCHcrYUNqKEOXUdGFECy+vEwaF5XArwR3l+UqyeMowDGcwBkEcAVVuIUa1IGBgmd4hTfPeC/eu/cxb13x8pkj+APv8wdqBZAI</latexit><latexit sha1_base64=\"xMovVCBOML7Z1VLF3K8Y9jsU4M4=\">AAAB8HicbVDLSgNBEOz1GeMr6tHLYBC8GHZF0GPAi8cI5iHJGmYnk2TIzOwy0yuEJV/hxYMiXv0cb/6Nk2QPmljQUFR1090VJVJY9P1vb2V1bX1js7BV3N7Z3dsvHRw2bJwaxusslrFpRdRyKTSvo0DJW4nhVEWSN6PRzdRvPnFjRazvcZzwUNGBFn3BKDrpgT7qbobnwaRbKvsVfwayTIKclCFHrVv66vRiliqukUlqbTvwEwwzalAwySfFTmp5QtmIDnjbUU0Vt2E2O3hCTp3SI/3YuNJIZurviYwqa8cqcp2K4tAuelPxP6+dYv86zIROUuSazRf1U0kwJtPvSU8YzlCOHaHMCHcrYUNqKEOXUdGFECy+vEwaF5XArwR3l+UqyeMowDGcwBkEcAVVuIUa1IGBgmd4hTfPeC/eu/cxb13x8pkj+APv8wdqBZAI</latexit><latexit sha1_base64=\"xMovVCBOML7Z1VLF3K8Y9jsU4M4=\">AAAB8HicbVDLSgNBEOz1GeMr6tHLYBC8GHZF0GPAi8cI5iHJGmYnk2TIzOwy0yuEJV/hxYMiXv0cb/6Nk2QPmljQUFR1090VJVJY9P1vb2V1bX1js7BV3N7Z3dsvHRw2bJwaxusslrFpRdRyKTSvo0DJW4nhVEWSN6PRzdRvPnFjRazvcZzwUNGBFn3BKDrpgT7qbobnwaRbKvsVfwayTIKclCFHrVv66vRiliqukUlqbTvwEwwzalAwySfFTmp5QtmIDnjbUU0Vt2E2O3hCTp3SI/3YuNJIZurviYwqa8cqcp2K4tAuelPxP6+dYv86zIROUuSazRf1U0kwJtPvSU8YzlCOHaHMCHcrYUNqKEOXUdGFECy+vEwaF5XArwR3l+UqyeMowDGcwBkEcAVVuIUa1IGBgmd4hTfPeC/eu/cxb13x8pkj+APv8wdqBZAI</latexit>rnt1<latexit sha1_base64=\"Y9gtPiZym8euQvLji49w3e56mrQ=\">AAAB8HicbVDLSgNBEOz1GeMr6tHLYBC8GHZF0GPAi8cI5iHJGmYnk2TIzOwy0yuEJV/hxYMiXv0cb/6Nk2QPmljQUFR1090VJVJY9P1vb2V1bX1js7BV3N7Z3dsvHRw2bJwaxusslrFpRdRyKTSvo0DJW4nhVEWSN6PRzdRvPnFjRazvcZzwUNGBFn3BKDrpwTzqbobnwaRbKvsVfwayTIKclCFHrVv66vRiliqukUlqbTvwEwwzalAwySfFTmp5QtmIDnjbUU0Vt2E2O3hCTp3SI/3YuNJIZurviYwqa8cqcp2K4tAuelPxP6+dYv86zIROUuSazRf1U0kwJtPvSU8YzlCOHaHMCHcrYUNqKEOXUdGFECy+vEwaF5XArwR3l+UqyeMowDGcwBkEcAVVuIUa1IGBgmd4hTfPeC/eu/cxb13x8pkj+APv8weEUZAZ</latexit><latexit sha1_base64=\"Y9gtPiZym8euQvLji49w3e56mrQ=\">AAAB8HicbVDLSgNBEOz1GeMr6tHLYBC8GHZF0GPAi8cI5iHJGmYnk2TIzOwy0yuEJV/hxYMiXv0cb/6Nk2QPmljQUFR1090VJVJY9P1vb2V1bX1js7BV3N7Z3dsvHRw2bJwaxusslrFpRdRyKTSvo0DJW4nhVEWSN6PRzdRvPnFjRazvcZzwUNGBFn3BKDrpwTzqbobnwaRbKvsVfwayTIKclCFHrVv66vRiliqukUlqbTvwEwwzalAwySfFTmp5QtmIDnjbUU0Vt2E2O3hCTp3SI/3YuNJIZurviYwqa8cqcp2K4tAuelPxP6+dYv86zIROUuSazRf1U0kwJtPvSU8YzlCOHaHMCHcrYUNqKEOXUdGFECy+vEwaF5XArwR3l+UqyeMowDGcwBkEcAVVuIUa1IGBgmd4hTfPeC/eu/cxb13x8pkj+APv8weEUZAZ</latexit><latexit sha1_base64=\"Y9gtPiZym8euQvLji49w3e56mrQ=\">AAAB8HicbVDLSgNBEOz1GeMr6tHLYBC8GHZF0GPAi8cI5iHJGmYnk2TIzOwy0yuEJV/hxYMiXv0cb/6Nk2QPmljQUFR1090VJVJY9P1vb2V1bX1js7BV3N7Z3dsvHRw2bJwaxusslrFpRdRyKTSvo0DJW4nhVEWSN6PRzdRvPnFjRazvcZzwUNGBFn3BKDrpwTzqbobnwaRbKvsVfwayTIKclCFHrVv66vRiliqukUlqbTvwEwwzalAwySfFTmp5QtmIDnjbUU0Vt2E2O3hCTp3SI/3YuNJIZurviYwqa8cqcp2K4tAuelPxP6+dYv86zIROUuSazRf1U0kwJtPvSU8YzlCOHaHMCHcrYUNqKEOXUdGFECy+vEwaF5XArwR3l+UqyeMowDGcwBkEcAVVuIUa1IGBgmd4hTfPeC/eu/cxb13x8pkj+APv8weEUZAZ</latexit><latexit sha1_base64=\"Y9gtPiZym8euQvLji49w3e56mrQ=\">AAAB8HicbVDLSgNBEOz1GeMr6tHLYBC8GHZF0GPAi8cI5iHJGmYnk2TIzOwy0yuEJV/hxYMiXv0cb/6Nk2QPmljQUFR1090VJVJY9P1vb2V1bX1js7BV3N7Z3dsvHRw2bJwaxusslrFpRdRyKTSvo0DJW4nhVEWSN6PRzdRvPnFjRazvcZzwUNGBFn3BKDrpwTzqbobnwaRbKvsVfwayTIKclCFHrVv66vRiliqukUlqbTvwEwwzalAwySfFTmp5QtmIDnjbUU0Vt2E2O3hCTp3SI/3YuNJIZurviYwqa8cqcp2K4tAuelPxP6+dYv86zIROUuSazRf1U0kwJtPvSU8YzlCOHaHMCHcrYUNqKEOXUdGFECy+vEwaF5XArwR3l+UqyeMowDGcwBkEcAVVuIUa1IGBgmd4hTfPeC/eu/cxb13x8pkj+APv8weEUZAZ</latexit>xnt<latexit sha1_base64=\"z1pJ4w9Pi+nIN6Fl1KSMaK855Xc=\">AAAB+3icbVDLSsNAFL2pr1pfsS7dDBbBVUlE0GXBjcsK9gFtDJPppB06mYSZibSE/IobF4q49Ufc+TdO2iy09cDA4Zx7uWdOkHCmtON8W5WNza3tnepubW//4PDIPq53VZxKQjsk5rHsB1hRzgTtaKY57SeS4ijgtBdMbwu/90SlYrF40POEehEeCxYygrWRfLs+jLCeBGE2y/1M54+ZyH274TSdBdA6cUvSgBJt3/4ajmKSRlRowrFSA9dJtJdhqRnhNK8NU0UTTKZ4TAeGChxR5WWL7Dk6N8oIhbE0T2i0UH9vZDhSah4FZrJIqla9QvzPG6Q6vPEyJpJUU0GWh8KUIx2jogg0YpISzeeGYCKZyYrIBEtMtKmrZkpwV7+8TrqXTddpuvdXjRYq66jCKZzBBbhwDS24gzZ0gMAMnuEV3qzcerHerY/laMUqd07gD6zPHyZOlQo=</latexit><latexit sha1_base64=\"z1pJ4w9Pi+nIN6Fl1KSMaK855Xc=\">AAAB+3icbVDLSsNAFL2pr1pfsS7dDBbBVUlE0GXBjcsK9gFtDJPppB06mYSZibSE/IobF4q49Ufc+TdO2iy09cDA4Zx7uWdOkHCmtON8W5WNza3tnepubW//4PDIPq53VZxKQjsk5rHsB1hRzgTtaKY57SeS4ijgtBdMbwu/90SlYrF40POEehEeCxYygrWRfLs+jLCeBGE2y/1M54+ZyH274TSdBdA6cUvSgBJt3/4ajmKSRlRowrFSA9dJtJdhqRnhNK8NU0UTTKZ4TAeGChxR5WWL7Dk6N8oIhbE0T2i0UH9vZDhSah4FZrJIqla9QvzPG6Q6vPEyJpJUU0GWh8KUIx2jogg0YpISzeeGYCKZyYrIBEtMtKmrZkpwV7+8TrqXTddpuvdXjRYq66jCKZzBBbhwDS24gzZ0gMAMnuEV3qzcerHerY/laMUqd07gD6zPHyZOlQo=</latexit><latexit sha1_base64=\"z1pJ4w9Pi+nIN6Fl1KSMaK855Xc=\">AAAB+3icbVDLSsNAFL2pr1pfsS7dDBbBVUlE0GXBjcsK9gFtDJPppB06mYSZibSE/IobF4q49Ufc+TdO2iy09cDA4Zx7uWdOkHCmtON8W5WNza3tnepubW//4PDIPq53VZxKQjsk5rHsB1hRzgTtaKY57SeS4ijgtBdMbwu/90SlYrF40POEehEeCxYygrWRfLs+jLCeBGE2y/1M54+ZyH274TSdBdA6cUvSgBJt3/4ajmKSRlRowrFSA9dJtJdhqRnhNK8NU0UTTKZ4TAeGChxR5WWL7Dk6N8oIhbE0T2i0UH9vZDhSah4FZrJIqla9QvzPG6Q6vPEyJpJUU0GWh8KUIx2jogg0YpISzeeGYCKZyYrIBEtMtKmrZkpwV7+8TrqXTddpuvdXjRYq66jCKZzBBbhwDS24gzZ0gMAMnuEV3qzcerHerY/laMUqd07gD6zPHyZOlQo=</latexit><latexit sha1_base64=\"z1pJ4w9Pi+nIN6Fl1KSMaK855Xc=\">AAAB+3icbVDLSsNAFL2pr1pfsS7dDBbBVUlE0GXBjcsK9gFtDJPppB06mYSZibSE/IobF4q49Ufc+TdO2iy09cDA4Zx7uWdOkHCmtON8W5WNza3tnepubW//4PDIPq53VZxKQjsk5rHsB1hRzgTtaKY57SeS4ijgtBdMbwu/90SlYrF40POEehEeCxYygrWRfLs+jLCeBGE2y/1M54+ZyH274TSdBdA6cUvSgBJt3/4ajmKSRlRowrFSA9dJtJdhqRnhNK8NU0UTTKZ4TAeGChxR5WWL7Dk6N8oIhbE0T2i0UH9vZDhSah4FZrJIqla9QvzPG6Q6vPEyJpJUU0GWh8KUIx2jogg0YpISzeeGYCKZyYrIBEtMtKmrZkpwV7+8TrqXTddpuvdXjRYq66jCKZzBBbhwDS24gzZ0gMAMnuEV3qzcerHerY/laMUqd07gD6zPHyZOlQo=</latexit>vnt<latexit sha1_base64=\"0fjG2cQx/rMWU+q6cx6mecOOfLw=\">AAAB+3icbVDLSsNAFL2pr1pfsS7dDBbBVUlE0GXBjcsK9gFtDJPppB06mYSZSbGE/IobF4q49Ufc+TdO2iy09cDA4Zx7uWdOkHCmtON8W5WNza3tnepubW//4PDIPq53VZxKQjsk5rHsB1hRzgTtaKY57SeS4ijgtBdMbwu/N6NSsVg86HlCvQiPBQsZwdpIvl0fRlhPgjCb5X6m88dM5L7dcJrOAmiduCVpQIm2b38NRzFJIyo04Vipgesk2suw1IxwmteGqaIJJlM8pgNDBY6o8rJF9hydG2WEwliaJzRaqL83MhwpNY8CM1kkVateIf7nDVId3ngZE0mqqSDLQ2HKkY5RUQQaMUmJ5nNDMJHMZEVkgiUm2tRVMyW4q19eJ93Lpus03furRguVdVThFM7gAly4hhbcQRs6QOAJnuEV3qzcerHerY/laMUqd07gD6zPHyM0lQg=</latexit><latexit sha1_base64=\"0fjG2cQx/rMWU+q6cx6mecOOfLw=\">AAAB+3icbVDLSsNAFL2pr1pfsS7dDBbBVUlE0GXBjcsK9gFtDJPppB06mYSZSbGE/IobF4q49Ufc+TdO2iy09cDA4Zx7uWdOkHCmtON8W5WNza3tnepubW//4PDIPq53VZxKQjsk5rHsB1hRzgTtaKY57SeS4ijgtBdMbwu/N6NSsVg86HlCvQiPBQsZwdpIvl0fRlhPgjCb5X6m88dM5L7dcJrOAmiduCVpQIm2b38NRzFJIyo04Vipgesk2suw1IxwmteGqaIJJlM8pgNDBY6o8rJF9hydG2WEwliaJzRaqL83MhwpNY8CM1kkVateIf7nDVId3ngZE0mqqSDLQ2HKkY5RUQQaMUmJ5nNDMJHMZEVkgiUm2tRVMyW4q19eJ93Lpus03furRguVdVThFM7gAly4hhbcQRs6QOAJnuEV3qzcerHerY/laMUqd07gD6zPHyM0lQg=</latexit><latexit sha1_base64=\"0fjG2cQx/rMWU+q6cx6mecOOfLw=\">AAAB+3icbVDLSsNAFL2pr1pfsS7dDBbBVUlE0GXBjcsK9gFtDJPppB06mYSZSbGE/IobF4q49Ufc+TdO2iy09cDA4Zx7uWdOkHCmtON8W5WNza3tnepubW//4PDIPq53VZxKQjsk5rHsB1hRzgTtaKY57SeS4ijgtBdMbwu/N6NSsVg86HlCvQiPBQsZwdpIvl0fRlhPgjCb5X6m88dM5L7dcJrOAmiduCVpQIm2b38NRzFJIyo04Vipgesk2suw1IxwmteGqaIJJlM8pgNDBY6o8rJF9hydG2WEwliaJzRaqL83MhwpNY8CM1kkVateIf7nDVId3ngZE0mqqSDLQ2HKkY5RUQQaMUmJ5nNDMJHMZEVkgiUm2tRVMyW4q19eJ93Lpus03furRguVdVThFM7gAly4hhbcQRs6QOAJnuEV3qzcerHerY/laMUqd07gD6zPHyM0lQg=</latexit><latexit sha1_base64=\"0fjG2cQx/rMWU+q6cx6mecOOfLw=\">AAAB+3icbVDLSsNAFL2pr1pfsS7dDBbBVUlE0GXBjcsK9gFtDJPppB06mYSZSbGE/IobF4q49Ufc+TdO2iy09cDA4Zx7uWdOkHCmtON8W5WNza3tnepubW//4PDIPq53VZxKQjsk5rHsB1hRzgTtaKY57SeS4ijgtBdMbwu/N6NSsVg86HlCvQiPBQsZwdpIvl0fRlhPgjCb5X6m88dM5L7dcJrOAmiduCVpQIm2b38NRzFJIyo04Vipgesk2suw1IxwmteGqaIJJlM8pgNDBY6o8rJF9hydG2WEwliaJzRaqL83MhwpNY8CM1kkVateIf7nDVId3ngZE0mqqSDLQ2HKkY5RUQQaMUmJ5nNDMJHMZEVkgiUm2tRVMyW4q19eJ93Lpus03furRguVdVThFM7gAly4hhbcQRs6QOAJnuEV3qzcerHerY/laMUqd07gD6zPHyM0lQg=</latexit>\u21e1nt(.)<latexit sha1_base64=\"cVeNfIfCNk9OlOgZcv92kD5PAUU=\">AAAB8XicbVBNS8NAEJ3Ur1q/qh69LBahXkIigh4LXjxWsB/YxrLZbtqlm03YnQil9F948aCIV/+NN/+N2zYHbX0w8Hhvhpl5YSqFQc/7dgpr6xubW8Xt0s7u3v5B+fCoaZJMM95giUx0O6SGS6F4AwVK3k41p3EoeSsc3cz81hPXRiTqHscpD2I6UCISjKKVHrqp6OGjqrrnvXLFc705yCrxc1KBHPVe+avbT1gWc4VMUmM6vpdiMKEaBZN8WupmhqeUjeiAdyxVNOYmmMwvnpIzq/RJlGhbCslc/T0xobEx4zi0nTHFoVn2ZuJ/XifD6DqYCJVmyBVbLIoySTAhs/dJX2jOUI4toUwLeythQ6opQxtSyYbgL7+8SpoXru+5/t1lpUbyOIpwAqdQBR+uoAa3UIcGMFDwDK/w5hjnxXl3PhatBSefOYY/cD5/AI3qkA8=</latexit><latexit sha1_base64=\"cVeNfIfCNk9OlOgZcv92kD5PAUU=\">AAAB8XicbVBNS8NAEJ3Ur1q/qh69LBahXkIigh4LXjxWsB/YxrLZbtqlm03YnQil9F948aCIV/+NN/+N2zYHbX0w8Hhvhpl5YSqFQc/7dgpr6xubW8Xt0s7u3v5B+fCoaZJMM95giUx0O6SGS6F4AwVK3k41p3EoeSsc3cz81hPXRiTqHscpD2I6UCISjKKVHrqp6OGjqrrnvXLFc705yCrxc1KBHPVe+avbT1gWc4VMUmM6vpdiMKEaBZN8WupmhqeUjeiAdyxVNOYmmMwvnpIzq/RJlGhbCslc/T0xobEx4zi0nTHFoVn2ZuJ/XifD6DqYCJVmyBVbLIoySTAhs/dJX2jOUI4toUwLeythQ6opQxtSyYbgL7+8SpoXru+5/t1lpUbyOIpwAqdQBR+uoAa3UIcGMFDwDK/w5hjnxXl3PhatBSefOYY/cD5/AI3qkA8=</latexit><latexit sha1_base64=\"cVeNfIfCNk9OlOgZcv92kD5PAUU=\">AAAB8XicbVBNS8NAEJ3Ur1q/qh69LBahXkIigh4LXjxWsB/YxrLZbtqlm03YnQil9F948aCIV/+NN/+N2zYHbX0w8Hhvhpl5YSqFQc/7dgpr6xubW8Xt0s7u3v5B+fCoaZJMM95giUx0O6SGS6F4AwVK3k41p3EoeSsc3cz81hPXRiTqHscpD2I6UCISjKKVHrqp6OGjqrrnvXLFc705yCrxc1KBHPVe+avbT1gWc4VMUmM6vpdiMKEaBZN8WupmhqeUjeiAdyxVNOYmmMwvnpIzq/RJlGhbCslc/T0xobEx4zi0nTHFoVn2ZuJ/XifD6DqYCJVmyBVbLIoySTAhs/dJX2jOUI4toUwLeythQ6opQxtSyYbgL7+8SpoXru+5/t1lpUbyOIpwAqdQBR+uoAa3UIcGMFDwDK/w5hjnxXl3PhatBSefOYY/cD5/AI3qkA8=</latexit><latexit sha1_base64=\"cVeNfIfCNk9OlOgZcv92kD5PAUU=\">AAAB8XicbVBNS8NAEJ3Ur1q/qh69LBahXkIigh4LXjxWsB/YxrLZbtqlm03YnQil9F948aCIV/+NN/+N2zYHbX0w8Hhvhpl5YSqFQc/7dgpr6xubW8Xt0s7u3v5B+fCoaZJMM95giUx0O6SGS6F4AwVK3k41p3EoeSsc3cz81hPXRiTqHscpD2I6UCISjKKVHrqp6OGjqrrnvXLFc705yCrxc1KBHPVe+avbT1gWc4VMUmM6vpdiMKEaBZN8WupmhqeUjeiAdyxVNOYmmMwvnpIzq/RJlGhbCslc/T0xobEx4zi0nTHFoVn2ZuJ/XifD6DqYCJVmyBVbLIoySTAhs/dJX2jOUI4toUwLeythQ6opQxtSyYbgL7+8SpoXru+5/t1lpUbyOIpwAqdQBR+uoAa3UIcGMFDwDK/w5hjnxXl3PhatBSefOYY/cD5/AI3qkA8=</latexit>SoftmaxWn<latexit sha1_base64=\"tbRvGnsdsQDMw9rly7QaZKhdu1A=\">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8eK9gPaWDbbSbt0swm7G6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTAXXxvO+ndLa+sbmVnm7srO7t3/gHh61dJIphk2WiER1QqpRcIlNw43ATqqQxqHAdji+mfntJ1SaJ/LBTFIMYjqUPOKMGivdtx9l3616NW8Oskr8glShQKPvfvUGCctilIYJqnXX91IT5FQZzgROK71MY0rZmA6xa6mkMeogn586JWdWGZAoUbakIXP190ROY60ncWg7Y2pGetmbif953cxE10HOZZoZlGyxKMoEMQmZ/U0GXCEzYmIJZYrbWwkbUUWZselUbAj+8surpHVR872af3dZrZMijjKcwCmcgw9XUIdbaEATGAzhGV7hzRHOi/PufCxaS04xcwx/4Hz+ACofjZk=</latexit><latexit sha1_base64=\"tbRvGnsdsQDMw9rly7QaZKhdu1A=\">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8eK9gPaWDbbSbt0swm7G6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTAXXxvO+ndLa+sbmVnm7srO7t3/gHh61dJIphk2WiER1QqpRcIlNw43ATqqQxqHAdji+mfntJ1SaJ/LBTFIMYjqUPOKMGivdtx9l3616NW8Oskr8glShQKPvfvUGCctilIYJqnXX91IT5FQZzgROK71MY0rZmA6xa6mkMeogn586JWdWGZAoUbakIXP190ROY60ncWg7Y2pGetmbif953cxE10HOZZoZlGyxKMoEMQmZ/U0GXCEzYmIJZYrbWwkbUUWZselUbAj+8surpHVR872af3dZrZMijjKcwCmcgw9XUIdbaEATGAzhGV7hzRHOi/PufCxaS04xcwx/4Hz+ACofjZk=</latexit><latexit sha1_base64=\"tbRvGnsdsQDMw9rly7QaZKhdu1A=\">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8eK9gPaWDbbSbt0swm7G6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTAXXxvO+ndLa+sbmVnm7srO7t3/gHh61dJIphk2WiER1QqpRcIlNw43ATqqQxqHAdji+mfntJ1SaJ/LBTFIMYjqUPOKMGivdtx9l3616NW8Oskr8glShQKPvfvUGCctilIYJqnXX91IT5FQZzgROK71MY0rZmA6xa6mkMeogn586JWdWGZAoUbakIXP190ROY60ncWg7Y2pGetmbif953cxE10HOZZoZlGyxKMoEMQmZ/U0GXCEzYmIJZYrbWwkbUUWZselUbAj+8surpHVR872af3dZrZMijjKcwCmcgw9XUIdbaEATGAzhGV7hzRHOi/PufCxaS04xcwx/4Hz+ACofjZk=</latexit><latexit sha1_base64=\"tbRvGnsdsQDMw9rly7QaZKhdu1A=\">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8eK9gPaWDbbSbt0swm7G6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTAXXxvO+ndLa+sbmVnm7srO7t3/gHh61dJIphk2WiER1QqpRcIlNw43ATqqQxqHAdji+mfntJ1SaJ/LBTFIMYjqUPOKMGivdtx9l3616NW8Oskr8glShQKPvfvUGCctilIYJqnXX91IT5FQZzgROK71MY0rZmA6xa6mkMeogn586JWdWGZAoUbakIXP190ROY60ncWg7Y2pGetmbif953cxE10HOZZoZlGyxKMoEMQmZ/U0GXCEzYmIJZYrbWwkbUUWZselUbAj+8surpHVR872af3dZrZMijjKcwCmcgw9XUIdbaEATGAzhGV7hzRHOi/PufCxaS04xcwx/4Hz+ACofjZk=</latexit>Learning networkEncoderDecoderInput sequencesLatent spacean1...anTn<latexit sha1_base64=\"fv2VJikq0gzlJIqz81/fun8OtTA=\">AAAB/XicbVDJSgNBEK2JW4xbXG5eGoPgKcyIoMeAF48RskGWoabTkzTp6Rm6e4Q4BH/FiwdFvPof3vwbO8tBEx8UPN6r6q56QSK4Nq777eTW1jc2t/LbhZ3dvf2D4uFRQ8epoqxOYxGrVoCaCS5Z3XAjWCtRDKNAsGYwup36zQemNI9lzYwT1o1wIHnIKRor+cUT9L2e7PRjown6Wa0nJz0rl9yyOwNZJd6ClGCBql/8si/QNGLSUIFatz03Md0MleFUsEmhk2qWIB3hgLUtlRgx3c1m20/IuVX6JIyVLWnITP09kWGk9TgKbGeEZqiXvan4n9dOTXjTzbhMUsMknX8UpoKYmEyjIH2uGDVibAlSxe2uhA5RITU2sIINwVs+eZU0LsueW/bur0oVsogjD6dwBhfgwTVU4A6qUAcKj/AMr/DmPDkvzrvzMW/NOYuZY/gD5/MHJA2U6A==</latexit><latexit sha1_base64=\"fv2VJikq0gzlJIqz81/fun8OtTA=\">AAAB/XicbVDJSgNBEK2JW4xbXG5eGoPgKcyIoMeAF48RskGWoabTkzTp6Rm6e4Q4BH/FiwdFvPof3vwbO8tBEx8UPN6r6q56QSK4Nq777eTW1jc2t/LbhZ3dvf2D4uFRQ8epoqxOYxGrVoCaCS5Z3XAjWCtRDKNAsGYwup36zQemNI9lzYwT1o1wIHnIKRor+cUT9L2e7PRjown6Wa0nJz0rl9yyOwNZJd6ClGCBql/8si/QNGLSUIFatz03Md0MleFUsEmhk2qWIB3hgLUtlRgx3c1m20/IuVX6JIyVLWnITP09kWGk9TgKbGeEZqiXvan4n9dOTXjTzbhMUsMknX8UpoKYmEyjIH2uGDVibAlSxe2uhA5RITU2sIINwVs+eZU0LsueW/bur0oVsogjD6dwBhfgwTVU4A6qUAcKj/AMr/DmPDkvzrvzMW/NOYuZY/gD5/MHJA2U6A==</latexit><latexit sha1_base64=\"fv2VJikq0gzlJIqz81/fun8OtTA=\">AAAB/XicbVDJSgNBEK2JW4xbXG5eGoPgKcyIoMeAF48RskGWoabTkzTp6Rm6e4Q4BH/FiwdFvPof3vwbO8tBEx8UPN6r6q56QSK4Nq777eTW1jc2t/LbhZ3dvf2D4uFRQ8epoqxOYxGrVoCaCS5Z3XAjWCtRDKNAsGYwup36zQemNI9lzYwT1o1wIHnIKRor+cUT9L2e7PRjown6Wa0nJz0rl9yyOwNZJd6ClGCBql/8si/QNGLSUIFatz03Md0MleFUsEmhk2qWIB3hgLUtlRgx3c1m20/IuVX6JIyVLWnITP09kWGk9TgKbGeEZqiXvan4n9dOTXjTzbhMUsMknX8UpoKYmEyjIH2uGDVibAlSxe2uhA5RITU2sIINwVs+eZU0LsueW/bur0oVsogjD6dwBhfgwTVU4A6qUAcKj/AMr/DmPDkvzrvzMW/NOYuZY/gD5/MHJA2U6A==</latexit><latexit sha1_base64=\"fv2VJikq0gzlJIqz81/fun8OtTA=\">AAAB/XicbVDJSgNBEK2JW4xbXG5eGoPgKcyIoMeAF48RskGWoabTkzTp6Rm6e4Q4BH/FiwdFvPof3vwbO8tBEx8UPN6r6q56QSK4Nq777eTW1jc2t/LbhZ3dvf2D4uFRQ8epoqxOYxGrVoCaCS5Z3XAjWCtRDKNAsGYwup36zQemNI9lzYwT1o1wIHnIKRor+cUT9L2e7PRjown6Wa0nJz0rl9yyOwNZJd6ClGCBql/8si/QNGLSUIFatz03Md0MleFUsEmhk2qWIB3hgLUtlRgx3c1m20/IuVX6JIyVLWnITP09kWGk9TgKbGeEZqiXvan4n9dOTXjTzbhMUsMknX8UpoKYmEyjIH2uGDVibAlSxe2uhA5RITU2sIINwVs+eZU0LsueW/bur0oVsogjD6dwBhfgwTVU4A6qUAcKj/AMr/DmPDkvzrvzMW/NOYuZY/gD5/MHJA2U6A==</latexit>rn1...rnTn<latexit sha1_base64=\"mD7D/vope84WJ97ZseRjUi+SqsI=\">AAAB/XicbVDJSgNBEK2JW4xbXG5eGoPgKcyIoMeAF48RskGWoafTkzTp6Rm6a4Q4BH/FiwdFvPof3vwbO8tBEx8UPN6r6q56QSKFQdf9dnJr6xubW/ntws7u3v5B8fCoYeJUM15nsYx1K6CGS6F4HQVK3ko0p1EgeTMY3U795gPXRsSqhuOEdyM6UCIUjKKV/OKJ9r2e6vRjNET7Wa2nJj0rl9yyOwNZJd6ClGCBql/8si+wNOIKmaTGtD03wW5GNQom+aTQSQ1PKBvRAW9bqmjETTebbT8h51bpkzDWthSSmfp7IqORMeMosJ0RxaFZ9qbif147xfCmmwmVpMgVm38UppJgTKZRkL7QnKEcW0KZFnZXwoZUU4Y2sIINwVs+eZU0LsueW/bur0oVsogjD6dwBhfgwTVU4A6qUAcGj/AMr/DmPDkvzrvzMW/NOYuZY/gD5/MHWWCVCg==</latexit><latexit sha1_base64=\"mD7D/vope84WJ97ZseRjUi+SqsI=\">AAAB/XicbVDJSgNBEK2JW4xbXG5eGoPgKcyIoMeAF48RskGWoafTkzTp6Rm6a4Q4BH/FiwdFvPof3vwbO8tBEx8UPN6r6q56QSKFQdf9dnJr6xubW/ntws7u3v5B8fCoYeJUM15nsYx1K6CGS6F4HQVK3ko0p1EgeTMY3U795gPXRsSqhuOEdyM6UCIUjKKV/OKJ9r2e6vRjNET7Wa2nJj0rl9yyOwNZJd6ClGCBql/8si+wNOIKmaTGtD03wW5GNQom+aTQSQ1PKBvRAW9bqmjETTebbT8h51bpkzDWthSSmfp7IqORMeMosJ0RxaFZ9qbif147xfCmmwmVpMgVm38UppJgTKZRkL7QnKEcW0KZFnZXwoZUU4Y2sIINwVs+eZU0LsueW/bur0oVsogjD6dwBhfgwTVU4A6qUAcGj/AMr/DmPDkvzrvzMW/NOYuZY/gD5/MHWWCVCg==</latexit><latexit sha1_base64=\"mD7D/vope84WJ97ZseRjUi+SqsI=\">AAAB/XicbVDJSgNBEK2JW4xbXG5eGoPgKcyIoMeAF48RskGWoafTkzTp6Rm6a4Q4BH/FiwdFvPof3vwbO8tBEx8UPN6r6q56QSKFQdf9dnJr6xubW/ntws7u3v5B8fCoYeJUM15nsYx1K6CGS6F4HQVK3ko0p1EgeTMY3U795gPXRsSqhuOEdyM6UCIUjKKV/OKJ9r2e6vRjNET7Wa2nJj0rl9yyOwNZJd6ClGCBql/8si+wNOIKmaTGtD03wW5GNQom+aTQSQ1PKBvRAW9bqmjETTebbT8h51bpkzDWthSSmfp7IqORMeMosJ0RxaFZ9qbif147xfCmmwmVpMgVm38UppJgTKZRkL7QnKEcW0KZFnZXwoZUU4Y2sIINwVs+eZU0LsueW/bur0oVsogjD6dwBhfgwTVU4A6qUAcGj/AMr/DmPDkvzrvzMW/NOYuZY/gD5/MHWWCVCg==</latexit><latexit sha1_base64=\"mD7D/vope84WJ97ZseRjUi+SqsI=\">AAAB/XicbVDJSgNBEK2JW4xbXG5eGoPgKcyIoMeAF48RskGWoafTkzTp6Rm6a4Q4BH/FiwdFvPof3vwbO8tBEx8UPN6r6q56QSKFQdf9dnJr6xubW/ntws7u3v5B8fCoYeJUM15nsYx1K6CGS6F4HQVK3ko0p1EgeTMY3U795gPXRsSqhuOEdyM6UCIUjKKV/OKJ9r2e6vRjNET7Wa2nJj0rl9yyOwNZJd6ClGCBql/8si+wNOIKmaTGtD03wW5GNQom+aTQSQ1PKBvRAW9bqmjETTebbT8h51bpkzDWthSSmfp7IqORMeMosJ0RxaFZ9qbif147xfCmmwmVpMgVm38UppJgTKZRkL7QnKEcW0KZFnZXwoZUU4Y2sIINwVs+eZU0LsueW/bur0oVsogjD6dwBhfgwTVU4A6qUAcGj/AMr/DmPDkvzrvzMW/NOYuZY/gD5/MHWWCVCg==</latexit>rN1...rNTN<latexit sha1_base64=\"srr8sEw4ycMopJQiQ1Ny4wzUqY4=\">AAAB/XicbVDLSsNAFL2pr1pf8bFzM1gEVyURQZcFN65Khb6gTcNkOmmHTiZhZiLUUPwVNy4Ucet/uPNvnLZZaOuBC4dz7p259wQJZ0o7zrdVWFvf2Nwqbpd2dvf2D+zDo5aKU0lok8Q8lp0AK8qZoE3NNKedRFIcBZy2g/HtzG8/UKlYLBp6klAvwkPBQkawNpJvn0jf7dd6g1grJP2s0a9N+zXfLjsVZw60StyclCFH3be/zAskjajQhGOluq6TaC/DUjPC6bTUSxVNMBnjIe0aKnBElZfNt5+ic6MMUBhLU0Kjufp7IsORUpMoMJ0R1iO17M3E/7xuqsMbL2MiSTUVZPFRmHKkYzSLAg2YpETziSGYSGZ2RWSEJSbaBFYyIbjLJ6+S1mXFdSru/VW5ivI4inAKZ3ABLlxDFe6gDk0g8AjP8Apv1pP1Yr1bH4vWgpXPHMMfWJ8/xZGUqg==</latexit><latexit sha1_base64=\"srr8sEw4ycMopJQiQ1Ny4wzUqY4=\">AAAB/XicbVDLSsNAFL2pr1pf8bFzM1gEVyURQZcFN65Khb6gTcNkOmmHTiZhZiLUUPwVNy4Ucet/uPNvnLZZaOuBC4dz7p259wQJZ0o7zrdVWFvf2Nwqbpd2dvf2D+zDo5aKU0lok8Q8lp0AK8qZoE3NNKedRFIcBZy2g/HtzG8/UKlYLBp6klAvwkPBQkawNpJvn0jf7dd6g1grJP2s0a9N+zXfLjsVZw60StyclCFH3be/zAskjajQhGOluq6TaC/DUjPC6bTUSxVNMBnjIe0aKnBElZfNt5+ic6MMUBhLU0Kjufp7IsORUpMoMJ0R1iO17M3E/7xuqsMbL2MiSTUVZPFRmHKkYzSLAg2YpETziSGYSGZ2RWSEJSbaBFYyIbjLJ6+S1mXFdSru/VW5ivI4inAKZ3ABLlxDFe6gDk0g8AjP8Apv1pP1Yr1bH4vWgpXPHMMfWJ8/xZGUqg==</latexit><latexit sha1_base64=\"srr8sEw4ycMopJQiQ1Ny4wzUqY4=\">AAAB/XicbVDLSsNAFL2pr1pf8bFzM1gEVyURQZcFN65Khb6gTcNkOmmHTiZhZiLUUPwVNy4Ucet/uPNvnLZZaOuBC4dz7p259wQJZ0o7zrdVWFvf2Nwqbpd2dvf2D+zDo5aKU0lok8Q8lp0AK8qZoE3NNKedRFIcBZy2g/HtzG8/UKlYLBp6klAvwkPBQkawNpJvn0jf7dd6g1grJP2s0a9N+zXfLjsVZw60StyclCFH3be/zAskjajQhGOluq6TaC/DUjPC6bTUSxVNMBnjIe0aKnBElZfNt5+ic6MMUBhLU0Kjufp7IsORUpMoMJ0R1iO17M3E/7xuqsMbL2MiSTUVZPFRmHKkYzSLAg2YpETziSGYSGZ2RWSEJSbaBFYyIbjLJ6+S1mXFdSru/VW5ivI4inAKZ3ABLlxDFe6gDk0g8AjP8Apv1pP1Yr1bH4vWgpXPHMMfWJ8/xZGUqg==</latexit><latexit sha1_base64=\"srr8sEw4ycMopJQiQ1Ny4wzUqY4=\">AAAB/XicbVDLSsNAFL2pr1pf8bFzM1gEVyURQZcFN65Khb6gTcNkOmmHTiZhZiLUUPwVNy4Ucet/uPNvnLZZaOuBC4dz7p259wQJZ0o7zrdVWFvf2Nwqbpd2dvf2D+zDo5aKU0lok8Q8lp0AK8qZoE3NNKedRFIcBZy2g/HtzG8/UKlYLBp6klAvwkPBQkawNpJvn0jf7dd6g1grJP2s0a9N+zXfLjsVZw60StyclCFH3be/zAskjajQhGOluq6TaC/DUjPC6bTUSxVNMBnjIe0aKnBElZfNt5+ic6MMUBhLU0Kjufp7IsORUpMoMJ0R1iO17M3E/7xuqsMbL2MiSTUVZPFRmHKkYzSLAg2YpETziSGYSGZ2RWSEJSbaBFYyIbjLJ6+S1mXFdSru/VW5ivI4inAKZ3ABLlxDFe6gDk0g8AjP8Apv1pP1Yr1bH4vWgpXPHMMfWJ8/xZGUqg==</latexit>aN1...aNTN<latexit sha1_base64=\"XYyd4N2putHSsXz1S7O8C9/K6Po=\">AAAB/XicbVDLSsNAFL2pr1pf8bFzM1gEVyURQZcFN65Khb6gTcNkOmmHTiZhZiLUUPwVNy4Ucet/uPNvnLZZaOuBC4dz7p259wQJZ0o7zrdVWFvf2Nwqbpd2dvf2D+zDo5aKU0lok8Q8lp0AK8qZoE3NNKedRFIcBZy2g/HtzG8/UKlYLBp6klAvwkPBQkawNpJvn2Df7dd6g1grhP2s0a9N+zXfLjsVZw60StyclCFH3be/zAskjajQhGOluq6TaC/DUjPC6bTUSxVNMBnjIe0aKnBElZfNt5+ic6MMUBhLU0Kjufp7IsORUpMoMJ0R1iO17M3E/7xuqsMbL2MiSTUVZPFRmHKkYzSLAg2YpETziSGYSGZ2RWSEJSbaBFYyIbjLJ6+S1mXFdSru/VW5ivI4inAKZ3ABLlxDFe6gDk0g8AjP8Apv1pP1Yr1bH4vWgpXPHMMfWJ8/kD6UiA==</latexit><latexit sha1_base64=\"XYyd4N2putHSsXz1S7O8C9/K6Po=\">AAAB/XicbVDLSsNAFL2pr1pf8bFzM1gEVyURQZcFN65Khb6gTcNkOmmHTiZhZiLUUPwVNy4Ucet/uPNvnLZZaOuBC4dz7p259wQJZ0o7zrdVWFvf2Nwqbpd2dvf2D+zDo5aKU0lok8Q8lp0AK8qZoE3NNKedRFIcBZy2g/HtzG8/UKlYLBp6klAvwkPBQkawNpJvn2Df7dd6g1grhP2s0a9N+zXfLjsVZw60StyclCFH3be/zAskjajQhGOluq6TaC/DUjPC6bTUSxVNMBnjIe0aKnBElZfNt5+ic6MMUBhLU0Kjufp7IsORUpMoMJ0R1iO17M3E/7xuqsMbL2MiSTUVZPFRmHKkYzSLAg2YpETziSGYSGZ2RWSEJSbaBFYyIbjLJ6+S1mXFdSru/VW5ivI4inAKZ3ABLlxDFe6gDk0g8AjP8Apv1pP1Yr1bH4vWgpXPHMMfWJ8/kD6UiA==</latexit><latexit sha1_base64=\"XYyd4N2putHSsXz1S7O8C9/K6Po=\">AAAB/XicbVDLSsNAFL2pr1pf8bFzM1gEVyURQZcFN65Khb6gTcNkOmmHTiZhZiLUUPwVNy4Ucet/uPNvnLZZaOuBC4dz7p259wQJZ0o7zrdVWFvf2Nwqbpd2dvf2D+zDo5aKU0lok8Q8lp0AK8qZoE3NNKedRFIcBZy2g/HtzG8/UKlYLBp6klAvwkPBQkawNpJvn2Df7dd6g1grhP2s0a9N+zXfLjsVZw60StyclCFH3be/zAskjajQhGOluq6TaC/DUjPC6bTUSxVNMBnjIe0aKnBElZfNt5+ic6MMUBhLU0Kjufp7IsORUpMoMJ0R1iO17M3E/7xuqsMbL2MiSTUVZPFRmHKkYzSLAg2YpETziSGYSGZ2RWSEJSbaBFYyIbjLJ6+S1mXFdSru/VW5ivI4inAKZ3ABLlxDFe6gDk0g8AjP8Apv1pP1Yr1bH4vWgpXPHMMfWJ8/kD6UiA==</latexit><latexit sha1_base64=\"XYyd4N2putHSsXz1S7O8C9/K6Po=\">AAAB/XicbVDLSsNAFL2pr1pf8bFzM1gEVyURQZcFN65Khb6gTcNkOmmHTiZhZiLUUPwVNy4Ucet/uPNvnLZZaOuBC4dz7p259wQJZ0o7zrdVWFvf2Nwqbpd2dvf2D+zDo5aKU0lok8Q8lp0AK8qZoE3NNKedRFIcBZy2g/HtzG8/UKlYLBp6klAvwkPBQkawNpJvn2Df7dd6g1grhP2s0a9N+zXfLjsVZw60StyclCFH3be/zAskjajQhGOluq6TaC/DUjPC6bTUSxVNMBnjIe0aKnBElZfNt5+ic6MMUBhLU0Kjufp7IsORUpMoMJ0R1iO17M3E/7xuqsMbL2MiSTUVZPFRmHKkYzSLAg2YpETziSGYSGZ2RWSEJSbaBFYyIbjLJ6+S1mXFdSru/VW5ivI4inAKZ3ABLlxDFe6gDk0g8AjP8Apv1pP1Yr1bH4vWgpXPHMMfWJ8/kD6UiA==</latexit>a11...a1T1<latexit sha1_base64=\"FghIiq3JCriLrng5S7nJ2TEvEcg=\">AAAB/XicbVDLSgMxFM3UV62v8bFzEyyCqzIRQZcFNy4r9AXtdMikaRuayQzJHaEOxV9x40IRt/6HO//GtJ2Fth64cDjn3uTeEyZSGPC8b6ewtr6xuVXcLu3s7u0fuIdHTROnmvEGi2Ws2yE1XArFGyBA8naiOY1CyVvh+Hbmtx64NiJWdZgk3I/oUImBYBSsFLgnNCA90u3HYDANsnqPTHskcMtexZsDrxKSkzLKUQvcL/sCSyOugElqTId4CfgZ1SCY5NNSNzU8oWxMh7xjqaIRN342336Kz63Sx4NY21KA5+rviYxGxkyi0HZGFEZm2ZuJ/3mdFAY3fiZUkgJXbPHRIJUYYjyLAveF5gzkxBLKtLC7YjaimjKwgZVsCGT55FXSvKwQr0Lur8pVnMdRRKfoDF0ggq5RFd2hGmoghh7RM3pFb86T8+K8Ox+L1oKTzxyjP3A+fwAKWJQx</latexit><latexit sha1_base64=\"FghIiq3JCriLrng5S7nJ2TEvEcg=\">AAAB/XicbVDLSgMxFM3UV62v8bFzEyyCqzIRQZcFNy4r9AXtdMikaRuayQzJHaEOxV9x40IRt/6HO//GtJ2Fth64cDjn3uTeEyZSGPC8b6ewtr6xuVXcLu3s7u0fuIdHTROnmvEGi2Ws2yE1XArFGyBA8naiOY1CyVvh+Hbmtx64NiJWdZgk3I/oUImBYBSsFLgnNCA90u3HYDANsnqPTHskcMtexZsDrxKSkzLKUQvcL/sCSyOugElqTId4CfgZ1SCY5NNSNzU8oWxMh7xjqaIRN342336Kz63Sx4NY21KA5+rviYxGxkyi0HZGFEZm2ZuJ/3mdFAY3fiZUkgJXbPHRIJUYYjyLAveF5gzkxBLKtLC7YjaimjKwgZVsCGT55FXSvKwQr0Lur8pVnMdRRKfoDF0ggq5RFd2hGmoghh7RM3pFb86T8+K8Ox+L1oKTzxyjP3A+fwAKWJQx</latexit><latexit sha1_base64=\"FghIiq3JCriLrng5S7nJ2TEvEcg=\">AAAB/XicbVDLSgMxFM3UV62v8bFzEyyCqzIRQZcFNy4r9AXtdMikaRuayQzJHaEOxV9x40IRt/6HO//GtJ2Fth64cDjn3uTeEyZSGPC8b6ewtr6xuVXcLu3s7u0fuIdHTROnmvEGi2Ws2yE1XArFGyBA8naiOY1CyVvh+Hbmtx64NiJWdZgk3I/oUImBYBSsFLgnNCA90u3HYDANsnqPTHskcMtexZsDrxKSkzLKUQvcL/sCSyOugElqTId4CfgZ1SCY5NNSNzU8oWxMh7xjqaIRN342336Kz63Sx4NY21KA5+rviYxGxkyi0HZGFEZm2ZuJ/3mdFAY3fiZUkgJXbPHRIJUYYjyLAveF5gzkxBLKtLC7YjaimjKwgZVsCGT55FXSvKwQr0Lur8pVnMdRRKfoDF0ggq5RFd2hGmoghh7RM3pFb86T8+K8Ox+L1oKTzxyjP3A+fwAKWJQx</latexit><latexit sha1_base64=\"FghIiq3JCriLrng5S7nJ2TEvEcg=\">AAAB/XicbVDLSgMxFM3UV62v8bFzEyyCqzIRQZcFNy4r9AXtdMikaRuayQzJHaEOxV9x40IRt/6HO//GtJ2Fth64cDjn3uTeEyZSGPC8b6ewtr6xuVXcLu3s7u0fuIdHTROnmvEGi2Ws2yE1XArFGyBA8naiOY1CyVvh+Hbmtx64NiJWdZgk3I/oUImBYBSsFLgnNCA90u3HYDANsnqPTHskcMtexZsDrxKSkzLKUQvcL/sCSyOugElqTId4CfgZ1SCY5NNSNzU8oWxMh7xjqaIRN342336Kz63Sx4NY21KA5+rviYxGxkyi0HZGFEZm2ZuJ/3mdFAY3fiZUkgJXbPHRIJUYYjyLAveF5gzkxBLKtLC7YjaimjKwgZVsCGT55FXSvKwQr0Lur8pVnMdRRKfoDF0ggq5RFd2hGmoghh7RM3pFb86T8+K8Ox+L1oKTzxyjP3A+fwAKWJQx</latexit>r11...r1T1<latexit sha1_base64=\"j/oGrWBiFlsQnjD2pJ2IRSMS3x8=\">AAAB/XicbVDLSgMxFM3UV62v8bFzEyyCqzIRQZcFNy4r9AXtdMikaRuayQzJHaEOxV9x40IRt/6HO//GtJ2Fth64cDjn3uTeEyZSGPC8b6ewtr6xuVXcLu3s7u0fuIdHTROnmvEGi2Ws2yE1XArFGyBA8naiOY1CyVvh+Hbmtx64NiJWdZgk3I/oUImBYBSsFLgnOiA90u3HYLAOsnqPTHskcMtexZsDrxKSkzLKUQvcL/sCSyOugElqTId4CfgZ1SCY5NNSNzU8oWxMh7xjqaIRN342336Kz63Sx4NY21KA5+rviYxGxkyi0HZGFEZm2ZuJ/3mdFAY3fiZUkgJXbPHRIJUYYjyLAveF5gzkxBLKtLC7YjaimjKwgZVsCGT55FXSvKwQr0Lur8pVnMdRRKfoDF0ggq5RFd2hGmoghh7RM3pFb86T8+K8Ox+L1oKTzxyjP3A+fwA/q5RT</latexit><latexit sha1_base64=\"j/oGrWBiFlsQnjD2pJ2IRSMS3x8=\">AAAB/XicbVDLSgMxFM3UV62v8bFzEyyCqzIRQZcFNy4r9AXtdMikaRuayQzJHaEOxV9x40IRt/6HO//GtJ2Fth64cDjn3uTeEyZSGPC8b6ewtr6xuVXcLu3s7u0fuIdHTROnmvEGi2Ws2yE1XArFGyBA8naiOY1CyVvh+Hbmtx64NiJWdZgk3I/oUImBYBSsFLgnOiA90u3HYLAOsnqPTHskcMtexZsDrxKSkzLKUQvcL/sCSyOugElqTId4CfgZ1SCY5NNSNzU8oWxMh7xjqaIRN342336Kz63Sx4NY21KA5+rviYxGxkyi0HZGFEZm2ZuJ/3mdFAY3fiZUkgJXbPHRIJUYYjyLAveF5gzkxBLKtLC7YjaimjKwgZVsCGT55FXSvKwQr0Lur8pVnMdRRKfoDF0ggq5RFd2hGmoghh7RM3pFb86T8+K8Ox+L1oKTzxyjP3A+fwA/q5RT</latexit><latexit sha1_base64=\"j/oGrWBiFlsQnjD2pJ2IRSMS3x8=\">AAAB/XicbVDLSgMxFM3UV62v8bFzEyyCqzIRQZcFNy4r9AXtdMikaRuayQzJHaEOxV9x40IRt/6HO//GtJ2Fth64cDjn3uTeEyZSGPC8b6ewtr6xuVXcLu3s7u0fuIdHTROnmvEGi2Ws2yE1XArFGyBA8naiOY1CyVvh+Hbmtx64NiJWdZgk3I/oUImBYBSsFLgnOiA90u3HYLAOsnqPTHskcMtexZsDrxKSkzLKUQvcL/sCSyOugElqTId4CfgZ1SCY5NNSNzU8oWxMh7xjqaIRN342336Kz63Sx4NY21KA5+rviYxGxkyi0HZGFEZm2ZuJ/3mdFAY3fiZUkgJXbPHRIJUYYjyLAveF5gzkxBLKtLC7YjaimjKwgZVsCGT55FXSvKwQr0Lur8pVnMdRRKfoDF0ggq5RFd2hGmoghh7RM3pFb86T8+K8Ox+L1oKTzxyjP3A+fwA/q5RT</latexit><latexit sha1_base64=\"j/oGrWBiFlsQnjD2pJ2IRSMS3x8=\">AAAB/XicbVDLSgMxFM3UV62v8bFzEyyCqzIRQZcFNy4r9AXtdMikaRuayQzJHaEOxV9x40IRt/6HO//GtJ2Fth64cDjn3uTeEyZSGPC8b6ewtr6xuVXcLu3s7u0fuIdHTROnmvEGi2Ws2yE1XArFGyBA8naiOY1CyVvh+Hbmtx64NiJWdZgk3I/oUImBYBSsFLgnOiA90u3HYLAOsnqPTHskcMtexZsDrxKSkzLKUQvcL/sCSyOugElqTId4CfgZ1SCY5NNSNzU8oWxMh7xjqaIRN342336Kz63Sx4NY21KA5+rviYxGxkyi0HZGFEZm2ZuJ/3mdFAY3fiZUkgJXbPHRIJUYYjyLAveF5gzkxBLKtLC7YjaimjKwgZVsCGT55FXSvKwQr0Lur8pVnMdRRKfoDF0ggq5RFd2hGmoghh7RM3pFb86T8+K8Ox+L1oKTzxyjP3A+fwA/q5RT</latexit>LSEP<latexit sha1_base64=\"TpRXs97k1Lz7FKPLZo8p6PuHeNQ=\">AAACAXicbVDLSsNAFJ34rPUVdSO4GSyCq5KIoMuCCC5cVLQPaEKYTCft0MmDmRuxhLjxV9y4UMStf+HOv3HSZqGtBy4czrmXe+/xE8EVWNa3sbC4tLyyWlmrrm9sbm2bO7ttFaeSshaNRSy7PlFM8Ii1gINg3UQyEvqCdfzRReF37plUPI7uYJwwNySDiAecEtCSZ+47IYEhJSK7zr3MAfYA2e1lM889s2bVrQnwPLFLUkMlmp755fRjmoYsAiqIUj3bSsDNiAROBcurTqpYQuiIDFhP04iETLnZ5IMcH2mlj4NY6ooAT9TfExkJlRqHvu4s7lWzXiH+5/VSCM7djEdJCiyi00VBKjDEuIgD97lkFMRYE0Il17diOiSSUNChVXUI9uzL86R9Uretun1zWmvgMo4KOkCH6BjZ6Aw10BVqohai6BE9o1f0ZjwZL8a78TFtXTDKmT30B8bnD0Uyl0s=</latexit><latexit sha1_base64=\"TpRXs97k1Lz7FKPLZo8p6PuHeNQ=\">AAACAXicbVDLSsNAFJ34rPUVdSO4GSyCq5KIoMuCCC5cVLQPaEKYTCft0MmDmRuxhLjxV9y4UMStf+HOv3HSZqGtBy4czrmXe+/xE8EVWNa3sbC4tLyyWlmrrm9sbm2bO7ttFaeSshaNRSy7PlFM8Ii1gINg3UQyEvqCdfzRReF37plUPI7uYJwwNySDiAecEtCSZ+47IYEhJSK7zr3MAfYA2e1lM889s2bVrQnwPLFLUkMlmp755fRjmoYsAiqIUj3bSsDNiAROBcurTqpYQuiIDFhP04iETLnZ5IMcH2mlj4NY6ooAT9TfExkJlRqHvu4s7lWzXiH+5/VSCM7djEdJCiyi00VBKjDEuIgD97lkFMRYE0Il17diOiSSUNChVXUI9uzL86R9Uretun1zWmvgMo4KOkCH6BjZ6Aw10BVqohai6BE9o1f0ZjwZL8a78TFtXTDKmT30B8bnD0Uyl0s=</latexit><latexit sha1_base64=\"TpRXs97k1Lz7FKPLZo8p6PuHeNQ=\">AAACAXicbVDLSsNAFJ34rPUVdSO4GSyCq5KIoMuCCC5cVLQPaEKYTCft0MmDmRuxhLjxV9y4UMStf+HOv3HSZqGtBy4czrmXe+/xE8EVWNa3sbC4tLyyWlmrrm9sbm2bO7ttFaeSshaNRSy7PlFM8Ii1gINg3UQyEvqCdfzRReF37plUPI7uYJwwNySDiAecEtCSZ+47IYEhJSK7zr3MAfYA2e1lM889s2bVrQnwPLFLUkMlmp755fRjmoYsAiqIUj3bSsDNiAROBcurTqpYQuiIDFhP04iETLnZ5IMcH2mlj4NY6ooAT9TfExkJlRqHvu4s7lWzXiH+5/VSCM7djEdJCiyi00VBKjDEuIgD97lkFMRYE0Il17diOiSSUNChVXUI9uzL86R9Uretun1zWmvgMo4KOkCH6BjZ6Aw10BVqohai6BE9o1f0ZjwZL8a78TFtXTDKmT30B8bnD0Uyl0s=</latexit><latexit sha1_base64=\"TpRXs97k1Lz7FKPLZo8p6PuHeNQ=\">AAACAXicbVDLSsNAFJ34rPUVdSO4GSyCq5KIoMuCCC5cVLQPaEKYTCft0MmDmRuxhLjxV9y4UMStf+HOv3HSZqGtBy4czrmXe+/xE8EVWNa3sbC4tLyyWlmrrm9sbm2bO7ttFaeSshaNRSy7PlFM8Ii1gINg3UQyEvqCdfzRReF37plUPI7uYJwwNySDiAecEtCSZ+47IYEhJSK7zr3MAfYA2e1lM889s2bVrQnwPLFLUkMlmp755fRjmoYsAiqIUj3bSsDNiAROBcurTqpYQuiIDFhP04iETLnZ5IMcH2mlj4NY6ooAT9TfExkJlRqHvu4s7lWzXiH+5/VSCM7djEdJCiyi00VBKjDEuIgD97lkFMRYE0Il17diOiSSUNChVXUI9uzL86R9Uretun1zWmvgMo4KOkCH6BjZ6Aw10BVqohai6BE9o1f0ZjwZL8a78TFtXTDKmT30B8bnD0Uyl0s=</latexit>LRE<latexit sha1_base64=\"Fi+YFI4waJojXSQ7iRhXlzhGs64=\">AAACAHicbVDLSsNAFJ3UV62vqAsXbgaL4KokIuiyIIILF1XsA5oQJtNJO3TyYOZGLCEbf8WNC0Xc+hnu/BsnbRbaeuDC4Zx7ufcePxFcgWV9G5Wl5ZXVtep6bWNza3vH3N3rqDiVlLVpLGLZ84ligkesDRwE6yWSkdAXrOuPLwu/+8Ck4nF0D5OEuSEZRjzglICWPPPACQmMKBHZTe5lDrBHyO6u8twz61bDmgIvErskdVSi5ZlfziCmacgioIIo1betBNyMSOBUsLzmpIolhI7JkPU1jUjIlJtNH8jxsVYGOIilrgjwVP09kZFQqUno687iXDXvFeJ/Xj+F4MLNeJSkwCI6WxSkAkOMizTwgEtGQUw0IVRyfSumIyIJBZ1ZTYdgz7+8SDqnDdtq2Ldn9SYu46iiQ3SETpCNzlETXaMWaiOKcvSMXtGb8WS8GO/Gx6y1YpQz++gPjM8fnAWW8A==</latexit><latexit sha1_base64=\"Fi+YFI4waJojXSQ7iRhXlzhGs64=\">AAACAHicbVDLSsNAFJ3UV62vqAsXbgaL4KokIuiyIIILF1XsA5oQJtNJO3TyYOZGLCEbf8WNC0Xc+hnu/BsnbRbaeuDC4Zx7ufcePxFcgWV9G5Wl5ZXVtep6bWNza3vH3N3rqDiVlLVpLGLZ84ligkesDRwE6yWSkdAXrOuPLwu/+8Ck4nF0D5OEuSEZRjzglICWPPPACQmMKBHZTe5lDrBHyO6u8twz61bDmgIvErskdVSi5ZlfziCmacgioIIo1betBNyMSOBUsLzmpIolhI7JkPU1jUjIlJtNH8jxsVYGOIilrgjwVP09kZFQqUno687iXDXvFeJ/Xj+F4MLNeJSkwCI6WxSkAkOMizTwgEtGQUw0IVRyfSumIyIJBZ1ZTYdgz7+8SDqnDdtq2Ldn9SYu46iiQ3SETpCNzlETXaMWaiOKcvSMXtGb8WS8GO/Gx6y1YpQz++gPjM8fnAWW8A==</latexit><latexit sha1_base64=\"Fi+YFI4waJojXSQ7iRhXlzhGs64=\">AAACAHicbVDLSsNAFJ3UV62vqAsXbgaL4KokIuiyIIILF1XsA5oQJtNJO3TyYOZGLCEbf8WNC0Xc+hnu/BsnbRbaeuDC4Zx7ufcePxFcgWV9G5Wl5ZXVtep6bWNza3vH3N3rqDiVlLVpLGLZ84ligkesDRwE6yWSkdAXrOuPLwu/+8Ck4nF0D5OEuSEZRjzglICWPPPACQmMKBHZTe5lDrBHyO6u8twz61bDmgIvErskdVSi5ZlfziCmacgioIIo1betBNyMSOBUsLzmpIolhI7JkPU1jUjIlJtNH8jxsVYGOIilrgjwVP09kZFQqUno687iXDXvFeJ/Xj+F4MLNeJSkwCI6WxSkAkOMizTwgEtGQUw0IVRyfSumIyIJBZ1ZTYdgz7+8SDqnDdtq2Ldn9SYu46iiQ3SETpCNzlETXaMWaiOKcvSMXtGb8WS8GO/Gx6y1YpQz++gPjM8fnAWW8A==</latexit><latexit sha1_base64=\"Fi+YFI4waJojXSQ7iRhXlzhGs64=\">AAACAHicbVDLSsNAFJ3UV62vqAsXbgaL4KokIuiyIIILF1XsA5oQJtNJO3TyYOZGLCEbf8WNC0Xc+hnu/BsnbRbaeuDC4Zx7ufcePxFcgWV9G5Wl5ZXVtep6bWNza3vH3N3rqDiVlLVpLGLZ84ligkesDRwE6yWSkdAXrOuPLwu/+8Ck4nF0D5OEuSEZRjzglICWPPPACQmMKBHZTe5lDrBHyO6u8twz61bDmgIvErskdVSi5ZlfziCmacgioIIo1betBNyMSOBUsLzmpIolhI7JkPU1jUjIlJtNH8jxsVYGOIilrgjwVP09kZFQqUno687iXDXvFeJ/Xj+F4MLNeJSkwCI6WxSkAkOMizTwgEtGQUw0IVRyfSumIyIJBZ1ZTYdgz7+8SDqnDdtq2Ldn9SYu46iiQ3SETpCNzlETXaMWaiOKcvSMXtGb8WS8GO/Gx6y1YpQz++gPjM8fnAWW8A==</latexit>\f{1, . . . , N} the subject performs T n trials, Dn \u2261 {(an\nchosen from a set C \u2261 {Ck}K\n\nk=1, and reward rn\n\nt \u2208 (cid:60).\n\nt )}T n\n\nt , rn\n\nt=1, with action an\n\nt \u2208 C on trial t\n\nEncoder and decoder. We treat decision-making as a (partly stochastic) dynamical system that\nmaps past experiences as input into outputs in the form of actions. As such, the dataset D may be\nconsidered to contain samples from the output of potentially N different dynamical systems. The aim\nis then to turn each sequence Dn into a vector, zn, in a latent space, in such a way that zn captures\nthe characteristic properties of the corresponding dynamical system (whilst avoiding over-\ufb01tting the\nparticular choices). In this respect, the task is to \ufb01nd a low-dimensional representation for each of the\ndynamical systems in an unsupervised manner. We take an autoencoder-inspired model to achieve\nthis, in which an encoder network is trained to process an entire input sequence into a vector in the\nlatent space. A decoder network then takes as input this vector and recovers an approximation to the\noriginal dynamical system. Along with additional factors that we describe below, the model is trained\nby minimising a reconstruction loss which measures how well the generated dynamical system can\npredict the observed sequence of actions. The latent representation is thus supposed to capture the\n\u201cessence\u201d of the input sequence. This is because the latent space is low-dimensional compared to the\noriginal sequence and acts as an information bottleneck; as such the encoder has to learn to encode\nthe most informative aspects of each sequence.\nThe model architecture is presented in Figure 1. The \ufb01rst part is an encoder RNN (shown in red), and\nis responsible for extracting the characteristic properties of the input sequence. It takes the whole\nsequence as input and outputs its terminal state. This state is mapped into the latent space through a\nseries of fully-connected feed-forward layers. The encoder is:\n\nM\u00d71 \u2261 enc(an\nzn\n\n1...T , rn\n\n(1)\nin which \u0398enc are the weights of the RNN and feed-forward layers, and M is the dimensionality of\nthe latent space (see Supplementary Material for the details of the architecture).\nThe second part of the model is a feed-forward decoder network (shown in blue) with weights \u0398dec,\nwhich takes the latent representation as input, and outputs a vector \u03a6n,\n\n1...T ; \u0398enc), n = 1 . . . N,\n\n\u03a6n \u2261 dec(zn; \u0398dec).\n\n(2)\nAs in a hyper-network, vector \u03a6n contains the weights of a second RNN called the learning network,\nitself inspired by [e.g., Dezfouli et al., 2019, 2018], and described below. The learning network apes\nthe process of decision-making, taking past actions and rewards as input, and returning predictions of\nthe probability of the next action. Making \u03a6n such that the learning network reconstructs the original\nsequence is what forces zn to encode the characteristics of each subject.\n\nLearning network. The learning network is based on the Gated Recurrent Unit architecture [GRU;\nCho et al., 2014] with Nc cells. This realizes a function f n, which at time-step t maps the previous\nstate xn\n\nt\u22121, into a next state,\n\nt\u22121, action an\n\nt\u22121 and reward rn\nt \u2261 f n(an\nxn\n\nt\u22121, rn\n\nt\u22121, xn\n\nt\u22121; \u03a6n),\n\n(3)\n\n(4)\n\nwith predictions for the probabilities of actions arising from a weight matrix W n \u2208 (cid:60)K\u00d7Nc\n\nt \u2261 W nxn\nvn\nt ,\n\nt (at \u2261 Ci; an\n\u03c0n\n\n1...t\u22121, rn\n\n1...t\u22121) =\n\n(cid:80)\n\nevn\n\nt [i]\n\nk=1...K evn\n\nt [k] ,\n\nwhere vn\nt represent the logit scores for each action (unnormalised probabilities), and \u03c0t(.) are the\nprobabilities of taking each action at time t. vn\nt [i] represents the ith element of vn\nt . For Nc GRU cells\nand K actions, function f n requires J = 3N 2\nc + 3KNc + 6Nc parameters. \u03a6n consists of these plus\nthe KNc parameters of W n. Note that this RNN, which models humans decision-making processes,\nshould not be confused with the RNN in the encoder, which extracts and maps each input sequence\ninto the latent space.\nSummary. The encoder network takes an input sequence and maps it into its latent representation.\nThe decoder network then takes the latent representation and generates the weights of an RNN that is\nable to predict the actions taken in the input sequence. The next section describes in detail how the\nnetwork weights \u0398enc and \u0398dec are learned end-to-end.\n\n3\n\n\f3 Training objective\n\nThe training loss function has three components: (i) a reconstruction loss which penalizes discrepan-\ncies between the predicted and actual input sequence, (ii) a group-level disentanglement loss which\nencourages sequences to spread independently across the dimensions of the input sequence, (iii) a\nseparation loss which favors dimensions of the latent space that have separate effects on the behavior\ngenerated by the learning networks.\nReconstruction loss. Each sequence Dn in D is passed through the encoder to obtain the cor-\nresponding latent representation. The latent representation is then passed through the decoder to\ngenerate the weight vector \u03a6n of the learning network. We then assess how well the n-th learn-\ning network can predict the actions taken in Dn. The total reconstruction loss is based on the\nnegative-log-likelihood:\n\nlog \u03c0n\n\nt (an\n\nt ; an\n\n1...t\u22121, rn\n\n1...t\u22121).\n\n(5)\n\nN(cid:88)\n\nT n(cid:88)\n\nn=1\n\nt=1\n\nLRE = \u2212 1\nN\n\nDisentanglement loss.\nThe second term in the loss function favors disentangled representations\nby seeking to ensure that each dimension of the latent space corresponds to an independent factor of\ncontribution in the variation across input sequences. This is achieved by maximising the similarity\nbetween the empirical encoding distribution (i.e., {zn}) and a prior distribution p(z), taken to\nbe isotropic Gaussian (see the Supplementary Material for details). Maximising this similarity\nencourages the dimensions of the latent representation to be independent across input sequences.\nDe\ufb01ne \u02c6q(z) as the empirical distribution of zn, and g(z) as a Gaussian with the same mean and\ncovariance as that of \u02c6q(z). We combine two popular measures of the dissimilarity between the\nencoding distribution and p(z):\n\nLDIS = \u03bb1MMD(\u02c6q(z), p(z)) + KL(g(z)(cid:107)p(z)),\n\n(6)\nwhere MMD is the Maximum Mean Discrepancy [Tolstikhin et al., 2017] and KL is the Kullback-\nLeibler divergence (the Gaussian approximation makes the KL computations tractable and robust) and\n\u03bb1 is a hyper-parameter. We expected the MMD term to suf\ufb01ce; however, we found that combining\nthe two yielded better results.\n\nSeparation loss. Crudely speaking, the disentanglement loss focuses on regularising the encoder;\nhowever, to make the latent representation behaviorally interpretable, it is also necessary to regularize\nthe decoder. Gaining insight into the effect of one dimension on the behavioral predictions of the\nlearning network is hard if this is affected by the other dimensions. Therefore, we introduce an\nadditional separation loss designed to discourage interactions.\nFor simplicity, assume the decision-making task involves only two actions, C1 and C2, and that the\nlatent space has two dimensions, z1 and z2 (M = 2; see Supplementary Material for the general\nt (C2) are the logits corresponding to the probability of taking\ncase). As noted before vn\naction C1 and C2 at trial t for input sequence n. Denote by un\nt the relative logits of the actions,\nun\nt = vn\n\nt changes by changing the \ufb01rst dimension, z1 is\n\nt (C2). The amount that un\n\nt (C1) \u2212 vn\n\nt (C1) and vn\n\nIdeally, the effect of changing z1 on behavior will be independent from the effect of z2 on behavior,\nwhich will allow us later to interpret z1 independently from z2. We capture the amount of interaction\n(inseparability) between the effect of z1 and z2 on changing un\n\nt as\n\nt\n\u2202z1\n\n(cid:12)(cid:12)(cid:12)(cid:12) \u2202un\n(cid:12)(cid:12)(cid:12)(cid:12) .\n(cid:12)(cid:12)(cid:12)(cid:12) \u22022un\nT n(cid:88)\nN(cid:88)\n\nt\n\n\u2202z1\u2202z2\n\nn=1\n\nt=1\n\n4\n\n(cid:12)(cid:12)(cid:12)(cid:12) .\n(cid:12)(cid:12)(cid:12)(cid:12) \u22022un\n\nt\n\n\u2202z1\u2202z2\n\n(cid:12)(cid:12)(cid:12)(cid:12) .\n\ndn\nt =\n\n\u02c6LSEP =\n\n1\nN\n\n(7)\n\n(8)\n\n(9)\n\nThis would be zero if the relative logit comprises additively separable functions, i.e., un\ng2(z2) for two functions g1 and g2. Even without this, minimizing dn\nhave a separate effect on behavior. We therefore consider the following loss term,\n\nt = g1(z1) +\nt can help make each dimension\n\n\fThe calculation of the above term is computationally intensive as it requires computing the second-\norder derivative for each time step and sequence pair (O(N T n) second-order derivative calculations).\nInstead, we use the following approximation,\n\n(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) T n(cid:88)\n\nt=1\n\nN(cid:88)\n\nn=1\n\n\u02c6LSEP \u2248 LSEP =\n\n1\nN\n\n(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) =\n\n(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)\n\nN(cid:88)\n\nn=1\n\n\u22022un\nt\n\u2202z1\u2202z2\n\n1\nN\n\nT n(cid:88)\n\nun\nt\n\n(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ,\n\n\u22022\n\n\u2202z1\u2202z2\n\nt=1\n\n(10)\n\nwhich can be calculated more ef\ufb01ciently (O(N ) second-order derivative calculations)1. We note that\nthe loss is de\ufb01ned using logits instead of the probabilities, since probability predictions are bounded\nand cannot be separated as \u03c0n\nWe now de\ufb01ne the combined loss function as follows,\n\nt (C2) = g1(z1) + g2(z2) in the general case.\n\nt (C1) \u2212 \u03c0n\n\nL = LRE + \u03bb2LDIS + \u03bb3LSEP,\n\n(11)\n\nin which \u03bb2 and \u03bb3 are hyper-parameter.\nThe model parameters \u0398enc and \u0398dec were trained based on the above objective function and using\ngradient descent optimisation method [Kingma and Ba, 2014]. See Supplementary Material for\ndetails.\n\n4 Results\n\nSYNTHETIC data. To illustrate that our method can learn the underlying ground truth dynamical\nsystem, we generated Q\u2212learning agents [Watkins, 1989] with various parameter values and simu-\nlated their behaviour on a bandit task involving two stochastically-rewarded actions, C1 and C2. The\nactions of the agents and the rewards they received comprise the dataset D. The Q\u2212learning agent\nwas speci\ufb01ed by two parameters, values for which were drawn randomly (see Supplementary Material\nfor more details). One parameter is the inverse temperature or reward sensitivity, \u03b2, which determines\nthe propensity to explore, equivalently controlling the impact of receiving a reward from an action on\nrepeating that action in future trials. The other parameter is \u03ba, which determines the tendency of an\nagent to repeat the last taken action in the next trial irrespective of whether it was rewarded [e.g., Lau\nand Glimcher, 2005]. Values \u03ba > 0 favor repeating an action in next trial (perseveration) and values\n\u03ba < 0 favor switching between the actions. We generated N = 1500 agents (saving 30% for testing).\nThe test data was used for determining the optimal number of training iterations (early stopping).\nEach agent selected 150 actions (T n = 150).\nWe trained our model using the data (see Figure S3 for the trajectory of the loss function and Figure S4\nfor the distribution of the latent representations). As shown in Figure 2(a), and as intended, these\nrepresentations turned out to have an interpretable relationship with the parameters that actually\ndetermined behavior: the exploration parameter \u03b2 is mainly related to z2 and the perseveration\nparameter \u03ba to z1. This outcome arose because of the LSEP term in the loss function. A similar graph\nbefore introducing the LSEP is shown in Figure S1(a), which shows that without this term, z1 and z2\nhave mixed relationships with \u03ba and \u03b2.\nWe then sought to interpret each dimension of the latent space in behavioral terms. To do this,\nwe used the decoder to generate learning networks corresponding to different values of the latent\nrepresentations and interpreted these dimensions by analysing the behaviour of the simulated networks\non a \ufb01xed input sequence. In the \ufb01xed input sequence, the agent always performs action C1 (see\nFigure S5 for the simulations using both C1 and C2 as the previous action), and receives just one\nnon-zero reward at the trial marked by the red vertical line. Using the same sequence for all networks\nrequires running simulations off-policy (i.e., the network predicts the next choice, but does not execute\nit). This setting provides the same input to the model in all conditions, allowing us to diagnose exactly\nwhat affects the output of the model. The simulations are shown in Figure 2(b). Each panel in the\n\ufb01gure shows the simulation of the generated network for 10 trials.\nIn Figure 2(b)-left panel, z2 is \ufb01xed at zero as z1 is allowed to vary. As the \ufb01gure shows, by changing\nthe z1 dimension, the probability of taking the same action (in this case C1) in the next trial is affected,\ni.e., high values of z1 are associated with the high probability of perseveration, and in the low values\nof z1 there is a high chance of switching to the other action in the next trial. This is consistent with\n\n1We use the automatic differentiation in Tensor\ufb02ow [Abadi et al., 2016].\n\n5\n\n\fFigure 2: SYNTHETIC data. (a) Relationship between the dimensions of the latent representations (z1,\nz2) and the parameters used to generate the data (\u03ba and \u03b2). z1 captures the perseveration parameter\n(\u03ba) and z2 captures the rate of exploration (\u03b2). The smoothed black lines were calculated using\nmethod \u2018gam\u2018 in R [Wood, 2011] and the shaded area represents con\ufb01dence intervals. (b) Off-policy\nsimulations of the model for different values of z1 (left-panel; z2 = 0) and z2 (right-panel; z1 = 0).\nThe plots show the probability of selecting C1 in each trial when C1 had actually been chosen on all\nthe previous trials. A single reward is provided, shown by the vertical red line. (c) Model simulations\nsimilar to the ones in panel (b) but using the actual Q\u2212learning model. In the left panel \u03b2 = 3 and in\nthe right panel \u03ba = 0.\n\nthe correspondence between z1 and the perseveration parameter \u03ba. Note that after receiving the\nreward, the probability of taking C1 increases (since C1 was the previous action and it was rewarded),\nhowever, the size of the increment is not affected by changes in z1, i.e., z1 is controlling perseveration\nbut not sensitivity of behaviour to reward.\nIn contrast in Figure 2(b)-right panel, z1 is \ufb01xed at zero as z2 is in turn allowed to vary. As the\npanels show, by changing z2 the probability of repeating an action is not affected, but the sensitivity\nof action probabilities to the reward is affected. That is, at the higher values of z2 there is a high\nprobability of repeating the rewarded action (C1 in this case). Therefore, z1 and z2 have separate\neffects on behaviour corresponding to the perseveration and reward sensitivity. This separation is a\nconsequence of introducing the LSEP term. In Figure S1(b) we show the same simulations but without\nintroducing the LSEP into the loss function, which shows that the effects of z1 and z2 on behaviour\nare not separated (see Figure S2 for how the behaviour of model changes during training). Figure 2(c)\nshows the same simulations as panel (b) but using the original Q-learning model which was used to\ngenerate the data (for different values of \u03b2 and \u03ba). As the \ufb01gure shows the model closely re\ufb02ects the\nbehaviour of Q\u2212learning model, as expected.\n\nBD dataset. This dataset [Dezfouli et al., 2019] comprises behavioural data from 34 patients with\ndepression, 33 with bipolar disorder and 34 matched healthy controls. As in the synthetic data above,\nsubjects performed a bandit task with two stochastically-rewarded actions (C1 and C2) (Figure 3(a)).\nEach subject completed the task 12 times using different reward probabilities for each action. The\ndataset thus contains N = 12 (sequences) \u00d7101 (participants) = 1212, which we used for training\nthe model. Out of the 12 sequences of each subject, 8 were used for training and 4 for testing\nto determine the optimal number of training iterations (see Figure S7 for the training curves and\nSupplementary Material for more details).\nWe considered a two-dimensional latent space z ={z1, z2}; the resulting coordinates for all sequences\nare shown in Figure 3(b). We made two predictions about z: \ufb01rst, we expected that the latent\nrepresentations for the sequences for a single subject should be mutually similar as they come from\nthe same decision-making system. We therefore compared the mean pairwise distances separating\n\n6\n\nrewardrewardc) Q-learning b) Modela)porabiblity of C10.00.51.0trial\u22121.00\u22120.50+0.00+0.50+1.00\u03ba\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u2212202\u2212202z1\u03ba\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019051015\u2212202z1\u03b2\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019051015\u22122024z2\u03b2\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u0019\u2212202\u22122024z2\u03baporabiblity of C10.00.51.0\u22121.00\u22120.50+0.00+0.50+1.00z1\u03b2+0.00+2.50+5.00+7.50trial\u22121.00\u22120.50+0.00+0.50+1.00z2\fthe latent representations within and between subjects (see Supplementary Material for details).\nFigure 4(a) shows that this prediction is indeed correct (p < 0.001 using Wilcoxon rank sum test).\nSecond, we expected that the decision-making differences between the three groups to be re\ufb02ected\nin their latent representations. Figure 4(b) shows the mean latent representations for the subjects\norganized by group. Although the groups differ along the z2 dimension (p < 0.001 comparing bipolar\nand healthy groups along z2 and using independent t-test, and p < 0.05 comparing depression and\nhealthy groups); the z1 dimension is evidently capturing variations that are unrelated to the existing\ndiagnostic categories (p > 0.1).\nThese results also highlight the substantial heterogeneity of the population. Some bipolar patients with\nhigh z2 are apparently more behaviorally similar to average depressed patients or healthy controls\nthan to the average bipolar patients. We would not have been able to extract this information by \ufb01tting\na single RNN to each whole group (as was done in previous work).\nWe then followed the same scheme as for the synthetic data to provide an interpretation for the\ndimensions of latent space. The results of off-policy simulations are shown in Figure 4(c) (see\nFigure S6 for simulations using both actions). First, both left and right panels show that, after\nreceiving the reward from an action, subjects show a tendency to switch to the other action. This\nis inconsistent with predictions of conventional Q\u2212learning, but was also found to be evident in\nmodel agnostic analyses [Dezfouli et al., 2019] (where it is also reconciled with gaining rather\nthan losing reward). The current model is able to capture this since it uses RNNs for representing\ndecision-making processes, which do not make such an assumption.\nSecond, as Figure 4(c)-left shows, the z1 dimension \u2013 which is not different between the groups \u2013\nis mostly related to reward sensitivity, as it controls (albeit rather weakly) the action probabilities\nafter receiving the reward. On the other hand, the z2 dimension \u2013 which is signi\ufb01cantly different\nbetween the groups \u2013 is more involved in perseveration/switching behaviour between the actions, i.e.,\nit controls the probability of staying on the same action. For low z2, the probability of repeating the\nprevious taken (in this case C1) is below 0.5, implying that the subjects in this part of the latent space\ntend to switch. In order to con\ufb01rm this, we simulated the model on-policy \u2013 in which the actions are\nselected by the model \u2013 for different values of z1 and z2. As the results in Figure 4(d) show, for low\nz2, the model indeed oscillates between the two actions. The z2 dimension is signi\ufb01cantly lower in\nthe bipolar group than healthy controls, which is consistent with the previous report indicating an\noscillatory behavioural characteristic in this group [Dezfouli et al., 2019].\nFinally, panel (c)-right shows that reward sensitivity and perseveration are not completely independent,\ni.e., when z2 is low, favoring switching, the effect of reward on probabilities is also more signi\ufb01cant,\nimplying that these two traits covary.\nIn summary, the model was able capture behavioural properties which are not consistent with cognitive\nmodels such as Q\u2212learning. It was also able to capture individual differences which could not be\nextracted by \ufb01tting a single RNN to the whole group.\n\n(a)\n\n(b)\n\nFigure 3: BD dataset. (a) The decision-making task. On each trial, subjects pressed a left (C1) or right\n(C2) key and had a chance of getting a reward (M&M chocolate or BBQ shapes). The task lasted for\n40 seconds and the responses were self-paced. Each participant completed the task 12 times with a\n12-second delay between them and different probabilities of reward. (b) Distribution of z values for\neach group. Each dot represents an input sequence.\n\n7\n\n40 seconds (free-responses)M&M=0Shapes=0M&M=1Shapes=0wassmall(thatis,atintermediatevaluesofDnearorequaltozero),wecomparedthepredictivevalueofPandDoverchoiceatdifferentlevelsofPandDinalogisticregression.Figure2bshowswewereablesuccessfullytoidentifyconditionsunderwhichPandDaredifferentiated:atsmalldifferencesinactionvalues(themiddletertileofDvalues),Pwasasigni\ufb01cantpredictor,whereasDwasnot.Conversely,Fig.2cshowsthatPandDweresigni\ufb01cantpredictorsacrossalltertilesofPvalues(pso0.001).Thisresultcon\ufb01rmsthatwhenchoicesweremadeinthepresenceofsmalldifferencesinactionvalue,Pvaluesbetterdiscriminatedthebestaction.Dorsolateralprefrontalcortextrackstherelativeadvantage.Toidentifytheneuralregionsinvolvedinthecomputationoftherelativeadvantagevaluesthatguidedchoice,wede\ufb01nedastickfunctionforeachresponseandparametricallymodulatedthisbyPinaresponse-by-responsefashionforeachparticipant.Asweusedafree-responsetaskandtheintervalbetweenchoiceswasnotsystematicallyjittered,wecannotdeterminewhetherthemodelvariableshadseparateeffectsatthetimeofeachchoice(orbetweenchoiceandfeedback).Wecanonlydeterminewhetherneuralactivitywasrelatedtothetimecourseofthemodelvari-ablesacrossthe40-sblockassubjectstriedtolearnthebestaction(forexample,Fig.2a).AnSPMone-samplet-testwiththeparametricregressorrepresentingPrevealedneuralactivitypositivelyrelatedtoPinasinglelargeclusterintherightmiddlefrontalgyrus,withthemajorityofvoxelsoverlappingBA9(dlPFC22,23;peakvoxel:44,25,37;t\u00bc5.98,family-wisecluster(FWEc)P\u00bc0.012).Figure2ashowsthecorticalregionswheretheBOLDresponsecovariedwiththePvaluesofeachresponse,implicatingtheseregionsinencodingtherelativelikelihoodthattheleftactionisbest(QLeft4QRight).M&Ms = 0Shapes = 0012340 s (free responses)12 sRatingResponse rateCausal rating10.07.55.02.50.0Responses s\u201310.250.080.1250.250.080.125M&Ms = 1Shapes = 00102345678910VeryunlikelyVerylikelyHow likely was it pressing the rightbutton earnt you an M&M?Figure1|Experimentalstimuli,behaviouralchoicesandcausalratings.(a)Beforethechoice,nostimuliindicatedwhichbuttonwasmorelikelytoleadtoreward.Whentheparticipantmadeachoice,thebuttonchosenwashighlighted(green)andonrewardedtrialstherewardstimuluswaspresentedfor1,000msduration.Aftereachblockoftrials,theparticipantratedhowcausaleachbuttonwas.(b)Meanresponserate(responsespersecond)washigherforthehigh-contingencyaction(blue)overlow-contingencyaction(red)ineachcondition.(c)Causalratingswerehigherforthehigh-contingencyaction(blue)overlow-contingencyaction(red)ineachcondition.Responserateandcausalratingsigni\ufb01cantlyvariedwithcontingency,Po0.001.Verticalbarsrepresents.e.m.Table1|Modelcomparisonsbetweenthehybridmodelanditsspecialcases.HybridQ-learningRelativeadvantageNegativeloglikelihood542155065558AggregateLRTfavouringhybrid\u2014X240\u00bc170***X220\u00bc274***No.offavouringhybrids\u2014138PseudoR20.6080.6020.597Shownforeachmodel:negativeloglikelihood;teststatisticandP-valueforalikelihoodratiotestagainstthehybrid(full)model,aggregatedacrosssubjects;thenumberofsubjectsfavoringthehybridmodelonalikelihoodratiotest(Po0.05);andthedegreetowhichthemodelexplainedthechoicedataaveragedovertheindividual\ufb01ts(pseudoR2).***Po1E-16.NATURECOMMUNICATIONS|DOI:10.1038/ncomms5390ARTICLENATURECOMMUNICATIONS|5:4390|DOI:10.1038/ncomms5390|www.nature.com/naturecommunications3&2014MacmillanPublishersLimited.Allrightsreserved.llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll\u22122\u22121012\u2212202z1z2lllHealthyDepressionBipolar\fFigure 4: BD dataset. (a) The distances between the latent representations for single subjects (Within)\nand between the subjects (Between). Each dot represents a subject; the bars show the means and the\nerror-bars, 1SEM. (b) The mean of the latent representations for each subject across the dimensions\nof the latent space. Each dot represents a subject and bars and error-bars show means and 1SEM. (c)\nOff-policy simulations of the model for different values of z1 (left-panel; z2 = 0) and z2 (right-panel;\nz1 = 0). The plots show the probability of selecting C1 in each trial when C1 had actually been\nchosen on all the previous trials. A single reward is provided, shown by the vertical red line. (d)\nOn-policy simulations of the model. The actions are selected by the model based on which action has\nthe higher probability of being taken (the \ufb01rst action was set to C1).\n\n5 Related work\n\nThere is a wealth of work using RNNs as models of decision-making, for unsupervised dimension\nreduction of dynamical systems, and for sequence-to-sequence mappings. For the \ufb01rst, a key focus\nhas been learning-to-learn [e.g., Wang et al., 2016, Song et al., 2017] \u2013 i.e., creating RNNs that can\nthemselves learn to solve a battery of tasks. However, these generative models have not been coupled\nwith recognition, for instance to capture individual differences. Techniques based on autoencoders\nhave explored low-dimensional latent spaces for modelling neural activity trajectories in the brain\n[e.g., Pandarinath et al., 2018]. However, like in\ufb02uential sequence-sequence models for translation\n[Bowman et al., 2015], these focus on building an open loop account for the state of the brain within\na trajectory, or the state characterizing a particular input sentence, whereas we focus on the trait\ncharacteristic of a closed-loop controller that has to interact with an external environment itself [see\nalso Fox et al., 2011, Johnson et al., 2016, as examples of other approaches]. Our combination of an\nautoencoder and hyper-network [Ha et al., 2016, Karaletsos et al., 2018] is, to our knowledge, novel,\nand might \ufb01nd other applications in the analysis of other dynamical systems such as time-series data.\n\n6 Conclusion and discussion\n\nWe proposed a \ufb02exible autoencoder-based framework for modelling individual differences in decision-\nmaking tasks. The autoencoder maps the sequence of actions and rewards taken and received by a\nsubject to a position in a low-dimensional latent space which is decoded to determine the weights of\na \u2018learning network\u2019 RNN that characterizes how that subject behaves as a plastic decision-maker.\nThe latent space was disentangled by adding a speci\ufb01c component to the loss function.\nThe model can be extended in various directions. One is to pin down (part of) the learning network as\na conventional, parameterized, RL agent, to facilitate further interpretation of the latent dimensions.\nAnother is to include tasks with non-trivial (Markovian or non-Markovian) state information partially\nsignalled by stimuli.\n\n8\n\nrewardd) On-policy c) Off-policya)b)0.00.51.0porabiblity of C1\u22121.00\u22120.50+0.00+0.50+1.00z1reward\u22121.00\u22120.50+0.00+0.50+1.00z2porabiblity of C10.00.51.0\u22121.00\u22120.50+0.00+0.50+1.00z1\u22121.00\u22120.50+0.00+0.50+1.00z2z1z2HealthyDepressionBipolarHealthyDepressionBipolar\u22122\u221210120123BetweenWithindistancetrialtrial\fAcknowledgments\n\nWe are grateful to Bernard W. Balleine for sharing BD dataset with us.\n\nReferences\nJohn B Carroll and Scott E Maxwell. Individual differences in cognitive abilities. Annual review of\n\npsychology, 30(1):603\u2013640, 1979.\n\nMichael J Frank, Bradley B Doll, Jen Oas-Terpstra, and Francisco Moreno. Prefrontal and stri-\natal dopaminergic genes predict individual differences in exploration and exploitation. Nature\nneuroscience, 12(8):1062, 2009.\n\nHanneke E M den Ouden, Nathaniel D Daw, Guill\u00e9n Fernandez, Joris A Elshout, Mark Rijpkema,\nMartine Hoogman, Barbara Franke, and Roshan Cools. Dissociable effects of dopamine and\nserotonin on reversal learning. Neuron, 80(4):1090\u20131100, 2013.\n\nDaniel J Navarro, Thomas L Grif\ufb01ths, Mark Steyvers, and Michael D Lee. Modeling individual\ndifferences using Dirichlet processes. Journal of mathematical Psychology, 50(2):101\u2013122, 2006.\n\nJerome R Busemeyer and Julie C Stout. A contribution of cognitive decision models to clinical\nassessment: decomposing performance on the Bechara gambling task. Psychological assessment,\n14(3):253, 2002.\n\nNathaniel D Daw. Trial-by-trial data analysis using computational models. In Mauricio R. Delgado,\nElizabeth A. Phelps, and Trevor W. Robbins, editors, Decision Making, Affect, and Learning.\nOxford University Press, 2011.\n\nEldad Yechiam, Jerome R Busemeyer, Julie C Stout, and Antoine Bechara. Using cognitive models\nto map relations between neuropsychological disorders and human decision-making de\ufb01cits.\nPsychological science, 16(12):973\u20138, dec 2005.\n\nAmir Dezfouli, Kristi Grif\ufb01ths, Fabio Ramos, Peter Dayan, and Bernard W Balleine. Models that\nlearn how humans learn: the case of decision-making and its disorders. PLoS computational\nbiology, 15(6):e1006903, 2019.\n\nAmir Dezfouli, Richard W Morris, Fabio Ramos, Peter Dayan, and Benrard W Balleine. Integrated\naccounts of behavioral and neuroimaging data using \ufb02exible recurrent neural network models. In\nAdvances in Neural Information Processing Systems (Neurips), 2018.\n\nGuangyu Robert Yang, Madhura R Joglekar, H Francis Song, William T Newsome, and Xiao-Jing\nWang. Task representations in neural networks trained to perform many cognitive tasks. Nature\nNeuroscience, 22(2):297\u2013306, 2019.\n\nHava T Siegelmann and Eduardo D Sontag. On the computational power of neural nets. Journal of\n\ncomputer and system sciences, 50(1):132\u2013150, 1995.\n\nDavid E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by\nerror propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science,\n1985.\n\nIlya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders.\n\narXiv preprint arXiv:1711.01558, 2017.\n\nDavid Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv:1609.09106, 2016.\n\nTheofanis Karaletsos, Peter Dayan, and Zoubin Ghahramani. Probabilistic meta-representations of\n\nneural networks. arXiv preprint arXiv:1810.00555, 2018.\n\nKyunghyun Cho, Bart Van Merri\u00ebnboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger\nSchwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for\nstatistical machine translation. arXiv preprint arXiv:1406.1078, 2014.\n\n9\n\n\fMart\u00edn Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S\nCorrado, Andy Davis, Jeffrey Dean, Matthieu Devin, and Others. Tensor\ufb02ow: Large-scale machine\nlearning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016.\n\nDiederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv preprint\n\narXiv:1412.6980, 2014.\n\nC.J.C.H. Watkins. Learning from Delayed Rewards. Ph.D. thesis, Cambridge University, 1989.\n\nBrian Lau and Paul W Glimcher. Dynamic response-by-response models of matching behavior in\n\nrhesus monkeys. Journal of the experimental analysis of behavior, 84(3):555\u201379, 2005.\n\nSimon N Wood. Fast stable restricted maximum likelihood and marginal likelihood estimation of\nsemiparametric generalized linear models. Journal of the Royal Statistical Society: Series B\n(Statistical Methodology), 73(1):3\u201336, 2011.\n\nJane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos,\nCharles Blundell, Dharshan Kumaran, and Matthew M Botvinick. Learning to reinforcement learn.\narXiv preprint arXiv:1611.05763, 2016.\n\nH. Francis Song, Guangyu R. Yang, and Xiao Jing Wang. Reward-based training of recurrent neural\n\nnetworks for cognitive and value-based tasks. eLife, 6:1\u201324, 2017.\n\nChethan Pandarinath, Daniel J O\u2019Shea, Jasmine Collins, Rafal Jozefowicz, Sergey D Stavisky,\nJonathan C Kao, Eric M Trautmann, Matthew T Kaufman, Stephen I Ryu, Leigh R Hochberg, and\nOthers. Inferring single-trial neural population dynamics using sequential auto-encoders. Nature\nmethods, page 1, 2018.\n\nSamuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio.\n\nGenerating sentences from a continuous space. arXiv preprint arXiv:1511.06349, 2015.\n\nEmily B Fox, Erik B Sudderth, Michael I Jordan, Alan S Willsky, et al. A sticky hdp-hmm with\n\napplication to speaker diarization. The Annals of Applied Statistics, 5(2A):1020\u20131056, 2011.\n\nMatthew J Johnson, David K Duvenaud, Alex Wiltschko, Ryan P Adams, and Sandeep R Datta.\nComposing graphical models with neural networks for structured representations and fast inference.\nIn Advances in neural information processing systems, pages 2946\u20132954, 2016.\n\n10\n\n\f", "award": [], "sourceid": 1331, "authors": [{"given_name": "Amir", "family_name": "Dezfouli", "institution": "Data61, CSIRO"}, {"given_name": "Hassan", "family_name": "Ashtiani", "institution": "McMaster University"}, {"given_name": "Omar", "family_name": "Ghattas", "institution": "University of Chicago"}, {"given_name": "Richard", "family_name": "Nock", "institution": "Data61, the Australian National University and the University of Sydney"}, {"given_name": "Peter", "family_name": "Dayan", "institution": "Max Planck Institute for Biological Cybernetics"}, {"given_name": "Cheng Soon", "family_name": "Ong", "institution": "Data61 and Australian National University"}]}