{"title": "Joint Active Feature Acquisition and Classification with Variable-Size Set Encoding", "book": "Advances in Neural Information Processing Systems", "page_first": 1368, "page_last": 1378, "abstract": "We consider the problem of active feature acquisition where the goal is to sequentially select the subset of features in order to achieve the maximum prediction performance in the most cost-effective way at test time. In this work, we formulate this active feature acquisition as a jointly learning problem of training both the classifier (environment) and the RL agent that decides either to `stop and predict' or `collect a new feature' at test time, in a cost-sensitive manner. We also introduce a novel encoding scheme to represent acquired subsets of features by proposing an order-invariant set encoding at the feature level, which also significantly reduces the search space for our agent. We evaluate our model on a carefully designed synthetic dataset for the active feature acquisition as well as several medical datasets. Our framework shows meaningful feature acquisition process for diagnosis that complies with human knowledge, and outperforms all baselines in terms of prediction performance as well as feature acquisition cost.", "full_text": "Joint Active Feature Acquisition and Classi\ufb01cation\n\nwith Variable-Size Set Encoding\n\nHajin Shim1, Sung Ju Hwang1,2, Eunho Yang1,2\n\nKAIST1, AItrics2, South Korea\n\n{shimazing, sjhwang82, eunhoy} @kaist.ac.kr\n\nAbstract\n\nWe consider the problem of active feature acquisition where the goal is to sequen-\ntially select the subset of features in order to achieve the maximum prediction\nperformance in the most cost-effective way at test time. In this work, we formu-\nlate this active feature acquisition as a joint learning problem of training both the\nclassi\ufb01er (environment) and the reinforcement learning (RL) agent that decides\neither to \u2018stop and predict\u2019 or \u2018collect a new feature\u2019 at test time, in a cost-sensitive\nmanner. We also introduce a novel encoding scheme to represent acquired subsets\nof features by proposing an order-invariant set encoding at the feature level, which\nalso signi\ufb01cantly reduces the search space for our agent. We evaluate our model\non a carefully designed synthetic dataset for the active feature acquisition as well\nas several medical datasets. Our framework shows meaningful feature acquisition\nprocess for diagnosis that complies with human knowledge, and outperforms all\nbaselines in terms of prediction performance as well as feature acquisition cost.\n\n1\n\nIntroduction\n\nDeep learning has shown remarkable growth in recent years mainly due to easier access to vast\namount of data from the internet, and demonstrated signi\ufb01cant improvements over classical and\nstandard algorithms on diverse tasks such as visual recognition [1, 2] and machine translation [3],\nto name a few. The fundamental assumption for training accurate deep networks is that data is\nreadily available at little or even no cost, such that the model can make predictions after observing\nall available features (in other words, feature acquisition is considered as an independent process\nagainst predictions). However, in some applications, information acquisition is not only affected by\nthe model (and vice versa) but it also incurs a cost. Consider, for instance, the task of diagnosing a\npatient with diseases. A human doctor, in this case, will start the diagnosis by starting with only a\nfew symptoms that the patient initially reported. From there, the doctor will either ask about more\nsymptoms or conduct some medical examinations to narrow down the set of potential diseases the\npatient might have until he or she has enough con\ufb01dence to make the \ufb01nal diagnosis. Acquiring all\nfeatures (via all medical tests) in this problem may cause a \ufb01nancial burden to patients and more\nseriously it may increase the risk of not receiving proper treatment at the right time. Furthermore,\ncollecting irrelevant features might add only noise and make the prediction unstable.\nIn this paper, we \ufb01rst formulate the feature acquisition problem that minimizes the prediction error as\nwell as the feature acquisition cost as an optimization problem.We then provide the sequential feature\nacquisition framework with the classi\ufb01er for predictions and the RL agent for feature acquisitions, in\norder to systematically solve the proposed optimization problem. We can understand the sequential\nfeature acquisition with the following example. As a human doctor diagnoses a patient in the\nprevious healthcare example, we need to decide which unknown features should be discovered in\norder to be suf\ufb01cient con\ufb01dent about our prediction. Given a new data point with missing entries,\nour RL agent sequentially chooses features to acquire based on the set of features acquired so far.\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fAt every examination (or feature acquisition), we pay the pre-de\ufb01ned inspection fee. This process\nis sequentially repeated until we have collected suf\ufb01cient but not redundant features to minimize\nacquisition cost. Once the agent decides to end the acquisition phase, the classi\ufb01er will make a\nprediction given the acquired features by the RL agent thus far. At the same time, the \ufb01nal rewards\nare set according to the prediction made by the classi\ufb01er, to signal the agent whether the current\nsubset of features is adequate for prediction or not. Interestingly, it turns out that the classi\ufb01er in our\noptimization framework can be understood as the estimated environment for the RL agent, which is\nintuitive in the sense that the reward to the RL agent should be based on how con\ufb01dent our classi\ufb01er\nis on its \ufb01nal decision.\nIt is worth noting that we do not assume a \ufb01xed classi\ufb01er that is pre-trained during a separate\nphase. Learning the optimal classi\ufb01er beforehand only by pre-training is unrealistic since the optimal\nclassi\ufb01er should be associated to the feature acquisition policy; if our acquisition policy changes,\nthe corresponding optimal classi\ufb01er does as well. Actually, the RL approach for feature selection\nproblem is challenging because it has a huge search space that includes all the variable-sized subset\nof features and each feature can be either discrete or continuous. To search this huge space ef\ufb01ciently\nand stably, we carefully devise the architecture with the recently proposed set encoding method [4].\nThe feature-level set encoding method helps to reduce the state space effectively, by making feature\nacquisition process to be order-invariant through the attention mechanism [3].\nOur contribution is threefold:\n\n\u2022 We formulate the feature acquisition problem as an optimization problem with cost-sensitive\npenalty in the objective function and optimize with a joint framework that simultaneously\ntrains the classi\ufb01er and the RL agent, for systematic learning of active feature acquisition\nmodel, without requiring a probabilistic model or pre-de\ufb01ned classi\ufb01er.\n\n\u2022 We propose a novel method to encode the subset of features by appropriately modifying\n[4] that can naturally handle missing entries and is shared by the classi\ufb01er and the agent.\nIn addition, we apply a synchronous variant of n-step Q-learning [5] to handle real-valued\nfeature space in feature acquisition problems.\n\n\u2022 We validate the superiority of the proposed framework on diverse simulated and medical\ndatasets and also verify the clinical signi\ufb01cance of the results with clinicians, showing that\nthe set of features obtained by our model is similar to what a clinician would use in practice.\n\n2 Related Work\n\nAs we mentioned in introduction section, we cannot guarantee that not all the features are given away,\nso we should consider which features to use for testing. First of all, we can select features in static\nmanners, that is, examine same predictors across all instances. It can be done in several different\nways including sparsity-inducing regularization [6], greedy forward and backward selections [7].\nInstead, we can perform active feature acquisition in more ef\ufb01cient way by considering the differences\nacross instances. Greiner et al. [8] investigate the problem of learning an optimal active classi\ufb01er\nbased on a variant of the probably-approximately-correct (PAC) model. The following works consider\nactive classi\ufb01cation under some constraint. Sheng et al. [9] propose sequential batch test algorithm\nthat acquires a batch of features iteratively until no more positive cost reduction occurs by utilizing\na pre-built cost-sensitive model. Kanani et al. [10] identify the data points whose missing features\nwill be completely acquired among all test instances with partially given features, based on the\nexpected utility. Kapoor et al. [11] bridge induction-time and test-time information acquisition in the\nrestrictive case that all attributes and labels are binary. They directly build a statistic model explaining\ndependency of the test label on its features and also the training data with Bayesian treatment of\nmodel parameters. Bilgic et al. [12] approach the problem by constructing graphical models to\nget the value of subsets of features with the given bayesian network that helps reducing a search\nspace. Trapeznikov et al. [13] consider the budget constraint and suggest a framework to analyze a\nmulti-stage and multi-class sequential decision system such that the K classi\ufb01ers make a prediction\nor pass to the next with incrementally acquired features in \ufb01xed order. Nan et al. [14] present the\nclassi\ufb01cation algorithm which acquires features greedily and sequentially by using partial margin\nof the k-nearest neighbors de\ufb01ned with the pre-trained linear classi\ufb01er. Xu et al. [15] construct and\ntrain a cost-sensitive classi\ufb01cation tree by using trained gradient boosted trees, who inspect different\nsubset of features, as weak learners.\n\n2\n\n\fSome existing works formulate the dynamic feature acquisition problem as Markov Decision Process\n(MDP) or partially observable MDP (POMDP) and try to learn the best feature acquisition policy\nthat gives maximum returns. Ruckstiess et al. [16] set this problem as partially observable MDP\n(POMDP) with the assumption that pre-learned classi\ufb01er and prediction result on each instance with\nmissing entries from this classi\ufb01er as the observation of the state. They solve the POMDP by Fitted\nQ-Iteration (FQI) but they fail to specify when to stop acquiring features. Dulac-Arnold et al. [17]\npropose an MDP formulation by directly modeling the state space with acquired features only. They\nalso incorporate the additional special actions corresponding to predictions, which leads to training\nthe classi\ufb01er (as well as when to stop) implicitly within MDP framework. [18, 19] follow similar\nMDP setting with ours and they adopt imitation learning approach that train the agent to follow the\nreference policy that is greedy policy of the oracle who exploit true labels and the classi\ufb01er. This can\nrestrict the performance of the agent under the oracle\u2019s performance that might be suboptimal. Our\nframework does not assume pre-learned classi\ufb01er to adapt classi\ufb01er to the feature acquisition policy\nand vice versa. Also, our framework learn its policy by directly interacting with and getting reward\nfrom the data and the external classi\ufb01er rather than imitate the suboptimal oracle. Due to the huge\nsearch space that is of subset of features, systematic architecture should be designed.\nAnother recent line of work [20, 21, 22] reduces the computational cost to process the high dimen-\nsional image data via the recurrent attention. [20, 21] use REINFORCE [23] to make the models\nlocalize informative part and Ba et al. [22] improve these methods with an additional inference\nnetwork and reweighted wake-sleep algorithm. These models acquire and aggregate observed features\nby using RNN and after \ufb01xed steps of acquisition they make a prediction. While these methods\nalso try to select the most informative subset of features (or area), the goal is different from active\nfeature acquisition: since these models are devised to reduce the computational cost, the number\nof acquisitions is pre-de\ufb01ned and \ufb01xed, depending on the target computational budget, leading to\ninstance-independent feature acquisition costs. Moreover, they do not consider the heterogeneous\nacquisitions costs across features since all features are in fact always available.\nReinforcement learning that is widely used to \ufb01nd an optimal policy in MDP is also be utilized for\ninformation extraction problem that is \ufb01lling the missing entries by extracting appropriate value from\nappropriate sources [24, 25]. Kanani et al. [24] consider resource-bounded information extraction\n(RBIE) as MDP with query, download, and extract as types of actions and learn Q-function by\ntemporal difference Q-learning. Narasimhan et al. [25] address two challenges of utilizing external\ninformation that are retrieving suitable external information sources and reconciling extracted entities\nby employing a deep Q-learning.\n\n3 Joint Learning Framework for Sequential Feature Acquisition and the\n\nClassi\ufb01cation\n\nConsider the standard K-class classi\ufb01cation problem where we learn a function f\u2713 that maps data\npoint x = (x1, x2, . . . , xp) 2 Rp with p features to a label y 2Y := {1, 2, . . . , K}. The basic\nassumption here is that the feature vector is \ufb01xed-dimensional, and given in full (but possibly with\nmissing entries). We instead consider the same problem under a slightly different condition.\nFor each data point x(i), we actively acquire the features in a sequential order. Speci\ufb01cally, at\nt = 0 we start with an empty acquired set O0 := ;. At every time step t, we choose the subset of\nunselected features, S (i)\nt \u2713{ 1, . . . , p} \\ O(i)\nat the\n:= Pj2S(i)\ncost c(i)\ncj. Hence, after the examination at time t, we have access to the values of\nt\nt [O (i)\n:= S (i)\nO(i)\nT (i) is not necessarily equal\nfor all data points i = 1, . . . , n) and classify x(i) given only the observed features O(i)\nT (i). Note that\nthe order of feature acquisitions and corresponding costs can be different across samples, but we\nsuppress the sample index i when it is clear from the context.\nIn order to learn the model that minimizes the classi\ufb01cation loss and the acquisition cost simultane-\nously, we formulate our framework in the following optimization problem:\n\nt1. We repeatedly acquire features up to time T (i) (O(i)\n\nt1 and examine the values of missing entries S (i)\n\nt\n\nt\n\nt\n\nminimize\n\n\u2713,#\n\n1\nN\n\nNXi=1\n\nL\u21e3f\u2713x(i), z(i)\n\n# , y(i)\u2318 + c>z(i)\n\n#\n\n(1)\n\n3\n\n\fwhere L is the cross-entropy loss as in the standard classi\ufb01cation problem and the binary vector\nz# 2{ 0, 1}p encodes whether each feature is acquired at the end (or at T (i)) when the sequential\nselection is performed by policy #. Note also that the classi\ufb01er f\u2713 is able to access only available\nfeatures with [z#]j = 1. The hyperparameter  controls the relative importances of the prediction\nloss and the acquisition cost.\nNatural way to solve (1) is to adopt an alternating minimization since the roles and properties of two\nparameter sets \u2713 and # are clearly distinguished in (1). Solving (1) with respect to \u2713k given #k1 is\ntrivial. Since the cost term c>z(i)\n# is a constant w.r.t \u2713, it is trained to minimize the prediction loss\ngiven the subset of features acquired by #k1. The only challenge here is to \ufb01nd a way of ef\ufb01ciently\nhanding missing features (since the unacquired features by #k1 are missing to the classi\ufb01er \u2713k).\nOn the other hand, solving (1) with respect to #k given \u2713k1 is not trivial. In the following subsection,\nwe will detail how we de\ufb01ne a Markov decision process (MDP) to solve (1) w.r.t. #.\n\n3.1 RL construction to solve (1) w.r.t. #\nMarkov decision process (MDP) consists of an agent in the environment, a set of states S and a set of\nactions A per state. In state st, the agent chooses the action according to its policy. Then the state\nchanges and environment gives a reward. The goal of the agent is to maximize the total rewards. In\nthis subsection, we construct an MDP corresponding to solving (1) w.r.t. # given \u2713k1.\nState. Since the informative features can be different across classes, the subset of features our RL\nagent should select will differ across data points. Without having any prior information on true class,\nthe importance of the missing features can be estimated from the currently available features Ot. To\nthis end, we construct the state st as the concatenation of zt and xt where the j-th entry of xt, [xt]j,\nis set to zero if j /2O t or to the value of the j-th feature otherwise. Note again that zt 2{ 0, 1}p is\nthe binary indicator encoding which features are acquired until time t.\n\nAction. The RL agent selects which features to examine. The set of all possible actions is de\ufb01ned\nas the power set of {1, . . . , p} including the empty set ;, which means to stop acquiring any more\nfeatures. Throughout the paper, we mainly assume that the agent gets one feature at a time for\nclarity, hence the size of action space is p + 1 = |{1, . . . , p,;}|. Some actions would be invalid if the\ncorresponding features have already been selected previously. ; is a special action that is valid at any\ntime corresponding to \u2018stop and predict\u2019 based on current state st.\n\nReward and environment. We naturally de\ufb01ne the reward as the negative acquisition cost. Speci\ufb01-\ncally, in the episode (s0, a0, r1, s1, . . . , sT , aT = ;, rT +1), rt+1 is set as cat for t = 0, . . . , T 1.\nNote that although the state transition from (xt, zt) to (xt+1, zt+1) is deterministic given at, it is\nstill not trivial since a new acquisition is unknown before actually observing it.\nUnlike \u2018feature acquisition\u2019 actions, the state transition by ; is trivial since no further feature value\nwill be revealed: i.e., (xt, zt) = (xt+1, zt+1). We also set the \ufb01nal reward rT +1 after selecting ; as\nLf\u2713k1(xT , zT ), y where \u2713k1 is given in the alternating minimization scheme.\nThen, we have the following result:\nTheorem 1. Consider some policy \u21e1#k parameterized by #. Suppose that this policy \u21e1#k is the\noptimal of Markov decision process described in Section 3.1, given the classi\ufb01er parameter \u2713k1.\nThen, \u21e1#k is also the optimal solution of (1) with respect to # given \u2713k1.\nThe theorem can be simply proven as follows:\n\n1\nN\n\nNXi=1\uf8ff L\u21e3f\u2713k1s(i)\n\n# \n# , y(i)\u2318  c>z(i)\n\n#\n\nargmax\n\n#\n\n1\nN\n\n= argmin\n\n#\n\n1\nN\n\nT (i)\n\nt1,\u21e1 #(s(i)\n\n# +1Xt=1\nrt\u21e3s(i)\nNXi=1\nNXi=1\uf8ffL\u21e3f\u2713k1s(i)\n\nt1)\u2318 = argmax\n# \n# , y(i)\u2318 + c>z(i)\n\nwhere s# and z# are respectively the \ufb01nal state and corresponding z by the policy \u21e1#. The selection\nof the \ufb01nal reward above is reasonable in the sense that it should measure how suf\ufb01cient information\nfor \u2713k1 has been provided so far for a prediction.\n\n4\n\n\fLearning policy.\nIn order to \ufb01nd an optimal policy, we use Q-learning [26] for our agent. Q-\nlearning is a value-based RL method which \ufb01nds the optimal policy by learning state-action value\nQ(s, a) from experience. To handle continuous state space, we use deep Q-learning that has shown\nto be successful in reinforcement learning with high dimensional, high order features and discrete\naction space A. More speci\ufb01cally, we adopt n-step Q-learning [5] but in synchronous style. [5] makes\nQ-learning more stable with delayed update of the target network and by running the multiple agents\nsimultaneously to decorrelate the running history. To avoid overestimation of state-action value, we\nalso adopt double Q-learning [27] instead of basic Q-learning.\n\n4 Feature-level Set Encoding for Joint Learning\n\nIn our framework, we need to learn the state-action value function Q (parameterized by #) and the\nclassi\ufb01er C (parameterized by \u2713); note that we rename it from f\u2713 to match with Q.\nSince both components share the input s, train-\ning them simultaneously can be understood as\nmulti-task learning. Intuitively, Q and C should\nshare certain degree of information between\nthem, since both aims to optimize a single joint\nlearning objective in (1). However, too much\nsharing could result in reducing the \ufb02exibility\nof each model, and we should \ufb01nd the appro-\npriate level of information sharing between the\ntwo. For instance, in case where Q and C are\nmulti-layer perceptrons (MLP), they may share\nthe \ufb01rst few layers. From our preliminary ex-\nperiment on the effects of information sharing\nin Figure 1 (see Section 5 for the detail exper-\nimental setup), we found out that the partially\nsharing model outperforms two other extremes\n(no sharing or complete sharing) in terms of ac-\ncuracy and the number of observed features, and\nalso experiences less variance. These shared layers can also be considered as a shared encoder,\nEnc(s), whose output is fed both into Q and C. Hence, our Enc-Q-C framework be formulated in the\nfollowing way. At every time t, the state st := (xt, zt) is fed to the shared encoder: ht := Enc(st).\nThen, the encoded representation ht is given to Q and C:\n\nFigure 1: Effect of sharing layers between Q and C:\nWe check the classi\ufb01cation accuracy and average num-\nber of features collected on the CUBE with 90 dummy\nfeatures for varied the number of shared layers from 0\n(completely separate) to 3 (completely shared). Both Q\nand C are MLP with 3 hidden layers whose sizes are\n50-30-50. The points and error bars are average and\n\ufb01rst/third quartile of 100 runs.\n\nqt := Q(ht), Q(st, a) := [qt]a\npt := C(ht), P(y|st) := softmax(pt).\n\nfor every action a,\n\nAs an example of the feature encoding Enc in our framework, we devise the feature-level set encoder\nbased on the recently proposed set encoding method [4]. It is composed of two main parts: i) MLP\ncalled reading block, which maps each set element xi to the real vector mi and ii) a process block that\nprocesses mi repeatedly with LSTM and the attention mechanism to produce the \ufb01nal set embedding.\nWe adopt this method to encode each state st, and individually treat the pair of feature index and\nits value, (j : xj), as the set element. We \ufb01rst represent each observed feature as uj = (xj,I(j))\nwhere I(j) = (0, ..., 0, 1, 0, ..., 0) is the one-hot vector with 1 for j-th coordinate and 0 elsewhere in\norder to incorporate the coordinate information. Then, via the set encoding mechanism (through the\nreading block to make {mj}j2Ot, followed by the process block) introduced above, we produce the\nset embedding of the observed features. The set encoding is well suited to Enc since it is invariant\naccording to feature acquisition order and naturally distinguishes two ambiguous cases: j-th entry is\ni) unexplored or ii) discovered but zero value.\n\n4.1 Learning and inference\n\nIn this subsection, we provide the details on how we can actually train Q and C equipped with the\nfeature-level set embedding Enc jointly in an end-to-end manner. Due to the space constraint, the\npseudocode that summarizes the learning algorithm is deferred to the supplementary material.\n\n5\n\n\fWe follow the n-step Q-learning procedure described in [5] with two key mechanisms: running\nmultiple agents in parallel and delayed update of target Q-network Q0 to prevent perturbation.\nSpeci\ufb01cally, in a training phase, each agent runs for n steps according to its policy based on current\nQ and gets the experience (st, at, rt, st+1, at+1, . . . , st+n). Q-values of invalid actions of each state\nare manually set to 1.\nAfter n steps running, Q and C are updated based on the running history. First, the target Q-\nvalue R of each state is computed by summing all given rewards after that state plus the approx-\nimated Q-value of the last state unless it is the terminal state (with discount factor 1). To avoid\noverestimation, we use double Q-learning method, so the approximated Q-value is de\ufb01ned as\nQ(st+n, argmaxa Q(st+n, a; #); #0) with the target Q network parametrized by #0 that only updated\nfor every Itarget steps to # for stability. All parameters of Q are updated by the gradient descent\nmethod to minimize the squared error (Q(st, at)  R)2.\nWhile Q is trained, C is also jointly trained. Since C is supposed to perform a classi\ufb01cation task with\nmissing values, it would be helpful to train it with incomplete dataset. Toward this, C is trained on\nthe experienced states that might be incomplete by the gradient descent method to minimize the cross\nentropy loss:  log Cytrue(st) where Cytrue(st) is the output (or probability after softmax layer)\ncorresponding to the true label. Q and C are alternatively updated until the stopping criteria are\nsatis\ufb01ed. While Q and C have their own loss functions, the shared function Enc can be learned both\nby Q and C, or only by one of them, depending on applications.\nBefore learning Q and C jointly, we pretrain our classi\ufb01er C with fully observed features and also\nrandomly dropped features with probability 0.5 to simulate random feature acquisition policy. This\ninitialization of C lets joint learning time until convergence be effectively reduced.\n\nInference. At test phase, the start state might be completely empty or partially known. Our RL\nagent determines which features to acquire by greedily selecting the action with the maximum\nQ-value until ; is chosen. When ; is selected, C makes a prediction based on the acquired features.\n5 Experiments\n\nOur code is available at https://github.com/OpenXAIProject/Joint-AFA-Classification.\nTo validate the versatility of our model, we perform the extensive experiments on simulated and\nmedical datasets.\n\nExperimental setup Throughout all experiments, we use Adam optimizer [28] with 0.001 learning\nrate and train the models for \ufb01xed number of iterations.\n\n5.1 Simulated dataset: CUBE-\n\nWe \ufb01rst experiment on a synthetic dataset CUBE- to see if the agent can identify few important\nfeatures that are relevant to the given classi\ufb01cation task. See Fig 2a and [29] for detailed description\nof the dataset. We train our model on CUBE dataset with 20 features, setting the Gaussian noise\n = 0.1 for the informative features. We train Q, C and Enc for 10000 iterations on 10, 000 training\ninstances. It is trained by 4-step Q-learning. Per iteration, 128 agents run in parallel for 4 steps.\nInstead of updating at once, we do mini-batch update with the size of 128 for 1 epoch (4 times update).\nWe assume uniform acquisition cost 0.05 and the \ufb01nal reward as the negative classi\ufb01cation loss\nL based on C. Both C and Q has two hidden layers of 32-32 units, and Enc consists of the MLP\nwith two hidden layers of 32-32 units which maps features to 16 dimensional real-valued vectors and\nLSTM whose hidden size is 16. For \u270f-greedy exploration, \u270f linearly decreases from 1 to 0.1 for the\n\ufb01rst 5000 iterations.\nThe overall results show that the agent can successfully learn which features are informative and\nhence it mostly selects the features from the ten informative ones. It consumes only 4.19 features on\naverage while obtaining 96.4% accuracy (even without \ufb01ne-tuning), which is comparable to the MLP\nclassi\ufb01er that uses all features (96.98%).\n\nEffect of Set encoding. We also evaluate the effect of using proposed feature level set encoding.\nGiven the same structure of C and Q, we compare the performances of the proposed set encoding\n\n6\n\n\f(a) The CUBE dataset\n\n(b) CUBE-0.3 with 20 features (c) CUBE-0.3 with 100 features\n\nFigure 2: (a) The CUBE dataset consists of p-dimensional real valued vectors in 8 classes. The \ufb01rst 10 features\nonly carry class information with three normally distributed entries in the different locations (dimmed in the\n\ufb01gure with written mean values) according to the classes. The rest of them are just uniformly random. (b),(c)\nComparisons of RAM, our proposed model and its variant on CUBE-0.3. We report averages with 1st and 3rd\nquantile as error bars of 10 times run.\n\n(a) Mortality\n\n(b) Length of stay < 3\n\n(c) Cardiac condition\n\n(d) Recovery from surgery\n\nFigure 3: The comparison on Physionet 2012 data. We report averages with 1st and 3rd quantile as\nerror bars of 10 times run. The black dotted line indicates the result of MLP with group norm.\n\nagainst the use of naive encoding with shared MLP which takes (x, z) as its input. We again use\nthe 20/100-dimensional CUBE data with  = 0.3. Fig 2 (b), (c) show the accuracy and the number\nof features acquired for both encodings. We use cross-validation to select the best setting, and\nrepeat the training ten times with different random seeds. The model with the proposed set encoding\noutperforms both in terms of accuracy and the number of acquired features, showing the resilience\nagainst the increase of dummy features.We conjecture that set encoding can reduce the noise coming\nfrom dummy features by attention mechanism so is accompanied by stable policy learning.\nMany previous works just assume that the pre-trained classi\ufb01er is given. However, this is not\nappropriate for dynamic feature acquisition problem because we need to consider the connection\nbetween the feature acquisition policy and the classi\ufb01er. We compare the jointly learned model with\nthe pre-trained classi\ufb01er variant in the Fig 2a. We train the classi\ufb01er with the partial information in\nthe trivial way that simply decides to choose features or not with equal probability and also with the\nfull feature. The result shows that this pre-trained variant has low accuracy with almost the same or\nmore amount of features meaning the classi\ufb01er cannot perfectly understand the agent\u2019s choice.\nWe additionally analyze how our agent acts on the CUBE dataset and the results are in the supplement.\n\nComparison against RAM Recurrent attention model (RAM) and its variations DRAM,\nWS_RAM [20, 21, 22] reduce the computational cost while obtaining high performance by tak-\ning only the informative parts of an image sequentially. Especially, RAM use REINFORCE to make\nthe models localize informative parts. However, they take glimpses for prede\ufb01ned n steps. It may\nlead redundant consumption for some cases. We compare RAM on CUBE-0.3 with 20/100 features.\nThe input of RAM is given in the form of uj = (xj,I(j)). We can observe in Fig 2(b), (c) that our\njointly trained model with set encoding can achieve higher accuracy with less features on average\nbecause our agent stops with enough informations and this is not the case for RAM.\n\n5.2 Case study: Medical Diagnosis\n\nAs our main motivation of this work is to make cost-effective predictions in the medical diagnosis,\nwe further examine how our model operates on real-world medical datasets.\n\n7\n\n\fTable 1: Statistics of Physionet 2012 data (# false: # true).\n\ndata\n\ntask\nMortality\nLength of stay < 3\nCardiac condition\nRecovery from surgery\n\ntrain\n\n2586 : 414\n2917 : 83\n2333 : 667\n2209 : 791\n\nvalidation\n443 : 57\n485 : 15\n401 : 99\n363 : 137\n\ntest\n\n417 : 83\n487 : 13\n392 : 108\n360 : 140\n\nTable 2: Result on the check-up data. We report prediction AUC (top) and the average number of\nacquired features (bottom) and cost (USD; in brackets).\n\nPre-trained w/ Set Encoding\nuniform cost\n\n0.774\n\n6.2 (127.9)\n\n0.721\n\nreal cost\n0.779\n4.8 (9.2)\n0.760\n\nuniform cost\n\nJoint w/ Set Encoding\nreal cost\n0.792\n\n0.795\n\n10.3 (149.8)\n\n7.5 (13.8)\n\n0.756\n\n0.750\n\nmodel\n\ntask\nFatty Liver\n\nIMT\n\nBaselines\n\nMLP\n0.812\n\n35 (577.7)\n\n0.755\n\nJoint w/ Naive Encoding\nreal cost\nuniform cost\n0.785\n\n0.775\n\n6.81 (167.0)\n\n8.7 (20.1)\n\n0.742\n\n0.704\n\n22 (1,629.8)\n\n5.12 (109.9) 14.22 (11.3)\n\n3.94 (191.5)\n\n23.71 (83.8)\n\n5.57(309.8) 4.79 (10.0)\n\nPhysioNet challenge 2012 dataset First, we conduct the experiment on EHR dataset from Phy-\nsionet challenge 2012 [30]. It has four binary classi\ufb01cation tasks, namely in-hospital mortality,\nwhether length-of-stay was less than 3 days, whether the patient had a cardiac condition, and whether\nthe patient was recovering from surgery. We only use the training set whose labels are available and\ntake the features only in the last timestep and split the data randomly into the training/validation/test\nset by 3000/500/500 ratio. The data is imbalanced (see Table 1), hence we use weighted cross entropy\nas a loss of C.\nAs baselines, we compare an MLP with group `1-norm regularizer for weights in the \ufb01rst layer,\nDWSC [17], joint training model with naive MLP encoding and the pre-trained classi\ufb01er variant with\nour model. DWSC is the model whose action space includes not only feature acquisition but also\nclassi\ufb01cation. We omit the result of DWSC in Fig 3 because they often fail to handle the imbalanced\ndata and just learn to predict as the majority. Fig 3 shows the ROC AUC and the number of acquired\nfeatures. MLP with regularizer also implicitly select features in a static manner. We report the number\nof nonzero columns of the \ufb01rst layer weight matrix as the number of acquired features shown as a\ndotted line in the \ufb01gure. The model in our framework achieves AUC close to that of MLP while\nacquiring fewer features and outperforms all the others for prediction task of length of stay. This\nshows that our model is able to select informative, task-related features well.\nFor in-hospital mortality prediction task, the \ufb01rst feature selected is Glasgow coma scale (GCS),\nwhich represents the level of consciousness. GCS is a decisive feature as having a very low GCS\nmeans that the patient is almost unconsciousness. Thus in such a case, the agent stop the examination\nand predicts that mortality is true. The other major features are blood urea nitrogen, serum creatine,\ngender and age.\n\nMedical check-up data In order to evaluate our model under a more realistic diagnosis situations\nwhere cost comes to play, we perform an additional experiment on check-up data provided by a major\nhospital. We consider two different binary classi\ufb01cation tasks: prediction of 1) fatty liver and 2)\nintima-media thickness (IMT). We label patents that have steatosis (dB/m) value larger than 280 as\nhaving fatty liver (1), and 0 otherwise. For IMT prediction, we label the patients with left or right\nthickness greater than or equal to 0.8 as positive, and 0 otherwise. Among a total of 3937 data points,\nsteatosis is available for all the data but IMT labels are only available for 1358 instances. Both tasks\nhave 96 features associated with different costs, including body measurement, blood test and CT\ncalcium score, etc. For reality, we consider some features examined together from the one test (such\nas blood test) as a one multidimensional feature and so make them achieved at the same time. We\nrandomly split the data into three folds with the ratio of 64:16:20 for train:validation:test.\nWe \ufb01rst compare the AUC to the MLPs with group `1-norm regularizer on the weight of the \ufb01rst\nlayer to leverage static feature selection effect. We here report the results selected by cross validation.\nSince we are concerned with cost-sensitive feature acquisition in this experiment, we report not only\nthe average number of examined features but also the average examine cost. We also compare two\ndifferent cost settings: 1) A uniform constant across all the features and 2) Actual examine cost\n\n8\n\n\fmultiplied by a constant . Examination costs of the features determining the labels are about 80 and\n1,564 USD for fatty liver and IMT respectively, and thus our aim is to make diagnosis with lower\ncosts. As shown in Table 2, an MLP and our model both achieve comparable AUC, while our model\nuse signi\ufb01cantly less cost (13.8 USD for fatty liver and 10 USD for IMT).\nNow, we brie\ufb02y describe the detail of diagnosis process of our model. The followings are veri\ufb01ed\nby the clinicians. MLP incurs large examination cost because it leverages coronary calcium score\nand Echo E/E\u2019 for prediction, and so does our model with uniform cost setting as it selects Echo\nEF value which are all expensive. However, our cost-sensitive model effectively acquires relatively\ncheaper and relevant features for fatty liver prediction such as age, BMI, low-density lipoprotein\n(LDL), etc. Similarly in the IMT prediction task, rather than taking expensive features such as Echo\nE/E\u2019 and steatosis (it is chosen because IMT is known as to be related to fatty liver disease) as done\nby MLP or our model with uniform cost, our cost-sensitive model makes a decision based on age,\nsmoking history, genetic factors, weight, BMI (obesity), glucose (diabetes), blood pressure, LDL\ncholesterol, TG (cholesterol). This result is meaningful because all these are well-known risk factors\nof atherosclerosis that are also often used by doctors.\n\n6 Conclusion\n\nA cost-aware sequential feature selection can be used in situations where the features are not provided\nin full and each collection of features incurs variable cost, such as with medical examination. To\nsolve this problem, we formulated it into an optimization problem of simultaneously minimizing\nthe prediction loss and the feature acquisition costs, and derived a joint learning framework for the\nclassi\ufb01er and the RL agent. We validated our model on both synthetic and real medical datasets to\ncon\ufb01rm the superiority of proposed feature level set embedding under our joint learning framework.\nHowever, there is still room for improvement of our model. First of all, in the synthetic dataset\nexperiment, we found that features are overly acquired for a few instances whose class is indistin-\nguishable because of the data generation rule. This problem may be due to the intrinsic overestimation\nbehavior of Q-learning because Q-learning update uses maximum approximated action value as a\ntarget. Therefore, Q-values of feature acquisition actions which are nonterminal are overestimated\ncompared to stop action. We plan to explore some tricks or other reinforcement learning methods to\nalleviate this problem as future work. Secondly, we can take model uncertainty into account, so that\nour agent collects the features that can help to reduce uncertainty as well as increase classi\ufb01cation\nperformance with as small as possible amounts of cost.\nWhen it comes to running time, training time is less than 20 minutes for reported experiments, and for\ntesting, it takes about 0.5 sec to evaluate 500 instances (on GTX 1070). In training phase, convergence\nis affected by decaying schedule for \u270f of \u270f-greedy policy in Q-learning algorithm. We found that\ndecaying it linearly from 1 to 0.1 from the \ufb01rst half of total iterations generally works well.\n\n7 Acknowledgements\n\nThis work was supported by Institute for Information &communications Technology Promotion(IITP)\ngrant (2017-0-01779, A machine learning and statistical inference framework for explainable arti\ufb01cial\nintelligence, 2017-0-00537, Development of Autonomous IoT Collaboration Framework for Space\nIntelligence), the Engineering Research Center Program through the National Research Foundation\nof Korea (NRF) (NRF-2018R1A5A1059921) funded by the Korea government(MSIT) and NAVER\nCorporation (Research on deep and structurally constrained machine learning).\n\nReferences\n[1] T. N. Sainath, A. r. Mohamed, B. Kingsbury, and B. Ramabhadran. Deep convolutional neural networks\n\nfor lvcsr. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, 2013.\n\n[2] K He, X Zhang, S Ren, and J Sun. Deep residual learning for image recognition. In Computer Vision and\n\nPattern Recognition, 2016.\n\n[3] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate.\n\nCoRR, abs/1409.0473, 2014.\n\n9\n\n\f[4] O. Vinyals, S. Bengio, and M. Kudlur. Order matters: sequence to sequence for sets. In International\n\nConference on Learning Representations, 2016.\n\n[5] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley,\nDavid Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In\nProceedings of The 33rd International Conference on Machine Learning, 2016.\n\n[6] R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society,\n\nSeries B, 58(1):267\u2013288, 1996.\n\n[7] Isabelle Guyon and Andr\u00e9 Elisseeff. An introduction to variable and feature selection. Journal of machine\n\nlearning research, 3:1157\u20131182, 2003.\n\n[8] Russell Greiner, Adam J. Grove, and Dan Roth. Learning cost-sensitive active classi\ufb01ers. Artif. Intell.,\n\n139(2):137\u2013174, 2002.\n\n[9] Victor S. Sheng and Charles X. Ling. Feature value acquisition in testing: a sequential batch test algorithm.\n\nIn International Conference on Machine learning (ICML), 2006.\n\n[10] Pallika Kanani and Prem Melville. Prediction-time active feature-value acquisition for cost-effective\n\ncustomer targeting. Advances In Neural Information Processing Systems (NIPS), 2008.\n\n[11] Ashish Kapoor and Eric Horvitz. Breaking boundaries: Active information acquisition across learning\nand diagnosis. In Proceedings of the 22nd International Conference on Neural Information Processing\nSystems, 2009.\n\n[12] Mustafa Bilgic and Lise Getoor. Voila: Ef\ufb01cient feature-value acquisition for classi\ufb01cation. In Proceedings\n\nof the National Conference on Arti\ufb01cial Intelligence, 2007.\n\n[13] Kirill Trapeznikov and Venkatesh Saligrama. Supervised sequential classi\ufb01cation under budget constraints.\n\nIn Arti\ufb01cial Intelligence and Statistics, 2013.\n\n[14] F. Nan, J. Wang, K. Trapeznikov, and V. Saligrama. Fast margin-based cost-sensitive classi\ufb01cation. In\n\n2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2014.\n\n[15] Zhixiang Xu, Matt J. Kusner, Kilian Q. Weinberger, and Minmin Chen. Cost-sensitive tree of classi\ufb01ers.\nIn Proceedings of the 30th International Conference on International Conference on Machine Learning -\nVolume 28, ICML\u201913, 2013.\n\n[16] T. R\u00fcckstie\u00df, C. Osendorfer, and P. Patrick van der Smagt. Sequential feature selection for classi\ufb01cation.\n\nIn Australasian Conference on Arti\ufb01cial Intelligence, 2011.\n\n[17] G. Dulac-Arnold, L. Denoyer, P. Preux, and P. Gallinari. Datum-wise classi\ufb01cation: A sequential approach\n\nto sparsity. In Proceedings of the ECML/PKDD 2011, 2011.\n\n[18] He He, Jason Eisner, and Hal Daume. Imitation learning by coaching. In Advances in Neural Information\n\nProcessing Systems, pages 3149\u20133157, 2012.\n\n[19] H. He, P. Mineiro, and N. Karampatziakis. Active Information Acquisition. ArXiv e-prints, 2016.\n\n[20] Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In Advances in\n\nneural information processing systems, 2014.\n\n[21] Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. Multiple object recognition with visual attention.\n\narXiv preprint arXiv:1412.7755, 2014.\n\n[22] Jimmy Ba, Ruslan R Salakhutdinov, Roger B Grosse, and Brendan J Frey. Learning wake-sleep recurrent\n\nattention models. In Advances in Neural Information Processing Systems, 2015.\n\n[23] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement\n\nlearning. Machine learning, 8(3-4):229\u2013256, 1992.\n\n[24] Pallika H. Kanani and Andrew K. McCallum. Selecting actions for resource-bounded information extraction\nusing reinforcement learning. In Proceedings of the Fifth ACM International Conference on Web Search\nand Data Mining, WSDM \u201912, 2012.\n\n[25] Karthik Narasimhan, Adam Yala, and Regina Barzilay. Improving information extraction by acquiring\n\nexternal evidence with reinforcement learning. arXiv preprint arXiv:1603.07954, 2016.\n\n[26] C. J. C. H. Watkins and P. Dayan. Q-learning. In Machine Learning, pages 279\u2013292, 1992.\n\n10\n\n\f[27] Hado V. Hasselt. Double q-learning. In Advances in Neural Information Processing Systems 23. 2010.\n\n[28] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980,\n\n2014.\n\n[29] T R\u00fcckstie\u00df, C Osendorfer, and P van der Smagt. Minimizing data consumption with sequential online\n\nfeature selection. Int. J. Machine Learning & Cybernetics, 4(3):235\u2013243, 2013.\n\n[30] A. L. Goldberger, L. A. N. Amaral, L. Glass, J. M. Hausdorff, P. Ch. Ivanov, R. G. Mark, J. E. Mietus,\nG. B. Moody, C.-K. Peng, and H. E. Stanley. PhysioBank, PhysioToolkit, and PhysioNet: Components of\na new research resource for complex physiologic signals. Circulation, 101(23):e215\u2013e220, 2000 (June 13).\n\n[31] Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision?\nIn I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors,\nAdvances in Neural Information Processing Systems 30, pages 5574\u20135584. Curran Associates, Inc., 2017.\n\n11\n\n\f", "award": [], "sourceid": 700, "authors": [{"given_name": "Hajin", "family_name": "Shim", "institution": "KAIST"}, {"given_name": "Sung Ju", "family_name": "Hwang", "institution": "KAIST, AItrics"}, {"given_name": "Eunho", "family_name": "Yang", "institution": "Korea Advanced Institute of Science and Technology; AItrics"}]}