This paper proposes a novel few-shot learning method, with a specific application focus to fine-tuning CV object classification models from pre-trained features. Different from previous few-shot learning or CNP work, this work tries to address a convincing real world use case. Its novelties include inference amortization for head models, adaptation of the feature network on each task using a novel autoregressive architecture. One point of improving the paper is to move details about the autoregressive model structure and the adaptation network into the main text. Too many relevant details are just in the supplemental material.