The paper describes a new method for selecting Homomorphic Encryption scheme parameters used for Linear layers in a secure inference setting. The paper explores an interesting combination of the particulars of homomorphic encryption and deep neural networks and is able to provide strong decryption correctness guarantees.