All (Poincaré + gauge)-symmetry-preserving lattice theories can be constructed using a discrete derivative and its inverse (lattice integral) that seem to satisfy the same calculi of the continuum. In this sense the discrete operations are exact for which there are certain preliminary evidences. Noting that the mathematical tool needed to construct a classical field theory in the continuum is that of the Schwartz class functions and tempered distributions, we embark on attacking the full-fledged problem at hand, i.e. to develop the tempered distribution theory on the lattice. Some of the key results our construction leads to are as follows: (1) Exact operations satisfy exactly the same calculi of the continuum. Kronecker delta satisfies the same continuum distributional identities of the Dirac delta on the lattice with respect to the exact derivative. This allows one to prove Noether's theorems and derive symmetry algebras for the lattice theories. (2) Lattice configuration space (Schwartz space) corresponds to a subspace of the continuum configuration space that carries lattice cutoff as a parameter such that: (A) Lattice path integral is same as the continuum path integral over this subspace. By construction, the latter is UV divergence free. This must be true non-perturbatively. (B) The subspace is closed under Poincaré and gauge transformations. This is the root cause for the symmetries being preserved on the lattice. (C) Preliminary observations indicate that this may be a dense subspace.