The author develops a rigorous second order analysis on the space of probability measures on a Riemannian manifold endowed with the quadratic optimal transport distance $W_2$. The discussion includes: definition of covariant derivative, discussion of the problem of existence of parallel transport, calculus of the Riemannian curvature tensor, differentiability of the exponential map and existence of Jacobi fields. This approach does not require any smoothness assumption on the measures considered.