We present the first results for masses of spin-3/2 N and Δ baryons in lattice QCD using fat-link irrelevant clover fermions. Spin-3/2 interpolating fields, providing overlap with both spin-3/2 and spin-1/2 states, are considered. In the isospin-1/2 sector we observe, after appropriate spin and parity projection, a strong signal for the J^P=3/2^− state together with a weak but discernible signal for the 3/2^+ state with a mass splitting near that observed experimentally. We also find good agreement between the 1/2^± masses and earlier nucleon mass simulations with the standard spin-1/2 interpolating field. For the isospin-3/2 Δ states, clear mass splittings are observed between the various 1/2^± and 3/2^± channels, with the calculated level orderings in good agreement with those observed empirically.