We present a study on the nuclear and magnetic structures of two iron-based garnets with magnetic cations isolated on tetrahedral sites. Ca$$_2$$YZr$$_2$$Fe$$_3$$O$$_{12}$$ and Ca$$_2$$LaZr$$_2$$Fe$$_3$$O$$_{12}$$ offer an interesting comparison for examining the effect of increasing cation size within the diamagnetic backbone of the garnet crystal structure, and how such changes affect the magnetic order. Despite both systems exhibiting well-pronounced magnetic transitions at low temperatures, we also find evidence for diffuse magnetic scattering due to a competition between the nearest-neighbor, next nearest-neighbor, and so on, within the tetrahedral sites. This competition results in a complex noncollinear magnetic structure on the tetrahedral sublattice creating a mixture of ferro- and antiferromagnetic interactions above the long-range ordering temperature near 20 K and suggests that the cubic site of the garnet plays a significant role in mediating the superexchange interactions between tetrahedral cations.