Spells are any skills that spend MP (mana points) upon casting. These include Mage offensive abilities and debuffs, Priest healing abilities and buffs, the Fighter's Self Heal skill and the like. Damage-dealing and healing spells are affected by a character's stats (Int for offensive magic, Piety for healing), but buffs and debuffs aren't. Keep in mind that, unlike OD, MP does not regenerate naturally - so unlike regular abilities, spells can only be cast so many times before a player must rest.
Spells also have a casting time, which is easily interrupted by any enemy attack. Keep spell casters away from the pointy sticks. Divine Armor can prevent skills from being interupted.