Beyond cryptography, there is the fundamental theorem of arithmetic, which plays an important role in encoding Gödel numbers in Gödel's theorem.
In algebra, the integers mod p are a finite field (addition, subtraction, multiplication, and division are defined) if and only if p is prime.
Primality in algebra also exists in a more general form with prime ideals. An ideal is the set of elements of a commutative ring in which any element of the ideal multiplied by any element of the ring is still in the ideal; there is a sort of 'closed' property. Even numbers form an ideal because if you multiply any number by an even number, you get an even number. For a prime ideal, if ab is in the ideal, a is in the ideal or b is (similar to how if a prime number divides ab, it divides a or b).
They have a number of interesting properties. For example, for a ring homomorphism (a function from one ring to another that preserves relationships between elements of the two rings), the pre-image of a prime ideal is also a prime ideal.