Residual Representations of Semistable Principally Polarized Abelian Varieties

Let $A$ be a semistable principally polarized abelian variety of dimension $d$ defined over the rationals. Let $\ell$ be a prime and let $\bar{\rho}_{A,\ell} : G_{\mathbb{Q}} \rightarrow \mathrm{GSp}_{2d}(\mathbb{F}_\ell)$ be the representation giving the action of $G_{\mathrm{Q}} :=\mathrm{Gal}(\bar{\mathrm{Q}}/\mathrm{Q})$ on the $\ell$-torsion group $A[\ell]$. We show that if $\ell \ge \max(5,d+2)$, and if image of $\bar{\rho}_{A,\ell}$ contains a transvection then $\bar{\rho}_{A,\ell}$ is either reducible or surjective. With the help of this we study surjectivity of $\bar{\rho}_{A,\ell}$ for semistable principally polarized abelian threefolds, and give an example of a genus $3$ hyperelliptic curve $C/\mathbb{Q}$ such that $\bar{\rho}_{J,\ell}$ is surjective for all primes $\ell \ge 3$, where $J$ is the Jacobian of $C$.


Introduction
Let A be a principally polarized abelian variety of dimension d defined over Q. Let ℓ be a prime and write A[ℓ] for the ℓ-torsion subgroup of A(Q). This is a 2ddimensional F ℓ -vector space, as well as a G Q -module, where G Q := Gal(Q/Q). The polarization induces the mod ℓ Weil pairing on A[ℓ], which is a bilinear, alternating, non-degenerate pairing , : A[ℓ] × A[ℓ] → F ℓ (1) that is Galois equivariant. The latter property means σv, σv ′ = χ(σ) v, v ′ for all σ ∈ G Q , and v, v ′ ∈ A[ℓ] where χ : G Q → F × ℓ is the mod ℓ cyclotomic character. In particular, the space (A[ℓ], , ) is a symplectic F ℓ -vector space of dimension 2d. We obtain a representation ρ A,ℓ : G Q → GSp(A[ℓ], , ) ∼ = GSp 2d (F ℓ ).
The study of images of representations ρ A,ℓ has received much attention recently (e.g. [Die02], [Hal11], [AdRAK + 14]). This study is largely motivated by the following remarkable result of Serre.
Theorem 1 (Serre, [Ser13, Theorem 3]). Let A be a principally polarized abelian variety of dimension d, defined over Q. Assume that d = 2, 6 or d is odd and furthermore assume that End Q (A) = Z. Then there exists a bound B A such that for all primes ℓ > B A the representation ρ A,ℓ is surjective.
For explicit (though large) estimates for the constant B A see [Lom15]. The conclusion of the theorem is known to be false for general d; a counterexample is constructed by Mumford [Mum69] for d = 4. The following is a tantalizing open question: given d as in the theorem, is there a uniform bound B d depending only on d, such that for all principally polarized abelian varieties A over Q of dimension d with End Q (A) = Z, and all ℓ > B d , the representation ρ A,ℓ is surjective? For elliptic curves an affirmative answer is expected, and this is known as Serre's Uniformity Question, which is still an open problem. Serre's Uniformity Question is much easier for semistable elliptic curves. Indeed, Serre [Ser72,Proposition 21] shows that if E/Q is a semistable elliptic curve, and ℓ ≥ 7 is prime, then ρ E,ℓ is either surjective or reducible. It immediately follows from Mazur's classification [Maz78] of isogenies of elliptic curves over Q that ρ E,ℓ is surjective for ℓ ≥ 11. It is natural to ask if a result similar to Serre's [Ser72,Proposition 21] can be established for semistable principally polarized abelian varieties. For now, efforts to prove such a theorem are hampered by the absence of a satisfactory classification of maximal subgroups of GSp 2d (F ℓ ) (indeed, Serre's result for semistable elliptic curves makes use of Dickson's classification of maximal subgroups of GL 2 (F ℓ ) = GSp 2 (F ℓ )). There is however a beautiful classification due to Arias-de-Reyna, Dieulefait and Wiese (see Theorem 3 below) of subgroups of GSp 2d (F ℓ ) containing a transvection. A transvection is a unipotent element σ such σ − I has rank 1. The main result of our paper, building on the classification of Arias-de-Reyna, Dieulefait and Wiese, is the following theorem.
Theorem 2. Let A be a semistable principally polarized abelian variety of dimension d ≥ 1 over Q and let ℓ ≥ max(5, d + 2) be prime. Suppose the image of ρ A,ℓ : G Q → GSp 2d (F ℓ ) contains a transvection. Then ρ A,ℓ is either reducible or surjective.
The proof of Theorem 2 is given in Section 2. We mention a well-known pair of sufficient conditions (see for example [Hal11, Section 2]) for the image of ρ A,ℓ to contain a transvection. Let q = ℓ be a prime and suppose that the following two conditions are satisfied: • The special fibre of the Néron model for A at q has toric dimension 1; • ℓ ∤ #Φ q , where Φ q is the group of connected components of the special fibre of the Néron model at q. Then the image of ρ A,ℓ contains a transvection. Now let C/Q be a hyperelliptic curve of genus d, given by a model is a squarefree polynomial. Let p be an odd prime not dividing the leading coefficient of f such that f modulo p has one root in F p having multiplicity precisely 2, with all other roots simple. Then the Néron model for the Jacobian J(C) (which is a principally polarized abelian variety) at p has toric dimension 1.
The remainder of the paper is concerned with semistable principally polarized abelian threefolds A/Q which possess a prime q such that the special fibre of the Néron model for A at q has toric dimension 1. Building on Theorem 2, we give in Section 3 a practical method which should in most cases produce an explicit (and small) bound B (depending on A) such that for ℓ ≥ B, the representation ρ A,ℓ is surjective. This method is inspired by the paper of Dieulefait [Die02] which solves the corresponding problem for abelian surfaces. Our method is not always guaranteed to succeed, but we expect it to succeed if End Q (A) = Z.
It is well known that GSp 2d (F ℓ ) is a Galois group over Q for all d ≥ 1 and all primes ℓ ≥ 3 (see for example [AdRAK + 14, Remark 2.5]). The proof of this statement in fact shows the existence of a genus d hyperelliptic curve C/Q such that ρ J,ℓ is surjective, where J is the Jacobian of C. The argument however relies on Hilbert's Irreducibility Theorem, and so does not produce an explicit equation for C. In [AdRAK + 14], a genus 3 hyperelliptic curve C/Q is given so that ρ J,ℓ is surjective for all 11 ≤ ℓ < 5 × 10 5 . As a corollary to our method for producing the bound B mentioned above, we prove the following.
Recently (and independently), Zywina [Zyw15] gives a genus 3 plane quartic curve over Q for which he proves that the Galois action on the torsion subgroup of the Jacobian J is maximal (in other words, ρ J : G Q → GSp 6 (Ẑ) is surjective).
We are grateful to Tim Dokchitser for providing us with a list of genus 3 hyperelliptic curves with small Jacobian conductors, from which we chose the curve C in Corollary 1.1. We thank Jeroen Sijsling for helpful remarks on an earlier version of this paper. We are indebted to the referees for their careful reading of the paper and for many helpful remarks.

Proof of Theorem 2
We shall make use of the following classification of subgroups of GSp 2d (F ℓ ) containing a transvection, due to Arias-de-Reyna, Dieulefait and Wiese.
Theorem 3 (Arias-de-Reyna, Dieulefait and Wiese, [ADW14]). Let ℓ ≥ 5 be a prime and let V a symplectic F ℓ -vector space of dimension 2d. Any subgroup G of GSp(V ) which contains a transvection satisfies one of the following assertions: (i) There is a non-trivial proper G-stable subspace W ⊂ V .
(ii) There are non-singular symplectic subspaces V i ⊂ V with i = 1, . . . , h, of dimension 2m < 2d and a homomorphism φ : We shall apply Theorem 3 to G = ρ A,ℓ (G Q ) where A and ℓ are as in the statement of Theorem 2. It follows from the surjectivity of the mod ℓ cyclotomic character Throughout we write I ℓ for the inertia at ℓ subgroup of G Q . We shall also make use of the following theorem of Raynaud.
Theorem 4 (Raynaud, [Ray74]). Let A be an abelian variety over Q. Let ℓ be a prime of semistable reduction for A. Regard A[ℓ] as an I ℓ -module and let V be a Jordan-Hölder factor of dimension n over F ℓ . Let ψ n : I ℓ → F × ℓ n be a fundamental character of level n. Then V has the structure of a 1-dimensional F ℓ n -vector space and the action of I ℓ on it is given by a character ̟ : We shall make use of the following elementary lemma in the proof of Theorem 2.
Lemma 2.1. Let k be an algebraically closed field and V = 0 be a finite dimensional vector space over k. Let T : V → V be a k-linear map, and suppose (the indices considered modulo r). Let α be an eigenvalue of T . Then ζα is also an eigenvalue of T for every ζ in k satisfying ζ r = 1.
Proof. Let v be an eigenvector corresponding to α and write and consider the action of G on the symplectic vector space V = A[ℓ]. Since G contains a transvection we may apply Theorem 3. To prove the theorem, it is sufficient to show that case (ii) of Theorem 3 does not arise. Suppose otherwise. Then we can write . Then H = G K for some number field K/Q. Moreover, ρ| GK is reducible as the V i are stable under the action of G K . We shall show that the extension K/Q is unramified at the finite places, and thus K has discriminant ±1. It then follows by a famous theorem of Hermite that K = Q, showing that π is trivial and contradicting the fact that φ(G) = π(G Q ) is transitive.
First let p = ℓ be a prime. As A is semistable, I p acts unipotently on V . Thus ρ(σ) has ℓ-power order for σ ∈ I p . However, the order of ρ(σ) is divisible by the order of π(σ) which in turn divides h!. As h = 2d/2m ≤ d < ℓ, we see that π(σ) = 1. Thus K/Q is unramified at p.
Next, consider σ ∈ I w ℓ , the wild subgroup of I ℓ . As I w ℓ is a pro-ℓ group, ρ(σ) has ℓ-power order, and we see that π(σ) = 1 as above. Finally, let σ ∈ I ℓ be an element whose image in the tame inertia group I t ℓ = I ℓ /I w ℓ is a topological generator.
It follows that W is stable under the action of I ℓ . Let α 1 ∈ F ℓ be an eigenvalue for σ acting on W . By Lemma 2.1, we know that α 2 = ζα 1 is also an eigenvalue for σ acting on W , where ζ ∈ F ℓ is a primitive r-th root of unity (observe that this exists as r ≤ h ≤ d < ℓ). By Raynaud's Theorem, there exist n 1 , n 2 and characters ̟ j : I ℓ → F × ℓ n j such that α j = ̟ j (σ). As σ is a topological generator for the tame inertia and the characters ̟ j are surjective, we see that α 1 and α 2 have orders ℓ n1 − 1, ℓ n2 − 1 respectively. Then ζ = α 2 /α 1 has order divisible by Suppose first that n 1 = n 2 . Without loss of generality n 1 < n 2 . Then gcd This contradicts the fact that the order of ζ is r < ℓ. Thus n 1 = n 2 = n (say). Now from Raynaud's Theorem, we know that where ψ n : I ℓ → F × ℓ n is a fundamental character of level n, and 0 ≤ a i , b i ≤ 1. Since ψ n (σ) has order ℓ n − 1 and ζ = ̟ 2 (σ)/̟ 1 (σ) has order r, we see that It follows that ζ has order 1. Since ζ is a primitive r-th root of unity, we have that r = 1. From the definition of r, we have that σ(V 1 ) = V 1 . Similarly, σ(V j ) = V j for j = 2, . . . , h. Hence π(σ) = 1. As we have shown that π(I w ℓ ) = 1, and as σ is a topological generator for the tame inertia, we have that π(I ℓ ) = 1, showing that K/Q is unramified at ℓ. This completes the proof.

Surjectivity for semistable principally polarized abelian threefolds
We now let A/Q be a principally polarized abelian threefold. We shall make the following assumptions henceforth: (a) A is semistable; (b) There is a prime q such that the special fibre of the Néron model for A at q has toric dimension 1. Let S be the set of primes q satisfying (b). For q ∈ S, write Φ q for the group of connected components of the special fibre of the Néron model of A at q. We shall suppose that (c) ℓ ≥ 5; (d) ℓ does not divide gcd({q · #Φ q : q ∈ S}). Thus [Hal11, Section 2] the image of ρ A,ℓ contains a transvection. It follows from Theorem 2 that ρ A,ℓ is either reducible or surjective. In this section we explain a practical method which should in most cases produce a small integer B (depending on A) such that for ℓ ∤ B, the representation ρ A,ℓ is irreducible and hence surjective.
Determinants of Jordan-Hölder factors. As before χ : G Q → F × ℓ denotes the mod ℓ cyclotomic character. We will study the Jordan-Hölder factors W of the G Q -module A[ℓ]. By the determinant of such a W we mean the determinant of the induced representation G Q → GL(W ).
Lemma 3.1. Any Jordan-Hölder factor W of the G Q -module A[ℓ] has determinant χ r for some 0 ≤ r ≤ dim(W ).
Proof. Let W be such a Jordan-Hölder factor, and let ψ : G Q → F × ℓ be its determinant. As J is semistable, for primes p = ℓ, the inertia subgroup I p ⊂ G Q acts unipotently on W and so ψ| Ip = 1. Moreover, by considering the Jordan-Hölder factors of W as an I ℓ -module, it follows from Raynaud's Theorem that ψ| I ℓ = χ r | I ℓ for some 0 ≤ r ≤ dim(W ). Thus the character ψχ −r is unramified at all the finite places. As the narrow class number of Q is 1, we see that ψχ −r = 1 proving the lemma.
Weil polynomials. For a prime p = ℓ of good reduction for A, we shall henceforth write for the characteristic polynomial of Frobenius σ p ∈ G Q at p acting on the Tate module T ℓ (A) (also known as the Weil polynomial of A mod p). The polynomial P p is independent of ℓ. It follows from (2) that the roots in F ℓ have the form u, v, w, p/u, p/v, p/w.
Lemma 3.2. If P p has a real root then (x 2 − p) 2 is a factor of P p .
Proof. By Weil, the complex roots have the form ω 1 , ω 2 , ω 3 , ω 1 , ω 2 , ω 3 where |ω i | = √ p and ω denotes the complex conjugate of ω. Suppose ω 1 is real. Then 1-dimensional Jordan-Hölder factors. Let T be a non-empty set of primes of good reduction for A. Let does not have any 1dimensional or 5-dimensional Jordan-Hölder factors.

Proof. As dim(A[ℓ]) = 6, if A[ℓ]
has a 5-dimensional Jordan-Hölder factor then it has a 1-dimensional Jordan-Hölder factor. Suppose W is a 1-dimensional Jordan-Hölder factor of A[ℓ]. Then the action of G Q on W is given by a character ψ : G Q → F × ℓ . It follows from Lemma 3.1 that ψ = 1 or χ. Let p be a prime of good reduction for A, and suppose ℓ = p. Thus P p has root 1 or χ(σ p ) = p ∈ F ℓ . Since the roots of P p have the form u, v, w, p/u, p/v, p/w, we know in either case that 1 is a root, and so #A(F p ) = P p (1) ≡ 0 (mod ℓ).
Thus if p is a prime of good reduction for A, then ℓ divides p · #A(F p ). This proves the lemma.
Since #A(F p ) > 0, we have B 1 (T ) = 0, and so we can always rule out 1dimensional and 5-dimensional factors for large ℓ. Proof. Suppose dim(U ) = 2. If the restriction of the Weil pairing to U is nondegenerate then det(U ) = χ and we can take W = U . Thus we may suppose that the restriction of the Weil pairing to U is degenerate. Thus U ∩ U ⊥ = 0, where The Galois invariance of the Weil-pairing implies that U ⊥ is a G Q -submodule of A[ℓ]. Since U is irreducible and U ∩ U ⊥ = 0 we have that U ⊆ U ⊥ . However U ⊥ is 4-dimensional. Thus each of the 2-dimensional quotients in the sequence 0 ⊂ U ⊂ U ⊥ ⊂ A[ℓ] is 2-dimensional, must be irreducible (as A[ℓ] does not have 1-dimensional factors) and has determinant 1 or χ or χ 2 by Lemma 3.1. Since det(A[ℓ]) = χ 3 we see that one the three quotients must have determinant χ. This completes the proof for the case dim(U ) = 2. Now suppose that dim(U ) = 4. If the restriction of the Weil pairing to U is degenerate, then U ⊆ U ⊥ as before; this is impossible as dim(U ⊥ ) = 2. It follows that the restriction of the Weil pairing to U is non-degenerate and so det(U ) = χ 2 . As det(A[ℓ]) = χ 3 , we have that A[ℓ]/U is an irreducible 2-dimensional G Q -module with determinant χ. This completes the proof.  Proof. By Lemmas 3.3 and 3.4 it is enough to rule out the existence of a 2dimensional Jordan-Hölder factor with character χ. This follows from the above discussion.
Of course we fail to bound ℓ in the above lemma if R(M, p) = 0 for all primes p of good reduction. Here are two situations where this can happen: • Suppose A is isogenous over Q to E × A ′ where E is an elliptic curve and A ′ an abelian surface. If we take M | N to be the conductor of the elliptic curve, and f to be the newform associated to E by modularity, then . Thus the resultant R(M, p) = 0 for all p ∤ N . • Suppose the abelian threefold A is of GL 2 -type. It is therefore modular by Khare and Wintenberger [KW09], and if we let f be the corresponding eigenform, then again , and so the resultant R(M, p) = 0 for all p ∤ N . Note that in both these situations End Q (A) = Z. We expect, but are unable to prove, that if End Q (A) = Z then there will be primes p such that R(M, p) = 0.

3-dimensional Jordan-Hölder factors.
Lemma 3.6. Suppose A[ℓ] has Jordan-Hölder filtration 0 ⊂ U ⊂ A[ℓ] where both U and A[ℓ]/U are irreducible and 3-dimensional. Moreover, let u 1 , u 2 , u 3 be a basis for U , and let G Q → GL 3 (F ℓ ), σ → M (σ) give the action of G Q on U with respect to this basis. Then we can extend u 1 , u 2 , u 3 to a symplectic basis u 1 , u 2 , u 3 , w 1 , w 2 , w 3 for A[ℓ] so that the action of G Q on A[ℓ] with respect to this basis is given by Proof. Any bilinear alternating pairing on an odd dimensional space (in characteristic = 2) must be degenerate. We thus deduce that U ⊆ U ⊥ as in the proof of Lemma 3.4. As both spaces have dimension 3, we have U = U ⊥ . Let u 1 , u 2 , u 3 be a basis for U . Then u i , u j = 0. Extend this to a symplectic basis u 1 , u 2 , u 3 , w 1 , w 2 , w 3 for A[ℓ]: meaning that in addition to u i , u j = 0 the basis satisfies w i , w j = 0, and u i , w j = δ i,j where δ i,j is the Kronecker delta. The lemma follows from the identity u i , σw j = χ(σ) σ −1 u i , w j for σ ∈ G Q . Now let U be as in Lemma 3.6. By Lemma 3.1, we have that det(U ) = χ r and det(A[ℓ]/U ) = χ s where 0 ≤ r, s ≤ 3. Moreover, as det(A[ℓ]) = χ 3 we have that r + s = 3.
Lemma 3.7. Let p be a prime of good reduction for A. For ease write α, β and γ for the coefficients α p , β p , γ p in (2). Suppose p + 1 = α (this is certainly true for p ≥ 36 as |α| ≤ 6 √ p). Let Write k for the greatest common divisor of the numerators of the coefficients in does not have a Jordan-Hölder filtration as in Lemma 3.6 with det(U ) = χ or χ 2 .
Lemma 3.8. Let p be a prime of good reduction for A. Write α, β and γ for the coefficients α p , β p , γ p in (2). Suppose p 3 + 1 = pα (this is true for p ≥ 5 as |α| ≤ 6 √ p). Let Write k ′ for the greatest common divisor of the numerators of the coefficients in does not have a Jordan-Hölder filtration as in Lemma 3.6 with det(U ) = 1 or χ 3 . Proofs of Lemma 3.7 and 3.8. For now let p be a prime of good reduction for A, and suppose that ℓ = p. Suppose A[ℓ] has a Jordan-Hölder filtration 0 ⊂ U ⊂ A[ℓ] where U and A[ℓ]/U are 3-dimensional (i.e. as in Lemma 3.6). Then det(U ) = χ r with 0 ≤ r ≤ 3. Let σ p ∈ G Q denote a Frobenius element at p. Let M = M (σ p ) as in Lemma 3.6. Then det(M ) = p r ∈ F ℓ . Moreover, from the lemma, If r = 1 or 2 then If r = 0 or 3 then We now suppose that ℓ ∤ K p and prove Lemma 3.7 which corresponds to r = 1 or 2. We thus suppose that (9) holds. Comparing the coefficients of x 5 in (9) we have that a ≡ b + α (mod ℓ). Substituting this into (9) and comparing the coefficients of x 4 and x 3 we obtain Eliminating b 2 we obtain the following congruence which is linear in b: As ℓ ∤ K p we have ℓ ∤ (p − 1)(p + 1 − α), and so we can solve for b mod ℓ. It follows that b ≡ δ and a ≡ b + α ≡ ǫ (mod ℓ) where δ and ǫ are given by (5). Substituting into (9), we see that P p ≡ g (mod ℓ) where g is given by (6). Thus ℓ divides the greatest common divisor of the numerators of the coefficients of P p − g showing that ℓ | k (in the notation of Lemma 3.7). As k | K p and ℓ ∤ K p we obtain a contradiction. Thus if ℓ ∤ K p then A[ℓ] does not have a Jordan-Hölder filtration as in Lemma 3.6 with det(U ) = χ or χ 2 . We need to show that K p = 0. We are supposing that p + 1 = α thus we need to show that P p = g. Suppose P p = g. As g is the product of two cubic polynomials, it follows that P p has at least two real roots. By Lemma 3.2, we see that (x 2 − p) 2 | P p . It follows that P p = g must have two rational roots. Since all the roots have absolute value √ p, this is a contradiction. We deduce that K p = 0 as required. This completes the proof of Lemma 3.7. The proof of Lemma 3.8 is practically identical.
Summary. The following theorem summarizes Section 3.
Theorem 5. Let A and ℓ satisfy conditions (a)-(d) at the beginning of Section 3. Let T be a non-empty set of primes of good reduction for A. Let where B 2 (T ) is as in Lemma 3.5. If ℓ ∤ B(T ) then ρ A,ℓ is surjective.
Proof. By Theorem 2 we know that ρ A,ℓ is either reducible or surjective. Lemmas 3.5, 3.7 and 3.8 ensure that ρ A,ℓ cannot be reducible. Hence it must be surjective.

Proof of Corollary 1.1
We implemented the method described in Section 3 in Magma [BCP97]. The model given in (1) for the curve C has good reduction at 2. Let J be the Jacobian of C. This has conductor N = 8907 = 3 × 2969 (the algorithm used by Magma for computing the conductor is described in [DDMM15]). As N is squarefree, the Jacobian J is semistable. Completing the square in (1) we see that the curve C has the following 'simplified' Weierstrass model. y 2 = x 8 + 2x 7 + 5x 6 + 6x 5 + 4x 4 + 2x 3 + x 2 + 2x + 1.
Here the non-linear factors in both factorizations are irreducible. As f has precisely one double root in F 3 and one double root in F 2969 with all other roots simple, we see that the Néron models for J at 3 and 2969 have special fibres with toric dimension 1. We found that #Φ 3 = #Φ 2969 = 1. Thus the image of ρ J,ℓ contains a transvection for all ℓ ≥ 3.
It remains to show that ρ J,3 is surjective. Denote ρ = ρ J,3 . Write G = ρ(G Q ). For a prime p of good reduction, let σ p ∈ G Q denote a Frobenius element at p and P p ∈ F 3 [t] be the characteristic polynomial of σ p acting on J[3]. Let N p be the multiplicative order of the image of t in the algebra F 3 [t]/P p . It is immediate that N p divides the order of ρ(σ p ) and hence divides the order of G. We computed N 2 = 2 3 × 5, N 5 = 2 × 13, N 19 = 7, N 37 = 2 × 3 2 .
Thus the order of G is divisible by 2 3 × 3 2 × 5 × 7 × 13. We checked that the only subgroups of GSp 6 (F 3 ) with order divisible by this are Sp 6 (F 3 ) and GSp 6 (F 3 ). As the mod 3 cyclotomic character is surjective on G Q , we have that G = GSp 6 (F 3 ). This completes the proof of the corollary.