Haystack hunting hints and locker room communication

We want to efficiently find a specific object in a large unstructured set, which we model by a random n$$ n $$‐permutation, and we have to do it by revealing just a single element. Clearly, without any help this task is hopeless and the best one can do is to select the element at random, and achieve the success probability 1n$$ \frac{1}{n} $$ . Can we do better with some small amount of advice about the permutation, even without knowing the target object? We show that by providing advice of just one integer in {0,1,…,n−1}$$ \left\{0,1,\dots, n-1\right\} $$ , one can improve the success probability considerably, by a Θ(lognloglogn)$$ \Theta \left(\frac{\log n}{\mathrm{loglog}n}\right) $$ factor. We study this and related problems, and show asymptotically matching upper and lower bounds for their optimal probability of success. Our analysis relies on a close relationship of such problems to some intrinsic properties of random permutations related to the rencontres number.


FIGURE 1
Consider the above randomly shuffled deck, one card per locker. What advice should Alice give to Bob-just by swapping the locations of at most one pair of cards-to increase the probability that Bob will find his randomly chosen card by opening at most two lockers?
x 0 , … , x n−1 according to a random permutation of S. This is the haystack, and you want to find one specific object, not surprisingly called the needle, by drawing from just one location. We will take the set S to be [n] = {0, 1, … , n − 1}.
Clearly, the probability of finding this object in a single draw is always 1 n , whichever location you choose. Since the permutation is random, the probability that your object is there is exactly 1 n . But can I give you any advice or hint about -without knowing which object you are seeking-to improve your chance of finding ? If I could tell you the entire , which can be encoded with log(n!) = Θ(n log n) bits, then this task is trivial and you would know the location of . But what if I give you just a small hint (on the basis of ), one number from [n] (or equivalently, one log n-bit sequence)-even when I know nothing about the target object?
Formally, the goal is to design a strategy to choose a hint = ( ) and an index = ( , ), with both , ∈ [n], such that for a given ∈ [n], Pr is maximized, where the probability is over the random choice of .

Related puzzle: Communication in the locker room
The needle in a haystack problem is closely related to the following locker room problem(see Figure 1): The locker room has n lockers, numbered 0, … , n − 1. A set of n cards, numbered 0, … , n − 1, is inserted in the lockers according to a uniformly random permutation . Alice and Bob are a team with a task. Before the game begins, Alice and Bob may communicate to decide on a strategy. Alice enters the locker room, opens all the lockers and can swap the cards between just two lockers, or may choose to leave them unchanged. She closes all the lockers and leaves the room. Bob is given a number ∈ [n] and his task is to find card . He can open at most two lockers. What is their optimal strategy, and how efficient is it? As in the needle in a haystack problem, without help from Alice, Bob can do no better than open lockers at random. If he opens one locker his probability of success is 1 n and if he opens two lockers this probability is 2 n . With the help of Alice, he can do better when opening one locker. For example, their strategy could be that Bob will open locker , where is his given number. Alice would then try to increase the number of fixed points in the permutation above the expected number of 1. If there is a transposition she can reverse it, increasing the number of fixed points by two, and if not she can produce one more fixed point (unless the permutation is the identity). This strategy succeeds with probability just under 12 5n . When Bob can open two lockers, the challenge is to increase the success probability to ( 1 n ) . The answer involves viewing Bob's first locker opening in a different way: not as looking for his card but as receiving a communication from Alice. The interest is in finding what kind of information Alice can send about the permutation which could help Bob in his search. Now, we invite the reader to stop for a moment: to think about this puzzle, to find any strategy that could ensure the success probability would be . It is easy to see that a solution to the needle in a haystack problem immediately yields a solution to the locker room problem: Alice just takes the card corresponding to the advice and swaps it into the first locker. For example, the shuffled deck from Figure 1 corresponds to the following permutation : (0, 1, … , 19) = (13,10,14,15,4,17,2,9,7,19,5,1,6,11,16,18,8,0,3,12).
If in the needle in a haystack problem the advice is a number ∈ [n], then Alice swaps the contents of locker 0 and the locker containing the card corresponding to number . This way, Bob gets the advice by opening locker 0.
For the strategy we propose in Theorem 7, Alice would swap cards 13 and 8. But can we do better?

Results for the needle in a haystack and locker roomproblems
We present a tight analysis of the needle in a haystack problem. While some basic examples suggest that it is difficult to ensure success probability , we will show that one can improve this probability considerably. Our main results are tight (up to lower order terms) lower and upper bounds for the maximum probability that with a single number hint one can find the target object. First, we will show that for any strategy, this probability is at most (1+o(1)) log n n log log n (Theorem 5). As the main result of this paper, we will complement this by designing a simple strategy that with a hint ensures that the target is found with probability at least (1+o(1)) log n n log log n (Theorem 7). Further, we demonstrate essentially the same results for the locker room problem. Theorem 7 for the needle in a haystack problem immediately implies that there is a simple strategy for Alice and Bob which ensures that Bob finds his card with probability at least (1+o(1)) log n n log log n . We will complement this claim, and extend in Theorem 21 the result from Theorem 5 for the needle in a haystack problem, to prove that for any strategy for Alice and Bob, the probability that Bob finds the required card is at most O ( log n n log log n ) .
Techniques. Our analysis exploits properties of random permutations to ensure that some short advice can reveal information about the input permutation, which can be used to increase the success probability substantially. Our approach depends on intrinsic properties of random permutations related to the rencontres number, the number of n-permutations with a given number of fixed points.
To show the upper bound for the success probability (Theorem 5), we observe that every deterministic strategy corresponds to a unique partition of S n (set of all permutations of [n]) into n parts, with part containing those permutations that cause the choice of hint . By a careful analysis of the properties of this partition, we devise a metric for the best possible accuracy of the prediction, counting instances in each part of the partition in which a permutation maps a given choice to . By combining these estimates with the bounds for the rencontres number, we prove the upper bound for the success probability in the needle in a haystack problem.
To show the lower bound for the success probability (Theorem 7), we present a simple shift strategy, and then provide a non-trivial analysis of random permutations that demonstrates desirable properties of this strategy. The analysis here is related to the maximum load problem for balls and bins, where one allocates n balls into n bins, chosen independently and uniformly at random (i.u.r.). However, the dependencies between locations of distinct elements in the random permutations make this analysis more complex (see Remark 11 for more detailed discussion).
Finally, while a solution to the needle in a haystack problem immediately yields a solution to the locker room problemwith the same success probability, we complement our analysis by showing (Theorem 21) that no strategy of Alice and Bob can do much better. We show that Alice can do little more than just to send a few numbers to Bob, which is essentially the setup of the needle in a haystack problem.

1.3
Background: Permutations, puzzles, and locker rooms Our locker room problemfollows a long line of combinatorial puzzles involving the analysis of properties of permutations. One such example is the following locker problem involving prisoners and lockers: There are n lockers into which a random permutation of n cards are inserted. Then n prisoners enter the locker room one at a time and are allowed to open half the lockers in an attempt to find their own card. The team of prisoners wins if every one of them is successful. The surprising result is that there is a strategy which wins with probability about 1 − ln 2. This problem was initially considered by Peter Bro Miltersen and appeared in his paper with Anna Gál [8], which won a best paper award at ICALP 2003. In that paper they refer to a powerful strategy approach suggested by Sven Skyum but it was left to the readers to find it for themselves. This is the idea of using the number contained in each locker as a pointer to another locker. Thus using a sequence of such steps corresponds to following a cycle in the permutation. Solutions to these problems are of a combinatorial and probabilistic flavor and involve an analysis of the cycle structure of random permutations. The original paper [8] stimulated many subsequent papers considering different variants (see, for example, [4,5,9]), including a matching upper bound provided in [6]. An early version giving the problem where each prisoner can open half the lockers was published by [15] (see also [16, p. 18]). If each prisoner begins with the locker corresponding to the number they seek then they will all succeed provided that there is no cycle in the permutation which is longer than n 2 . It is easy to show that a helpful prison warder, Alice, can always find an appropriate transposition of the contents of two lockers so that the resulting permutation has no cycle longer than n 2 . We were told of this observation recently by Kazuo Iwama and this stimulated the current paper, in which we subvert the locker problem tradition with a problem which has little to do with the cycle structure of permutations and is more concerned with some basic communication complexity and rather different properties of permutations.
Various results about permutations have found diverse applications in computer science, especially for sorting algorithms (e.g., see [11, chapter 5]). In this article, we are particularly interested in rencontres numbers. Firstly, to apply known results concerning their asymptotic growth, in order to approximate the optimal success probabilities in both the needle in a haystack problem and the locker room problem. Secondly, to use them to examine the way in which the sizes of "shift sets" (sets of elements which a permutation displaces by the same number of positions "to the right") are distributed in permutations of S n for a fixed natural number n. In particular, to determine the mean size of the largest shift set of a permutation chosen uniformly at random from S n , as well as to show that it is typical, that is, that the variance of the size of the largest shift set is small. These results are useful for providing a concrete optimal strategy for both of the titular search problems.

Formal framework and justification about worst-case versus random
We consider the problem with two inputs: a number ∈ [n] and a permutation ∈ S n . We are assuming that is a random permutation in S n ; no assumption is made about . For the needle in a haystack problem (a similar framework can be easily set up for the locker room problem), a strategy (or an algorithm) is defined by a pair of functions, = ( ) and = ( , ), with both , ∈ [n].
For a fixed strategy, let ( ) be the success probability for a given and for a randomly chosen ∈ S n . That is, where the probability is over taken i.u.r. from S n . The goal is to design an algorithm (find a strategy) that will achieve some given success probability for every ∈ [n]. That is, we want to have a strategy which maximizes In our analysis for the upper bounds in Sections 2 and 3 (Theorem 5) and Section 6 (Theorem 21), for simplicity, we will be making the assumption that (the input to the needle in a haystack problem and to the locker room problem) is random, that is, is chosen i.u.r. from [n]. (We do not make such assumption in the lower bound in Section 4 (Theorem 7), where the analysis is done explicitly for arbitrary .) Then the main claim (Theorem 5) is that if we choose i.u.r. then ( ) ≤ (1+o(1)) log n n log log n . Observe that one can read this claim equivalently as that ∑ ( ) n ≤ (1+o(1)) log n n log log n . However, notice that this trivially yields and therefore Theorem 5 yields Pr[] ≤ (1+o(1)) log n n log log n , as required. Note that such arguments hold only for the upper bound. Indeed, since min ∈[n] { ( )} may be much smaller than ∑ ( ) n , in order to give a lower bound for the success probability, Theorem 7 proves that there is a strategy that ensures that ( ) ≥ (1+o(1)) log n n log log n for every ∈ [n]; this clearly yields Pr[] ≥ (1+o(1)) log n n log log n , as required.

Describing possible strategies for needle in a haystack
In this section, we prepare a framework for the study of strategies to prove an upper bound for the success probability for the needle in a haystack problem (see Section 3). First, we rephrase the original problem as an equivalent communication game between Alice and Bob. Bob, the seeker, has as his input a (random) number ∈ [n]. Alice, the adviser, sees a permutation chosen i.u.r. from S n , and uses to send advice to Bob in the form of a number ∈ [n]. Bob does not know , but on the basis of and , he picks some ∈ [n] trying to maximize the probability that ( ) = .
First we will consider deterministic strategies, and will later argue separately that randomized strategies cannot help much. For deterministic strategies, the advice sent is a function S n → [n], which can be defined by a partition of S n into n sets.

Definition 1.
A strategy C for S n is a partition of S n into n sets C 0 , C 1 , … , C n−1 . Such a strategy C is denoted by C = ⟨C 0 , C 1 , … , C n−1 ⟩.
Given a specific strategy C, we examine the success probability. Let  be the event that the target is found,  the event that is the received advice, and  the event that is the target. (1) The magneticity of an element i for an element k in the class C j is defined as The element with the greatest magneticity for k in the class C j is called the magnet in C j of k and is denoted max-mag(C j , k); ties are broken arbitrarily. The magneticity of max-mag( Let us discuss the intuitions. Firstly, the magneticity in the class C j of an element i for an element k, mag(C j , i, k), denotes the number of permutations in C j with k in position i. Therefore, the magnet in C j of k is an index i ∈ [n] such that, among all permutations in C j , k is most likely to be in position i.
The intensity in C j of k denotes just the number of times (among all permutations in C j ) that k appears in the position of the magnet i.
In the needle in a haystack problem, Alice sends to Bob a message which points to a class C of their agreed strategy C, and Bob has to choose a number in order to find whether ( ) is the number ∈ [n] which he seeks. The maximum probability that they succeed is |C | , realized if Bob opts for the magnet of in C . Thus, by (1), we obtain int(C , ).
With this definition, a strategy which yields the field of S n is called optimal, and We will use this bound to prove Theorem 5 in Section 3, that whatever the strategy, we always have Pr[] ≤ (1+o(1))⋅log n n log log n .

Derangements
We use properties of random permutations related to derangements and rencontres numbers.

Definition 4.
A permutation ∈ S n with no fixed points is called a derangement. The number of derangements in S n is denoted D n . A permutation ∈ S n with exactly r fixed points is called an r-partial derangement. The number of r-partial derangements in S n (also known as the rencontres number) is denoted D n,r .
Definition 4 yields D n,0 = D n and it is easy to see that D n,r = ( n r ) D n−r . It is also known (see, e.g., [10, p. 195]) that D n = ⌊ n! e + 1 2 ⌋, and hence one can easily show D n,r ≤ n! r! .

UPPER BOUND FOR THE SUCCESS PROBABILITY FOR NEEDLE IN A HAYSTACK
We will use the framework set up in the previous section, in particular the tools in Definition 2 and inequality (2) and that is chosen i.u.r. from [n], to bound from above the best possible success probability for the needle in a haystack problem.

Theorem 5. For any strategy in the needle in a haystack problem, the success probability satisfies
Proof. Consider a strategy C = ⟨C 0 , … , C n−1 ⟩. For a permutation ∈ C j to contribute at least r to ∑ i∈[n] int(C j , i), −1 must map at least r elements to their magnets in C j . Hence, there are at most ( , so all in all there are at most n ⋅ n! r! permutations that contribute at least r to F(n). Let X be the random variable measuring the contribution to F(n) of a permutation chosen uniformly at random from S n . Then for any natural , We choose some = (1+o(1)) log n log log n to ensure that n = o( !), yielding ( + s)! ≥ ( + 1) s ! ≥ ( + 1) s n for every s ∈ N, whence we obtain We can combine (2) and (13) to obtain the following, ▪ Remark 6. The upper bound of (1+o(1)) log n n log log n is valid not only for deterministic strategies, but also for randomized strategies. Let c(C, ( , i)) be the indicator function of the event that the strategy C fails to guess the image of i under the permutation . Let us consider a probability measure P over the set D of all deterministic strategies, and the distribution U = ( , where U S denotes the uniform probability measure over the set S. Let S be a random strategy chosen according to P, and let X be a random set-up chosen according to U. Then, by Yao's principle, max . That is, the probability that a randomized strategy fails for the worst-case input exceeds the probability that an optimal deterministic strategy fails. Hence, the worst-case probability that a randomized strategy succeeds is also bounded above by (1+o(1)) log n n log log n .

LOWER BOUND: SOLUTION FOR THE NEEDLE IN A HAYSTACK SEARCH
In Theorem 5, we showed that whatever strategy we use in the needle in a haystack problem, the best success probability we can hope for is (1+o(1)) log n n log log n . In this section we will show that such success probability is achievable by a simple strategy, which we call the shift strategy.
(Observe that our choice of is equivalent to maximizing |{ ∈ [n] ∶ ( − (mod n)) = ( )}|. ) We will prove that the shift strategy ensures a success probability of at least (1+o (1)) log n n log log n . Notice that this is equivalent to saying that Pr (1)) log n n log log n , and hence, by the definition of , that with probability This also implies, by Theorem 5 (Section 3), that the shift strategy is asymptotically optimal.
In order to prove Theorem 7, we introduce some notation.
{S }. Therefore, our goal is to study basic properties of the distribution of S , and in particular, to estimate the largest value of S j over all j ∈ [n]. Example 1. Using the example presented in Figure 1 with  Lemma 8 tells us that in expectation, there is at least one value j such that S j ≥ (1+o(1)) log n log log n . Notice however that in principle, we could have that this expectation is high but only because with small probability the random variable takes a very high value. Therefore the bound in Lemma 8 is fairly weak. We will now prove, using the second moment method, that with high probability there is some j such that S j ≥ (1+o(1)) log n log log n . This yields Theorem 7. Proof.
With this notation, our goal is to show that R t = 0 is unlikely for our choice of some t = (1+o(1)) log n . We use the second moment method relying on a standard implication of Chebyshev's inequality, Let us recall that Next, since every Z t j is a 0-1 random variable, we obtain the following, Our main technical claim is that the covariance of random variables Z t j , Z t i is small. Although the proof of Lemma 10 is the main technical contribution of this section, for clarity of presentation, we defer its proof to Section 5.

Lemma 10. Let t ≤ O(log n). Then, the following holds for any
Therefore, if we combine (6) and Lemma 10 in identity (5), then (assuming t ≤ O(log n)) If we plug this into (4), we will get the following (assuming t ≤ O(log n)), > n 2et! , which is (1) for any t such that t! = o(n). This in particular holds for some t = (1+o(1)) log n log log n , and thus concludes Lemma 9. ▪ Remark 11. A reader may notice a close similarity of the problem of estimating max i∈[n] S i to the maximum load problem for balls and bins, where one allocates n balls into n bins i.u.r. Indeed, random variables S 0 , … , S n−1 have similar distribution to the random variables B 0 , … , B n−1 , where B i represents the number of balls allocated to bin i. However, the standard approaches used in the analysis of balls-and-bins processes seem to be more complicated in our setting. The main reason is that while every single random variable S i has approximately Poisson distribution with mean 1, as has B i , the analysis of max i∈[n] S i is more complicated than the analysis of max i∈[n] B i because of the intricate correlation of random variables S 0 , … , S n−1 . For example, one standard approach to show that max i∈[n] B i ≥ (1+o(1)) log n log log n with high probability relies on the fact that the load of a set of bins B i with i ∈ I decreases if we increase the load of bins B j with j ∈ J, I ∩ I = ∅. However, the same property holds only approximately for S 0 , … , S n−1 (and in fact, the o(1) error term in Lemma 10 corresponds to this notion of "approximately"; for balls and bins the covariance is known to be always non-positive). To see the difficulty (see also the classic reference for permutations [14, chapters 7-8]), notice that, for example, if (i) = i + then we cannot have (i + 1) = i + , meaning that there is a special correlation between S (which counts i with (i) = i + ) and S −1 (which counts i with (i + 1) = i + ). In particular, from what we can see, random variables S 0 , … , S n−1 are not negatively associated [7]. In a similar way, we do not expect the Poisson approximation framework from [1] (see also [12, chapter 5.4]) to work here. Our approach is therefore closer to the standard second moment method, see, for example, [2, chapter 3] and [13].

PROOF OF LEMMA 10: BOUNDING THE COVARIANCE OF Z t i AND Z t j
The main technical part of the analysis of the lower bound for the needle in a haystack problem in Section 4 (see Theorem 7) relies on the proof of Lemma 9. This proof, in turn, is quite simple except for one central claim, Lemma 10, bounding the covariance of Z t i and Z t j . The proof of Lemma 10 is rather lengthy, and therefore for the convenience of the reader the proofs of some lemmas are deferred to Section A.
Let Z t j be the indicator random variable that S j = t. Since Z t i and Z t j are 0-1 random variables, we have E , to complete the proof of Lemma 10, we only have to show that, for i ≠ j, We will prove this claim with Lemma 19 in Section 5.2.4 below.

Notation and key intuitions
For any set I ⊆ [n] and any integer ∈ [n], let  I, = { ∈ S n ∶ (i) = i + (mod n) iff i ∈ I} and Next, with this notation and for i ≠ j, we also have Notice that in the sum above one can restrict attention just to I ∩ J = ∅, since  I,i ∩  J,j = ∅ otherwise.
In view of this, our goal is to estimate In what follows, we will consider sets S i and S j with i = 0 and j = s for some s ∈ [n] ⧵ {0}. By symmetry, we can consider the first shift to be 0 without loss of generality; s is an arbitrary non-zero value. As required in our analysis (see Lemma 10), we will consider t ≤ O(log n).
Our approach now is to focus on a typical pair I and J, and consider some atypical pairs separately. We will show in Lemma 13 that almost all pairs of disjoint sets I and J are so-called compatible for shift s. As a result, the contribution of pairs I and J that are not compatible for s is negligible, and so we will focus on pairs compatible for s. Then, for the pair of indices I and J we will estimate | I,i ∩ J,j | using the Principle of Inclusion-Exclusion. For that, we must consider the contributions of all possible sets K ⊆ [n] ⧵ (I ∪ J) to the set of permutations in  * I,i ∩  * J,j . As before, contributions of some sets are difficult to be captured and so we will show in Lemma 15 that almost all sets K ⊆ [n] ⧵ (I ∪ J) are so-called feasible for I, J, and s. As a result, the contribution of sets K that are not feasible for I, J, and s is negligible, and so we will focus on sets that are feasible for I, J, and s. The final simplification follows from the fact that we do not have to consider all such sets K, but only small sets K, of size O(log n). Once we have prepared our framework, we can use the Principle of Inclusion-Exclusion to estimate | ⋃ I,J⊆[n],|I|=|J|=t  I,i ∩  J,j | in Lemmas 18 and 19.

The analysis
For an integer and subset L ⊆ [n], we use L + to denote the set of elements in L shifted by , in arithmetic modulo n, that is, However, our main goal, to compute the size of Φ 0,s (I, J), is significantly more complicated, as this quantity cannot be reduced to an intersection test and a simple formula over n, |I|, |J|, and s. Proof. If I and J are compatible for shift s then any permutation ∈ S n with (i) = i for all i ∈ I, (j) = j + s (mod n) for all j ∈ J and complemented by an arbitrary permutation [n] ⧵ (I ∪ J) is in Φ * 0,s (I, J). Hence the claim follows from the fact that since I, J, and J + s are pairwise disjoint, such permutations always exist. ▪ The following lemma shows that almost all pairs of disjoint sets of size t ≤ O(log n) are compatible (see A.2 for a proof).

Lemma 13. Let s be an arbitrary nonzero integer in [n]. If we choose two disjoint sets I, J ⊆ [n]
of size t i.u.r., then the probability that I and J are compatible for shift s is at least To justify this definition, we begin with the following simple lemma (see A.3 for a proof). Next, similarly to Lemma 13, we argue that almost all suitably small sets are feasible for pairs of disjoint small sets (see A.4 for a simple proof). [n]. Let I and J be a pair of compatible sets for s with |I| = |J| = t. Let k be a positive integer with 2k ≤ n − 4t. If we choose set K ⊆ [n] ⧵ (I ∪ J) of size k i.u.r., then the probability that K is feasible for I, J, and s is at least

5.2.3
Approximating To approximate the size of Φ 0,s (I, J) for sets I and J compatible for shift s, we first notice that Therefore, since we know that |Φ * 0,s (I, J)| = (n − (|I| + |J|))! by (12) where the last inequality follows from the auxiliary Claims 16 and 17. If we combine (11) and (12), then we get the following lemma. .
This immediately gives, Hence, we can conclude that for i ≠ j we have,

ANALYSIS OF THE COMMUNICATION IN THE LOCKER ROOM SETTING
A lower bound for the success probability in the locker room problemis provided by a straightforward adaptation of the shift strategy: Alice enters her message relaying the most common shift to locker 0, then Bob opens locker 0 and uses Alice's message to check location ( + ) mod n for his card. This strategy ensures a success probability of (1+o(1)) log n) n log log n . As in Sections 2 and 3, we will consider the case when is chosen i.u.r. from [n] (see Section 2.1). In order to obtain an upper bound for the success chance in the locker room problem, we shall introduce some intermediate settings, or "protocols". In the CLR protocol, the rules which govern the locker room problem, Alice views the contents of all the lockers, interchanges the contents of two lockers, then Bob is given a number and can open two lockers in search of it. In the NH protocol, the rules which govern the needle in a haystack game, Alice views the contents of all the lockers, communicates a message of length log n to Bob, then Bob is given a number and can open one locker in search of it. Moreover, we could append the modifier "-with-r-bits" to NH, which substitutes r for log n in the above description.
We write Pr[()] for the optimal probability of success in protocol  and Pr [ (C, ) ] for the probability of success for strategy C in protocol . For example, we have already shown that Pr[(NH)] = (1+o(1)) log n n log log n .
Proof. We will interpolate between CLR and NH-with-(4 log n)-bits with two other protocols.
In the protocol CLR0, Alice views the contents of all the lockers and interchanges the contents of two lockers, then Bob is given a number and can open two lockers in search of it, and he can recognize upon seeing the content of the first locker whether it has been altered by Alice.
In the protocol CLR1, Alice views the contents of all the lockers, interchanges the contents of two lockers, and leaves these two lockers open with their contents visible to Bob, then Bob is given a number and can open one locker in search of it.
Also, let Sim be the strategy in NH-with-(4 log n)-bits in which Alice uses her message to communicate to Bob the cards whose positions she would exchange, and the positions of these cards, if she encountered the permutation while working in the CLR1 protocol, simulating an optimal strategy C in CLR1. Since this is an ordered quadruple in [n] 4 , it can indeed be communicated in at most 4 log n bits.
The proof is in four parts: ] .
(i), (iii), (iv) are straightforward and so we only have to show (ii). Let p t be the maximum probability that Bob finds his target in the t th locker that he opens, t ∈ {1, 2}.
Firstly, we bound p 1 . Suppose that Alice and Bob have settled on a specific strategy. Let e x,w be the probability that is such that Alice's transposition sends the card x to locker w. Evidently, 0 ≤ e x,w ≤ ( . Then, we bound p 2 . If Bob opens first one of the lockers whose contents have been altered by Alice, then there is one remaining locker for him to open, and he has at most as much information as in the CLR0 protocol. Hence, in this case, p 2 ≤ Pr[(CLR0)].
Alternatively, Bob first opens one of the lockers whose contents have not been altered by Alice. This requires a more detailed analysis of the CLR0 protocol.
Alice's choice of a transposition is informed solely by the initial permutation of the cards inside the lockers. Hence, there should be a function a ∶ S n → Let F w = b −1 (w) be the event that Bob opens the w th locker first, so Pr[F w ] = |b −1 (w)| n . Let G y be the event that the initial content of Bob's first locker is y, so Pr [ G y ] = 1 n . If s(y, w) ⊆ S n is the set of permutations which map w to y, and E u,v = a −1 ({u, v}) is the event that Alice transposes the contents of the u th and v th lockers, then Pr The probability that Bob finds his target in his second attempt given that his first locker was not altered by Alice is Thus, in this case, p 2 ≤ 4 n . Ultimately Proof. We use Lemma 20 along with the fact that Pr [ (NH-with-(4logn)-bits) ] ≤ (4+o(1)) log n n log log n , which can be immediately derived from Theorem 22 in Section 7.1 by setting m = n 4 . ▪

GENERALIZATIONS
There are several natural generalizations of the problem studied in this paper and related questions about properties of random permutations, which we will discuss here.

Simple generalization: Longer message
In the needle in a haystack problem, when Alice sends the message to Bob, there is no reason why she must choose a number in [n]; instead, she could transmit a number ∈ [m] for an arbitrary integer m. One can easily generalize the analysis from Theorems 5 and 7 in this setting for a large range of m. Let us denote the maximum attainable sum of intensities received from partitioning S n to m parts the m-field of S n , and denote it by F(n, m). Fields are simply diagonal m-fields (fields of the form F(n, n)).
We have F(n, 1) = n! (yielding a success probability of 1 n , corresponding to not receiving advice) and F(n, m) = n ⋅ n! for every m ≥ n! (yielding a success probability of 1, corresponding to obtaining full information). For other values of m we can follow the approach used in Theorem 5. First, notice that there is = (1+o(1)) log m log log m , such that m = o( !). Then, using the techniques from the proof of Theorem 5, we obtain for a random variable X recording the contribution of a random permutation to F(n, m) that By (2), this yields the success probability of (1+o (1)) log m n log log m , giving the following theorem. .
Since the algorithm given in Theorem 7, using the shift strategy with hint ∈ [n], has success probability Ω ( log n n log log n ) , Theorem 22 implies that this shift strategy is asymptotically optimal to within a constant factor for any hint which is polynomial in n. A similar conclusion holds also for the communication in the locker room setting: even if Alice leaves Bob a message by altering the contents of a constant number c of lockers rather than just one, this message is c log n bits long, and hence the success probability is still at most O ( log n n log log n ) . Asymptotic results for several other interesting domains of m could be found in a similar way. However, for super-polynomial domains, the upper bound derived above is far from the lower bound in Theorem 7. Determining some properties of the rate of growth of F(n, m) for fixed n would be a good step towards determining its values. With this in mind, we have the following natural conjecture.

Optimal strategies
Although we have successfully calculated the maximum field and the maximum success probability for the needle in a haystack problem, the problem of determining a characterization of, or even some major properties for, optimal strategies remains. Indeed, the only optimal strategy that we have explicitly described so far is the shift strategy (which is in fact a set of different strategies, since, for permutations which have several S 's of maximum size, there are multiple legitimate options for their class). A natural generalization of shift strategies are latin strategies; in these, Alice and Bob decide on a n × n latin square S, and Alice's message indicates the row of S which coincides with at the maximum number of places.
We present a couple of interesting questions concerning the optimal strategies for S n in needle in a haystack.

Conjecture 2.
For every natural number n, there is an optimal strategy for S n whose parts all contain exactly (n − 1)! permutations.

Conjecture 3.
Optimal strategies are exactly the latin strategies.

Alice-In-Chains
Let us explore another specific strategy. The naive strategy is to group permutations according to the content of location 0. That is, , ′ belong to the same class if and only if (0) = ′ (0). This is a natural strategy to conceive, and it agrees with the common (but erroneous) notion that efficiency in the lockers game cannot be improved beyond O . Indeed, straightforward calculations yield a success probability of 2 n for the naive strategy in the needle in a haystack problem.
Intuitive though it is, in the preceding sections we have proven the naive strategy to be suboptimal. In fact, the naive strategy fails to fully utilize the possibilities provided by the problem's framework. In this subsection, we show that, with only a minor restriction for our problem, the naive strategy becomes optimal. This demonstrates that strategic efficiency is very sensitive to changes in our assumptions about the needle in a haystack.
Suppose that Alice and Bob face a challenge similar to the needle in a haystack, but with this restriction: the chosen strategy C = ⟨C 0 , … , C n−1 ⟩ must satisfy that is, "there exists a needle s such that for each location i there is a corresponding message h from Alice which tells Bob that s is not in i". We call this the Alice-In-Chains (AIC) variant.
Theorem 23. The naive strategy is optimal in Alice-In-Chains.

Proof.
It is easy to see that the naive strategy satisfies the AIC rules. For instance, we can take s = 0, in which case the message h = 1 informs Bob that s is not in locker 0, and the message h = 0 informs Bob that s is not in locker i for any i > 0.
We proceed by induction. For n ≤ 3, it is easy to see that the naive strategy is optimal, even without the restriction.
Suppose that it is optimal for n ≤ N. Let C = ⟨C 0 , … , C N ⟩ be an optimal strategy for S N+1 in the AIC variant. Without loss of generality, let s = N.
Let A m be the subset of S N+1 which contains every permutation that maps N to m. To bound the field F AIC (N + 1), we will try to maximize the sum of the intensities produced by distributing the members of A m across the N + 1 classes. That is, we partition each A m into a collection C (m) = [A m,0 , … , A m,N ] which maximizes the sum ∑ 0≤s,h≤N int(A m,h , s). We claim that To see that, observe that partitioning one set of permutations to several does not decrease the sum of the intensities. Indeed, Hence, However, each A m is a copy of S n , and one of its parts must be empty (because of the restriction of AIC, and the fact that all of the members of A m agree on the image of N). Therefore, ∑ 0≤s,h≤N int(A m,h , s) = F AIC (N) for all m ∈ [N], and so (14) yields From our inductive hypothesis, the naive strategy is an optimal strategy for S n in the AIC variant, so F AIC (N) = 2N!, which from (15) implies F AIC (N + 1) ≤ 2(N + 1)!. Since the yield of the naive strategy for S N+1 is exactly 2(N + 1)!, we have that the naive strategy is optimal for S N+1 in the AIC variant.
Remark 24. The above implies that the Alice-In-Chains variant has a maximum attainable probability of 2 n . It also proves an interesting result about the form of optimal strategies: every optimal strategy in the needle in a haystack setting is such that every element c ∈ [n] has an image which is present in all of the strategy ' We present an elementary proof of Lemma 8 (following standard arguments) showing that the expected number of j ∈ [n] with S j ≥ (1+o(1)) log n log log n is at least one.
Proof of Lemma 8. Let us recall Definition 4 for derangements and r-partial derangements. The probability that a random permutation in S n is a derangement is D n ∕n! = ∕ n! e and note that D n = u(n) n!∕e, that u(n) = 1 + o(1), and u(n) > 0.9 for all n > 1. Since the permutation ∈ S n is chosen i.u.r., we have The same bound can be obtained for S j for every j ≥ 0. For any permutation ∈ S n and any integer ∈ [n], define the permutation ∈ S n given by For any permutation ∈ S n and any , the operator  → is a bijection from S n to S n , and a permutation ∈ S n with ∈ [n] has exactly k fixed points if and only if permutation has exactly k points with (i) = i + (mod n). Hence for every j, j ′ ∈ [n] and k ∈ [n], we have Pr . Therefore, for any integers j ∈ [n] and k ∈ [n − 2], Let k(n) be the largest k such that 2ek! ≤ n. We will give a construction of sets I and J, each of size t, such that I and J are compatible for shift s. We begin by selecting t elements from I one by one. We will ensure that sets I, I − s, and I − 2s are pairwise disjoint. The first element i 1 is arbitrary, and we can select it in n ways. We choose the second element i 2 from [n] ⧵ {i 1 , i 1 − s (mod n), i 1 − 2s (mod n)} in at least n − 3 ways, the third element i 3 in at least n − 6 ways, and so on; since the elements in I can be ordered arbitrarily, the number of choices is at least n(n−3) … (n−3(t−1)) t! . Next, we choose t elements from J. We will ensure that J is pairwise disjoint from sets I and I − s, and J + s is pairwise disjoint from sets I and I − s; notice that the latter means that J is pairwise disjoint from sets I − s and I − 2s. The first element j 1 is selected in at least (n − 3t) ways, since  2)) ways, and so on. Since the elements in J can be ordered arbitrarily, the number of choices is (n−3t)(n−3(t+1)) … (n−3(2t−1)) t! . Therefore, we have presented a way of selecting at least n(n − 3) … (n − 3(t − 1)) t! ⋅ (n − 3t)(n − 3(t + 1)) … (n − 3(2t − 1)) t! distinct pairs of sets I and J of size t that are compatible for shift s. This implies that if we choose two disjoint sets I, J ⊆ [n] of size t i.u.r., then the probability that I and J are compatible for shift s is at least Next, we use  . ▪
Let PC I,J,0,s (K) be the set of all permutations consistent with I, J, s, K, and . We notice that  I,J,0,s (K) is the union over all 2 |K| functions ∶ K → {0, 1} of the sets of all permutations consistent with I, J, s, K, and , that is,  I,J,0,s (K) = ⋃ ∶K→{0,1} PC I,J,0,s (K). First, let us note that if K is feasible for I, J, and s, then for any two distinct functions , ′ ∶ K → {0, 1} the set of all permutations consistent with I, J, s, K, and and the set of all permutations consistent with I, J, s, K, and ′ are disjoint, that is, PC I,J,0,s (K) ∩ PC ′ I,J,0,s (K) = ∅. Indeed, let us take two distinct , ′ ∶ K → {0, 1} and let be an arbitrary permutation in PC I,J,0,s (K). We will show that ∉ PC ′ I,J,0,s (K). Since and ′ are distinct, there is some such that ( ) ≠ ′ ( ); without loss of generality let ( ) = 0. But then, for any permutation ′ ∈ PC Next, assuming that t, k ≤ O(log n), we have .