We claim that the maximum value of $k'$ is $\frac{p-3}{2}$. First, a well known lemma: Lemma. For all $0 \leq x \leq p-2$, we have $\sum_{i=1}^{p-1} i^x \equiv 0 \pmod{p}$. So we can rewrite the condition as : $$2\sum_{a\in X} a^k \equiv \sum_{j=1}^{p-1} j^k \equiv 0 \pmod{p} \implies \sum_{a\in X} a^k \equiv 0 \pmod{p}$$for all $k = 1,2, \dots , k'$. Further, if we plug in $k = 0$ into the original, we get that $|X| = |Y| = \frac{p-1}{2}$. Let's assume $k' \geq \frac{p-1}{2}$. Then we have $\sum_{a\in X} a^k \equiv 0 \pmod{p}$ for all $k = 1,2, \dots , \frac{p-1}{2}$. By Newton's Identities, this means $\prod_{a \in X} a \equiv 0 \pmod{p}$, a contradiction. Thus $k' \leq \frac{p-3}{2}$. Now let's show that $k' = \frac{p-3}{2}$ is possible. To prove this, let $X$ be the set of quadratic residues modulo $p$ and let $Y$ be the set of non-quadratic residues modulo $p$. If we let $g$ be a primitive root mod $p$ , then $X = \{g^0 , g^2 , \dots , g^{p-3}\} , Y = \{g^1 , g^3 , \dots , g^{p-2}\}$. It is easy to see that for all $k = 1,2, \dots , \frac{p-3}{2}$, we have $$\sum_{a\in X} a^k \equiv \frac{g^{(p-1)k} - 1}{g^{2k} - 1} \equiv 0 \pmod{p}$$since $k < \frac{p-1}{2}$. Thus, our condition is satisfied and $k' = \frac{p-3}{2}$ works.

I have two solutions. One of them is the same as hakN's, which considers the symmetric sums of the polynomial $P(x)=\prod\limits_{j\in X}(x-j)$ in $\mathbb{Z}_p[x]$ (oops, I ended up proving a property about Newton polynomials, and basically proving the fundamental theorem of symmetric polys). One of them uses the legendary(re) symbol! All congruences are done modulo $p$. Rephrasing the problem condition, we need $|X|=\frac{p-1}{2}$ and $2\sum\limits_{j\in X} j^k \equiv \sum\limits_{j=1}^{p-1} j^k (\bmod\; p)$ for all $1\le k\le k'$. First I prove $k'=\frac{p-3}{2}$ works. Take $X$ to be the set of nonzero quadratic residues, and $g$ to be a primitive root mod $p$. For all $1\le k\le \frac{p-3}{2}$, we have $$\sum\limits_{j=1}^{p-1} j^k \equiv \frac{g^{(p-1)k}-1}{g^k-1} (\bmod\; p)$$. Since $ord_p(g)=p-1$, the denominator is not a multiple of $p$ while the numerator is, so this sum is 0. We also have the multiset with two copies of $A$ is equivalent to $1^2,2^2,\cdots,(p-1)^2$ mod $p$, so $$2\sum\limits_{j\in X} j^k = \sum\limits_{j=1}^{p-1} j^{2k} \equiv \frac{g^{2(p-1)k}-1}{g^{2k}-1}$$, which vanishes mod $p$ for the same reason when $k=1,\cdots,\frac{p-3}{2}$. It remains to show that $\frac{p-1}{2}$ fails. For brevity let $n=\frac{p-1}{2}$ Method 1. Legendary symbol. First, for all $t=0,\cdots,p-1$, $$\sum\limits_{j\in X} (j+t)^n = \sum\limits_{j\in X} \sum\limits_{m=0}^n \binom nm j^m t^{n-m} = \sum\limits_{m=0}^n \binom nm t^{n-m} \left(\sum\limits_{j\in X} j^m \right)$$ Observe the inner sum vanishes whenever $j>0$. Therefore this is simply equal to $\frac{p-1}{2} \left(\frac tp \right)$. We look at this sum in another way. This sum is simply $$\sum\limits_{j\in X} \left( \frac{j+t}{p} \right)$$. Select two $t$ such that $-t\notin S$. Observe each term $\left( \frac{j+t}{p} \right) \in \{-1,1\}$ and their sum is either $\frac{p-1}{2}$ or $\frac{1-p}{2}$. This means all of $j+t$'s are QRs or NQRs. Picking two of them gives a contradiction because the set of QRs $S$ satisfy $S+t$ having $\frac{p-1}{2}$ elements in common for any $t$. Method 2. Symmetric Polynomials. Basically, noting $|X|$ is equal to the number of nontrivial symmetric polynomials. We first prove a lemma. Lemma: Let $P(x)=\prod\limits_{j=1}^n (x-r_j)$. Let $S_k=\sum\limits_{j=1}^n r_j^k$ and $T_k=[x^{n-k}]P(x)$. Then $S_k$ can be written as a polynomial $Q_k(T_1, T_2, \cdots, T_n)$ with coefficients not divisible by any prime greater than $k$ or $n$. Proof: We first address the scenario for $n=2$. By binomial theorem, $(x+y)^k=\sum\limits_{j=0}^k \binom kj x^jy^{k-j}$. The idea is to note for $k$ odd, it can be written as $\sum\limits_{j=0}^{\frac{k-1}{2}} \binom kj (xy)^j (x^{k-2j}+y^{k-2j})$ Thus, $x^k+y^k=(x+y)^k-\sum\limits_{j=1}^{\frac{k-1}{2}} \binom kj (xy)^j (x^{k-2j}+y^{k-2j})$ The coefficients of this are $\binom kj$ times stuff with prime factors less than $k$ by inductive hypothesis, so the $n=2$ scenario is fully addressed. Now, we induct on $n$. Let $S$ be the $n-1$ element subsets of $[n]$. Observe that $\sum\limits_{j=1}^n r_j^k =\frac{1}{n-1} \sum\limits_{S} \sum\limits_{j\in S} r_j^k$. By inductive hypothesis, each $\sum\limits_{j\in S} r_j^k$ has coefficients with a nonnegative $\nu_p$ value for all $p>\max(k,n)$. Say $\sum\limits_{j=1}^{n-1} r_j^k$ contains the term $c\prod_{j=1}^l T'_{a_j}$ where $T'$ is the coefficient of $Q(x)=\prod\limits_{j=1}^{n-1} (x-r_j)$. Then cyclic summing gives $c \sum_{cyc} \prod_{j=1}^l T'_{a_j} = c \sum_{cyc} \prod_{j=1}^l (T_{a_j}-r_nT'_{a_j-1})$ If I expand this, the $T_{a_j}$'s form a symmetric polynomial. Taking out other symmetric terms give a polynomial of smaller degree because the $T_{a_j}$'s can be treated with ease and so can the $\prod_{j=1}^n r_n$'s, so I can induct on degree to prove the desired result. Going back to the problem. Let $P(x)=\prod\limits_{j\in X} (x-j)$. We know that $S_k$ vanish for all $k=1,\cdots,n$. By our lemma, we can induct on $k$ to prove that $[x^{n-k}]P(x)$ is 0 mod p for all $k=1,\cdots,n$ because no coefficients vanish mod $p$ and noting the existence of a $\prod r_j$ term exists. But this implies $p\mid \prod\limits_{j\in X} j$, absurd!

To different k,whether the set X keeps the same or not?