This problem was proposed by Bosnia and Herzegovina

This is in fact true for any positive integer $p>1$. It is well known that the last digit of $p^n$ is eventually periodic $\pmod {10}$, so it suffices to show that the first digit isn't periodic (we immediately rule out the case of $p$ being a power of $10$). If the first digit of $p^n$ is $a\in \{1,...,9\}$, this means that $a\cdot 10^k\leq p^n<(a+1)10^k$ for some integer $k$. Taking $\log$s, we get $\log a+k\leq n\log(p)<\log(a+1)+k\implies \{n\log p\}\in [\log a,\log(a+1))$. Now assume that $a_n$ (the initial digit corrisponding to $p^n$) is periodic with period $T$, i.e. $\{n\log p\}=\{(n+kT)\log_p\}$ both belong to the same set of the form $[\log a,\log(a+1))$ for any nonnegative integers $k$ and $n$. Now since $p$ isn't a power of $10$, $\log p$ is irrational (in particular also $T\log p$), and so we can apply Kronecker theorem to see that the number $y_k=\{k(T\log p)\}$ is equidistributed in $[0,1)$ and so by translation also $x_k=\{(n+kT)\log_p\}$. Therefore, since $[0,1)-[\log a,\log (a+1))$ is nonempty, there exist a $k$ such that $\{(n+kT)\log p\}\not\in [\log a,\log(a+1))$, or equivalently $a_{n+kT}\neq a$, and so $a_n$ can't be periodic. Therefore, since the last digit is eventually periodic, while the first isn't, they can't always match and thus $p^n$ can't be always a palindrome. Remark: the argument immediately applies to all other bases (except for base 2 in which the first digit is always $1$, but it suffices to look at the first two and last two digits).

Divisibility criteria for $11$ tells us that $11$ divides every palindromic number with even digits. Now we can easily check, that for $p=11$, $p^5=11^5=161051$, which is not palindromic. So now we look at other primes. We can check primes up to 10 easily by hand (probably not necessary, but I wanted to get rid of 2 and 5 which divide 10) Now suppose there exist such prime $p$ such that every power of $p$ is a palindromic number. Then every power of $p$ must have an odd number of digits, which means that for each positive intiger $i$, $p^i$ has an odd number of digits, which means that $\lfloor \textrm{log}_{10} (p^i) \rfloor = \lfloor i \cdot \textrm{log}_{10} (p) \rfloor$ is an even number for all $i$. But $\textrm{log}_{10} (p)$ is irrational so now by some theorem (to which I forgot the name lol) we are finished because it states that for any irrational number $a$ and any positive real number $\epsilon$ there exist intigers $m$ and $n$ such that $|ma-n|< \epsilon$. But also this theorem itself is not very difficult to prove and may be let as an exercise to the reader. It turns out that wee actually don't need this theorem, but we can look at $ i \cdot \textrm{log}_{10} (p)$ mod $2$ and we know that $ i \cdot \textrm{log}_{10} (p) \neq 0$ or $1$ mod $2$ for any $i$. Now if $\textrm{log}_{10} (p) > 1$ (we look everything mod $2$ now) we are finished, because then $p$ itself has got an even number of digits, a contradiction since $p\neq 11$, otherwise $\textrm{log}_{10} (p) < 1$ and there exists a positive intiger $k$ for which it holds that $k \cdot\textrm{log}_{10} (p) < 1 < (k+1) \cdot \textrm{log}_{10} (p) < 2$ and now we have proved that there exists a power of $p$ which has got an even number of digits, which is a contradiction.