Putting $n=1$ into (a) we get $f(1)\left(f(-1)-1\right)=0$ Case 1. $f(1)=0$ Then putting $n=1$ into (b) yields $f(m+1)=f(m)+2m\implies f(m+1)=f(1)+2\cdot{m(m+1)\over 2}$. Hence $f(m+1)=m^2+m\iff f(m)=m^2-m$. Checking shows that the function satisfies all the requirements. Case 2. $f(-1)=1$ Then puttiing $n=-1$ into (b) yields $f(m-1)=f(m)+1-2m\implies f(m)=f(m-1)+2m-1$, therefore $f(m)=f(-1)+2\cdot{m(m+1)\over 2}-(m+1)=m^2$. Checking shows that the function also satisfies all the requirements. Conclusion. The solutions are $f(n)=n^2$ and $f(n)=n^2-n$

I am not entirely sure if this is correct, so please correct me if I am wrong. It is clear that $f(x)=x^2$ is one solution. Define a function $g$, such that $g(n)=f(n)-n^2$. Then, $f(n)=g(n)+n^2$, and by plugging this into $b)$ we obtain that $g(m+n)=g(m)+g(n)$. Since the set of integers is a subset of the set of rationals, the solution to this equation is $g(x)=kx$, where $k$ is an arbitrary integer constant. Therefore, $f(x)=x^2+kx$. Now, by plugging this into $a)$, we obtain that $(n^2+kn)(n^2-kn)=n^4+kn^2$, i.e. $-k^2n^2=kn^2$, so $k=-1$ or $k=0$. By checking, we verify that $f(n)=n^2$ and $f(n)=n^2-n$ are indeed solutions.

Farenhajt wrote: Putting $n=1$ into (a) we get $f(1)\left(f(-1)-1\right)=0$ Case 1. $f(1)=0$ Then putting $n=1$ into (b) yields $f(m+1)=f(m)+2m\implies f(m+1)=f(1)+2\cdot{m(m+1)\over 2}$. Hence $f(m+1)=m^2+m\iff f(m)=m^2-m$. Checking shows that the function satisfies all the requirements. Case 2. $f(-1)=1$ Then puttiing $n=-1$ into (b) yields $f(m-1)=f(m)+1-2m\implies f(m)=f(m-1)+2m-1$, therefore $f(m)=f(-1)+2\cdot{m(m+1)\over 2}-(m+1)=m^2$. Checking shows that the function also satisfies all the requirements. Conclusion. The solutions are $f(n)=n^2$ and $f(n)=n^2-n$ Can you explain the first case again?

ruslan41717 wrote:

Can you explain the first case again? Here putting $n=1$ in the 2nd given equation we get \[f(m+1) = f(m) + 2m.\]And also \[f(m) = f(m-1) + 2(m-1).\]Therefore, \[f(m+1) = f(m-1) + 2m + 2(m-1) = f(m-1) + 2(m+(m-1)).\]Similarly, putting the value of $f(m-1),f(m-2),\cdots f(1),$ we get that \[f(m+1) = f(1) + 2(m + (m-1) + \cdots + 2 + 1) = f(1) + 2\cdot \frac{m(m+1)}{2}.\]