Hint: set $a=0$ and differentiate with respect to $b$ twice (use the Fundamental Theorem of Calculus to show that $f'$ exists).

Let $P(a,b)$ be the assertion $\left( {{a}^{2}}+ab+{{b}^{2}} \right)\int\limits_{a}^{b}{f\left( x \right)dx=3\int\limits_{a}^{b}{{{x}^{2}}f\left( x \right)dx,}}$ and $t\in \mathbb{R}$, we have \[P(0,t) \implies {{t}^{2}}\int\limits_{0}^{t}{f\left( x \right)dx=3\int\limits_{0}^{t}{{{x}^{2}}f\left( x \right)dx}}\]since $f$ is continuous here, we know that $f'$ exists, and differentiating with respect to $t$ on both sides gives \[t^2f(t)+2t\int\limits_{0}^{t} f\left(x\right) dx=3t^2f(t) \implies \int\limits_{0}^{t} f\left(x\right) dx=tf(t)\]and differentiating with respect to $t$ again gives \[f(t)=f(t)+tf'(x)\implies f'(t)=0\]and so $f$ is a constant function. It is easy to check that $f(x)=C $ where $C\in \mathbb{R}$ works. $\blacksquare$

Whoa this was easy Just manipulate the terms a tiny bit to get that $3 \int\limits_{a} ^ {b} x^2 dx \cdot \int\limits_{a}^{b} f(x) dx = (b-a) \int\limits x^2f(x)dx$ Putting $a= 0 $ and differentiating both side by FTC we get $ 3b^2 \int\limits_0 ^b f + \cancel{b^3 f(b) } = \cancel{b^3f(b)}\implies \int\limits_0 ^b f(x) dx = 0 $ so the integral is constant differentiating we get $f(x) = 0 \forall x \in \mathbb {R}$

ionbursuc wrote: Determine continuous functions $f:\mathbb{R}\to \mathbb{R}$ such that $\left( {{a}^{2}}+ab+{{b}^{2}} \right)\int\limits_{a}^{b}{f\left( x \right)dx=3\int\limits_{a}^{b}{{{x}^{2}}f\left( x \right)dx,}}$ for every $a,b\in \mathbb{R}$ . I think i have a different solution from others.. For $a,b,a+b \neq 0$ we have $(a^2+ab+b^2)\int_{a}^{b} f(x)dx=(a^2+ab+b^2)(\int_{a}^{0} f(x)dx+\int_{0}^{b} f(x)dx )=3(\int_{a}^{0} x^2 f(x)dx +\int_{0}^{b} x^2f(x)dx)$ So after cancelling term we get $(a+b)(b\int_{a}^{0} f(x)dx +a\int_{0}^{b} f(x)dx)=0\implies \frac{1}{a} \int_{0}^{a} f(x)dx =\frac{1}{b}\int_{0}^{b} f(x)dx $ $\frac{1}{x}\int_{0}^{x}f(t)dt=C$ where $C$ is a constant for all $x\in R-\{0\}$ $\implies \int_{0}^{x} (f(t)-C)dt =0\implies f(t)=C \forall x\in R$