All solutions here are SUGGESTED. Mr. Teng will hold no liability for any errors. Comments are entirely personal opinions.

(i)
$\vec{OC} = \lambda \vec{OA} + \mu \vec{OB}$

$=\lambda \begin{pmatrix}1\\-1\\1\end{pmatrix} + \mu \begin{pmatrix}1\\2\\0\end{pmatrix}$

Area $= \frac{1}{2}|\vec{OA} \times \vec{OC}|$

$= \frac{1}{2} |\begin{pmatrix}1\\-1\\1\end{pmatrix} \times \begin{pmatrix}{\lambda + \mu}\\{-\lambda + 2 \mu}\\{\lambda}\end{pmatrix}|$

$= \frac{1}{2} |\begin{pmatrix}{-2\mu}\\{\mu}\\{3\mu}\end{pmatrix}|$

$= \frac{1}{2} \mu \sqrt{14}$

$\Rightarrow \frac{1}{2} \mu \sqrt{14} = \sqrt{126}$

$\mu = 6$

(ii)
$|\vec{OC}| = |\begin{pmatrix}{\lambda + 4}\\{8-\lambda}\\{\lambda}\end{pmatrix}| = 5 \sqrt{3}$

$3 \lambda^2 - 8 \lambda + 5 = 0$

$\lambda = 1 \mathrm{~or~} \frac{5}{3}$

Thus required coordinates are $(5, 7, 1)$ and $(\frac{17}{3}, \frac{19}{3}, \frac{5}{3})$