To determine which of the given systems has a non-trivial solution, let's analyze each option individually:
\(AX=0\), \(|A|=4\)
- If \(|A| \neq 0\), matrix \(A\) is invertible, indicating that the only solution to \(AX=0\) is the trivial solution \(X=0\). Therefore, this system has only the trivial solution.
\(AX=0\), \(|A|=-4\)
- Similar to the first case, since \(|A| \neq 0\) (even though it is negative), matrix \(A\) is invertible, and the system has only the trivial solution \(X=0\).
\(AX=0\), \(|A|=0\)
- In this case, since \(|A|=0\), matrix \(A\) is singular. This implies there are infinitely many solutions, meaning the system has a non-trivial solution (a solution other than the zero vector).
\(AX=B\), \(|B|=5\)
- The determinant of \(B\) being non-zero is irrelevant when determining if \(AX=B\) has a non-trivial solution. The solution depends on whether \(A\) is invertible (i.e., \(|A|\neq 0\)).
Based on the analysis above, the system that has a non-trivial solution is option 3: \(AX=0\), \(|A|=0\). This occurs because a singular matrix (\(|A|=0\)) implies an infinite number of solutions.