Hint: Telescoping product.
Let
an=n6−64(n3+3n)2.
Notice that
an=(n3−8)(n3+8)(n3+3n)2=(n−2)(n2+2n+4)⋅(n+2)(n2−2n+4)n2(n2+3)2=n−2n⋅n+2n⋅(n−1)2+3n2+3⋅(n+1)2+3n2+3.
Hence, for N≥3 we have
n=3∏Nan=(n=3∏Nn−2n)(n=3∏Nn+2n)(n=3∏N(n−1)2+3n2+3)(n=3∏N(n+1)2+3n2+3)=1⋅2N(N−1)⋅(N+1)(N+2)3⋅4⋅22+3N2+3⋅(N+1)2+332+3=772⋅(N+1)(N+2)((N+1)2+3)N(N−1)(N2+3)=772⋅(1+N1)(1+N2)((1+N1)2+N23)(1−N1)(1+N23),
so
n=3∏∞an=N→∞limn=3∏Nan=N→∞lim(772⋅(1+N1)(1+N2)((1+N1)2+N23)(1−N1)(1+N23))=772.