Do ground fault trip units and relays still work on a 277/480V 3 wire 3 phase grounded system without a neutral?
Yes. On a 3 phase 3 wire 277/480V grounded system with no neutral the trip unit ground fault function still works. The trip unit still uses the internal phase CT`s to vectorally sum the phase currents like it does with a 4 wire system with a neutral. The only difference is that you do not have single phase loads and the trip unit does not have input from the NCT since there is no neutral current.
The trip unit is not looking for imbalance, since a two phase load would cause that. It is looking for current leaving the system which only a GF can cause, and therefore cause the current to vectorally sum to a value greater than zero, the value of the GF current.
On a 4 wire system with neutral current, you have single phase loads with neutral current. This neutral current is measured with the neutral CT and is also part of the vectoral summing process along with the internal phase CT`s. The current sums to zero when no GF exists. If a ground fault exists then the current sums to the value of the current going to ground. Again not looking for imbalance, as a single phase load would cause an imbalance, but would still properly sum to zero.