The fishing village of Long Hai, 30km northeast of Vung Tau, is one such place, but can really be only be recommended if you aren’t interested in meeting many other Western travelers.
Long Hai has little to offer besides its rustic beach and local flavour – which might be all you need. If you require nightlife (or, well, daylife), head to Mui Ne instead.