Though not officially sanctioned, there is a crew of swimmers to be found around 10-11am at Grimaldo’s chair on the weekends, and there’s often people there before and after.
You can find the local Brighton beach swim info at http://www.shallweswim.today which includes the water temperature over the last few years by day. It’s usually in the mid50-low 60s, definitely swimmable.