Connectors were not designed to be used with Grouped objects.
If you Group several objects and then try to attach a Connector (Right-angle or Straight Connector) you will have difficulty finding a good attachment point with the PointGuide. If you do find a location where the Arrow tool shows a plus sign ("+"), the Connector may not be attached to any particular object in the Group.
The best solution is to Ungroup the objects and attach the Connector to just one of the objects. Also, Grouping objects that have Connectors attached will detach the Connector.
Explanation:
The connector is connecting to the center or corners of the group. It treats the group of objects as if it were a single object.
Here's an easy way to see it.
1. Draw a diagonal line.
2. Duplicate the line, rotate it 90¡ and align the center points of the two lines so that they look like the letter "X."
3. Group the lines.
4. Now try attaching a Connector.