Round Post Advantages
Round posts are used with rail fences. Because the rails go through the holes in the posts, there is no need for a flat surface on the post to which you attach the rails. If you have a large property with cedar trees on it, cut your own cedar posts and save money on fencing lumber. The trunks from appropriately sized cedar trees make round fence posts with no need for milling.
Round Post Disadvantages
Round posts do not offer a flat surface to which you can attach boards, so if you are constructing a fence that is of a different style than a rail fence, its strength may be compromised by round posts. A board that is attached to a square 6-by-6 will have 6 inches of contact with the post, while one that is attached to a round post only contacts the post for a fraction of an inch at the point where the curve of the round post meets the board.
Square Post Advantages
Square fence posts provide a more finished and formal look for fences that are doubling as aesthetic statements. In loose soil, the flat faces of a square post provide a firmer base and are less likely to work loose than a round post. If you are working with commercial lumber, it is easier to make square posts than round posts. Square posts are easy to measure and cut to create accurate joints for meeting the fence rails.
Square Post Disadvantages
Square posts provide corners that soil and water get into. The exposed corners and edges of a square post are more likely to rot than the more contained surface of a round post. Boards or rails that are nailed to a square post are less resistant to blows or kicks from animals than rails that are passed through the center of a round post. Large square posts require larger excavations to bury their bottoms, creating more work during the fence building process.