I always like the old saying BB stands for "Big Bird". I think another variable is how close your shots will be, and you might not know that until you get there. I shot one 2 years ago with a 10 gauge Parker and black powder shells loaded with BBs. My shot was 15-20 yards and I dropped the swan first shot. In that situation, you could have used a much smaller shot and gauge. Though I was mighty proud to have done it with a Parker hammer gun and brass shells...
Last year, there was another blind close by that messed up the shooting. Shots this time were 40-50 yards. I brought an original sxs muzzleloader(with cylinder chokes), because by now I thought swan hunting was easy. I took 2 shots at a 40 yard bird and didn't come close. My friends with autoloaders had open chokes too and they didn't get any either.
So next time I will be prepared for both situations(close and far). I think #2s would work all day long. I have a Sandhill Crane hunt in December and RST loaded me 10 gauge 2 7/8" 1 3/8oz Bismuth #2's at 1225fps. I would use those shells swan hunting as well(not in my hammer gun, but in a DH #3 frame.
In a 12 sxs I would go with #2s. These are big birds and if you have to make a farther shot I would want bigger sized shot and tighter chokes. In close, open chokes(would help) and I still like the bigger shot sizes. However, in close even a 28 gauge with #4s would certainly work(as Craig mentioned)...If I can ever save up for a 28 gauge Parker I will try it
I should be going this year too and I will have all bases covered this time...
Good Luck!