I'm not saying you're completely wrong.
However noone here mentioned issues related to link saturation, yet
Theoretically array with 8x1Gbps ports can produce 8Gbps throughput. Agreed?
Would you buy an 8port 1Gbps iSCSI array if you're designing for this type of load? How many spindles would you need for that?
Your'e talking about RR PSP. How many paths per LUN are active (serving I/Os) at a time? (hint: number is the same for RR, MRU and Fixed). What is the bandwidth of a single path if I'm having 1Gbps ports on host? Will it change if I'm having 2 or 8 ports on host?
Yes, link aggregation might help if there is a link saturation between the array and switch (multiple hosts producing load) and if that is supported by the array, but that's a few steps forward. First of all we need to connect to that array and if I'm not mistaken, currently can't change IPs on array ports.
Just out of curiosity, have you seen a lot of environments where 1Gbps iSCSI bandwidth is the bottleneck? From my experience the first thing that customers usually hit is the number of IOs that array can produce (first of all # of spindles and then CPU on array). Throughput is rarely the case. When it is, it's either something very specific (i.e. public streaming video servers) or it's a system that due to normal evolution is about to migrate to higher storage tier (FC or 10Gbps). Otherwise what I'm seeing typically is that if we place ~20-30 average / regular VMs on a single host, those 1Gbps iSCSI links are rarely loaded above 30% throughout the day. Utilization only increases during backup window.
Again, it depends heavily on the case and on VM IO profiles. This is what I see every day. Maybe I'm living under a rock so I'd be happy to hear about your experience.
WBR
Imants