you have mentioned 3 nics in the server, 2 for the storage and 1 for the management.So there is no redundancy in the management that is in the vswitch.
if i disconnect one cable from the Team the vSpare is done / lost connection - it looks like the whole trafic goes over port 8. - So this is the expected thing.any way the storage traffic goes through it..so why you have surprised?
If you select "allow management traffic in the portgoups" of the iscsi then you can connect to the esxi....but normally it is not recommended to share the storage traffic with management traffic in the same pnics.
1. i have read i need to configure VLAN for it / on the switch side ?
a, how to configure VLAN ? do i need to put port 5 and 8 in VLAN4 and 6 and 7 in VLAN6 or does it needs to be all 4 ports in the same VLAN ?
the storage and iscsi ports groups and vswitch for the storage should have the same VLAN.. why you need different VLAN ?
b, then i need to configure VLAN on ESXI Team too ?
yes
c, why does the networks for teaming have 2 IPs ?
for iscsi you need to use 2 vmkernel portgroups and 2 ip and all of them have same vlan.
First you need create a VLAN in the switch for the switch ports 5,6,7,8 and assign to that physical ports. Then create a vswitch with 2 nics, and create 2 vmkernel portgroups with the same VLAN. Thats it...Storage part is over.
Now for the management. you only have one nic so there is no redundancy, so buy one more dual port nic or single port nic and make redundancy.Here also create another VLAN in the switch and in the vmkernel portgoups for the management..and create a team.
In the iscsi port goups check if you have enabled the management traffic to pass