Packets' Losses

Hi everyone!
Please help to resolve the great issue in my local net. About three weeks ago I noticed the packets’ losses when they go from one switch to another in my server room. The biggest complexity in this case for me that there are no losses at all when all computers are connected to the same switch, but if I connect PC1 to Switch1, PC2 to Switch2 and try "ping hostname "“t" I’m starting to get packet losses every 5-20 minutes on the average.
I replaced almost all my switches (3Com 48+2 ports) with other 3Com switches, the result is quite interesting: some of them (for example, Switch3) are working good (no packet losses) although they’ve got 40-45 users connected to them. The other (Switch 4) has only 1 or 2 users connected and is experiencing packets losses every 1-10 minutes.
There are no viruses in the local net.
What might be the cause of it?

 

How are the 2 switches connected together?

 

Switch1/random port (not the uplink one) to Switch2 uplink 1Gb port or vice versa.

 

Are the switches "managed "? If so you can usually get into the switches management system and get an error report. That should identify the port or ports where the errors are happening. It might also be worth using something like wireshark to sniff the connects to see what errors are causing the packets to be dropped.

My guess is that there are three likely causes of the problem:

• Hardware fault. Start with cables as they are the most likely items to fail. However, it could be a switch fault. It is less likely to be an individual PC's network card (as your report of the problem seems to indicate that a number of PC are affected), but it could be if one NIC is sending out a lot of rouge packets. If you've got managed switches it will be easier to isolate the hardware most likely to be at fault, but otherwise I'm afraid it is a case of trial and error to isolate the problem kit.
• It is a distance, signal strength, noise issue. I notice you are using gigabit connection. The faster the connection, the more suseptible it is to signal degradation. So infrastructure that works at 100Mb/s can fall over badly at 1000Mb/s. How far is one switch from the other? If it is tens of metres you may well be getting distance problems. Also look at cable runs - are they running along side power cables or running near electrically noisy equipment? Also are there loose wires visible at the terminations, are bends too tight, can you move bindings (tie-wraps and the like) back and forth along the cables (you should be able to or the bindings are too tight causing the cable twisting to be disrupted thereby making the noise cancellation less effective).
• A latency issue. You mention two switches. Are there more? If you daisy chain switches together, you start getting latency problems where the delay occuring as the packets are processed by each switch, start to become significant. If there are more switches in the line, this could well cause packets to be dropped.

 

ReggieB, thank you very much for your suppositions.

My switches are managed (3Com 2250) but I did not see any error in their reports. Looks like everything works fine. I failed to notice any deviance watching our local net traffic by Wireshark either.

In my production environment there are 4 switches (48 100Mb ports and 2 uplink 1000Mb ports) connected to a single 16-port Gb switch. All of them are in the single rack.

On the one hand there's the 48-ports swich that is not affected by the issue at all, so it should be the hardware fault to be blamed. On the other hand as soon as I take any of faulty switches to my testing laboratory they work fine and I'm inclined to think the problem is in one of PC's.

 

You could install 2 Gbic's and a MM patch cord to take the copper out of the equation.

Why don't you have them connected by the 1000Mb ports? Already taken for something else?

 

All switches are connected by 1Gb ports: each 1Gb uplink port of 48-port switch is connected to one of 16 Giga-byte ports of main switch (16* 1000Mb ports).

 

Does this mean that any packet going from one 48 port switch to another, must go through the 16 port switch?

If so, I think your problem may be latency. This arrangement means that any packet travelling from one 48 port switch to another must travel over three switches. That's not a good idea.

Your arrangement can work if all your servers are connected to the 16 port switch and users usually only connect to the servers or other users on their 48 port switch. But if they need regular access to one of the other 48 port switches you can get odd behaviour going over three switches.

I'd be tempted to remove the 16 port switch, put all your servers and company wide resources on one of the 48 port switches, and use that as the network core. Then the other two 48 port switches connect directly into the core 48 port, and arrange it so that users on the these two switches only need occassional connection between the two.

 

What's the make and model of the 16 port?
The back plane capacity may be suspect.

 

16 1000Mb-port Switch: 3Com Baseline 2816

I'm already thinking of Baseline 2816... Maybe its ports became out of sync...

 

Since the backplane is not even mentioned in the specs I would be suspicious.

http://www.3com.com/prod/pl_PL_EMEA/detail.jsp?tab=prodspec&sku=3CBLUG16

 

Yes, it does, but it is the 16 port switch that makes up the core of the net. Only servers and the other three switches are connected to it.

The main user traffic is directed from 48 port switch1-4 to one of the servers on 16 port switch. So the longest path for a packet is "48 port switch - 16 port switch ". Moreover, even "48 port switch - 16 port switch - 48 port switch" pathes worked good for about 8 months.

What could have changed since then???

 

A new fact: if I change 48 port switch with two 1000Mb uplink ports (3Com 2250) to 48 port switch with two 100Mb uplink ports (3Com 4250) the problem disappears (these uplink ports are used for connections between 48 port switches and 16 port 1000Mb switch)!!!

 

Maybe... I'll try to figure out...

 

I have replaced the 16 port 3Сom 2816 with small 5 1000Mb-port 3Com 3CGSU05 switch and the problem is gone!

If there was an issue of latency (or the lack of the backplane) how could this little switch handle all packets propely whereas the big one had packet losses every 3-5 minutes???

 

Sounds like the 16 port switch was faulty.

If you have spare capacity on your 48 port switches, you'll get better performance dropping the "backbone switch ". With your servers connected to one of the switches, all users connnected to that switch will be separated from the servers via their dedicated link and the switch backplane (the backplane being the fastest part of the network). This makes a difference when more than one user tries connecting to the a server at the same time.

Of course the users on the other switch will still need to connect over an uplink, but the performance should be no worse than they currently have.

The main point is that at the moment, all the users have to connect to the servers via an uplink. If the servers were connected to one of the 48 port switches, only about half of the users would have to connect via an uplink. Gigabit uplinks are always significantly slower than switch backplanes (which have speeds in the tens of Gb/s)

 

Now if you want a back bone switch this is one.

http://netgear.com/Products/Switche...00Switches/GSM7212.aspx?detail=Specifications
12 auto-sensing 10/100/1000 Mbps ports

12 Small Form-factor Pluggable (SFP*) fiber interfaces

Full wirespeed Gigabit Ethernet throughput


RJ-45 Connectors for 10BASE-T, 100BASE-TX, and 1000BASE-T (Auto Uplinkâ„¢ on all ports):
GSM7212: 12 ports

Bandwidth:
GSM7212: 24 Gbps; 17.8 Mpps

Switch latency: 20 μs for 64-byte frames (1G to 1G)

System memory: 64 MB

# Packet buffer memory: 122 KB embedded memory per port
# Code storage (flash): 8 MB

 

Ok, thanks to everyone! I'll try to use another switch.