Skip to content

December 12, 2010

Serial AKA RS-232 and Relearning

During some of that work on my network I’ve been talking about over the last few posts, I stumbled across a realization that I wanted to share. Before I get there, I have to explain that I was turning on VLANs for my wireless network. My network has an inner ring where my files are stored and which has a site-to-site VPN to the colocation center. However, there’s an outer ring behind the U-Verse gateway. I wanted to allow guest wireless to this network – thus they could get to the Internet and I didn’t have to worry about them on my network. I setup VLAN 1 as my private network and VLAN 2 as my U-Verse (public) network. Once that was setup I went to my access point, a NetGear WAG302 which I’ve discussed before. When I turned on VLAN tagging the device became unresponsive.

These particular APs have a console port on the back so I started trying to make a console connection. I pulled out my USB to Serial adapter and tried to make a connection. Based on the day I was having I realized that this wasn’t going to be easy – and it wasn’t. The console port had 9600, N, 8, 1 under it which I knew meant 9600 baud, no parity, eight data bits, and one stop bit. I started up a communications client called Putty and tried to connect. However, that wasn’t working.

At this point I’ve got to explain how I even knew anything about the console port. You see I cut my teeth on networks of serial based terminals (like the VT100). I was used to wiring up serial cables. We’d run them through offices and wire them up ourselves with a soldering iron. Of course, since then I have installed modems that used serial connections and have connected all sorts of devices like CNC machines. That seems like a million years ago. However, there are still things that have stuck with me. Obviously, I recognized the pattern of the baud rate, parity, data bits, and stop bits. However, I also can still recall from memory the pin out of a DB25 RS-232 serial connection – I had to use it all the time for several years so it’s not exactly surprising that I remember it. You see Pins 1 and 7 are grounds. Pin 1 is a frame ground and pin 7 is a signal ground. Pins 2 and 3 are send data and receive data. Pins 4 and 5 are ready to send and clear to send – the hardware flow controls. I remember that pins 6, 8, and 20 are also signaling pins for various states including data set ready, data carrier detect, and data terminal ready. I remember them as a set because we were often jumpering the three of these together. You see the RS-232 standard was really designed to work across modems but we often were using it for connecting without modems. Don’t ask me what the pin outs are for a DB9 connection – like the ones that are used often today because I never had to make enough of those connections to remember the pin outs.

All of this is important because connecting to the console port didn’t work. I knew that the settings were right – something I only knew because of years of experience. I was struck by how this knowledge that in most cases is useless – you don’t see much RS-232 these days – was something useful. Sure I couldn’t figure out initially why it wasn’t working, but I realized that someone coming out of school today may be barely introduced into what a serial port is. No computers come with them any longer. Why would someone teach about serial ports these days? However, in this case knowing how serial works meant that I could identify that I had things right on the software side and allowed me to move quickly from that as a possible cause.

One of the common problems with RS-232 back in the day was the null modem problem. That is that RS-232 was intended for a terminal or modem to be connected to a host system. In order to do this the wires need to be crossed. One device used pin 2 to transmit and 3 to receive and the other device had those reversed. So it was a common issue that we’d have to do some pin flipping. Today you can buy some cute and tiny null modems. We used to make them in the cables. That’s a great suspicion but that suspicion doesn’t do you any good unless you’ve got a way to test it. As it turns out, I still had my RS-232 breakout box from all of those years ago. The one I have is hard to find and still relatively expensive even today. Take a look at what it:

It’s got rows of green and red LEDs which light to indicate what signals are on which lines, a set of switches down the center to allow you to open or close the connection between the two sides. This allows you to separate the two signals and if necessary remap them. The brass posts are designed to have jumpers attached to them so that you can connect various pins with others. On the bottom there’s a convenient set of switches to cross pins 2 and 3 – the most frequent need.

In this case I did in fact need to have a null modem – which I used the breakout box to accomplish since the need was very short term – just getting the device back online.

So why am I telling you this story? Well, I realized that in addition to the learning which most folks aren’t getting today there are also tools that are necessary that most folks won’t have today – a breakout box was a standard tool back then. I should say in defense of the learning that serial communications are still officially listed as a part of the objectives for the CompTIA A+ certification – so there’s some hope that the skill is being taught somewhere. Ultimately, the folks that we’re trying to teach how to use the tools today don’t necessarily have the background or tools to solve the problems we’re creating.

And I’d be the first person to kill off serial communications in a review of useful information today. I’d argue that technologies like USB and working with them is much more important than serial communications – but I’d be wrong. The problem is technicians today have to know BOTH about serial and USB. In other words, they need to know MORE and not less in order to work today. Some have argued that It professionals today have it easy. Everything just works. The argument is that you don’t have to worry about fitting drivers into 1 MB of RAM or how to get a large spreadsheet on a hybrid 8/16 bit processor. However, I’d argue that there are places where this knowledge is essential to success.

And at the same time I’d say if I were looking at learning a 30 year old technology today I’d probably turn my nose up. I’d think that I could surely ignore that technology – but again I’d be wrong. Why the long post about a long perceived dead technology? Because the point is that you never know what you’re going to need to know. I deal with SharePoint every day and I can tell you that there are tons of things that you’ll find helpful if you know – and those are the things that you don’t teach in a SharePoint course. Why not? They’re foundational. They should be known. In the Microsoft Learning 10232 course we make a point of spending the first module talking about advanced ASP.NET concepts like caching and memory management because these things are really helpful to understand for SharePoint development – but things that are often skipped or overlooked as folks try to focus on test driven design, LINQ, ORD, and other more interesting items.

In the end, knowing about serial communications – and packet sniffing – made it possible to figure out what was wrong with the access point and resolve it. I’ll explain the need for packet sniffing in another post.

Virtual Switch Loops

For some time I’ve had some issues on my network. They don’t happen very often so they’ve been hard to track down. I know for sure that one of the issues I have is that one of my Linksys SGE2024 refuses to hold on to its configuration if it loses power. This is made more difficult by the fact that the power supply in the unit has a very low capacitance. As a result of that it will appear to lose its configuration randomly. Take a look at my last post What is a UPS, Really? for the details on that.

However, I also had these weird “storms” where I’d start losing packets and despite my best efforts to diagnose it, I wouldn’t find a problem and suddenly it would be fixed. Honestly there’s nothing that’s more frustrating than a problem that fixes itself before you can put your finger on it — and then comes back later when you’re not ready.

This problem and the lack of diagnostic capabilities on the Linksys SGE2024 switches lead me to replace the switches with some new HP V1910 switches. However, immediately during the deployment of them I started seeing similar — but not identical issues. While not perfect the enhanced diagnostics of the V1910 switches allowed me to sort out that there was a problem with the connection between my virtualization server and the switch.

The virtualization server is a Dell R710 which has four 1 GB Ethernet ports on it. Since I expected that overtime I’d want to get more than 1GB of traffic to the virtual servers, I decided to team the Ethernet ports together to aggregate the total capacity of three of the four ports. The fourth port I’d used for the virtualization host. Wat is called Teaming on the windows side is known on the switch side as Link Aggregation. Generally speaking this aggregation is done through the Link Aggregation Control Protocol (LACP) — although it doesn’t have to be done that way. It defines how the communication should happen to form up an aggregated channel.

When I looked at the LACP status on the switch it told me that the ports weren’t properly bonding together. In more research I found that the infrastructure guy that I had helping me — and a friend of mine — had used network bridging to connect the NICs together and had not in fact actually used network teaming. The teaming had to be setup with a special utility that Dell didn’t see fit to include in the latest driver package. When my friend didn’t see it he tried to do a workaround.

The problem was that each of those adapters was operating independently. The Windows was doing a software bridge of them. This means that every multicast frame that one of the adapters would get would be copied to the other two adapters. This is what a bridge does; multicast (public) traffic is sent everywhere.

So imagine a scenario where each of these three adapters is in turn bound to a virtual switch inside the server. Well the virtual switch itself will send out multicast traffic on every port. Then imagine that the physical switche which the physical network adapters are connected to is also replicating the packets. What you end up with is the same packet getting sent over and over again. I have no idea whether it was three times per iteration or six times per iteration or how many it was but I do know that as soon as I plugged in the second adapter into the physical switch I got a lot of traffic very quickly.

So the mystery was somewhat solved. The packet loss was because I was accidentally flooding the switches. I had created a wiring loop — even if part of that loop was completely virtual. Fix the teaming and the problem went away. The network started working just fine.

So why did the network ever work? Well, eventually the switches were doing flood protection and shutting down some of the ports — but this took 15 minutes or more in some cases. So about the time I’d really get into looking at the problem the switch would have solved the problem — again back to diagnostics, the relatively poor diagnostics made it impossible for me to figure out what was going on with the Linksys — that’s really disappointing for a product that’s supposed to be marketed to people that can make mistakes.

power tower

What is a UPS, Really?

What most of us believe — that uninterruptable power supplies (UPS) today are uninterruptable – isn’t exactly true. What companies advertise as UPSs are in fact standby power supplies (SPS). What’s the difference? Well, in a true UPS the power never falters when the utility power is interrupted. There are several ways to do this but sometimes it’s done with a charger and an inverter. The charger converts the alternating current (AC) that the utility provides into the direct current (DC) that the battery can store. The inverter converts the DC into AC. It’s also been done with flywheels and other approaches too but the result is the same. The problem with this approach is that it’s inefficient. You lose some of the power by doing the two sets of conversions. The benefit is the output power is “as smooth as a baby’s bottom.” Because of the efficiency loss, only the most specialized applications use UPS.

Most of the “UPS devices” that we see in the market are really SPSs. The SPS work by keeping the load (usage) on the utility power while monitoring it. It’s also running a charger to ensure that the internal battery is always fully charged and ready to deliver power. Once the utility power falls out of range because of over voltage, under voltage (brown out), or outage (black out) the SPS transfers the load to an inverter. The inverter keeps the load going while the utility power is out. After a minimum switchover time and the utility power reentering a safe zone the load is transferred back to the utility.

The minimum switchover time is designed to prevent rapid transitions between utility power and inverter if the utility power isn’t stable. That’s important when you consider that during the transition to the inverter and back to utility power there’s a small gap in power. Most devices today can handle a small gap every now and then but they can’t take many of them in a small period of time.

Most devices (as opposed to lights or machines) today are actually internally running on DC current. From the computer to the DVD player internally they’re converting the AC power back into DC power for consumption. This can be done a few different ways but for the most part this is done through a switching power supply (sometimes called a switched mode power supply). This type of design has a small amount of capacitance depending upon the specific design. Capacitance is like a small battery. It’s the ability to keep the output side going with small fluctuations in the input power.

Some switching power supplies have a large amount of capacitance and others have a smaller capacitance. The problem is that sometimes when you couple a SPS with a switching power supply you can have a problem when you have a switching power supply with a low capacitance and a SPS with a high switching time you can run out of power before the SPS has made it to transfer the load to the inverter.

The impact of this is the device has a short power outage and reboots. If the process of rebooting is quick you may not even notice it. However, if your device takes a while to reboot it will be frustrating — and very hard to find the source of the problem. Most of the SPSs in use today don’t have monitoring or logging. Without monitoring or logging you don’t really know that the SPS switched the load to the inverter — and caused the momentary power outage. Larger units typically have this sort of logging like the RU2200 with the network management (Ethernet) card in it that I use to keep my virtual machine host operational. Take a look at the output of an event earlier this year.

You’ll see that several times the system transitioned to battery power for two seconds – the minimum switching time. You’ll also notice that even on these devices it’s somewhat frustrating because the events are scattered in with other messages – like the DHCP release being renewed. I’ve blogged before that many issues with technology are the result of bad power. That’s true even when you’re trying to ensure that the power is good.

In the case of the virtualization server I’ve got plenty of capacitance in the power supplies (they’re oversized for the load on them which tends to increase capacitance) and the RU2200 has a relatively quick switching time so I’m good. However, my Linksys/Cisco SGE2000 connected to a CyberPower CPS1500AVR doesn’t work so well. It has a tendency to reboot – and lose its configuration but that’s another story… I can figure out what happened because I check the other SPSs to see that they transferred – and I can infer that the CyberPower transferred power since they’re on the same power source.

Keep an eye out when you’re buying a SPS in the clothing (or branding) of a UPS.

Recent Posts

Public Speaking