Recently in networking Category

This is inspired from last night's LOPSA DC meeting. The topic was IPv6 and we had a round-table.

One of the big questions brought up was, "What's making me go IPv6?"

The stock answer to that is, "IPv4 addresses are running out, we'll have to learn at some point or be left behind."

That's all well and good, but for us? Most of us are working in, for, or with the US Government, an entity that is not going to be experiencing v4 address scarcity any time soon. What is going to push us to go v6 (other than the already existing mandate to have support that is)?

In my opinion, it'll come from the edges. IPv6 is a natural choice for rapidly expanding networks such as wireless networks, and extremely large networks like Comcast/Verizon run for their kit. These are two areas where sysadmins in general don't deal with much at all (VPN and mobile-access being the two major exceptions).

If your phone has an IPv6 address and accesses the IPv4 internet through a carrier-grade NAT device, you may never notice. Joe Average User is going to be even less likely to notice so long as that widget just works. Once v6 is in the hands of the "I don't care how it works so long as it works" masses, it'll start becoming our problem.

Once having a native v6 site means slightly better perceived mobile performance (those DNS lookups do cause a bit of latency you know), you can guarantee that hungry startups are going to start pushing v6 from launch. Once that ecosystem develops it'll start dragging the entrenched legacy stuff (the, er, government) along with it.  Some agency sites are very sensitive to performance perception and will adapt early. Others only put their data online because they were told to and will only move when the pain gets to be too much.

Business-to-business links (or those between .gov agencies, and their .com suppliers) will likely stay v4 for a very, very long time. Those will also be subject to pain-based mitigation strategies.

But the emergence of v6 on mobile will likely push a lot of us to get v6 to at least our edges. Internal use may be long time coming, but it'll show up at all because of the need to connect with others.

I've wondered why IPSec VPNs do that for a while, but hadn't taken the time to figure out why that is. I recently took the time.

The major reason comes down to one very big problem: NAT traversal.

When IPSec VPNs came out originally, I remember there being many problems with the NAT gateways most houses (and hotels) had. It eventually cleared up but I didn't pay attention; it wasn't a problem that was in my area of responsibility, so I didn't do any troubleshooting for it.

There are three problems IPSec VPNs encounter with NAT gateways. One is intrinsic to NAT, the other two are specific to some implementations of NAT.

1. IPv4 IPSec traffic uses IP Protocol 50, which is neither TCP (proto 6) or UDP (proto 17), and protocol 50 uses no ports on the packet. Therefore, a VPN concentrator can only support a single VPN client behind a specific NAT gateway. This can be a problem if four people from the same company are staying in the same hotel for a conference.
2. IPv4 IPSec traffic uses IP Protocol 50, which is neither TCP or UDP. Some NAT gateways drop anything that isn't TCP or UDP, which will be a problem for IPSec VPNs.
   
3. NAT gateways rewrite certain headers and play games with packet checksums, which IPSec doesn't like. So if IPSec is going to tunnel via TCP or UDP, there will be issues.

These are some of the reasons SSL VPNs became popular.

This is where RFC 3751 comes in. It's titled, "IPsec-Network Address Translation (NAT) Compatibility Requirements" oddly enough. It turns out that packet checksums are not required for IPv4 UDP packets, which makes them a natural choice to tunnel an IPSec VPN through a stupid NAT gateway. The VPN concentrator pulls the IPSec packet out of the UDP packet, and thanks to the cryptographic nature of IPSec it already has ways to detect packet corruption and will handle that (and any required retransmits) at the IPSec layer.

Part of the blog-silence the past few weeks is because I was on vacation.

Nice, nice vacation.

However, it was at my parent's place and as with any technically savvy sprog who comes home, there be questions. In my case, it was an ongoing problem of slowness with the network. The first few days I didn't have time to delve, but I did eventually get into it.

The symptoms:

• Wifi download speeds were about 75KB/s, and upload about 3x that. Yes, upload was faster. Yes, my 4G phone had faster downloads.
  
• Wired speeds were at the rated speeds for the broadband connection.

Clearly, something was wrong with the wireless.

My little spectrum analyzer showed remarkably clean airwaves for a residential area. There was some congestion on their frequency, but moving it didn't help much.

And then I noticed something when I ran iwconfig on my laptop:

wlan0     IEEE 802.11abgn  ESSID:"dadhome"  
          Mode:Managed  Frequency:2.462 GHz  Access Point: 06:1D:61:FF:BB:00   
          Bit Rate=54 Mb/s   Tx-Power=15 dBm   
          Retry  long limit:7   RTS thr:off   Fragment thr:off
          Encryption key:on
          Power Management:off
          Link Quality=70/70  Signal level=-31 dBm  
          Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
          Tx excessive retries:2942  Invalid misc:1892   Missed beacon:0

Specifically, that "Tx excessive retired" statistic was incrementing, and I'd never seen that other than zero before. Most odd.

The router was a Cisco/Linksys, and a pretty new one at that. The firmware was latest, so that wasn't it. After a bit of poking about I found out that I could get vastly better throughput by setting the Wifi to G-Only, instead of B/G/N. In fact, setting it to N-only made the problem worse! Clearly, this router's N implementation is a bit off.

Wifi wasn't at the broadband speed yet, but it was still a vast improvement. That's where I left it, and they're happy.

I've been talking about this one for a while around the new .xxx top-level-domain. Simply put, some organizations such as my old employer consider a certain string of characters to be their trademark regardless of what follows the . in the name. This is why WWU had at the time I left mid twenties of domains that all redirect to wwu.edu. The same will happen with .xxx

And also with .sex, .adult and .porn.

Each new TLD means yet another domain to buy defensively.

For the extremely cash-strapped non-profit, now entering year 5 of shrinking or stagnating budgets, such forced trademark defense expenses are highly resented.

The new .XXX top-level-domain is coming on line later this year. Apparently the organizers of the domain are having a hard time convincing the porn industry to get on-board.

Most left feeling like there were few reasons to buy a .XXX domain other than in self-defense - which may whisper the future of what an unlimited number of vanity gTLDs will look like.

I totally agree. Back on March 28th, 2007 I said this regarding this very TLD:

Which is very true. They won't be reaping the big bucks by signing over the seamy side of the internet, they'll be making the big bucks from reputation protection buys from the likes of us. Higher Education domains like ours, WWU.EDU, are PRIME PICKINGS for .XXX. You can fully imagine that "WWU.XXX" will be full of ***Hot CoEd Action***, if we don't get it first. So you can further guarantee that our Telecom group will dilligently pick up "WWU.XXX" in order to maintain our reputation. So whoever is granted the authority to register ".XXX" domains will be getting our money, and most other .EDU domains as well.

That's what the gold-rush of the now wide open gTLD system looks like to organizations with a brand to protect. As it happens, WWU is also sensitive to alcohol consumption by students, so a hypothetical "WWU.BEER" would be yet another defensive buy on the part of the University. Each new TLD that comes up will have to be vetted by trademark and copyright owners to determine if a defensive buy is in order to protect their brandname.

I haven't been doing that lately. This is due in no small part to a week's vacation I recently took. But it's also because I've been dealing with Product Next and... that's company IP right there, and complaining/explaining the problems I'm facing around it would count as intelligence to our competitors so... I can't talk about it. So blog posts, no SF-questions even though they'd help. Maybe I need a sock-puppet over there... hm.

This is crimping my style somewhat, but I expected this when I moved out of the public sector. I do apologize for that.

There is one thing I can talk about though, phones. Specifically, changing ours out. The system we have was purchased some time in the 4-6 years ago range and is probably a good example of the state of IP telephony of the time. One of the first things I was half-asked to do (half-asked = a higher up complained about it but didn't demand action on it) was find something more modern. I was full-asked last week.

I can't blame them, really. As an IT professional, even one with a previously casual relationship with the networking and phones world (my previous two jobs were tightly siloed, so I never got to play with routers or 110-blocks) I can recognize that this phone system is chock-full of what I call "TelCo crap".

TelCos. You know, those people who built continent spanning networks before digital switching was invented. And after digital switching was invented, encoded all of the standards in the mode of the era while carrying over some analog-switching thoughts because the edges still had analog switches. Someone used to Google Voice or Skype will take one look at that and see, "A maze of twisty passages, all alike." Or worse, "I was designed for mainframes, here is my 7300 page manual."

I am not at all surprised that big money can be made by providing phone-simplification services to companies. In olden days that meant outsourcing the PBX maintenance and calling a service to turn on new extensions. These days with Office Communication Server, Asterisk, Skype, Google Voice, and whatever else promising to get rid of the PBX entirely there are a lot of options.

So, yeah. I need a small-business friendly phone system that isn't OCS or Skype and can use phone-like devices. With a pretty-looking web-portal that allows checking of voice mail, preferably mobile-friendly. I'm open to recommendations and experiences. In the mean time, I'll slip the research into my non Product-Next time.

We decided to change our hold music off of the stock music on our UC520 to something a bit less... like default hold music (the "MOH file"). The instructions for doing so were pretty straight forward, the hard part was creating the music file. The work can be done in Audacity, but the options are hidden. And I didn't find much out there for generating this, so this is my attempt to help people out.

Before I get into this, a warning:

---

There are copyright issues surrounding hold music. You can't just plug a loaded iPod into the External Music port and hit play, shunt a SiriusXM feed into it, or feed a random Pandora station into that port. Nor can you hunt the web for the perfect tune that is both inoffensive and yet un-sucky. As I understand it, the music has to be free for commercial use, or you must hold a license to use it for commercial use.

---

Back to the How-To.

Step 1: Open the file in Audacity

I do review our firewall logs, and do what I can to keep them clear of stuff coming from my network. Even so, there is still a lot of stuff there. One particular IP address had been port-scanning us very persistently for weeks. Last Thursday I finally had enough and looked into it.

The weird thing about the log-lines was that the source TCP port was TCP/443. The target ports were the usual assortment of high-order ports. Most odd.

The reverse lookup of that IP didn't give me anything. However, an openssl query to see what SSL certificate it was offering did give me information.

One of our cloud-app providers.

Wha?

Time to sniff packets. And sniff I did. I got a couple minutes and it was long enough to get a few port-scan attempt log-entries. After correlating the source/target port numbers one pattern was pretty clear:

The cloud-app provider was issuing double RST packets after receiving a FIN packet from our side. The first RST tore down the connection as far as the Firewall was concerned, which made the second one seem like a packet from nowhere with strange flags on it and therefore suspicious.

I did helpfully tell said cloud-app provider about this, and they send back a, "situation normal, nothing to see here," reply, and googling around does show some systems do use a double-RST method for tearing down connections. I'm not sure why this is a good idea in some cases, but it's pretty clear our firewall doesn't like it. At least this is a bit of log-spam I can now safely ignore.

Tonight I'll be doing a new thing. I'll be going to a sporting event, tickets provided by a trio of vendors. There will be box seats. Ostensibly the #1 companies in the Fibre Channel Storage, Virtualization, and Networking spheres (I think you know who they are) are trying to convince me that their cloud/grid/massively-scaled-out solutions really are the best out there and they deserve our business. We'll see about that.

What's new is that this is not the kind of thing I could accept at WWU, or at least it was a big gray area. My boss most definitely couldn't. Since I'm actually a decision maker here (or at least a strong influencer) for IT purchases, and having spent my entire career thus far in the civil service, this seems... strange to me. I know it happens, but to have it happen to me is still kind of odd.

I was able to talk 'em into letting me take my Intern along. It'll be a nice thing all around for him, since he's pretty network focused and they'll be talking about the kind of grand high geekery that even the good tech schools don't really cover.

Too bad (or perhaps it would be a good thing) it isn't a DC United game.

You'd think I'd be doing something, but I'm not, really. The full extent of what I'll be doing is dealing with users on our network troubleshooting why access some now-v6-available services don't work out there on the greater internet. It'll be interesting data to have, and may be useful more widely. But...

Our network isn't really v6-ready. Our core network infrastructure just doesn't have everything it needs, or if it does it's hiding from me. Also, our firewall software is just old enough I don't trust it with v6.

As for the home, I've long had a project to get a v6 tunnel going but that's been blocked since I moved out east. In large part because I suddenly have a lot less time at home, and I also need better hardware for the device that'd have to be our IPv6 gateway; the 8yo Linux server is technically good enough but I no longer trust the hardware to stay up.

So, I'm going to be spending this World IPv6 day on resolutely v4 networks. Alas.