Recently in networking Category

The new .XXX top-level-domain is coming on line later this year. Apparently the organizers of the domain are having a hard time convincing the porn industry to get on-board.

Most left feeling like there were few reasons to buy a .XXX domain other than in self-defense - which may whisper the future of what an unlimited number of vanity gTLDs will look like.

I totally agree. Back on March 28th, 2007 I said this regarding this very TLD:

Which is very true. They won't be reaping the big bucks by signing over the seamy side of the internet, they'll be making the big bucks from reputation protection buys from the likes of us. Higher Education domains like ours, WWU.EDU, are PRIME PICKINGS for .XXX. You can fully imagine that "WWU.XXX" will be full of ***Hot CoEd Action***, if we don't get it first. So you can further guarantee that our Telecom group will dilligently pick up "WWU.XXX" in order to maintain our reputation. So whoever is granted the authority to register ".XXX" domains will be getting our money, and most other .EDU domains as well.

That's what the gold-rush of the now wide open gTLD system looks like to organizations with a brand to protect. As it happens, WWU is also sensitive to alcohol consumption by students, so a hypothetical "WWU.BEER" would be yet another defensive buy on the part of the University. Each new TLD that comes up will have to be vetted by trademark and copyright owners to determine if a defensive buy is in order to protect their brandname.

I haven't been doing that lately. This is due in no small part to a week's vacation I recently took. But it's also because I've been dealing with Product Next and... that's company IP right there, and complaining/explaining the problems I'm facing around it would count as intelligence to our competitors so... I can't talk about it. So blog posts, no SF-questions even though they'd help. Maybe I need a sock-puppet over there... hm.

This is crimping my style somewhat, but I expected this when I moved out of the public sector. I do apologize for that.

There is one thing I can talk about though, phones. Specifically, changing ours out. The system we have was purchased some time in the 4-6 years ago range and is probably a good example of the state of IP telephony of the time. One of the first things I was half-asked to do (half-asked = a higher up complained about it but didn't demand action on it) was find something more modern. I was full-asked last week.

I can't blame them, really. As an IT professional, even one with a previously casual relationship with the networking and phones world (my previous two jobs were tightly siloed, so I never got to play with routers or 110-blocks) I can recognize that this phone system is chock-full of what I call "TelCo crap".

TelCos. You know, those people who built continent spanning networks before digital switching was invented. And after digital switching was invented, encoded all of the standards in the mode of the era while carrying over some analog-switching thoughts because the edges still had analog switches. Someone used to Google Voice or Skype will take one look at that and see, "A maze of twisty passages, all alike." Or worse, "I was designed for mainframes, here is my 7300 page manual."

I am not at all surprised that big money can be made by providing phone-simplification services to companies. In olden days that meant outsourcing the PBX maintenance and calling a service to turn on new extensions. These days with Office Communication Server, Asterisk, Skype, Google Voice, and whatever else promising to get rid of the PBX entirely there are a lot of options.

So, yeah. I need a small-business friendly phone system that isn't OCS or Skype and can use phone-like devices. With a pretty-looking web-portal that allows checking of voice mail, preferably mobile-friendly. I'm open to recommendations and experiences. In the mean time, I'll slip the research into my non Product-Next time.

We decided to change our hold music off of the stock music on our UC520 to something a bit less... like default hold music (the "MOH file"). The instructions for doing so were pretty straight forward, the hard part was creating the music file. The work can be done in Audacity, but the options are hidden. And I didn't find much out there for generating this, so this is my attempt to help people out.

Before I get into this, a warning:

---

There are copyright issues surrounding hold music. You can't just plug a loaded iPod into the External Music port and hit play, shunt a SiriusXM feed into it, or feed a random Pandora station into that port. Nor can you hunt the web for the perfect tune that is both inoffensive and yet un-sucky. As I understand it, the music has to be free for commercial use, or you must hold a license to use it for commercial use.

---

Back to the How-To.

Step 1: Open the file in Audacity

I do review our firewall logs, and do what I can to keep them clear of stuff coming from my network. Even so, there is still a lot of stuff there. One particular IP address had been port-scanning us very persistently for weeks. Last Thursday I finally had enough and looked into it.

The weird thing about the log-lines was that the source TCP port was TCP/443. The target ports were the usual assortment of high-order ports. Most odd.

The reverse lookup of that IP didn't give me anything. However, an openssl query to see what SSL certificate it was offering did give me information.

One of our cloud-app providers.

Wha?

Time to sniff packets. And sniff I did. I got a couple minutes and it was long enough to get a few port-scan attempt log-entries. After correlating the source/target port numbers one pattern was pretty clear:

The cloud-app provider was issuing double RST packets after receiving a FIN packet from our side. The first RST tore down the connection as far as the Firewall was concerned, which made the second one seem like a packet from nowhere with strange flags on it and therefore suspicious.

I did helpfully tell said cloud-app provider about this, and they send back a, "situation normal, nothing to see here," reply, and googling around does show some systems do use a double-RST method for tearing down connections. I'm not sure why this is a good idea in some cases, but it's pretty clear our firewall doesn't like it. At least this is a bit of log-spam I can now safely ignore.

Tonight I'll be doing a new thing. I'll be going to a sporting event, tickets provided by a trio of vendors. There will be box seats. Ostensibly the #1 companies in the Fibre Channel Storage, Virtualization, and Networking spheres (I think you know who they are) are trying to convince me that their cloud/grid/massively-scaled-out solutions really are the best out there and they deserve our business. We'll see about that.

What's new is that this is not the kind of thing I could accept at WWU, or at least it was a big gray area. My boss most definitely couldn't. Since I'm actually a decision maker here (or at least a strong influencer) for IT purchases, and having spent my entire career thus far in the civil service, this seems... strange to me. I know it happens, but to have it happen to me is still kind of odd.

I was able to talk 'em into letting me take my Intern along. It'll be a nice thing all around for him, since he's pretty network focused and they'll be talking about the kind of grand high geekery that even the good tech schools don't really cover.

Too bad (or perhaps it would be a good thing) it isn't a DC United game.

You'd think I'd be doing something, but I'm not, really. The full extent of what I'll be doing is dealing with users on our network troubleshooting why access some now-v6-available services don't work out there on the greater internet. It'll be interesting data to have, and may be useful more widely. But...

Our network isn't really v6-ready. Our core network infrastructure just doesn't have everything it needs, or if it does it's hiding from me. Also, our firewall software is just old enough I don't trust it with v6.

As for the home, I've long had a project to get a v6 tunnel going but that's been blocked since I moved out east. In large part because I suddenly have a lot less time at home, and I also need better hardware for the device that'd have to be our IPv6 gateway; the 8yo Linux server is technically good enough but I no longer trust the hardware to stay up.

So, I'm going to be spending this World IPv6 day on resolutely v4 networks. Alas.

Yesterday we were caught up in the Media Temple DNS server outage. Details about what happened are found here. We were out name resolution for two and a half hours. Paying customers couldn't get at our hosting, and our email stopped flowing.

This is not an outage I'd dealt with before, as both WWU and my earlier job self-hosted their DNS servers. WWU had one off campus, but the first two records were on campus.

This was a case of, "a lot of fire, nothing I can do about it," which is one of the hardest major events to endure as one of the prime fire-fighters. Happily, people figured out fairly quickly that this was an external event we had no control over. Until that time, the statusing was pretty intense.

And unfortunately, there is nothing much you can do about a DNS domain outage. The 24-48 hour DNS propagation time pretty much shoots down any short term quick-fixes. I went several rounds with a couple people here about ways to work around the problem and there was much educating going on as a result. We did give some end-users the IP address of the servers in question. And then the SSL based services stopped working, of course.

As with any major failure of a system you never thought to think about before, there is discussion about moving our DNS to somewhere else. We'll see if this holds up.

(Now coming to you from the East Coast of the US)

One of the things that struck me during the Cascadia IT Conference was the impact of IPv6 on IP reputation services. I've blogged about this in the past, but IP reputation is a very key spam-fighting technique. DNS RBLs have been around for over a decade now and remain the free option. In the paid anti-spam realm, the big vendors manage IP reputation databases to determine whether or not an incoming connection is worth of their time, and usually provide better granularity than the RBLs do. The same applies to blog comment-spam, as it happens.

The DNS-RBL functions very simply:

1. A connection is made from the Internet.
2. The mailer/blog-engine performs a lookup of the IP in the black-hole list.
3. The RBL returns a value.
4. The mailer/blog-engine acts on that value.

In an era where Comcast is passing out whole /64's to end-users, which in turn means end users can have more IP addresses than are available on the IPv4 Internet, this one-to-one style of lookup breaks. Obviously, a one-to-one port of the IPv4 RBL code to IPv6 will be not nearly as effective as it is with just IPv4.

The solution is fairly obvious, start blacklisting subnets, but the code-changes are non-trivial. Right now a stock RBL can be made with BIND and a standard Zone file filled with A records. Classful IPv4 subnets can be blacklisted with wildcard DNS entries. The same can be done for IPv6 zone-files, but the granularity is a lot better. Of course, RBL-clients need to be updated to handle RBL-lookups with v6 addresses.

Which is to say, that in the IPv6 future, subnet will matter more than discrete IP Address for many things. This is one of the areas that everything that relies on IP addresses for access decisions will have to start taking into consideration (as well as the people who encode the rules).

Summary:
The end-user environment is becoming heterogeneous at a time when our central computing environment is becoming more homogeneous due to cost constraints. This has impacts to how we do business. The latest features may not be able to be deployed until the lagging platforms get sufficient support. Formerly ignored platforms will need central resources to effectively manage them.

Comcast, the United States largest ISP, has a vested interest in IPv6. They've been posting their progress on a web-page:

http://www.comcast6.net/

For those wondering when they'll get a v6 capable ISP, and I know many of my readers are technically inclined enough to desire such, it is coming. I haven't dropped by the site in a while, it doesn't update much, but they did present at IETF recently and posted their slides. One slide-deck in particular is interesting.

Down on page 5:

* Delegated prefix is minimally a /64 by default for trial (LAN-­‐side)

That is a subnet being allocated to customers, not a single IPv6 address. Clearly, Comcast is leaving it up to customers to figure out if they want to use some kind of address-translation or just use straight-up public IPv6 addresses on their home network. This kind of thing is exactly what the authors of IPv6 had in mind, and not coincidentally is one of the biggest critiques of IPv6... the role of NAT.

NAT has been a network security 'best practice' for years. While firewalls provide robust network security, hiding the IP-space behind the firewall makes planning an attack on that IP-space take longer. A fact that has been enshrined in the PCI-DSS standards.

IPv6 NAT exists now, which it didn't shortly after IPv6 was ratified. The question remains over what addresses you use to replace the obsolete-in-v6 RFC1918 addresses we all know and love. The answer to that is one of two types of addresses:

• If your home network is completely flat, you only have one subnet, then Link Local addresses (fe80:/8) will work just fine. These addresses are explicitly non-routed, so a NAT gateway at the edge of the network is required.
  
• If your home network has multiple subnets, Unique Local addresses are what you want. Unlike Link Local, ULA's are routeable. NAT is optional, but your home-ISP is vanishingly unlikely to agree to route those addresses.

However, running without any kind of address translation is not as insane as it sounds. Keep in mind a few points.

• Comcast is handing out a /64 subnet to you, so your attacker already knows what your IP space looks like.
• A /64 provides a mind bogglingly huge number of potential addresses. 2^64 worth! That's vastly, vastly larger than the entire 2^32 IPv4 address space. While the nature of IPv6 autoprovisioning reduces the actual number of addresses that are likely to be provided, scanning it is still very infeasible.
  
• Unless you set up your own domain to provide it, Comcast will not be providing forward or reverse DNS lookups to your /64-worth of IP addresses. This greatly reduces the ability of attackers to recon your network.

Running without a firewall is still just as insane as it sounds, though. Happily, you can do firewalling without having to NAT.

Running 'in the clear' will be an absolute boon to server-in-the-home folks. Get a domain from registrar-of-choice and set up your DNS records to point to the server on your home /64 and away you go. Want to play with multiple web-servers? No problem! Everyone can use port 80 on their very own IPv6 address, no need to monkey with :8080 :8008 and so on. Of course, terms-of-service violations may get this behavior whacked, but I can guarantee it'll happen regardless of what the TOS forbids.

Whole suites of protocols and applications just plain work better when there isn't a NAT gateway getting in the way. Peer-to-peer networks of all kinds are a LOT easier to maintain without NAT. We're still years and years from a fully featured pure-v6 Internet experience, but I like the way it looks from here.