My MediaOne Experiences

Wierd Network Slowdowns

For a day or two late in 1999, my connection to MediaOne became so slow when I tried to download large files that it was unusable. Ping times to the local gateway, normally 2ms, shot up to several thousands of milliseconds, often nearly a multiple of one second. The problem seemed to disappear after I rebooted. It also went away if I disabled then reenabled the ethernet card, but would come back in two seconds if I downloaded a large file from any one of a number of sites. Two friends had similar problems. We were all using Linux with Masq support enabled - Clay was using kernel 2.2.9, I was using Red Hat 6's 2.2.5. After thinking for a minute, I recalled seeing a patch, http://kernelnotes.org/lnxlists/linux-kernel/lk_9909_04/msg00406.html, on the Linux Kernel mailing list that solved a similar problem. We all applied it, and by gosh, it solved our problems.

This patch is included in the kernel as of 2.2.13pre15. It's also in Red Hat 6.1.

Not sure if I should tell MediaOne about this, since they don't seem to want people to use Linux.

Wierd Problems Caused by IP Address Ending in .255

For about two weeks early in 1999, I had a very odd problem connecting to web pages at www.caltech.edu, www.jpl.nasa.gov, www.usc.edu, www.hotmail.com, and probably many other sites.

My problems started when my IP address was changed by the ISP's DHCP server to 24.130.78.255. Note that it ends in .255. Many system administrators block access from addresses that end in .255, to guard against a security problem called smurfing. (This is a simple and effective defense against smurfing, but it isn't the officially recommended one, as it hurts innocent people.)

About two weeks after I reported the problem, my computer received a new IP address that did not end in .255, and the problem went away.

I don't know how many ISPs still hand out .255 addresses to other people. If you get a .255 address, and have trouble accessing a few sites on the web, complain, and demand a non-.255 address; it can't hurt, and it might solve your problem.

Tracking down the problem

The following was written while the problem still existed. Eventually, I'll rewrite it in the past tense.

The other people at my ISP who have tested this do not have the same problem. The support staff at mediaone.net can't reproduce this problem and have not provided any useful support (beyond sympathy).

Note that even someone on my same subnet but with a non-.255 address does not have the same problem!

Here are logs from traceroute exploring the problem. Note especially that I cannot reach 198.32.146.10 nor 130.152.60.1, two routers which are only four or so hops away from me.

The most likely explanation is given at the bottom of this page- I may have fallen victim to a rough defense against smurf attacks. If this is the case, MediaOne (and any other domain with bigger than class-C networks) should probably reserve .255 and .0 addresses, and not hand them out to customers, as routers which use this defense will ignore them! Alternately, they should work with router admins who use this defense to set up less drastic defenses.

Given two computers, one with a normal-looking address and from an address ending in .255, one could easily implement a scanner that pinged known routers from both computers. Routers that respond to the normal probe but don't respond to the probe from the .255 address probably suffer from this problem. It would be in MediaOne's interest to implement such a scan.

My conversations with MediaOne Support and other NOCs

  1. On 4 Feb 99, I reported the problem initially via email.
  2. I received the following autoresponse: 
    From: "MediaOne Express Technical Support" (mailsupport@ne.mediaone.net)
Date: 04.02.99 14:29
Subject: RE: hotmail?
To: "Dan Kegel" 

We have received and logged your E-mail message. Our team of Technical
Support Specialists is working to respond to your information request
within 24 hours.
  3. On 6 Feb 99, I reported the problem in more detail on via phone and email.
  4. On 13 Feb 99, I reported the problem in more detail via phone and email, and gave them this URL (http://www.kegel.com/mediaone.html).
  5. I received the following autoresponse: 
    From: "MediaOne Express Technical Support" 
Date: 13.02.99 11:14
Subject: RE: routing problem at mae-la-CCIBRT.mediaone.net?
To: "Dan Kegel" (dank@alumni.caltech.edu)

We have received and logged your E-mail message. Our team of Technical
Support Specialists is working to respond to your information request
within 24 hours.
  6. On 13 Feb 99, I emailed the technical contact for ln.net alerting them to the odd behavior of lap.ln.net.
  7. On 14 Feb 99, I started a discussion in comp.security.firewalls about the problem. My post is here. A reply from Tony Rall said:
    There basically isn't anything an origin site or intermediate router can do to prevent smurfs. What it would have to do is filter directed broadcasts, but it needs to know the destination subnet mask to do this; in general, there is no way to know this.

    So the lap.ln.net router should not have been filtering .255 source addresses if it was trying to block smurfing. What it is doing is simply denying service to all systems using such an address.

  8. I received the following real response from MediaOne: 
    From: "MediaOne Express Technical Support" (mailsupport@mediaone.net)
Date: 16.02.99 08:53
Subject: RE: routing problem at mae-la-CCIBRT.mediaone.net?
To: "'Dan Kegel'" (dank@alumni.caltech.edu)

Thanks for writing.
I will pass along your information to the appropriate parties.
Thank you for the information.
  9. I received the following real response from a different group at MediaOne: 
    From: WE-BDSFeedback (WE-BDSFeedback@MediaOne.com)
Date: 17.02.99 08:44
Subject: RE: hotmail?
To:  "'Dan Kegel'" (dank@alumni.caltech.edu)

Thank you I will forward this information to our Network Admin for
follow-up. I am sorry our IP assignment is causing you problems such as
this.
  10. On 18 Feb 99, my computer was assigned a non-.255 address, and the problem went away.
  11. About two hours later, I received the following note from MediaOne: 
    From: WE-BDSFeedback (WE-BDSFeedback@MediaOne.com)
Date: 18.02.99 08:10
Subject: RE: hotmail?
To:  "'Dan Kegel'" (dank@alumni.caltech.edu)

The problem has been resolved. You will need to recycle your modem to
receive a IP update. There are several ways to achieve this:
1.      Log out of MediaOne.
2.      Turn off your computer.
3.      Power cycle the modem.
Thank you,
MediaOne Express

    A Likely Explaination

    Out of the blue, a friend I haven't heard from for years wrote me in response to my web page (which was created 12 hours ago!): 
    Hi Dan - this is Mike Newton in yet another existence....

Just saw your page:
        http://www.kegel.com/mediaone.html

due to a random walk.  I can shed some light on it....:

I do lots of network security consulting.  One of the common problems
is DOS (denial of service) attacks.  Smurf's are particularly a problem
at some sites.  Also, very few sites actually run with class A or B
subnets internally.  For example, the non-exist class B of 128.0/16
(128.0.0.0) would usually be subnetted internally to 128.0.0/24 or
some such.  So, because of smurfs (forged icmp echos to broadcast
addresses - a luser on a dialup line can fill a T-3 easily) most
places block broadcast addresses both in and out (so they don't
get in trouble from rambunctious internal people).

Depending upon the place/equipment (router/firewall/version) a
common way of doing this is to prevent all '.0' and '.255' addresses
from going into/out of an org.  Some places even do it at the ISP
or not-quite-backbone level.

I'd get a different IP address.  You are always going to have problems
as it is impossible to tell how people are going to subnet and so
many places will play it safe and block as above.  For large sites
the necessary rules to block it in a cisco are so large as to 
slow the router down (depending upon model, path, ...).

Still lost in space and time...
- mike

ps: you're welcome to use any of the above.  Don't put an e-mail
(or other) pointer to me on the web though -- you can leave my name
off or on as you feel if you do use it.
    http://www.kegel.com/mediaone.html
    Dan Kegel
    dank@alumni.caltech.edu