Saturday, March 9, 2013

of the pulling of plugs and dead fios connections...

Oh no!  The internet is down!!!

So we have had FiOS for about five years now. It's been great until today. We have a business line, pay good money for it, and it has worked mostly fine, and I've never had to call support until today. More of that later on...

First sign, the TiVo was not connecting... rotroh! How are we going to survive!  I thought nothing much of it, there were after all multiple recorded episodes of Bearing Sea Gold to be watched! :-)

I did eventually surface, and do the regular reboot house routine - power off wifi, pull the little green glass fiber optic. Doh! Still nothing much going on there, still dead, they are probably working on something, no biggy. Went shopping and decided to generally get on with my life.

Came back from the shopping mission and started to look at why things were busted. Very odd, the wifi/wan box actually had an IP address, but things were very still much fail, IPs could resolve, but traffic was like molasses or non existent.  This *has* to be an upstream problem I thought to myself...

Decided to call Verizon - in retrospect that was my first big mistake.

An hour plus of back and forth and many powering off and on of "the router" resulted in nothing more than the technician managing to release my DHCP address. I knew I had IP connectivity, that was not the issue, the network was clearly broken...

I'm not a CCIE, but I know that this response from a ping ain't right! Me trying to explain this to the technician was probably worth hearing, and could make for a good comedy sketch at some point, but I digress... :-)

jcuff-air:~ jcuff$ ping www.google.com
PING www.google.com (173.194.75.103): 56 data bytes
Request timeout for icmp_seq 0
64 bytes from 173.194.75.103: icmp_seq=1 ttl=250 time=120.565 ms
64 bytes from 173.194.75.103: icmp_seq=1 ttl=250 time=122.475 ms (DUP!)
64 bytes from 173.194.75.103: icmp_seq=1 ttl=250 time=122.661 ms (DUP!)
64 bytes from 173.194.75.103: icmp_seq=1 ttl=250 time=122.671 ms (DUP!)
64 bytes from 173.194.75.103: icmp_seq=1 ttl=250 time=122.677 ms (DUP!)
64 bytes from 173.194.75.103: icmp_seq=1 ttl=250 time=122.684 ms (DUP!)
64 bytes from 173.194.75.103: icmp_seq=1 ttl=250 time=122.693 ms (DUP!)
64 bytes from 173.194.75.103: icmp_seq=1 ttl=250 time=122.699 ms (DUP!)
64 bytes from 173.194.75.103: icmp_seq=1 ttl=250 time=122.705 ms (DUP!)

[SNIPPED 80 LINES]

64 bytes from 173.194.75.103: icmp_seq=1 ttl=250 time=133.318 ms (DUP!)
64 bytes from 173.194.75.103: icmp_seq=1 ttl=250 time=133.322 ms (DUP!)
64 bytes from 173.194.75.103: icmp_seq=1 ttl=250 time=133.326 ms (DUP!)
64 bytes from 173.194.75.103: icmp_seq=1 ttl=250 time=133.422 ms (DUP!)
Request timeout for icmp_seq 2
^C
--- www.google.com ping statistics ---
4 packets transmitted, 1 packets received, +73 duplicates, 75.0% packet loss
round-trip min/avg/max/stddev = 120.565/128.757/133.422/3.811 ms


Ok, so clearly this is wrong.  What really got me (as I was clearly reaching the end of my tether) at the end of a very long phone call to Verizon, having been passed through five departments, each separately asking me for my details, then each also could not find them!  That part alone was particularly awesome, I just do so love repeating myself!

Then finally the awesome service desk tells me...

"Apple" products are not supported!





I hung up.

A nice cup of tea and a sit down later, I decided to take matters into my own hands....

First up... this bad boy:



I'm clearly throwing all caution to the wind here, I'm not qualified... never have been. "Telco Use Only Do Not Remove"... well it is in my house, so I'm pulling the plug.

Next up, I pull the wire on the battery, the machine is beeping quite a lot now because of the removed "Telco Use Only" plug!  I picked the red wire you can see here, hoping it will stop the beeping!



Gave it a tug, and all the lights went out.  I also pulled the fiber optic just incase for added awesomeness.  I'm not qualified, but I've been known to know a little bit about how computers and networks "work".  Then I went off to finish my tea... and left this thing to understand the jolly good lesson I'd just taught it!

As I drink my tea, I'm thinking about how to make double sure that this is going to work.  So I plugged a laptop right into the network jack on the terminal adapter, (by passing any possibly crappyness in the home network, I did wire it after all!) quickly cloned the MAC address from the airport express... I'm not going back to call them again to release the MAC!  ifconfig to the rescue!

jcuff-air:~ jcuff$ ifconfig en0 ether aa:ff:ee:ff:ee:aa

Ok - now time to put the battery back in, reconnect the glass fiber connection (green), plug back in the "naughty boy - do not remove!" plug back into the wall...

Oh look, see what on earth do we have here?  Oh yes, a fully functioning internet!

jcuff-air:~ jcuff$ ping www.google.com
PING www.google.com (173.194.75.99): 56 data bytes
64 bytes from 173.194.75.99: icmp_seq=0 ttl=250 time=35.918 ms
64 bytes from 173.194.75.99: icmp_seq=1 ttl=250 time=40.758 ms
64 bytes from 173.194.75.99: icmp_seq=2 ttl=250 time=39.774 ms
^C
--- www.google.com ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 35.918/38.817/40.758/2.089 ms
jcuff-air:~ jcuff$ 

Disconnected the laptop, plugged the Airport Express back in - all better!

Something in that terminal adapter (Verizon's equipment) clearly got messed up, not that Verizon could tell me.  They were much more concerned that I had an unsupported "router", when in fact it was their equipment that was funky.

Hope this helps any other wayward souls on the internet.

Pull plugs first, ask questions later :-)




[any opinions here are all mine, and have absolutely nothing to do with my employer]
(c) 2011 James Cuff