www.neil.blog Just another WordPress weblog

December 12, 2018

Cisco Catalyst not passing traffic after upgrade

Filed under: Uncategorized — Tags: — npd @ 8:37 am

I typically go onsite for switch software updates. They’re just about the only thing that I don’t have a good failback mechanism for in most of the networking stacks that I support. If a host server update fails, I can reset it through iLO or iDRAC. If a firewall update fails, I mostly have High Availability configurations so a single failure won’t ruin my night. However, I always am present for Cisco Catalyst updates. The failure scenarios are too many, and my recovery options too few. 

This past Friday I was doing a simple update, from 15.1 to 15.2.4(E6) on a pair of non-stacked Catalyst 2960X’s. I’d done two previous updates on this environment without issue, and after my onsite maintenance windows had been delayed a few times, I had to just schedule it to be done remotely. What could go wrong?

I backed up all my configurations and downloaded the latest Cisco-recommended software on my switch, set it to /overwrite and /reload. I watched the upgrade status proceeding normally, remembering that there is often a long period where the switch is unresponsive due to console display errors during upgrades. Then I saw it start to reboot. And I waited.

After 20 minutes my remote session didn’t come back up. I connected to the VPN and found that I could ping and ssh to the switch, but couldn’t ping any connected network devices. Logging in to the switch and running terminal monitor I started looking for what the problem could be. show ver shows me that the upgrade was successful. I can ping other switches and servers from inside this switch. So what’s wrong?

After a few minutes, the following message comes up in the terminal:

%ILET-1-DEVICE_AUTHENTICATION_FAIL: The FlexStack Module inserted in 
this switch may not have been manufactured by Cisco or with Cisco's
authorization. If your use of this product is the cause of a support
issue, Cisco may deny operation of the product, support under your
warranty or under a Cisco technical support program such as
Smartnet. Please contact Cisco's Technical Assistance Center for
more information.

But I’m not using any FlexStack modules, and all my hardware is legitimate. What’s going on? I search this message in Cisco support forums and find the link to Bug ID CSCur56395. Which states:

If this issue is seen AFTER UPGRADE, then hard power-cycle is required


You can try a reload but this won’t work. You can try a downgrade back to the previous version, but I don’t know if this will work (let me know if it does). Seemed too risky to me, and I’ve never done it, hope to try it in the lab if I can recreate the issue. In my case I had to call a coworker who lives nearby to go onsite and power the switch down. 

Sorry if you read this far hoping for a quick solution to this problem. Time to call your datacenter smart hands, or lace up your boots and head onsite yourself. If you are lucky, you are onsite already, laptop balanced on top of the KVM, reading this post, in which case you are very lucky! Just unplug the switch for 5 minutes, do some stretches, plug it back in, and all will be well again.

Postmortem notes for next time:

  • My hosts should be balanced between switches. Fix that next time I’m onsite. This outage wouldn’t have required repair at 11pm on a Friday if the host had just failed over to the other switch.
  • UPS should have had a network card in it. Not sure I would have done it in this scenario, but in some cases it would be helpful to be able to reset one of the power banks in the UPS using telnet from inside the failed switch. In this case there was no management card in the switch, and I would rather not risk a dirty shutdown of Exchange. But had I been prepared for this, I could arrange servers and switches accordingly into each of the APC’s power banks to minimize unsafe shutdowns while still allowing remote reboots.

November 29, 2018

Gang Gang Dance – Kazuashita

Filed under: Uncategorized — npd @ 9:19 pm

Gang Gang Dance are back after a long break following the release of Eye Contact, one of my favorite albums of 2011 and still in regular rotation for me. I haven’t been able to see them live since that year at a live show that blew me away. Their new record Kazuashita picks up where you’d hope they’d be after 7 years: shows growth, but dials the intensity back and feels to me a little more introspective and serious. From the first few mourning lines on album opener J-TREE (“I’m not ready / I’m not ready to go”) set to familiar pulsing rhythms, the vocal samples and chanting evolved to clips of protestors at the Dakota Pipeline, moving the listener from soaring Dakota plains and beyond into outer space.. man.

September 6, 2018

Two Signholders, NYC, Summer 2018

Filed under: Uncategorized — npd @ 8:18 pm

Maybe the Ayn Rand Walking Tour is a Looney Tunes-style trick, into the side of a wall painted to look like a tunnel, or over the edge of a cliff.

February 28, 2017

Cisco Catalyst switch software upgrade hangs at Extracting

Filed under: Uncategorized — Tags: , — npd @ 11:03 am

While upgrading the software from 12 to 15 last week on a Catalyst 2960-X over TFTP, I noticed that the upgrade hung at “Extracting” for a very long time. I had just finished a similar update on a 2960S that was a few years older and did not notice a similar delay. 5 minutes passed and I started to get nervous. 10 minutes later I started Googling to see if this is normal. I didn’t find anything and wondered if something had gone wrong.

Of course, the prior action on the screen was this:

Old image for switch 2: flash:/c2960x-universalk9-mz.120-2.EX5
  Old image will be deleted before download.

Deleting `flash:/c2960x-universalk9-mz.120-2.EX5' to create required space
Extracting images from archive into flash...

So I was very nervous that any attempt to kick this thing back into action would leave me with no image on the switch. It’s 9pm on Friday night and I’d like to go home! After about 15 minutes of waiting, the rest of the process finally kicked back into action and showed this message on screen. Would have been helpful if it had come up when it was actually relevant!

Warning: Unable to allocate memory to display the tar extraction of files, however upgrade process is still continuing. If you would like to see the tar extraction output, try upgrading one switch at a time.
Installing (renaming): `flash:update/c2960x-universalk9-mz.152-2.E6' ->
New software image installed in flash:/c2960x-universalk9-mz.152-2.E6

In conclusion: don’t sweat it! It is normal for the Catalyst upgrade process to hang at Extracting.

May 19, 2015

Veeam Backup & Replication 8.0 Hangs at “Deleting Helper Snapshot”

Filed under: vmware — npd @ 8:57 am

During a replication job to move a few VMs to new infrastructure (ESXi 5.0 to ESXi 5.5 in 2 different VCenters), watching the progress bar anxiously as the client is very sensitive to I/O interruptions and I have bad luck with snapshot deletions, we hang here, at “Deleting helper snapshot.”



I do have patience for these things but when we look at the previous job for DC2, we can see that this task finished in a reported 9 seconds and the job completed shortly thereafter.

Anxiously I review the logs in C:\ProgramData\Application Data\Veeam\Backup\<Job_Name> and confirm that there have been no additions to the log since this last entry.

Next I review disk usage graph in VSphere console and verify that there is no I/O intensive operation occurring. There is nothing to see here.


Finally, as I prepare to open a case with Veeam, the job finally finishes successfully as shown here:


Success! Strange that it says it only took 5 seconds when it really took 2 hours.

To compare results, I looked at the log file for the job which says it took 9 seconds, and see this in the logs. Look at the discrepancy between the highlighted lines.

19.05.2015 00:58:26] <12> Info [Soap] Removing snapshot ‘snapshot-46’
[19.05.2015 00:58:26] <12> Info [VimApi] RemoveSnapshot, type “VirtualMachineSnapshot”, ref “snapshot-46”, removeChildren “False”
[19.05.2015 01:03:08] <44> Info [AP] (0339) output: –asyncNtf:Received external stop signal.
[19.05.2015 01:03:08] <73> Info [AP] (049e) state: closed
[19.05.2015 01:34:09] <53> Info [AP] (0339) state: closed
[19.05.2015 01:35:55] <12> Info [Soap] Outgoing connection ‘client01vcenter:443:DOMAIN\itadmin:False::0:12’.
[19.05.2015 01:35:55] <12> Info [Soap] Connection ‘client01vcenter:443:DOMAIN\itadmin:False::0:12’ is provided from the cache.
[19.05.2015 01:35:55] <12> Info [Soap] Loading ‘vm-45:VirtualMachine’ hierarchy
[19.05.2015 01:35:55] <12> Info [Soap] Loaded 12 elements
[19.05.2015 01:35:55] <12> Info [Soap] Connection ‘client01vcenter:443:DOMAIN\itadmin:False::0:12’ is disposing.

So, don’t worry if you are hanging at “deleting helper snapshot” in Veeam. There will be no progress on the screen and I don’t see any way to measure the progress. Relax, don’t stare at it, don’t cancel the job. There should be no impact because of this. I had no dropped pings to my server and had no I/O lockup as when deleting other snapshots.

Hope this helps!



September 2, 2013

Fixing the U2 and U3 errors on Pentax point-and-shoot 1QZoom EZY-R

Filed under: cameras — npd @ 1:05 pm

In need of a new 35mm P&S, I picked up this slightly aged but clean Pentax 1QZoom EZY-R at a local thrift shop for $4. The battery in it was dead, but I replaced it for another $4 at a local electronics store and loaded it up with an expired roll of Fuji 400 to test. I spooled up the film, popped in the battery, and was greeted by a small clicking sound and a flashing “U3” on the camera’s LCD before it shut off. One detail that seemed noteworthy was that the lens cover (two retracting triangles) was slightly open and the zoom lens protruded just a bit.

I spent a few minutes browsing the internet for repair manuals, user manuals, or other helpful hints that might help diagnose this problem but found very little, save for those SEO spam sites that seem to log every error condition on every device ever built- but fail to offer solutions. The user manual for a similar Pentax P&S model points to problems with the zoom lens, and this was corroborated by a number of confused newsgroup posts, but no actual solutions.

So, on a cloudy Saturday afternoon I sat down with a small screwdriver and decided to dive in to see what was wrong with the zoom lens. I don’t know what I was looking for exactly- I was reminded of a an old Jerry Seinfeld bit about opening the hood of your broken-down car and not knowing what you’re looking for. A big red button that says “ON” that you can flip?

I wasn’t careful when disassembling the camera, and minutes after making a comment that I needed to put it back together without electrocuting myself, I took a jolt from the capacitors. Try your best to only touch the plastic on the camera while you’re working on it or you’re likely to get zapped. It didn’t kill me, but I did get a blister and my girlfriend told me I screamed and jumped across the room. Be careful!

Anyhow, by removing the two screws on the left side, the two screws holding on the strap, the 3 on the bottom, and the 1 screw in the film compartment on the right side where the full spool sits (plus 1 or 2 more on top in the back that holds the top plastic in place), you’ll be able to remove the top and front of the camera, exposing the innards. 

Pentax gear cover plate

Take a look at the metal plate on the left, with the small box labeled “M934”. That’s the motor that drives the gear assemblage (obscured by the metal plate) for the zoom lens. Remove those few screws and pull away the motor and plate, exposing the gears.

Pentax gear assembly

Gently grab the gear assembly and set it aside. Be careful not to drop it, as all the gears could fall out of place. Take your finger and make sure they all spin freely. If they don’t, bump them back into place and add a small drop of lubricant if necessary.

Pentax zoom gear

What you’re really looking for is the black cog which drives the zoom. It’s circled here in red. With your screwdriver or finger, gently spin it until the zoom lens closes and retracts back into the body (in the full “off” position). If it’s already there, go the other way and ensure you can fully open and close it.

In my case all I had to do was close it. I then popped the front case back on and inserted the battery. It whirred to lift for the first time, the motor squealed back and forth before flashing the “U2” error on the screen. This is a different zoom lens error code, and it’s expected as you haven’t put your gears back in yet!

So, do that. Remove the battery, remove the case again, Pop the cogs back in and set the motor back on top. Screw everything back together and load up with film!

All together this took me about 30 minutes. If you’re having trouble with your Pentax P&S I recommend giving this procedure a shot before tossing it out. Good luck!

June 18, 2012

Mothers News 2012 Fundraiser Still In Full Effect

Filed under: friends,Uncategorized — npd @ 8:23 pm

mothers newsI keep neglecting to write about my own fundraising efforts (maybe later tonight) but instead, would like to remind you about the Mothers News 2012 Fundraiser. Mothers News is a Real Newspaper with Real News, comics, ideas, and is generally the only good reason to check the mail, if your friends are like most of mine and don’t return postcards.

Mothers News is edited by Jacob Berendes, whose endeavors and projects have continued to impress and inspire me, and I look forward to the 3rd year of the Paper of Record. The fundraiser has reached its goal, so any money donated beyond this point will only further enable the project’s stated mission of “design[ing] and manufactur[ing] a craft that would take a human occupant into outer space.” I’m happy to support this project!

There are 12 days left, as of this writing, so donate now, which is really a Subscription at most levels, and let me tell you, it is well worth it.

April 1, 2012

Truck drivin adventures

Filed under: farming,learning — npd @ 5:16 pm

Truck driversShortly after this photo was taken I ran the truck into my own car. Never having driven a 20′ box truck before I may have been out of my league being behind the wheel. Fortunately, at the moment when I was about to drive away in this thing, Ryan came by and offered to pilot it.

We were on our way to Locust, NJ, where Meg and I will move this summer to try our hand at farming. The girls at Domestic Construction had $10,000 in plexiglass in 4×10 frames painted in CMYK colors for some famous printer company’s promotion. It was destined for the scrap heap unless we could rescue it, so with plans of a colorful greenhouse in mind, we headed south.

tom with plexitom and ryan unloading trucktom with a black frame

walking on the dockI’d completely miscalculated the scale of this job- the truck was jammed full of wood, plexiglass, and other junk on top of it. We pulled everything out, stacked it in the garage and put the extra under a tarp outside the garage. Tom and Ryan were both great sports about it. I’d toldTom earlier on the phone that this would only be a 30-minute job. Clearly I had no idea. All told, it probably took us 2 hours to unload the truck, plus travel time. Along the way, the guys pointed out all the great thins about the area I’ll soon be moving to- mostly strip clubs, liquor stores, and bait-and-tackle shops, with lots of ice cream stands along the road. I’m sure we can find some hidden gems here, at least with the ice cream.

after loading in- garage closed I really hope I didn’t use up too many favors with this job- because it’s only the beginning of our farm adventure, and I know that Meg and I will need all the help we can get from our friends to make it work.

March 25, 2012

Building a nuc at Hayseed’s

Filed under: beekeeping,learning — npd @ 11:04 am

building a nucleus hive for Hayseed’s Whenever I have to break out even the simplest of tools, I feel like I’m in over my head. Whether it’s racking servers, fixing a loose bicycle wheel, or build a compost bin, I start to wonder if there are skills that I just never picked up from my father (if he even had them). It’s a frustrating situation- I need to drill 5 holes, evenly spaced, on each side of 4 boards, and I can’t even remember how to calculate the distance between each hole. I take shortcuts and as a result, things end up kind of sloppy. In the end, all the pieces fit together and I’m sure this will be a perfectly adequate demo unit for the shop.

This nucleus hive (used for transporting or capturing swarms, splitting a hive off to a friend, etc) took about 90 minutes to assemble. Someone better endowed than I could have done it in 60, I suspect. I learned a few important things here: even being able to follow along with the build diagram felt like a success. Oof. I have got a lot of work to do!

March 24, 2012

8 of your friends posted about “Rush Limbaugh”

Filed under: rants — npd @ 11:01 am

Draft of reply to someone on facebook re: douchebag radio jock’s misogynist comments- though the 2 issues which seem relevant here are the anti-contraception attitude pushed by Catholic institutions and red states in general, and the anti-woman rhetoric which would shame and punish a woman for having the gall to express herself sexually- the more relevant political issue (if we have to pick) is the former, so that is what I’m commenting on.

Though of course all women benefit from sex-positive legislation, and the entire country benefits from healthier, safer women, the real victims of Blunt’s amendment and Rubio’s bills are those with no other choices.

Ms. Fluke’s organization does important work for reproductive rights and justice, which includes standing up for all women’s reproductive rights and health. The issue she brought before the House (see transcript [PDF]) directly affects more than college women- but just as her school’s health care plan or religiously-affiliated employer’s health care plan can deny her birth control, so can those of much poorer women with less money for BC and less education on other options. Birth control is as important an issue now as ever, and any organization standing in the way of a woman’s right and privilige to stay safe and healthy should seriously reconsider what it stands for, and who it really represents.

Don’t be fooled by some antagonistic and inflamitory radio pundit’s slut-shaming rhetoric. The real beneficiaries of the Affordable Care Act bill’s contraception provisions are not college “sluts” who just can’t get enough sex. It’s poor women who lack affordable family planning knowledge and tools. Our last president completely decimated comprehensive sex ed in schools in favor of the abstinence-only garbage (proven to not work)- on religious grounds- and now you want to take away the other options- on religious grounds?

The Catholic church’s constant intrusion (and recent call to arms by NY’s cardinal Dolan) tell us that this vocal minority is not going anywhere, and may get stronger. Please write to your Senators and thank them for striking down the Blunt bill on Thursdsay. It was a close vote, and it won’t be the last.

Older Posts »

Powered by WordPress