Playing with Solarwinds Orion NPM – How to recover from a corrupted database


I can’t believe it’s been that long, but I recently realized that my Solardwinds SCP has actually slipped. The SCP was one of the first certifications focused on network management and, as I’m sure you can imagine, I was in there as an early adopter. The training was really good ( I still miss Josh Stevens!) and the test was one of the best tests I’ve every taken in IT. It had some REALLY evil questions on there. You know the kind… the ones that prove you either know your stuff or you don’t.  No messing around with ambiguities. Ahh… good times.  On to the present though.

Open Disclosure

I’m assuming because of the major focus of my blog is network management, I was approached by Solarwinds and offered an NFR license for a couple of their products to run in my labs. As with them, I think it’s important for my readers to understand that I work for HP and sometimes find myself in competition directly against these products. I do also find myself giving some guidance to customers who are using Solarwinds products and trying to manage their HP Networking products through the Orion console. It’s the experience of using NPM and NCM to manage HP Networking equipment that I’m going to try to focus on.  Please don’t ask me to compare products.  I told you where I get paid and you can guess what my official opinion is going to be. 🙂

Orion NPM

The first product I wanted to play with is Solarwinds NPM. Solarwinds has a great following and has been around for a lot of years. There were some things that I really didn’t like about this a few years back when I passed the SCP and it will be interesting to see how the product has improved overtime and whether my old issues have been fixed.

Specifically, I was never happy with the half-enabled web-console.  The fact that I had to bounce back and forth between the windows console and the web browser to get anything done was frustrating to say the least. I know there were a lot of improvements made in Orion 10, and I’ve heard good things about 10.5 specifically.  I downloaded 10.5 and will be upgrading to the 10.6 with hot fix 3 tonight. I’m really excited to see the improvements that Solarwinds has made in the years since I last had my hands dirty with the platform.  WIsh me luck!

Before we get started…

So this is detailing some the issues I had getting NPM up and running.  To say the least, I had some issues. ( as detailed below ).  I’ve written down the symptoms and the fixes that I went through, but to be honest, this was just a REALLY bad Windows build. Sometimes, there’s just nothing you can do when the base operating system gets corrupted right from the initial install.

 

Installing

To be honest, I had some issues getting it running. The licensing actually crashed and somehow it was assigned in the Solarwinds system, but never applied to my system. I also made the mistake of downloading the package that didn’t have SQL installed ( wasn’t clear and I didn’t read the documentation closely enough ).  On the bright side, Solarwinds support actually helped me through this one in about 24 hours. Sometimes thing happen during an install, so I can’t complain too much. Plus, I should have paid more attention when flipping through the documentation. My bad.

Unscheduled Interuption.

Ahhh… well… Sometimes things don’t go as planned.  I had an unscheduled power outage tonight and it seems something has gone wrong with my installation.

NewImage

Google didn’t come up with anything. So I’m off to follow the SQL Management studio where the SolarWindsOrion database is marked as suspect…. hmmm… that’s not good.

A couple of scooby snacks and some super-sluething later and I come up with this link

In a nutshell, it looks my SQL database has been corrupted somehow and it’s now showing up as suspect in the Microsoft SQL server management console. ( While I was banging my head against this problem, I didn’t take a good screen capture. So this is where I ask you to imagine a big yellow exclamation mark of DOOM over the SolarwindsOrion database in the following image.  )

NewImage

Looks like the power outage REALLY messed up the SQL database.  But GoogleTechnician to the rescue!

Solarwinds Configuration Wizard – Attempt #1

So now I’m off to the Solarwinds Configuration tool ( on the console of the windows server ). For this attempt, I run the database configuration only. Thinking, I’ve got a database, issue, let’s just run the database configuration wizard and that should fix it, right?

NewImage

Nope… doesn’t look like this is going to work either

NewImage

Solarwinds Configuration Wizard – Attempt #2

So now I’m off to the Solarwinds database.  Hmm.. nothing on this error.

At this point, I just try what any good network guy does. I start clicking things and seeing if anything will work.

So this is what I did

  •  Logged into the Microsoft SQL Management console and reset the password on the SolarWindsOrionDatabaseUser account to something I knew.
  • Re-ran the Solarwinds Configuration Wizard. This time, instead of just the database, I’m going to re-run this for the Database, Web Site, and the Services.

note: Normally at this point, I would pull the plug, call the patient dead and re-install. But this was supposed to be a learning experience, right? We’re certainly learning now, aren’t we?

NewImage

Look like I’m back in business! Good to go right?

NewImage

Nope. now it’s time to remove the license, delete the VM and start from scratch. I don’t want a known corrupted system monitoring my network, even in a lab.

Hopefully, this blog will help someone with a production Solarwinds deployment who gets this same nasty SQL suspect database error.

Lesson to Learn

In a lab, sometimes things happen. Take the opportunity for the full learning experience when things go wrong. It’s always fun to see if we can bring a system back from the dead. But remember, once you’re done with the learning. Scrap it. This is not the system that I want to be evaluating as I will always be wondering “Hmmm… I wonder if this is normal or if this is a result of that bad install.”

Things go wrong. Known good clone images just have something funky. I’ve seen registry issues on brand new windows installation. SQL strangeness etc… None of which I feel like dealing with for longer than necessary. With how easy it is now to deploy a new VM from a template. There’s just no need to subject myself to this kind of long term pain.

So before I go to bed tonight, I’m going to start cloning a new windows image so that I can re-do the entire install tomorrow night on a clean VM.

FOR THE RECORD :  I’m 100% sure this is not a normal Solarwinds Orion NPM installation tale. I just happened to be the lucky one who was hand-selected by the universe as it thought ” Hmmm…  who can I REALLY mess with today? “.

Can’t wait for tomorrow.

@netmanchris

Advertisements

One thought on “Playing with Solarwinds Orion NPM – How to recover from a corrupted database

  1. This design is incredible! You most certainly know how to keep a reader entertained.
    Between your wit and your videos, I was almost moved to start my own blog (well, almost…HaHa!) Great job.
    I really loved what you had to say, and more than that, how you presented it.
    Too cool!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s