At OpenWorld 2016, one of the presentations I gave was entitled “My 13 DBA mistakes in 13 years“. Seldom I see so many smiley faces in the audience.
Earlier, in 2009 at the BGOUG, Plamen Zyumbyulev and I gave a very similar talk called “Don’t do like they do. People would make fun of you“. We were both quoting mistakes we have done while working with the databases.
In my opinion, it ok for DBAs to make mistakes – it just happens from time to time. It can happen during the day, during oncall and it happens also to the very best ones.
I have heard from some DBAs that they have never made any mistakes. May be. Who knows. But when spending years and years of database administration once in a while it is OK to press the wrong button or type the wrong command. Or the right command and press the right button but … in the wrong window
For me, the transition from DBA to Senior DBA comes at the moment when you start admitting your mistakes. I believe this is the borderline.
Here is a good collection of mistakes DBAs have made but first my top 3 ones I mean my mistakes.
New patcheset – I think 126.96.36.199 or something. I patched the RMAN catalog database running on a standalone server. So far, so good. All done. Perfect! I wanted to remove the patchset binaries afterwards. Simple rm –rf * Then I got a phone call from the Unix admin: Julian: are you connected to ..? Cold sweat. I noticed screens were changes one after each other. Then even before typing pwd I knew I was root and I would see that single slash – live and learn. We had to reinstall the OS and recreate the RMAN catalog database.In a production database I saw a user called ABC – not too many tables, I thought it was some test schema remained from who knows what and where. Then: drop user cascade; Why didn’t I asked what that schema was and who created it and what it was used for! The answer I got (after the schema was dropped) was the worst I could expect. Like the worst of the worst. Luckily, for some reason I ran a schema export before the drop. I have never ever as DBA run an import so quickly in my life Heavily loaded 24×7 OLTP database – I got call that a filesystem was 99% full – it had only UNDO tablespace files. One of them rather huge. Something I have done so many times: recreate the UNDO online:
Only one problem: I deleted with rm the wrong UNDO file – imagine what happened – all databases which work within one business application got messed up – at least transitions were failing. I still remember the reaction of the datacenter manager when with a pale face I went to report what happened: “This is just a website Julian – no human lifes are at stake – go and fix it”. We managed to fix it without shutting the databases down but I had another DBA behind me following what I am typing. Afterwards, most of us were often watching each other when doing something important – 4 eyes are always better than 2.
What is the biggest mistake you made in production? by Tara Kizer
5 DBA Mistakes That Can Cost You Your Job by Robert Davis
Confessions of a DBA: My worst mistake by Phil Factor
Don’t Just Do Something, Stand There! Avoiding Junior DBA Mistakes by Jim Czuprynski
Top 6 MySQL DBA Mistakes by Rob Gravelle
Common Mistakes of DBA in MS SQL Server by Evgeniy Gribkov
The 3 DBA Mistakes You Don’t Know You Are Making by Thomas LaRock
And here are interesting videos to watch: Top 10 DBA Mistakes: Horror Stories!
Read MoreJulian Dontcheff’s Database Blog