Four Areas to Focus on When You Start A New DBA Role

 

You’ve just been hired into a DBA role at a new company, or you’ve been given the DBA keys at your current company. Maybe you’re a SysAdmin and your boss has informed you that you are now supposed to manage the SQL Servers as well as everything else on your plate. In any of these situations, you may have some confidence in your skills, but especially in the case of being a new hire, you have absolutely no true idea of what you’re walking into.

In these scenarios, where do you start? Start with these four areas.

  1. Backups
  2. SQL Agent Job Notifications
  3. Building an environment inventory
  4. Security

 

1. SQL Server backups: You have to be able to recover the data.

My experience has been that many companies aren’t doing an adequate job of managing the backup routines for their databases. I’ve been in companies where two or even three products were running backups on the same SQL Servers. I’ve been in situations where important databases weren’t backed up frequently enough and with the right set of backup types to recover appropriately, which would result in losing more data than the management was comfortable with. I’ve also seen scenarios where there were no backups at all.

So, what do you do to tackle the issue of SQL Server backups in your environment? First, connect to your SQL Servers and run the below query. This will give you the database name and the date of the last full backup if the last full backup was over 7 days ago . If the database has never been backed up, then you’ll get that information as well. Any results from this query should be investigated immediately.

SELECT  d.[name] AS DatabaseName ,
COALESCE(CAST(MAX(b.backup_finish_date) AS VARCHAR(25)),'No Backup Ever Made') AS LastFullBackup
FROM master.sys.databases AS D
LEFT OUTER JOIN msdb.dbo.backupset AS B ON D.name COLLATE SQL_Latin1_General_CP1_CI_AS = B.database_name COLLATE SQL_Latin1_General_CP1_CI_AS

AND B.type = 'D' /*Full backup*/
AND B.server_name = SERVERPROPERTY('ServerName') /*Backupset ran on server you're currnetly connected to. */

WHERE D.database_id <> 2  /* Eliminate TempDB. No need to back that up */
AND D.state_desc NOT IN ('RESTORING', 'OFFLINE', 'OFFLINE_SECONDARY') /* Exclude databases that are offline or involved in log shipping, for example */
AND D.is_in_standby = 0 /* Database is read-only to restore a log backup as in Log Shipping. */
AND D.source_database_id IS NULL /* Excludes database snapshots */

GROUP BY d.name
HAVING MAX(B.backup_finish_date) <= GETDATE()-7 /*Full backup older than 7 days ago.*/
OR MAX(B.backup_finish_date) IS NULL;										    

 

Review the SQL Server to see if there are any native backup methods in place, such as a Maintenance Plan or Ola Hallengren scripts.  Each of these backup methods will produce one or more SQL Server Agent jobs. Check to see if the jobs associated with those options are enabled.

If you don’t see either of those options set up, your company may be using a 3rd party backup software. these are typically, but not always, deployed by System Administrators. Check with them to see what 3rd party software is in place for backups and let them know your findings for the server in question. this could be a situation where this SQL Server was just not enrolled in the 3rd party software and so is not being backed up.

If none of these things turn up a backup solution, then get one in place ASAP. See this post for options on how to create backups.

2. SQL Agent Job Notifications: If a job is worth creating, it’s worth knowing if it failed.

This is in position #2 because it’s been an easy win in every job I’ve taken. In every new role I’ve had, I have discovered many SQL Agent jobs with no notifications or notifications set up to wrong people. If a job is worth creating, it’s worth knowing if it failed.

As you work through finding failed jobs that never notified anyone, you may find that you’re fixing important agent jobs that should have been working. This process fixes business and data related processes and also helps you find stake-holders for data processes. When jobs fail and you investigate, you often need to know who is impacted by the job failure. This further leads to identifying who should be notified if it fails. This process then, actually helps connect you to key players in the organization very early. Those people on the business side almost always appreciate your efforts to make sure that data processes that affect them are working properly. This gives you allies as you move forward in your role.

So how do you proceed if you find jobs that have no notification set up if they should fail?

  • First, locate jobs that are failing. I have a post that will show you two ways to do that. You can either look at the SQL Agent Job Activity Monitor on a SQL instance or query the msdb database.
  • Second, locate enabled jobs that have no email operators assigned. These will be jobs that no one will know whether they are working or not. This post will tell you about how to find these jobs.
  • Third, work through a process to make sure jobs with no notification set up will have an email operator assigned to them. This will ensure the right “someone” will know if the job fails.

3. Build an environment Inventory: You can’t manage what you don’t know about.

The process of gathering information about your environment tells you about the make up of versions and editions in your environment. This, in turn, leads to the answer to the question, “What SQL Server’s are out of support and likely need to be upgraded?”

“How many SQL Servers are there at company X?”

Sure, the hiring manager told you the company only has a dozen SQL Servers, but I’ve often found that hiring managers often don’t know the real answer to the question, “So, how many SQL Servers do you have at company x?” That’s a question you should always ask when interviewing by the way. When you hear the answer, I’d encourage you to add at least 50% to that answer. At one job, I was told there were about a dozen SQL Servers. Less than 6 months in I had located about 50 SQL Servers, and that was before I ran the MAP Toolkit.

If you are not familiar with the MAP Toolkit, when you run the utility it does a search of your network or Active Directory to find SQL Servers. You will want to ensure that your security team and/or System Administrators know you are going to run this utility. This is because it will likely set off security software or even be blocked, in some cases.

The tool produces an Excel format list of plenty of information for you to know and this is the simplest way to start creating a SQL Server inventory. This scan is likely to find a lot of SQL Servers that no one seems to remember existing. Managing all the SQL Servers within your purview is important. You can’t do that if you don’t know about them.

You will also probably find individual employees who have installed SQL Server Standard or even Enterprise Edition on their local PCs. That’s a licensing no-no. This situation gives you the chance to help your company avoid licensing entanglements with Microsoft by addressing these scenarios.

If you find that you aren’t allowed to use the MAP Toolkit, as you hear about other SQL Servers that exist in your environment, or as you discover them for yourself as you look through Active Directory Users and Computers, make a Central Management Server and add SQL Servers to it. Ensure you have access to those SQL Servers and then query them to find out all you can about them. Review the information available to you from SERVERPROPERTY and build your own queries to discover information about the company SQL Servers. Here is an example query from the MS Docs link above. As you discover information about the company SQL Servers, create your own spreadsheet of information, or create a database and tables to hold the information you find.

SELECT  
  SERVERPROPERTY('MachineName') AS ComputerName,
  SERVERPROPERTY('ServerName') AS InstanceName,  
  SERVERPROPERTY('Edition') AS Edition,
  SERVERPROPERTY('ProductVersion') AS ProductVersion,  
  SERVERPROPERTY('ProductLevel') AS ProductLevel;  
GO

 

4. Security: Who has access to SQL Server and what type of access do they have?

I generally focus on this one after the others because security is often a political subject within an organization. You don’t want to become embroiled in power struggles right away by trying to take away someone’s access, because you will lose. Additionally, if you’ve done the previous work mentioned, you have likely built some good business relationships and a few allies to help you have the discussion about SQL Server access and security. As a DBA, you and the business need to know who has SysAdmin level access. The access that they possess allows their credentials to make any change they want to. In truth, it’s often not so much about trusting the person who has the access as it is realizing that every person who has SA permission is a person whom a hacker could exploit to gain access to the company’s data. You will want to make the business aware of how many credentials have this level of permission and whose credentials they are.

How do you find who has SysAdmin level access on your SQL Server? Here is a query for that from MS Docs. I have added a WHERE clause to exclude the commonly found SQL Server accounts that start with NT SERVICE.

 

SELECT	roles.principal_id AS RolePrincipalID,	
roles.name AS RolePrincipalName,	server_role_members.member_principal_id	AS MemberPrincipalID,	
members.name AS MemberPrincipalName

FROM sys.server_role_members AS server_role_members
INNER JOIN sys.server_principals AS roles
    ON server_role_members.role_principal_id = roles.principal_id
INNER JOIN sys.server_principals AS members 
    ON server_role_members.member_principal_id = members.principal_id
WHERE members.name NOT LIKE 'NT SERVICE%';

This query shows you what credentials have Server level SA access. You also will want to know what accounts have the db_owner role at the database level.

I would strongly encourage you to review the permissions scripts written and maintained by Kenneth Fisher. These will provide a wealth of information about your server level logins and your database level users.

 

Next Steps To Take

1.If you have a comment or question about this post specifically, leave a comment and I’ll get back to you.

2. If you would like help with something else related to SQL Server, reach out to me on Twitter, and I’ll be glad to offer assistance.

 

Three Keys to SQL Server Performance

 

Everyone wants good performance from their database systems. Some even expect and need a high performing Ferrari all the time. How is this achieved though? What do you need to understand about SQL Server specifically in order to make your company’s applications hum along like a well tuned car? We will look at three keys to SQL Server performance.

 

SQL Server Speed
SQL Server Performance

Here’s the short list of performance keys.

1. SQL Server must be able to cache sufficient data in memory

2. SQL Server must be able to retrieve data from disk efficiently

3. You must write “good” T-SQL

 

SQL Server Memory

 

1. SQL Server must be able to cache sufficient data in memory

SQL Server does not use memory like most applications.  People unfamiliar with SQL Server’s use of memory are surprised to see SQL Server using 80% or more of a server’s RAM. “I have 128 GB  of RAM on this machine why is SQL Server set to consume 115 GB of that?!” Then they decide to change Max Server Memory down to like 64 GB or less. Suddenly they might find that disk IO shoots up. You will also likely see that execution plans are rapidly aging out of the plan cache, which will burn up CPU with new compiles for query plans and potentially lead to parameter sniffing issues. When all this comes into play people start complaining about applications not working as well. They start saying the magical phrase, “It’s slow.”

When a query is issued and the  data requested is not in memory, SQL Server must use CPU and disk IO to get the data into memory first. While it does this a PAGEIOLATCH_* wait is registered in SQL Server because the current query is waiting on the data to be retrieved. This wait causes the query to be put on hold.  Reading data off the disk is always, always going to be slower than working with data that’s already in memory.

So, how do you determine what is a sufficient amount of RAM for your SQL Server? The short answer is that I would encourage you to look at  my post here and use the DBATools cdmlet Set-DBAMaxMemory. The post will help you understand more about what the Max Server memory setting does and the PowerShell cmdlet will recommend a good starting place for setting Max Server Memory. Here are some examples from the documentation from DBATools and from the Help commands available in PowerShell.

<# If you have a Central Management Server for your SQL environment, consider using this command to loop through all the SQL Servers and set the Max Server Memory where it is set to something larger than the total amount of RAM assigned to the server. #> 

Get-DbaRegServer -SqlInstance sqlserver | Test-DbaMaxMemory | Where-Object { $_.MaxValue -gt $_.Total } | Set-DbaMaxMemory 

<# If you have a Central Management Server for your SQL environment, consider using this command to loop through all the SQL Servers and set the Max Server Memory to this accepted formula created by a SQL Server expert. #> 

Get-DbaRegServer -SqlInstance sqlserver | Test-DbaMaxMemory | Set-DbaMaxMemory 

<# If you don't have a registered server then just use the below #> 

Test-DbaMaxMemory -SQLinstance SQLServerInstanceNameHere | Set-DbaMaxMemory

The longer answer is this. Take a look at three metrics in the SQL Server:BufferManager category in Performance Monitor.  First, if you observe Page Life Expectancy over time and the value has long stretches where it plummets and stays low without recovering, then you have some heavy hitting queries running that could likely benefit from having more RAM available. This will likely be accompanied by SQL Server wait stats showing a high average wait for PAGEIOLATCH_* type waits. This is SQL Server reading pages from disk because the data pages it needed were not in memory.

Second, review the Perfmon counter Lazy Writes/Sec. If SQL Server is having to free up memory in between checkpoints, this value will be above zero. If it is regularly above zero, then SQL Server is regularly under memory pressure and is having to write data pages in memory back to the disk so that it can load different data pages to satisfy queries.

Third, look at Free list Stalls/sec. The value of this Performance Monitor counter indicates the number of requests per second that had to wait for a free data page in memory. If Page Life Expectancy is often low for long stretches and both Lazy Writes/sec and Free List Stalls/sec are greater than zero, then you need to either adjust Max Server Memory up (as long as you don’t go too high based on the above information), add memory or, take a hard look at your indexes and queries involved when these PerfMon metrics are out of balance.

SQL Server Indexing and Disk Performance

 

2. SQL Server Must Be Able to Retrieve Data From Disk Efficiently

Look at your SQL Server wait stats information and if you see PAGEIOLATCH_* very prominent then there could be a good chance that the indexes in your database need attention or those long IO waits could mean a problem with the disk subsystem.

PAGEIOLATCH_* type waits are all wait types related to reading 8KB data pages from the disk into memory. Inefficient indexes could be making these reads longer because SQL Server can’t quickly get the data it needs from the existing indexes. This can happen over time as query patterns and data patterns change. For example, your company might introduce a new category of products Suddenly people are querying that category and its associated products far more and older, mainstay products less. The distribution of that data may affect the execution plan generated.

The company may have re-factored an existing application into new tables and missed indexing the Category column. Now when people are searching for things like “Bike Accessories” there is no supporting index. This results in long table scans of millions of rows.

As a start to determine if there is inefficient indexing, run and save the output of sp_Blitzindex to examine your tables and indexes. Review its recommendations and make adjustments. Then re-measure index usage with sp_BlitzIndex. Some time after a restart of SQL Server re-run sp_BlitzIndex and compare the output to the previously saved run looking to see if SQL Server is using the adjusted indexes.

To review whether you have a disk IO subsystem issue, look at the DMV called sys.dm_io_virtual_file_stats. You can also review the SQL Server Error Log looking for messages indicating IO that took longer than 15 seconds. These could be an indication of an IO subsystem issue. Review this article and this article on these topics.

Both of these articles provide information on understanding the DMV and the error message. There is also information about Performance Monitor counters to use to measure potential problems.

If you are in the cloud, such as AWS, be sure to review settings for ebs and fsx storage to ensure that the IOPs and throughput are set up appropriately. Also, be sure to take into consideration how the AWS ec2 instance type might be throttling your IO and throughput capabilities. Just because your storage is set up with a certain IOPs and throughput doesn’t mean that the ec2 instance can support the storage settings.

T-SQL Anti-Patterns to Avoid

 

3. You Must Write “Good” T-SQL

Poorly written T-SQL can cause poor application and SQL Server performance. Here is a brief list of things to avoid.

A. Writing T-SQL that makes it non-SARGable. This will cause SQL Server to have to scan entire tables instead of using an existing index. For details and examples see this, this and this. there is no need for me to explain this in depth, as it has been written about quite frequently.

B. Overuse of cursors. SQL is a set based language so making it do operations row by row, is far less efficient than using set-based logic.  Take a look at this post and this post.

C. Overuse of SELECT *. Some tables are very wide and have millions or even billions of rows. If you don’t truly need every column, then do SQL Server and your application a favor and only return the columns that are actually needed!

D. Be careful with scalar user defined functions. This goes back to the idea of SQL being a set-based language. A scalar udf returns a single value for each value passed to it. when you use this sort of logic and pass into it large numbers of rows, then each row is processed one at a time inside the function to return a value. Also, SQL Server doesn’t do a good job of “seeing inside” Scalar UDFs and show you that one is present in a query plan. Asa result, you might not see this sort of thing if you’re looking at a query plan. Additionally, scalar UDFs kill SQL Server’s ability to go parallel. For more, take a look at this, this , and this.

This list could be much, much longer. the point here is that how you write your T-SQL often has a direct impact on performance. If you write T-SQL for applications, I strongly encourage you to look at the blogs of folks like Erik Darling and Kendra Little as well as sites that have a large number of entries on T-SQL like this one.

Next Steps To Take

If you would like help with anything in this post, or with something else related to SQL Server, reach out to me here, on Twitter, or by email at Leem@leemarkum.com and I’ll be glad to offer assistance.

 

How I Became a Database Administrator

I recently saw a Tweet from Kendra Little where she asked the simple question, “What got you into databases?”  I’m intrigued by people’s stories of how they got into their current careers so I looked through the thread. There were lots of interesting paths shared and even a blog post was shared by someone who had recently written about how they got into a career involving databases. Kendra’s question and all of the interesting responses inspired me to write a post about my story.

 

It was a Dark and Dreary Night

 

Actually, it was February 2008. For a number of years I had been a team Supervisor at a physical security company called Interface Security; think ADT only much smaller. I was good at leading the team of 8-12 people as we used a critical piece of software to interact with customer accounts. I understood a lot about this software and was often the “lucky” person to be selected to do application failovers. This involved using a custom GUI from the vendor to shut down all the services of the application and start them back up, pointed to a new server. I haven’t worked there in almost 7 years, but I can still see this GUI in my mind.  Anyway, the woman who was the liaison between our company and the software maker was leaving her job to go work for the software maker, Bold Technologies.

I had reached about the top of the pay scale in my position and, at that point, there really was nowhere else for me to go in the company. The next few positions above me were taken and those people weren’t going to leave any time soon. The employee who was leaving asked me to apply for her job. At first I was adamant that I didn’t want to do that and that I didn’t know enough. She insisted that I did know enough and that I should apply. I gave it some more thought and decided to throw my hat in the ring. After a conversation with my boss two levels above me (Dan Reynolds) and about a month later, I had the job and the title of Manitou System Administrator. Manitou was the name of the software.

Interface Security was in the planning phases of a major version upgrade for this software. I took the job thinking I was just going to manage software. However, we were also going to be doing hardware upgrades for the servers this application was running on. My desire to understand everything even tangentially connected to the software got the better of me. We completed the software upgrade later that year and the server hardware upgrade as well.

Lee, Meet SQL Server

As part of that application upgrade process, I learned that there was this thing called SQL Server installed on the servers we were going to replace. I dove in head first to learn all I could about SQL Server. I learned that the process I was controlling with this company’s custom GUI was actually transactional replication under the covers. I loved learning about it and it was an important process to understand for my company and for my role.

About 18 months after starting the new role I began learning T-SQL using whatever free tutorials I could find on the internet. I started experimenting with SQL Server Reporting Services. I began to augment the application’s reporting capabilities with SSRS, at first just building the reports and making them available via the Report Manager and then later by scheduling them to be delivered to various people. In Jan 2012 I was officially moved into the I.T. department.

During that time I discovered the Microsoft Certified Master video training that Kimberly Tripp and Paul Randall recorded. I watched all the videos, some of them multiple times. There was so much I didn’t understand, but as I kept watching them and searching for information on the internet, the more the concepts made sense. I was by no means on my way to being an MCM for SQL Server, but I was learning rapidly.

Somewhere in the following years I started experimenting with what at the time were brand new features in Excel to help with reporting – Power Query and Power Pivot. I used those new tools to pull in and visualize data so that I had both SSRS and these new Excel tools at my disposal.

I also found Brentozar.com during those years. At the time, it was Brent, Kendra Little, and Jeremiah Peschka. Later, as I continued to learn from all of them, they added Jes Borland. This post was one of my first forays into performance tuning. I found it after the I.T. Director asked me to look into the performance problems that the financial system was having. It was also on SQL Server.

Earning a Microsoft Certification

In May of 2012 I completed a certificate in database technology from the Continuing Education Center University of Missouri at Saint Louis. The course I took there were concentrated two day courses on various parts of SQL Server. They advanced my career greatly. Somewhere along the way, I bought a book called SQL Server 2008 R2 Unleashed. I read that book, watched the aforementioned MCM videos and all the blog posts I could from BrentOzar.com.  I bought a Microsoft practice test and then a test from another vendor as well. At night around 9pm, after my three kids and my wife went to bed, I would study for about 2 hours a night, 5 nights a week. I studied probably 4-8 hours on the weekends too. This went on for months.

In May 2013 I got my first certification. It was for Microsoft SQL Server 2008 Implementation and Maintenance. I took the test on the eighth anniversary of my dad’s passing. I took it for me to advance my career, but I took the test on that day to attempt to honor him as well. The test was challenging, but I passed and I was ecstatic! I knew he would have been so proud.

A Change of Title

I held the Manitou System Administrator title until March 2014 and continued to grow more and more as a data professional. In March 2014 I got a title change to Database Administrator. I was elated! I had achieved a remarkable career change from a Call Center Supervisor role to a Database Administrator!

 

Acknowledgments

Wife and Kids –

My wife, Dacia, missed time with me on the weekends when I was studying for my first certification to help me break into the world of database administration. She handled the kids while I was sequestered away in my basement office studying like crazy. My kids, of course, missed time with me because of that too. I missed being with them, but it was necessary to help all of us as I was trying to advance my career and support a family.

Dan Reynolds – Interface Security

My previous boss, Director and then later Vice President of Customer Operations, Dan Reynolds was instrumental in my career development and I’d be remiss if I didn’t mention him. He believed in me from years previous when I had been working under him in the Call Center. I continued to work closely with him even after I  officially went into I.T. He took me to the user conference for the software I was managing. It wasn’t cheap and I was grateful. I learned a lot at those conferences and met a lot of great people at Bold Technologies while I was at those conferences. I met a lot of people from other companies working in the same role I was in. Dan helped me pursue education for SQL Server at the Continuing Education and Training Center at the University of Missouri at Saint Louis.

Amy Spurgeon – Interface Security/Bold Technologies

Amy was the support person who encouraged me to apply for her job before she left. She pushed me to consider the possibilities.

Josh Tafoya – Bold Technologies

Josh was a person in the support department at Bold Technologies when I started my application support role. He knows the Manitou software very well. He helped me understand the software better and he taught me a lot about what it meant to be an I.T. professional.

Matthew Narowski – Bold Technologies

Matthew held a number of roles at Bold while I was at Interface and was eventually named either CEO or President at Bold. He was patient and taught me a lot about I.T. support. He explained a great many things to me about the world of I.T. in general.

I am sure there are others who made contributions to my transition into a new career. I thank all of you.

 

I hope my story helps and encourages you along your own journey.