Introduction To PowerShell For The DBA Part 3

Hello dear reader! This is the third post in a series to help you get started automating things with PowerShell. You will be looking at how to start and stop SQL Server using PowerShell as well as setting the Max Server Memory setting, measuring free disk space and more.

This series is just an introduction to PowerShell and basic functionality you might be interested in as a SQL Server Database Administrator or as a System Administrator responsible for SQL Server. Because it is introductory I don’t have examples of long scripts that do a series of complex things as part of these posts. What the series does have are one or two line scripts that are still powerful and help you explore what is available to you in PowerShell.

With that in mind, let’s delve into a few more PowerShell commands that can help you with common tasks.

How To Start And Stop SQL Server Using PowerShell

Open PowerShell as an Administrator and run these commands. These are simple ways to search for commands related to the SQL Server services. Since we’re looking for information related to services, the first command searches for cmdlets containing the word “service”.  Since we know we want to start or stop the service, we’re looking for commands containing “start” or “stop” in those second and third commands.

GCM *service* -Module DBATools, SQLServer

GCM *start* -Module DBATools, SQLServer

GCM *stop* -Module DBATools, SQLServer

Service Related PowerShell Commands

PowerShell Commands with Start

Commands With Stop in PowerShell DBATools SQLServer

I’m showing you how to find interesting commands related to what you’re trying to accomplish, but I also want you to see how much else you can learn about the capabilities of the DBATools and SQLServer PowerShell modules. As you can see from the output, DBATools has Stop-DBAService, Start-DBAService, Restart-DBAService. The SQLServer module has Stop-SqlInstance and Start-SqlInstance.

I encourage you to look through the output of the screenshots, or in your own PowerShell session, to see what else you can do.  Particularly inside the DBATools these Get-Command outputs show many things are available. For example, you can start a SQL Agent job, start a trace, start an extended events session, you can stop an endpoint or process as well as do several other things. By the way, there is no Stop-DbaAgentJob because you can actually do that from the Start-DbaAgentJob cmdlet in DBATools using a switch.

So, now that you know the types of commands available how are you going to find out what they do? Remember your friend Get-Help or its alias Help?

Help Get-DBAService -Full
Help Restart-DbaService -Full

The output of these commands is too much to put into the post. So, I’ll just point out a few things about each one.

For Get-DBAService notice that you can pass it a computer name value to get the services for all SQL Server instances installed on a given computer.  This will be useful if you know your environment tends to do what’s called “instance stacking.” This is were 2 or more SQL instances are installed on the same computer. If this is done in your environment, you will want to know that information before you start sending commands to restart SQL Server services or you could end up restarting more SQL Server instances than you planned on. That sounds like unplanned down time and you want to avoid that!

<#Returns all SQL Server instances on a given computer. Default columns 
show things like the computerhname, instancename, service name and whether
the current status of the service.
#>
Get-DBAService -ComputerName MyComputerNameHere

Setting Max Server Memory With PowerShell

We’re going to look at Set-DBAMaxMemory from the DBATools module. If you would like more information about SQL Server memory settings. I have a blog post that describes the topic in more detail.

Here is what the DBATools website has to say about this cmdlet in their documentation.

“Inspired by Jonathan Kehayias’s post about SQL Server Max memory (http://bit.ly/sqlmemcalc), this uses a formula to determine the default optimum RAM to use, then sets the SQL max value to that number.”

Here are some examples from the documentation from DBATools and from the Help commands available in PowerShell.

<#
If you have a Central Management Server for you SQL environment, consider using this command to loop through all the SQL Servers and set the Max Server Memory where it is set to something larger than the total amount of RAM assigned to the server.
#>
Get-DbaRegServer -SqlInstance sqlserver | Test-DbaMaxMemory | Where-Object { $_.MaxValue -gt $_.Total } | Set-DbaMaxMemory

<#
If you have a Central Management Server for you SQL environment, consider using this command to loop through all the SQL Servers and set the Max Server Memory to this accepted formula created by a SQL Server expert.
#>

Get-DbaRegServer -SqlInstance sqlserver | Test-DbaMaxMemory | Set-DbaMaxMemory

<#
If you don't have a registered server then just use the below
#>

Test-DbaMaxMemory -SQLinstance SQLServerInstanceNameHere | Set-DbaMaxMemory

Measuring Free Space with PowerShell

DBAs should have some way of knowing and tracking free space in the database files and for the drives of the SQL Server machines. The DBATools module has two commands for doing just that. Check out the below examples and be sure to use the Help cmdlet on these to look at other examples.

<#Gets just certain parts of the output of get-dbaDbspace. Can add | Out-File C:\DBATools\SpaceOutput.txt 
to the end of this to output this information to a file for review.
#>
Get-DbaDbSpace -SqlInstance SQLinstanceNameHere | SELECT Database, FileName, UsedSpace, FreeSpace, PercentUsed, AutoGrowth

#Resturns drive letters, total space and free space on drives on a computer.
Get-dbadiskspace -ComputerName MyComputerNameHere

Detecting IO Latency With PowerShell

One of the things that DBAs want to check for is IO performance of the storage as seen by SQL Server.  This is often done using sys.dm_io_virtual_file_stats. You can view this information in the DBATools cmdlet Get-DbaIoLatency.

The DBATools output from the Help command says that the output of this commands is based on two articles by Paul Randal of SQLSkills.com. Those articles are listed below. If you don’t know who Paul Randal is, you need to find out. Your career with SQL Server will be greatly enhanced by reading his stuff.

How to examine IO subsystem latencies from within SQL Server

Capturing IO latencies for a period of time

I also have a post on finding queries experiencing waits related to SQL Server IO latency.

So here are a couple of examples of how to run this PowerShell cmdlet.

#Outputs the IO latency information of two different SQL Servers

Get-DbaIoLatency -SqlInstance SQLServerInstance1, SQLServerInstance2

#Outputs the IO latency to a GridView UI for visual examination
$output = Get-DbaIoLatency SQLServerInstance1 | Select-Object * | Out-GridView -PassThru

#Writes the output to a file.
Get-DbaIoLatency SQLServerInstance1 | Select-Object * | Out-File C:\DBATools\IOLatency.txt

 

Next Steps To Take

  1. On a Dev environment, practice using the DBATools commands related to stopping and starting SQL Server. This gives you another option when things go south and you have to restart the SQL instance. If the machine suddenly has issues showing the MMC snap-ins then SQL Server Configuration Manager may not be available. And yes, this has happened to me!
  2. Use the Test-DBAMaxMemory command to see if your SQL Servers are misconfigured in terms of the RAM allocated to them. Once you’ve reviewed the output of the command and are comfortable with it, then use the PowerShell examples in here to make the changes.
  3. Consider how you can leverage the cmdlets about diskspace, database space and IO latency to examine your environment. Write some test scripts and try them out.
  4. If you would like help with anything in this post, or with something else related to SQL Server, reach out to me here, or on Twitter, and I’ll be glad to offer assistance.

 

5 Ways to Make SQL Server Backups

Last time we discussed 5 backup types for SQL Server.  Now I want to introduce you to five ways to make backups in SQL Server. I can’t possibly explore all the options available with each of these six methods. There is a lot to cover for even a “simple” topic like backups.

Method 1: Using the SSMS GUI to Make a SQL Server Back up

 

For those readers who like the SSMS GUI, there is good news.  SQL Server Management Studio offers a fairly straightforward method for backing up a database. Once you connect to the SQL Server instance containing the database you want to back up, left click on the “+” sign next to the Databases folder.  Then right click the name of the database you want to backup, choose Tasks, then choose the Backup option in the fly out menu.

That brings you to the General Backup Menu page in SSMS.  In this menu you can access a number of settings related to the backup that will be generated.

With the “Backup type” drop down you can control whether this will take a Full, Differential or Log backup.

The “Backup component” section can be adjusted to take either a backup of files and filegroups or the default selection of “Database.”

In the Destination section you can backup to the default of “Disk” or choose “URL” from the drop down to make a backup to Azure as the target. When the destination of Disk is left you will be supplied with the destination of the backup and a filename for the backup.  This directory will be the default backup directory selected during the SQL Server installation.  If this is not where you want the backup, simply click “Remove” then click “Add” to navigate to the location you want to use.  The “Add” menu does accept share paths.

The Media Options selection on the “Select a Page” allows you to choose options for things like whether you want to append this backup to the same media set or start over with a new one.

The media options seem to me to be from an earlier day when backups were written to physical tapes.  Those tapes then had to be rotated from time to time.  It’s not a best practice to append backups to the media set because you’re adding backup files into a single media set.  If something happens to the media set and it is unusable, then the backups are not accessible.

In the Reliability section, you should check the options for “Verify backup when finished” and “Perform checksum before writing to media.” Thes options will cause your backup to take longer to complete, but they do help with validating the integrity of the backup at the time it was written.

The Backup Options are in the “Select a page” menu has what I think is one very important feature to note.

On this page there is an option related to backup compression.  Back in older versions of SQL Server, like 2005 and 2008 this was an Enterprise Edition only feature.  As of SQL Server 2008R2 it is available in Standard Edition.  To make using this compression the default for all of your backups, simply run the code below on your SQL Server.  Then, when you get to this option in the SSMS GUI, just leave it set to “Use the default server setting.”  You’ll want the space savings that compression offers.  Why use more space on your separate storage for backups than is necessary? I mean, you are storing your backups somewhere other than on the SQL Server, right?!

EXEC sys.sp_configure N'backup compression default', N'1'
GO
RECONFIGURE WITH OVERRIDE
GO

Once you work through these meu options, simply click “Ok” and SQL Server will make the backup for you.  You can also click on the “Script” option at the top middle of the wizard to have SQL Server show you the T-SQL it is about to run.  You could save this as an example script to review later.

Method 2: Using T-SQL To Make a SQL Server Backup

T-SQL is a tried and true method for backing up databases.  You have more options available when backing up with T-SQL versus the GUI.  Most of those options are going to be more advanced options. A very basic example of the backup command that produces a full SQL Server backup is below. That is followed by examples for a differential backup and a log backup.

BACKUP DATABASE [MyDB] TO DISK = 'C:\Program Files\Microsoft SQL Server\MSSQL14\MSSQL\Backup\MyDB_Full.bak'

BACKUP DATABASE [MyDB] TO DISK = 'C:\Program Files\Microsoft SQL Server\MSSQL14\MSSQL\Backup\MyDB_Differntial.bak' WITH DIFFERENTIAL

BACKUP LOG [MyDB] TO DISK = 'C:\Program Files\Microsoft SQL Server\MSSQL14\MSSQL\Backup\MyDB_log.trn'

Buffer Count and maxtransfersize are two options worth noting. You can experiment with these T-SQL options to take backups faster. The buffer count value conrols how many I/O buffers are used for processing the backup and the maxtransfersize controls how much data is moved at any one time.

Below I’ve provided three examples from my testing on my home lab PC. The initial buffercount and maxtransfersize data was captured by turning on trace flags 3605 and 3213 then looking in the error log after the first backup was taken.  After that I simply experimented with values.  Be aware that increasing the buffer count too much can cause an out of memory error.

As you can see, the initial throughput was 219.412 mb/sec and the elapsed time for that part was 39 seconds. This was using SQL Server defaults.

Increasing the buffer count to 8 increased the throughput to 258.653 mb/sec and the elapsed time dropped about 6 seconds. Combining the second change with a maxtransfersize of 4MB increased the throughput to 270.095 and pushed the time down another 1.4 seconds. I shaved 8 seconds off the backup time.  This was for a small database of about 14 GB.  For larger databases the increased throughput could represent a significant time savings.

BACKUP DATABASE [StackOverflow2010] TO  DISK = N'C:\Program Files\Microsoft SQL Server\MSSQL14.KRONOS2017\MSSQL\Backup\StackOverflow2010\StackOverflow2010_backup_2020_09_20_180836_9111962.bak' WITH NOFORMAT, INIT,  NAME = N'StackOverflow2010-Full Database Backup', 
SKIP, NOREWIND, NOUNLOAD,  STATS = 10, CHECKSUM
GO
/*
buffercount = 4 and maxtransfersize = 1024kb
no options
Processed 1096456 pages for database 'StackOverflow2010', file 'StackOverflow2010' on file 1.
100 percent processed.
Processed 2 pages for database 'StackOverflow2010', file 'StackOverflow2010_log' on file 1.
BACKUP DATABASE successfully processed 1096458 pages in 39.041 seconds (219.412 MB/sec).
The backup set on file 1 is valid.
*/

BACKUP DATABASE [StackOverflow2010] TO  DISK = N'C:\Program Files\Microsoft SQL Server\MSSQL14.KRONOS2017\MSSQL\Backup\StackOverflow2010\StackOverflow2010_backup_2020_09_20_180836_9111962.bak' WITH NOFORMAT, INIT,  NAME = N'StackOverflow2010-Full Database Backup', 
SKIP, NOREWIND, NOUNLOAD,  STATS = 10, CHECKSUM, BUFFERCOUNT = 8
/*
Buffer Count = 8
Processed 1096456 pages for database 'StackOverflow2010', file 'StackOverflow2010' on file 1.
100 percent processed.
Processed 2 pages for database 'StackOverflow2010', file 'StackOverflow2010_log' on file 1.
BACKUP DATABASE successfully processed 1096458 pages in 33.118 seconds (258.653 MB/sec).
The backup set on file 1 is valid.
*/

BACKUP DATABASE [StackOverflow2010] TO  DISK = N'C:\Program Files\Microsoft SQL Server\MSSQL14.KRONOS2017\MSSQL\Backup\StackOverflow2010\StackOverflow2010_backup_2020_09_20_180836_9111962.bak' WITH NOFORMAT, INIT,  NAME = N'StackOverflow2010-Full Database Backup', 
SKIP, NOREWIND, NOUNLOAD,  STATS = 10, CHECKSUM, BUFFERCOUNT = 8, maxtransfersize = 4194304

/*
Buffercount = 8 and maxtransfersize = 4MB
Processed 1096456 pages for database 'StackOverflow2010', file 'StackOverflow2010' on file 1.
100 percent processed.
Processed 2 pages for database 'StackOverflow2010', file 'StackOverflow2010_log' on file 1.
BACKUP DATABASE successfully processed 1096458 pages in 31.715 seconds (270.095 MB/sec).
The backup set on file 1 is valid.
*/

 

Method 3: Using Powershell to Make a Backup

 

If you’re not using PowerShell with your SQL Servers, you should be.  If you’re not using the DBATools module with your SQL Servers, get it now. PowerShell can do fantastic, wonderful things and DBATools can do powerful, amazing things for you with respect to all things SQL Server. Below is a simple example of using the DBATools command Backup-DbaDatabase to make a full backup.  The command has a full range of options available, including backing up all the databases on a SQL Server when you don’t pass the -Database parameter. You should check into it right away.

Backup-DbaDatabase -SqlInstance MySQLServer -Database Stackoverflow2010 -path 'C:\Program Files\Microsoft SQL Server\MSSQL14\MSSQL\Backup'

 

Method 4: Using Maintenance Plans to Make a Backup

 

Because this post is already running long and there are a lot of other places that show how to use maintenance plans, I’m not going to step through how to do this here. You can go here and here to see how to use maintenance plans.  Also Jonathan Kehayias has a Pluralsight course on using them.

I am going to give you some quick thoughts on using maintenance plans though. First, Maintenance Plans are another GUI method to setting up backups.  In that respect, they are an easy point and click method to handle taking backups and handling backup retention, which we haven’t even talked about. Secondly, because of the nature of this method, which allows you to pick Backups as an option for a plan and then steps you through each part of a workflow wizard, Maintenance Plans can be a common approach for IT professionals coming from the SysAdmin side of things. No need to know or understand the options presented in the SSMS Backup wizard, for example.

Some downsides exist in backups inside maintenance plans prior to SQL Server 2016. Namely, the below options don’t exist.

– Perform checksum
– Continue on error
– Block size
– Max transfer size

Method 5: Using Ola Hallengren’s Maintenance Solution to Make A Backup

 

You can get help with Ola’s scripts here. Ola’s scripts are known and used world wide by DBAs.  They just work.  When you go to his page, you’ll download the file maintenancesolution.sql.  That file contains not only the backup routine but routines for CHECKDB and index maintenance.  You can download each components scripts individually as well. On his support page for the backup script, he explains all the available options and provides a laundry list of real world examples for how to set up the scripts for various scenarios.

Ola has options for the buffer count and maxtransfersize I demonstrated earlier.  He has options for using his scripts with AlwaysOn Availability Groups and he has options for backup retention. His scripts even work with third party vendor applications like LiteSpeed and Red Gate SQL Backup Pro.

What to Do Next:

  1. Review your current backup strategy and look for servers that aren’t being backed up at all.
  2. If you are a SysAdmin or an “Accidental DBA”, look at those links about setting up maintenance plans and get busy backing up your servers.
  3. If you didn’t know about DBATools, then stop what you’re doing and go to their web page.

5 Types of SQL Server Backups

One of the foundational responsibilities of a DBA is making sure that backups are available.  SQL Server has several foundational types of backups that can be made. Before I dive into that though, I want to ask a question that seems to have a fairly obvious answer.

Why take backups?

  1. You want to be able to recover from data corruption or mistakes made when people make changes directly to the data – like when they write and run a script in SSMS, outside of an application that’s using a set of tested stored procedures.
  2. You want to be able to refresh you non-prod environments so developers have data that looks like prod  to develop against.
  3. You want to have backups so you can test whether you can restore and how long it will take
  4. You want backups as they are often an essential part of the migration process from one SQL Server to another – usually newer SQL Server, or to Azure.

SQL Server Full Backups

What does a full backup accomplish?  A full backup, creates a backup of the entire database. It is all-inclusive, containing the entire database structure and all the data.  When a full backup starts, the Log Sequence Number is recorded and when the full backup completes, the Log Sequence Number is recorded.  This Log Sequence Number is the mechanism used by SQL Server to know what order INSERTS, UPDATES or DELETES occurred in.  As such, having the beginning and ending Log Sequence number recorded as part of the full backup allows for a transactionally consistent backup because the full backup is aware of the changes that took place while the backup was being made.  This allows for recovery of those transactions during the restore process. This full backup serves as the base backup, or starting point for all differential and log backups after it.  Each time a new full backup is taken, a new base backup point is created.

SQL Server Differential Backups

What does a differential backup do for you?  A differential backup works off the concept of a differential bitmap.  SQL Server is composed of eight kilobyte pages. These pages are collected in groups of eight called “Extents.” There is a bitmap that is used to mark any Extents that have experienced data changes since the last full backup. When a differential backup is executed, it is this bitmap that determines what data is backed up.

Based on the size and rate of change in a database, differential backups are often going to complete faster than a full backup.  Additionally, the differential backups will be smaller in size since only changed extents are backed up, rather than the entire database.

When the full recovery model is used, the power of the differential can be very important.  The differential backup reduces the number of log backups that have to be restored. For example, if there is a full backup taken on Sunday night, and transaction logs every 15 minutes after that, when a DBA needs to restore the database to Thursday at 5:53 AM, she needs to construct a restore process based on the Full backup and every transaction log since Sunday evening!  That’s a lot of transaction logs to have to include.  With a daily differential occurring at 5 AM, the DBA now only needs to restore the full backup from Sunday evening, the one differential that was created at 5 AM on Thursday, and only the transaction log backups that occurred after 5 AM Thursday. Being able to skip over using all those log backups is powerful.

This makes for much simpler script creation and readability, and can save a lot of time if the DBA is trying to write this script manually.  You should have an automated process to create the backup script, but some DBAs don’t have that set up yet. We’ll talk about automating the backups script creation  later in the series.

SQL Server File and FileGroup Backups

What are File and FileGroup backups? Well to start with, this is a more advanced topic than these other backup types. As such, it is less likely that you will need this type of backup. However, knowing what your options are is always a good idea.

File and FileGroup backups allow a more granular backup, and by extension, a more granular restore process. It does what it sounds like.  A file or filegroup backup backs up an individual file or a collection of files contained in a filegroup. This allows for restoring a small part of a database to fix a problem rather than having to restore the entire database. This option, however, does introduce complexity and, in my opinion, is really only useful for large databases, particularly in the range of 500 GB or more.

These file and filegroup backup types can be used when a larger database has been divided into multiple files.  For example, a large database containing sales information might be divided into various Sales files, perhaps by year, or even by month if there are a high volume of sales recorded in the database.  These files are then aggregated into FileGroups. See below for an example of creating files and filegroups in a database. This example was taken from the  MS docs page here.

USE master;
GO
-- Create the database with the default data
-- filegroup, filestream filegroup and a log file. Specify the
-- growth increment and the max size for the
-- primary data file.
CREATE DATABASE MyDB
ON PRIMARY
  ( NAME='MyDB_Primary',
    FILENAME=
       'c:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\data\MyDB_Prm.mdf',
    SIZE=4MB,
    MAXSIZE=10MB,
    FILEGROWTH=1MB),

--Here is where a separate file is created and assigned to a FileGroup, which is little more than a container.
FILEGROUP MyDB_FG1
  ( NAME = 'MyDB_FG1_Dat1',
    FILENAME =
       'c:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\data\MyDB_FG1_1.ndf',
    SIZE = 1MB,
    MAXSIZE=10MB,
    FILEGROWTH=1MB),

--Here is where another separate file is created and assigned to a FileGroup.
  ( NAME = 'MyDB_FG1_Dat2',
    FILENAME =
       'c:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\data\MyDB_FG1_2.ndf',
    SIZE = 1MB,
    MAXSIZE=10MB,
    FILEGROWTH=1MB)
LOG ON
  ( NAME='MyDB_log',
    FILENAME =
       'c:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\data\MyDB.ldf',
    SIZE=1MB,
    MAXSIZE=10MB,
    FILEGROWTH=1MB);
GO
ALTER DATABASE MyDB 
  MODIFY FILEGROUP MyDB_FG1 DEFAULT;
GO

-- Create a table in the user-defined filegroup.
USE MyDB;
CREATE TABLE MyTable
  ( cola int PRIMARY KEY,
    colb char(8) )
ON MyDB_FG1;
GO

SQL Server Transaction Log Backups

What is a transaction log backup? A transaction log backup will take a backup of SQL Server’s transaction log so that data changes stored there can be recovered during the restore process.  Again the Log Sequence Number concept mentioned earlier is key.  Because transactions occur in a certain order, the log backups contain that order of transactions.  Transaction log backups must be restored in order.  If you try to restore a sequence of log backups in the wrong order, SQL Server will generate an error.

Having these log backups allows you to do what is called a point in time restore.  Having the option for a point in time restore can be critical for a business. Let’s suppose someone is making changes directly to one or more tables.  If a mistake is made and an unexpected data change occurs, point in time restore will allow you as the DBA to recover the data up to the moment just before the erroneous change was made.

SQL Server Copy Only Backups

What are Copy-only backups?  Copy only backups are backups that do not affect or track the Log Sequence Number information.  Why would you use this? Well, let’s suppose that you are already taking full, differential and log backups.  You are asked to provide a backup file of a database to another department or perhaps just a single developer who is working on a project. That Developer, hopefully, doesn’t have access to the backup directory for your backup processes.  Rather than you trying to copy a set of backups to a share or other location that the Developer does have access to, you could simply create a Copy Only full backup that is written to a shared location.  The Developer can then use that backup to restore locally to her machine, and you receive the benefit of not changing the base backup for your backup chain.

What to Do Next

  1. Examine your environment to see if you are leveraging the “Power of the Diff” by taking differential backups to help simplify your recovery process.
  2. Consider the size of your databases and determine if you have a situation that might benefit from having a database with multiple files and filegroups.
  3. If you would like help with anything in this post, or with something else related to SQL Server, reach out to me here, or on Twitter, and I’ll be glad to offer assistance.