29 Aug 2024

Parameter sniffing in SQL Server

 Parameter sniffing in SQL Server is a feature where the query optimizer uses the specific parameter values passed to a stored procedure or query to generate an execution plan. While this can be beneficial for performance, it can also cause issues when the chosen execution plan is not optimal for other parameter values. This can lead to queries performing poorly for some inputs, particularly when the data distribution is uneven.

Why Parameter Sniffing Can Be a Problem

When SQL Server compiles a stored procedure or a parameterized query for the first time, it creates an execution plan based on the initial parameter values provided. If those initial values are atypical or represent edge cases, the generated plan might not perform well for more common parameter values.

How to Resolve Parameter Sniffing Issues

  1. Use OPTION (RECOMPILE)

    • Adding OPTION (RECOMPILE) to a query forces SQL Server to generate a new execution plan every time the query is executed.
    • Pros: Ensures the plan is optimized for the specific parameter values at runtime.
    • Cons: Recompiling the plan for every execution can add overhead, especially for frequently run queries.


    SELECT * FROM Orders WHERE CustomerID = @CustomerID OPTION (RECOMPILE);
  2. Use WITH RECOMPILE in Stored Procedures

    • Adding WITH RECOMPILE when creating or executing a stored procedure forces SQL Server to recompile the procedure each time it is executed.
    • Pros: Ensures the execution plan is tailored to the specific parameters each time.
    • Cons: Similar to OPTION (RECOMPILE), this can introduce overhead.


    CREATE PROCEDURE GetOrders @CustomerID INT WITH RECOMPILE AS BEGIN SELECT * FROM Orders WHERE CustomerID = @CustomerID; END;
  3. Optimize with OPTION (OPTIMIZE FOR @parameter)

    • Use the OPTIMIZE FOR hint to instruct SQL Server to optimize the query for a specific parameter value, which might be more representative of typical use cases.
    • Pros: Can lead to a more consistent execution plan for typical cases.
    • Cons: May still be suboptimal for other edge cases.

    SELECT * FROM Orders WHERE CustomerID = @CustomerID OPTION (OPTIMIZE FOR (@CustomerID = 123));
  4. Use OPTIMIZE FOR UNKNOWN

    • This option tells SQL Server to generate a "generalized" execution plan rather than one based on the specific parameter values, as if the parameter values were not known at compile time.
    • Pros: Useful when you want a more generic plan that doesn't overly favor any particular parameter value.
    • Cons: The resulting plan might not be optimal for any specific case but can provide more stable performance across a range of values.

    SELECT * FROM Orders WHERE CustomerID = @CustomerID OPTION (OPTIMIZE FOR UNKNOWN);
  5. Manually Create Multiple Plans with Different Parameters

    • You can create separate stored procedures or queries optimized for different parameter ranges.
    • Pros: Each version can be tailored to a specific type of query or set of parameter values.
    • Cons: Increases maintenance complexity as you manage multiple versions of the same logic.

    IF @CustomerID BETWEEN 1 AND 100 BEGIN EXEC GetOrders_SmallCustomers @CustomerID; END ELSE BEGIN EXEC GetOrders_LargeCustomers @CustomerID; END;
  6. Use Dynamic SQL

    • Writing your query using dynamic SQL inside a stored procedure ensures the query plan is compiled fresh for each execution based on the actual parameter values.
    • Pros: Tailors the execution plan to the exact values being passed.
    • Cons: Dynamic SQL can make code harder to read and maintain and may have security implications (e.g., SQL injection risks).
    DECLARE @SQL NVARCHAR(MAX); SET @SQL = 'SELECT * FROM Orders WHERE CustomerID = @CustomerID'; EXEC sp_executesql @SQL, N'@CustomerID INT', @CustomerID;
  7. Index Tuning

    • Sometimes, parameter sniffing issues are exacerbated by suboptimal indexes. Reviewing and optimizing indexes can mitigate these issues.
    • Pros: Can resolve the root cause by ensuring the most efficient data access methods.
    • Cons: Requires analysis and might involve significant changes to the indexing strategy.

Monitoring and Diagnosing Parameter Sniffing

  • Query Store: SQL Server's Query Store feature can help identify queries that suffer from parameter sniffing by tracking query performance and execution plans over time.
  • Execution Plan Analysis: Comparing execution plans for different parameter values can reveal if parameter sniffing is causing suboptimal plans.

By applying these strategies, you can manage and mitigate the effects of parameter sniffing, leading to more consistent and reliable query performance in SQL Server.

26 Aug 2024

SQL EXISTS vs IN vs JOIN Performance Comparison

 

Problem

I built a query that utilizes a subquery that needs to be compared back to a main query. I want to know how to best achieve this task. Should I use an IN statement? An EXISTS? Or maybe a JOIN? I need to know which options will be valid for my use case and which one will perform the best. I also need to be able to prove it.

Solution

As with many situations within SQL Server the answer depends on the circumstances. This tip will look at the pros and cons of each method and use a repeatable methodology to determine which method will offer the fastest performance. The best part about this tip is that the performance comparison methodology can be applied to any TSQL coding situation!

Compare SQL Server EXISTS vs. IN vs JOIN T-SQL Subquery Code

All of the demos in this tip will use the WideWorldImporters sample database which can be downloaded for free from here and will be run against SQL Server 2019. The images might be different, but the methodology should still work on older versions of SQL Server.

The subquery to be used will be a list of the top 3 sales people for a single quarter based on invoice count. For the simplicity of the example queries, this subquery will be stored as a view. As seen in this tip, for a simple query like this one, there likely isn't a difference in performance between using a view, CTE, or traditional subquery.

CREATE VIEW vTop3SPs2013Q2
AS
SELECT TOP 3 SalespersonPersonID
FROM Sales.Invoices
WHERE InvoiceDate BETWEEN '4/1/2013' AND '6/30/2013'
GROUP BY [SalespersonPersonID]
ORDER BY COUNT(*) DESC

SQL IN Code

The IN statement can be used to find rows in a query where one column can be matched to a value in a list of values. The list of values can be hard coded as a comma-separated list or can come from a subquery as it does in this example.

IN statements are easy to write and understand. The only downside is that they can only compare a single column from the subquery to a single column from the main query. If 2 or more values need to be compared then the IN statement cannot be used.

Below is a query that returns some invoices that belonged to our top group of salespeople. Notice that the subquery returns exactly one row. This is a requirement for use of the IN statement. Also note that the query inside the parentheses is a fully functional query on its own. It can be highlighted and executed by itself.

SELECT Invoices.InvoiceID, Invoices.TotalDryItems, People.FullName
FROM Sales.Invoices
  INNER JOIN [Application].People ON Invoices.SalespersonPersonID = People.PersonID
WHERE SalespersonPersonID IN (SELECT SalespersonPersonID FROM vTop3SPs2013Q2)
  AND InvoiceDate BETWEEN '4/1/2013' AND '6/30/2013'
  AND TotalDryItems >= 4;

Executing that query with both STATISTICS IO and STATISTIC TIME enabled, outputs this information. This output will give us some metrics for performance to compare to the other options. If unfamiliar with how to get this output, please consult this tip.

This screenshot of the output shows that the query returned 681 rows and used 13,116 reads from Invoices and 6 reads from People.  It executed in 39ms.

SQL EXISTS Code

The EXISTS statement functions similarly to the IN statement except that it can be used to find rows where one or more columns from the query can be found in another data set, usually a subquery. Hard coding isn't an option with EXISTS.

Below is the same query as above except that the IN has been replaced by EXISTS. The format for EXISTS and its subquery is a little bit different than for IN. In this case, the subquery references a column, I.SalespersonPersonID, that does seem to be available to the subquery. For this reason, the subquery cannot be executed on its own and can only be executed in the context of the entire query. This can sometimes be difficult to understand.

Logically, think of it as having the subquery run once for every row in the main query to be determined if a row exists. If a row exists upon executing the subquery, then the Boolean return value is true. Otherwise, it is false. The selected column(s) of the subquery does not matter as the result is tied only to the existence or non-existence of a row based on the FROM/JOIN/WHERE clauses in the subquery.

SELECT I.InvoiceID, I.TotalDryItems, People.FullName
FROM Sales.Invoices I
  INNER JOIN [Application].People ON I.SalespersonPersonID = People.PersonID
WHERE EXISTS (SELECT 1 FROM vTop3SPs2013Q2 WHERE SalespersonPersonID = I.SalespersonPersonID)
  AND I.InvoiceDate BETWEEN '4/1/2013' AND '6/30/2013'
  AND I.TotalDryItems >= 4;

Executing this query returns the following statistical output which is virtually identical to the IN statement.

sql server execution

SQL INNER JOIN Code

A regular JOIN can be used to find matching values in a subquery. Like EXISTS, JOIN allows one or more columns to be used to find matches. Unlike EXISTS, JOIN isn't as confusing to implement. The downside to JOIN is that if the subquery has any identical rows based on the JOIN predicate, then the main query will repeat rows which could lead to invalid query outputs. Both IN and EXISTS will ignore duplicate values in a subquery. Take extra precaution when joining to a table in this fashion. In this example, the view will not return any duplicate SalespersonPersonID values, so it is a safe implementation of a JOIN.

SELECT I.InvoiceID, I.TotalDryItems, People.FullName
FROM Sales.Invoices I
  INNER JOIN [Application].People ON I.SalespersonPersonID = People.PersonID
  INNER JOIN vTop3SPs2013Q2 ON I.SalespersonPersonID = vTop3SPs2013Q2.SalespersonPersonID
WHERE InvoiceDate BETWEEN '4/1/2013' AND '6/30/2013'
  AND TotalDryItems >= 4;

Executing this query returns the following statistical output which is, once again, virtually identical to the IN and EXISTS statement versions of the query.

sql server execution time

Why are all of the statistics the same?

The statistics for each of these 3 options are virtually identical because the optimizer is compiling all 3 options into the same query plan. This can be seen by running all three queries together while viewing the actual execution plans. The screenshot below shows one plan, but the exact same plan appears in each of the 3 query options.

create nonclustered index

Each copy of the query plan shows a missing index recommendation. Acting on that recommendation and creating the index will modify the plans and query performance statistics. Will it modify them all the same way? Let's find out. First, make the index, then rerun the 3 queries.

CREATE NONCLUSTERED INDEX mssqltips ON [Sales].[Invoices] ([InvoiceDate],[TotalDryItems]) 
   INCLUDE ([InvoiceID],[SalespersonPersonID]);

Now execute all 3 statements together again. Something interesting happens. All 3 plans have changed from the versions they were before the index creation, but they are not identical this time. The IN and EXISTS got the same new plan, but the JOIN gets a different plan.

The plan for the IN and EXISTS used the new index twice and performed a SEEK on the People table.

nested loops

This plan was generated for the JOIN version of the query. It used the new index twice, but performed a SCAN on the people table.

index scan

Checking the IO and TIME statistics for the 3 queries shows identical statistics for the 2 queries that shared a plan, but improved statistics and execution time for the JOIN version. If this were a query getting ready to be promoted to production, going with the JOIN would probably be the best bet.

sql server execution times

Conclusion

This query is a great example that while the optimizer strives to treat each option the same, it won't always do that. Using this performance verification methodology, along with understanding the value and limitations of each query option, will allow the programmer to make the best choice in each situation.

Many Uses of SQL CASE Expression in a JOIN Clause

 

Problem

Have you ever heard: "Never place a CASE expression in a JOIN clause?" I know I have. It's one of those best practices handed down by the SQL elders of old. I'm all for following guidelines, but sometimes I like to know why I'm doing something. Not just because someone wrote a blog article 20 years ago and it's the first one that shows up in search results. Stay tuned if you're like me, and don't follow advice blindly.


Solution

In this tutorial, we'll look at the many uses of a CASE expression. Additionally, why would you use a CASE expression in a JOIN clause? Next, I'll explore a common alternative that may or may not provide better performance. Finally, we'll answer the questions like "Should you never use a CASE expression in a JOIN clause?" My answer might surprise you.

Exploring SQL CASE

You're missing out if you've never used a CASE expression. It's the next command to learn after SELECT *. You might describe it as the MacGyver of T-SQL. Microsoft defines CASE as an expression that evaluates a list of conditions and returns one of the multiple possible result expressions. Said another way: Tell SQL that if this condition is true, then return this other thing.

SELECT CASE
           WHEN 1 = 1 THEN
               1
           WHEN 2 = 2 THEN
               2
           WHEN 3 = 3 THEN
               3
           ELSE
               0
       END AS [My Favorite Number];

Before moving on, try and answer the questions in the example above. What results will SQL Server return and why?

If you feel extra motivation, you can nest CASE expressions for added control. I'm not a fan of doing this if it makes code harder to follow. I would rather repeat certain parts. I know there is an adage of DRY ("Don't Repeat Yourself."), which, for the most part, I agree with.

Below is an example of a nested CASE expression. Doesn't that look lovely? If I put more effort in, I could clean it up. However, I use Redgate SQL Prompt to format all my code, including the example below.

SELECT CASE
           WHEN 1 = 1 THEN
               CASE
                   WHEN 2 = 2 THEN
                       CASE
                           WHEN 3 = 3 THEN
                               3
                           ELSE
                               0
                       END
                   ELSE
                       0
               END
           ELSE
               0
       END AS [My Second Favorite Number];

There are a few other places where you see CASE expressions; the first is in a WHERE clause:

SELECT Column3
FROM dbo.SpecialTable
WHERE CASE
          WHEN Column3 = 'Hello' THEN
              1
          WHEN Column3 = 'Goodbye' THEN
              2
          ELSE
              0
      END > 0;

And finally, in the example below, I've added it to the JOIN clause:

SELECT t1.Column1,
       t1.Column2,
       t2.Column3
FROM Table1 t1
    INNER JOIN SpecialTable t2
        ON CASE
               WHEN t2.Column3 = 'Hello' THEN
                   1
               WHEN t2.Column3 = 'Goodbye' THEN
                   2
               ELSE
                   0
           END = t1.JoinColumn;

As I mentioned above, people might frown on the second more than the first. I've seen it written on several "you should never do" lists.

Build Our Dataset

Let's create a dataset where using a CASE expression in a JOIN makes sense. I'll add two tables: Person and Sales.

CREATE TABLE dbo.Person
(
    Id INT NOT NULL,
    FirstName NVARCHAR(200) NOT NULL,
    LastName NVARCHAR(200) NOT NULL,
     IsActive BIT NOT NULL
    CONSTRAINT PK_Person_Id
        PRIMARY KEY CLUSTERED (Id)
);
CREATE TABLE dbo.Sales
(
    Id INT IDENTITY(1, 1) NOT NULL,
    SalesDate DATE NOT NULL,
    OrderCode NVARCHAR(3) NOT NULL,
    Quantity INT NOT NULL,
    Price DECIMAL(6, 0) NOT NULL,
    SalesPersonId INT NOT NULL,
    AccountManagerId INT NULL,
    CONSTRAINT PK_Sales_Id
        PRIMARY KEY CLUSTERED (Id),
    CONSTRAINT FK_Sales_SalesPerson
        FOREIGN KEY (SalesPersonId)
        REFERENCES dbo.Person (Id),
    CONSTRAINT FK_Sales_AccountManager
        FOREIGN KEY (AccountManagerId)
        REFERENCES dbo.Person (Id)
);
GO

Now, let's add some data to each of them. For the Person table, we'll insert 20 rows. For the Sales table, we'll insert 100,000 rows. I want at least one table to be on the larger side.

INSERT INTO dbo.Person
(
    Id,
    FirstName,
    LastName,
    IsActive
)
VALUES
(1, 'John', 'Smith', 1),
(2, 'Sarah', 'Johnson', 1),
(3, 'Michael', 'Williams', 1),
(4, 'Emily', 'Brown', 1),
(5, 'David', 'Jones', 1),
(6, 'Emma', 'Garcia', 1),
(7, 'Daniel', 'Martinez', 1),
(8, 'Olivia', 'Davis', 1),
(9, 'Matthew', 'Rodriguez', 1),
(10, 'Ava', 'Miller', 1),
(11, 'Christopher', 'Gonzalez', 1),
(12, 'Sophia', 'Wilson', 1),
(13, 'Andrew', 'Anderson', 1),
(14, 'Isabella', 'Thomas', 1),
(15, 'Joshua', 'Jackson', 1),
(16, 'Mia', 'White', 1),
(17, 'William', 'Harris', 1),
(18, 'Charlotte', 'Clark', 1),
(19, 'Ethan', 'Lewis', 1),
(20, 'Amelia', 'Lee', 1);



WITH SalesData AS (
    SELECT TOP 100000
        DATEADD(DAY, RAND(CHECKSUM(NEWID())) * (1 + DATEDIFF(DAY, '2023-04-01', '04-30-2023')), '2023-04-01') AS SalesDate,
        CASE
            WHEN (ROW_NUMBER() OVER (ORDER BY (SELECT NULL))) % 3 = 0 THEN 'ABC'
            WHEN (ROW_NUMBER() OVER (ORDER BY (SELECT NULL))) % 3 = 1 THEN 'DEF'
            ELSE 'GHI'
        END AS OrderCode,
        ABS(CHECKSUM(NEWID()) % 100) + 1 AS Quantity,
        ABS(CHECKSUM(NEWID()) % 1000) + 100 AS Price,
        ABS(CHECKSUM(NEWID()) % 10) + 1 AS SalesPersonId,
        CASE
            WHEN (ROW_NUMBER() OVER (ORDER BY (SELECT NULL))) % 100 = 0 THEN NULL
            ELSE ABS(CHECKSUM(NEWID()) % 10) + 11
        END AS AccountManagerId
      FROM sys.all_columns cl1
    CROSS JOIN sys.all_columns cl2
)
-- Insert the data into the dbo.Sales table
INSERT INTO dbo.Sales
(
    SalesDate,
    OrderCode,
    Quantity,
    Price,
    SalesPersonId,
    AccountManagerId
)
SELECT SalesDate,
       OrderCode,
       Quantity,
       Price,
       SalesPersonId,
       AccountManagerId
FROM SalesData;

We always populate the SalesPersonId in the Sales table. The AccountManagerId column is sometimes populated. You can see that the SalesPersonId and AccountManagerId are foreign keys back to the Person table.

Why Use a CASE Expression in a JOIN?

Why would you use a CASE in a JOIN clause in the first place? One reason is if you have some complex join logic you're trying to implement, essentially joining a table based on multiple conditions. For example, we want to join our Person and Sales tables primarily on the SalesPersonId. However, when the OrderCode has a specific value, we want to join on the AccountManagerId.

SELECT CONCAT(   p.LastName,
                 ', ',
                 p.FirstName
             ) AS PersonName,
       SUM(s.Price) AS TotalSales,
       s.SalesDate
FROM dbo.Sales s
    INNER JOIN dbo.Person p
        ON p.Id = CASE
                      WHEN s.OrderCode = 'ABC'
                           AND s.AccountManagerId IS NOT NULL THEN
                          s.AccountManagerId
                      ELSE
                          s.SalesPersonId
                  END
GROUP BY s.SalesDate,
         p.FirstName,
         p.LastName;

Another reason for using CASE is to handle NULLs returned from one of the tables. Of course, this is when an ISNULL() doesn't provide the results you're looking for.

SELECT CONCAT(   p.LastName,
                 ', ',
                 p.FirstName
             ) AS PersonName,
       SUM(s.Price) AS TotalSales,
       s.SalesDate
FROM dbo.Sales s
    INNER JOIN dbo.Person p
        ON p.Id = CASE
                      WHEN s.AccountManagerId IS NULL THEN
                          1
                      WHEN s.AccountManagerId IS NULL
                           AND s.SalesDate > '2023-04-05' THEN
                          2
                      ELSE
                          s.SalesPersonId
                  END
GROUP BY s.SalesDate,
         p.FirstName,
         p.LastName;

I'm sure there are other examples, but these two seem to come up the most.

Alternatives to CASE

We have yet to look at the performance of CASE versus another method. However, how can we accomplish the same results differently by not using a CASE expression? The one I see recommended the most is using a UNION ALL operator. That's the one we'll compare against.

Below is how you would write a statement to avoid using the CASE. The UNION ALL creates two datasets that match the join criteria and then combines them.

SELECT CONCAT(   p.LastName,
                 ', ',
                 p.FirstName
             ) AS PersonName,
       SUM(s1.Price) AS TotalSales,
       s1.SalesDate
FROM dbo.Sales s1
    INNER JOIN dbo.Person p
        ON p.Id = s1.SalesPersonId
WHERE s1.OrderCode <> 'ABC'
      OR s1.AccountManagerId IS NULL
GROUP BY s1.SalesDate,
         p.FirstName,
         p.LastName
UNION ALL
SELECT CONCAT(   p.LastName,
                 ', ',
                 p.FirstName
             ) AS PersonName,
       SUM(s2.Price) AS TotalSales,
       s2.SalesDate
FROM dbo.Sales s2
    INNER JOIN dbo.Person p
        ON p.Id = s2.AccountManagerId
WHERE s2.OrderCode = 'ABC'
      AND s2.AccountManagerId IS NOT NULL
GROUP BY s2.SalesDate,
         p.FirstName,
         p.LastName;

One thing to remember is that the above method goes against the DRY philosophy. Not that I advocate following the philosophy to a T.

Comparing Performance

Now it's time to compare both methods and see if one performs better. Before we do that, I'm going to create three indexes. The first is mainly for the statement with the CASE expression, and the other two are for the UNION ALL.

CREATE NONCLUSTERED INDEX IX_SalesPerson_AccountManager_OrderCode
ON dbo.Sales (
                 SalesPersonId,
                 AccountManagerId,
                 OrderCode
             )
INCLUDE (
            Price,
            SalesDate
        );


CREATE NONCLUSTERED INDEX IX_AccountManager_OrderCode
ON dbo.Sales (
                 AccountManagerId,
                 OrderCode
             )
INCLUDE (
            Price,
            SalesDate
        );

CREATE NONCLUSTERED INDEX IX_SalesPerson_OrderCode
ON dbo.Sales (
                 SalesPersonId,
                 OrderCode
             )
INCLUDE (
            Price,
            SalesDate
        );
GO

First, turn STATISTICS IO on and enable the actual execution plan. Then execute both statements in the same batch and see what the performance markers say.

SET STATISTICS IO ON;

SELECT        CONCAT(   p.LastName,
                 ', ',
                 p.FirstName
             ) AS PersonName,
       SUM(s.Price) AS TotalSales,
       s.SalesDate
FROM dbo.Sales s
    INNER JOIN dbo.Person p
        ON p.Id = CASE
                      WHEN s.OrderCode = 'ABC'
                           AND s.AccountManagerId IS NOT NULL THEN
                          s.AccountManagerId
                      ELSE
                          s.SalesPersonId
                  END
GROUP BY s.SalesDate,
         p.FirstName,
           p.LastName;

/* Second method using a UNION ALL operator*/
SELECT CONCAT(   p.LastName,                 ', ',
                 p.FirstName
             ) AS PersonName,
       SUM(s1.Price) AS TotalSales,
       s1.SalesDate
FROM dbo.Sales s1
    INNER JOIN dbo.Person p
        ON p.Id = s1.SalesPersonId
WHERE s1.OrderCode <> 'ABC'
      OR s1.AccountManagerId IS NULL
GROUP BY s1.SalesDate,
         p.FirstName,
         p.LastName
UNION ALL
SELECT CONCAT(   p.LastName,
                 ', ',
                 p.FirstName
             ) AS PersonName,
       SUM(s2.Price) AS TotalSales,
       s2.SalesDate
FROM dbo.Sales s2
    INNER JOIN dbo.Person p
        ON p.Id = s2.AccountManagerId
WHERE s2.OrderCode = 'ABC'
      AND s2.AccountManagerId IS NOT NULL
GROUP BY s2.SalesDate,
         p.FirstName,
           p.LastName;

SET STATISTICS IO OFF;
GO
Statistics IO results. Page Reads.
Execution Plan from both queries in one batch.

You can see from the images above that the UNION ALL makes up more than 50% of the batch, along with additional page reads on both tables. Ultimately, the difference isn't going to be life-changing. However, if someone says using a CASE expression in a JOIN is a lousy practice, ask them why. I don't advocate using a CASE expression in the join for every query. But there are instances where it comes in handy.

I'm curious. Do you ever use a CASE expression in a JOIN clause? Please let me know in the comments below.

Key Points

  • You can use a CASE expression in almost any part of a SQL statement, including the WHERE and JOIN.
  • Given the example, the CASE expression performed better in this tutorial than the UNION ALL. However, this isn't an absolute rule.
  • If someone says adding a CASE expression to a JOIN clause is a bad practice, ask them to explain why.

Union Budget 2024-25 – All You Need to Know

  Union Budget 2024-25 – All You Need to Know The Union Budget is a yearly financial plan presented by the Finance Minister of India for the...