studia korolevae Int © Library of CCU / studiakorolevae SQL Window Functions Nate Rosidi Nate Rosidi is a data scientist and in product strategy. He's also an adjunct professor teaching analytics, and is the founder of StrataScratch, a platform helping data scientists prepare for their interviews with real interview questions from top companies. Connect with him on Twitter: StrataScratch or LinkedIn. https://www.kdnuggets.com/2022/04/sql-window-functions.html In this article, we’ll go over SQL window functions and how to use them when writing SQL queries*. 1. Introduction Writing well-­‐structured, efficient SQL queries is no easy task. It requires a thorough knowledge of all SQL functions and statements, so that you can apply them in your everyday job to solve problems efficiently. In this article, we’ll talk about SQL window functions, which offer a lot of utility to solve common problems, but they are often overlooked. SQL window functions are very versatile, and can be used to address many different problems in SQL. Almost all companies ask interview questions that require at least some knowledge of window functions to answer. So if you’re preparing for a data science interview, it’s a good idea to refresh your knowledge of window functions in SQL. In this article, we will focus on the basics. If you’d like to gain a deeper understanding, read this ultimate guide to SQL window functions. Having a good grasp of window functions can also help you write efficient and optimized SQL queries to address problems you encounter in your everyday job. 2. What is a Window Function in SQL? In simple words, window functions in SQL are used to access all other rows of the table to generate a value for the current row. * By Nate Rosidi, KDnuggets on April 5, 2022 in SQL https://www.kdnuggets.com/tag/sql 1 2 Nate Rosidi SQL Window functions get their name because their syntax allows you to define a specific portion, or window of data to work on. First, we define a function, which will run on all rows, and then we use the OVER clause to specify the window of data. Window functions are considered an ‘advanced’ feature of SQL. At first glance, junior data scientists might be scared by the syntax, but with a little practice, SQL window functions can become much less scary. 2.1.SQL Window Function Types Aggregate Window Functions are necessary to do calculations or find the lowest or highest extremes of data within the window of data. These are the same as regular aggregate functions, but they are applied to specific windows of data, so their behavior is different. Ranking Window Functions give us the ability to assign rank numbers to a window of data. Each of the 6 major functions ranks rows differently. The ranking also depends on the use of ORDER BY statement. Value Window Functions allow us to find the values based on their position relative to the current row. They are useful for getting the value from previous or following rows, and for analysis of time-­‐series data. This is just a short overview of three types of SQL window functions. We’ll discuss them in detail in later parts of the article. 2.2.How and When To Use Them? Once you understand all the window functions and their use cases, it will become a powerful tool at your disposal. They can save you from writing many unnecessary lines of code to solve the problems that can be solved by a single window function. SQL Window Functions VS Group By Statement Beginners who read the description of window functions are often confused about what it’s supposed to do and how they’re different from the Group By statement, which seems to work in the exact same way. However, the confusion will go away if you write window functions and see their actual output. The most significant difference between the Window functions and the Group By statement is that the former allows us to summarize the values while keeping all of the original data. The GROUP BY statement also lets us generate the aggregate values, but the rows are collapsed into a few groups of data. studia korolevae 2022-04-06 SQL Window Functions 3 Use Cases Getting a job as a data scientist is only the beginning. In your everyday job you’ll encounter problems that need to be solved efficiently. Window functions are very versatile and they will be invaluable as long as you know how to use them. For instance, if you’re working for a company like Apple, you might need to analyze inner sales data to find the most popular, or the least popular products in their portfolio. One of the most common use cases of window functions is to track time-­‐series data. For instance, you might have to calculate month-­‐over-­‐month growth or decline of specific Apple products, or bookings on Airbnb platform. Data scientists who work for SaaS companies are often tasked with calculating user churn rate, and track its changes over time. As long as you have user data, you can use window functions to keep track of churn rate. Aggregate window functions, such as SUM() can be useful for calculating running totals. Let’s imagine that we have sales data for all 12 months of the year. With window functions, we can write a query to calculate a running total (current month + the total of previous months) of the sales data. Window functions have many other use cases. For example, if you’re working with user data, you can order users by when they signed up, the number of sent messages or other similar metrics. Window functions also allow us to keep track of health statistics, such as changes in the virus spread, the severity of cases, or other similar insights. With Other SQL Keywords In order to effectively use window functions, you must first understand the order of operations in SQL. You can only use window functions with the operations that come after the window functions, not before them. In accordance with this rule, it’s possible to use window functions with the SELECT and ORDER BY statements, but not with others, such as WHERE and GROUP BY. Typically SQL developers use window functions with the SELECT statement, and the main query can include the ORDER BY statement. 2.3.Ranking Window Functions in SQL These functions allow SQL developers to assign numerical rank to rows. 4 Nate Rosidi There are six functions of this type: ROW_NUMBER() simply numerates the rows starting from 1. The order of rows depends on the ORDER BY statement. If there is none, the ROW_NUMBER() function will simply numerate the rows in their initial state. RANK() is a more nuanced version of the ROW_NUMBER() function. RANK() considers if the values are equal and assigns them the same rank. For instance, if the values in the third and fourth rows are equal, it will assign them both ranks of three, and, starting from the fifth row, it will continue counting from 5. DENSE_RANK() works just like the RANK() function, except for one difference. In the same example, if the values in third and fourth columns are tied, both will be assigned the rank of three. However, the fifth row will not start counting from 5, but from 4, considering the fact that previous rows have a rank of three. To learn more about the differences, check this post ? An Introduction to the SQL Rank Functions. PERCENT_RANK() uses a different approach to ranking. It creates a new column to display the rank values in percentages (from 0 to 1). NTILE() is a function that takes one numerical argument, which will create batches to divide the data. For instance, NTILE(20) will create 20 buckets of rows, and assign the rank accordingly. CUME_DIST() function calculates cumulative distribution of the current row. In other words, it goes through every record in the window and returns the portion of the rows with values that are less than or equal to the value in the current row. The relative size of these rows is between 0 (none) and 1 (all). Ranking Data With RANK() Ranking Window Functions often come up during the interviews at major tech companies. For example, interviewers at Facebook/Meta often ask candidates to find the most active users on Messenger. To answer this question, we must first look at the available data. We have one table with multiple different columns. The values of user1 and user2 columns are usernames, and the msg_count column represents the number of messages exchanged between them. According to the name of the question, we must find the users with the highest number of recorded activities. To do that, we must first think about what the activity is: in this context, both sending and receiving a message counts as an activity. studia korolevae 2022-04-06 SQL Window Functions 5 After looking at the data, we see that the msg_count does not represent a total number of messages sent and received by each user in the record. There may be other users they are chatting with. In order to get the total number of activities for each user, we must get the value in the msg_count column where they are at least on the sending or receiving end of the messages. Let’s take a look at the sample of data from this task: As you can see, the user called sgoodman is a part of two conversations -­‐ one is with the username wangdenise and the other with wolfelizabeth. In real life, people can have online conversations with dozens of people. Our query should capture the number of messages exchanged between them. Solution Step 1: Combine users in one column First, we select usernames in the user1 column with their corresponding msg_count value. Then we do the same for users in the user2 column and combine them in one column. We use the UNION ALL operator to do so. This will ensure that all of the users, with their corresponding, sent or received msg_count values are kept in place. SELECT user1 AS username, msg_count FROM fb_messages UNION ALL SELECT user2 AS username, msg_count FROM fb_messages We must keep in mind that in order for UNION ALL statements to combine the values, the number of columns in both SELECT statements and their respective value types must be the same. So we use the AS statement to rename them to username. If we run this code, we’ll get the following result: Step 2: Order the users in decreasing order Once we have the list of all the users, we must select the username column from the above table and add up the msg_count values of every individual user. Then we’ll use the RANK() window function to enumerate every record. In this case, we want to use this specific function, instead of 6 Nate Rosidi DENSE_RANK() because of possible ties in the number of messages within the TOP 10. The accuracy of ranking window functions depends on the ORDER BY statement, which is used to arrange the values within the window of input data, not the output of the function. In this case, we must use the DESC keyword to make sure that the number of messages is arranged in descending order. This way, RANK() function is applied to the highest input values first. The OVER keyword is an essential part of window functions syntax. It is used to connect the ranking function to the window of data. So far, our SQL query should look something like this: WITH sq AS (SELECT username, sum(msg_count) AS total_msg_count, rank() OVER ( ORDER BY sum(msg_count) DESC) FROM (SELECT user1 AS username, msg_count FROM fb_messages UNION ALL SELECT user2 AS username, msg_count FROM fb_messages) a GROUP BY username) To solve our question, we must find the 10 most active users. Using the RANK() window function is necessary to handle the cases when there are any ties within that group of 10 users. Step 3: Display the TOP 10 In the final step, we should get username and total_msg_count values from the sq subquery, and display the ones that have a rank value of 10 or less. Then arrange them in descending order. (SELECT username, sum(msg_count) AS total_msg_count, rank() OVER ( ORDER BY sum(msg_count) DESC) FROM (SELECT user1 AS username, msg_count FROM fb_messages studia korolevae 2022-04-06 SQL Window Functions 7 UNION ALL SELECT user2 AS username, msg_count FROM fb_messages) a GROUP BY username) SELECT username, total_msg_count FROM sq WHERE rank <= 10 ORDER BY total_msg_count DESC If we run this code, we’ll see that it works as it should. And we potentially avoid any errors in case some of the users had identical total_msg_count values. Finding the Top 5 Percentile Values Here is an example of another interview question asked at Netflix. A fictional insurance company has developed an algorithm to determine the chances of an insurance claim being fraudulent. Candidates must find the claims in the TOP 5 percentile that are the most likely to be a fraud. As you might’ve noticed, this question revolves around finding percentile values. The easiest way to do so is using the NTILE() ranking window function in SQL. In this case, we are looking for a percentile value, so the argument to NTILE() would be 100. The instructions say that we have to identify the top 5 percentile of fraudulent claims from each state. To do that, our window definition should include the PARTITION BY statement. Partition is a way to specify how to group values within the window. For instance, if you had a spreadsheet of users, you could partition them based on the month they signed up. In this case, we must partition the values in the state column. This means calculating percentiles of each claim from each state. We use the ORDER BY statement to arrange the values in the fraud_score column in descending order. Note that because the ORDER BY and PARTITION BY statements are used within the window definition, they only apply to each ‘group’ of records, each group representing one state. For instance, the records from California are arranged based on the value in their fraud_score column, the rows with highest values coming first. As soon as there are no more rows for California, the order is reset and starts over from the highest scoring record in another state, Florida. 8 Nate Rosidi Finding the Nth Highest Value There is another question, often asked at Amazon to gauge the candidate’s proficiency in Ranking Window Functions. The task is simple: you are given a single table with many different columns. The question asks us to find the second highest salary of all employee records. After analyzing the available data, it becomes obvious that the most important is the salary column. In this case, the wording of the question tells us to find the second highest salary at the company. So if five employees all have a salary of 100 000$ per year, and it is the highest salary, we’ll have to access the sixth employee, who is next in the descending order of salaries. If you look at the current solution on StrataScratch, we use the DENSE_RANK() window function to get the second-­‐highest value. Also, we use the DISTINCT keyword to weed out duplicates, in case multiple employees have the same salary. We want to rank every remaining record individually, so there’s no need to use the PARTITION BY statement to separate groups of employees. 2.4.Aggregate Window Functions in SQL The default behavior of aggregate functions in SQL is to aggregate the data of all records into a few groups. However, when used as window functions, all rows are kept intact. Instead, aggregate window functions create a separate column for storing the results of aggregation. There are five aggregate window functions: AVG() -­‐ returns the average of values in a specific column or subset of data MAX() -­‐ returns the highest value in a specific column or subset of data MIN() -­‐ returns the lowest value in a specific column or subset of data SUM() -­‐ returns the sum of all values in a specific column or subset of data COUNT() -­‐ returns the number of rows in a column or a subset of data Interview questions often revolve around aggregate window functions. For instance, it’s a common task to calculate a running sum and create a new column to display the running sum for every record. For common uses of this SQL window function during interviews, refer to this ultimate guide to SQL aggregate functions. Finding the Latest Date studia korolevae 2022-04-06 SQL Window Functions 9 To better understand aggregate window functions, let’s look at one interview question from Credit Karma. In this question, we have to find and output the most recent balance for every user’s ‘Refinance’ submission. To better understand the question, we must analyze the available data, made up of two tables: loans and submissions. Let’s take a look at the loans table: Next, the submissions table: To answer this question successfully, it’s essential to analyze both tables and the data within them. Then we can use aggregate window functions to solve key pieces of the puzzle: find the most recent submission for each user. To do this, a candidate must understand that the MAX() aggregate function will return the ‘highest’ date, which in SQL is equivalent to the latest date. MAX() window function must be applied to the created_at column in the loans table, where every record represents a single submission. Another key piece of the puzzle is that the rows should be partitioned by user_id value, to make sure we generate the latest date for every unique user, in case they’ve made multiple submissions. The question specifies that we should find the latest submission of the ‘Refinance’ type, so our SQL query should include that condition. 2.5.Value Window Functions in SQL SQL developers can use these functions to take values from other rows in the table. Like the other two types of window functions, there are five distinct functions of this kind. These are exclusively for window functions: LAG() function allows us to access values from previous rows. LEAD() is the opposite of LAG(), and allows us to access values from records that come after the current row. FIRST_VALUE() function returns the first value from the dataset and allows us to set the condition for ordering the data, so the developer can control which value will come first. The LAST_VALUE() function works the same way as the previous function, but it returns the last value instead of the first. NTH_VALUE() function allows developers to specify which value in the order should be returned. 10 Nate Rosidi Time Series Analysis Functions like LAG() and LEAD() allow you to extract values from the rows that follow or precede each row. For this reason, SQL developers often use them to work with time-­‐series data, tracking daily or monthly growth, and other use-­‐cases. Let’s look at a question asked at Amazon interviews that can be solved using the LAG() function. In this question, candidates have a fairly simple task: calculate monthly revenue growth based on the provided data. Ultimately, the output table should include a percentage value that represents month-­‐over-­‐month growth. The LAG() window function allows us to solve this difficult question in just a few lines of code. Let’s take a look at the actual recommended solution: SELECT to_char(created_at::date, 'YYYY-­‐MM') AS year_month, round(((sum(value) -­‐ lag(sum(value), 1) OVER w) / (lag(sum(value), 1) OVER w)) * 100, 2) AS revenue_diff_pct FROM sf_transactions GROUP BY year_month WINDOW w AS ( ORDER BY to_char(created_at::date, 'YYYY-­‐MM')) ORDER BY year_month ASC The date values in the table are ordered from earlier to later. Therefore, all we have to do is calculate the total revenue for every month, and use the LAG() function to access the income value from the previous month and use it, along with current month’s revenue to calculate the monthly difference expressed in percentages. In the solution above, we use the round() function to round the results of our equation. First, we define the window of data, where we arrange the date values and organize them in a specific format. We could do this directly in the window functions, but we will have to use it in multiple places. It’s more efficient to define the window once, and simply reference it as w. First, by subtracting lag(sum(value), 1) from sum(value) we find the numerical difference between each month and its previous month (except for the first, which doesn’t have a previous month). We divide this number by the previous month’s revenue, which we find using the lag() function. Finally, we multiply the result by 100 to get the percentage value, and specify that the value needs to be rounded to two decimal points. studia korolevae 2022-04-06 SQL Window Functions 3. 11 Final Words It shouldn’t be a surprise that many interview questions test the candidate’s knowledge of SQL window functions. Employers know that to perform at the highest level, data scientists must understand this part of SQL very well. If you aspire to the role where you’ll be writing advanced SQL queries, a thorough understanding of SQL window functions can help you find easy solutions to complicated problems. Nate Rosidi is a data scientist and in product strategy. He's also an adjunct professor teaching analytics, and is the founder of StrataScratch, a platform helping data scientists prepare for their interviews with real interview questions from top companies. Connect with him on Twitter: StrataScratch or LinkedIn. More On This Topic • KDnuggets™ News 22:n03, Jan 19: A Deep Look Into 13 Data Scientist… • Top Five SQL Window Functions You Should Know For Data Science Interviews • KDnuggets Top Posts for January 2022: Is Data Science a Dying Career? • Top Posts Jan 31 -­‐ Feb 6: 7 Steps to Mastering Machine Learning with Python… • Top Posts Jan 10-­‐16: Is Data Science a Dying Career? • Top Posts Jan 24-­‐30: The High Paying Side Hustles for Data Scientists Get the FREE collection of 50+ data science cheatsheets and the leading newsletter on AI, Data Science, and Machine Learning, straight to your inbox. By subscribing you accept KDnuggets Privacy Policy <= Previous post Next post => Top Posts Past 30 Days 1. The Complete Collection of Data Science Cheat Sheets – Part 2 2. Decision Tree Algorithm, Explained 3. The Complete Collection of Data Science Cheat Sheets – Part 1 4. Top Programming Languages and Their Uses 12 Nate Rosidi 5. Build a Machine Learning Web App in 5 Minutes 6. Why Are So Many Data Scientists Quitting Their Jobs? 7. Telling a Great Data Story: A Visualization Decision Tree 8. Best Data Science Books for Beginners 9. Calculus: The hidden building block of machine learning 10. Linear vs Logistic Regression: A Succinct Explanation Latest News • Uncertainty Quantification in Artificial Intelligence-­‐b... • SQL Window Functions • Logistic Regression for Classification • DBSCAN Clustering Algorithm in Machine Learning • The Complete Collection Of Data Repositories – Pa... • How to Design Experiments for Data Collection 3.1. Top Posts Last Week 1. Why Are So Many Data Scientists Quitting Their Jobs? 2. Decision Tree Algorithm, Explained 3. Linear vs Logistic Regression: A Succinct Explanation 4. Best Data Science Books for Beginners 5. Top Programming Languages and Their Uses More Recent Posts • How to Design Experiments for Data Collection • What is an MLOps Engineer? • Connect With the Data Science Community at Rev 3 in NYC, the #... • A Bug That Can Make You a Data Science Hero • Machine Learning Pipeline Optimization with TPOT • Loss Functions: An Explainer • Introductory Pandas Tutorial • Accelerating Distributed AI Applications • People Management for AI: Building High-­‐Velocity AI Teams • 8 Free MIT Courses to Learn Data Science Online studia korolevae 2022-04-06 SQL Window Functions • • 13 Related Posts o Top Posts Jan 17-­‐23: The High Paying Side Hustles for Data Scientists o Building a Tractable, Feature Engineering Pipeline for Multivariate Time… o Data Science Tools Illustrated Study Guides o A Simple Way to Time Code in Python o SQL Interview Questions for Experienced Professionals o Crack SQL Interviews Get The Latest News! Get the FREE collection of 50+ data science cheatsheets and the leading newsletter on AI, Data Science, and Machine Learning, straight to your inbox. By subscribing you accept KDnuggets Privacy Policy KDnuggets Home » » SQL Window Functions © 2022 KDnuggets. | About KDnuggets | Contact | Privacy policy | Terms of Service
