When a web server interacts with many different browsers at the same time, it needs to identify which browser a specific request came from.

Because the HTTP request/response is stateless, all web browsers look identical. To identify the web browsers, the web server uses cookies.

Technically, cookies are text files with a small piece of data that the web server sends to a web browser. The web browser may store the cookie and send it back to the web server in subsequent requests.

Note that the web browser only sends back cookies that were originally set by the same web server.

By comparing the cookies, the web server can identify that the requests come from the same web browser.

Cookies have expiration dates. Some may last for years while others are short-term and expired as soon as you close the web browsers.

Django Cookies

Django allows you to set, read, and delete cookies via methods of the HttpResponse object.

Setting a cookie

To set a cookie, you use the set_cookie() method of the HttpResponse object:

set_cookie(key, value='', max_age=None, expires=None, path='/', domain=None, secure=False, httponly=False, samesite=NoneCode language: Python (python)

In this method:

• key is the cookie name.
• value is a string that represents the cookie’s value.
• max_age can be a timedelta object or an integer that denotes the number of seconds that specifies how long the cookie should expire. It defaults to None that expires cookie once you close the browser.
• expires should be either a datetime object in UTC or a string in the format "Wdy, DD-Mon-YY HH:MM:SS GMT".
• Use secure=True when you want the web browser to send the cookie to the server if the request is HTTPS only.
• Use httponly=True if you don’t want the client-side JavaScript to access the cookie.
• Use samesite='None' (string) to allow the cookie to be sent with all same-site and cross-site requests.

Deleting a cookie

To delete a cookie, you use the delete_cookie() method of the HttpResponse object:

delete_cookie(key, path='/', domain=None, samesite=None)Code language: Python (python)

The delete_cookie() method deletes a cookie with a specified name. It fails silently if the key doesn’t exist.

Note that the path and domain must have the same values as you used in the set_cookie() method or the cookie will not be deleted.

Reading cookies

To access all cookies sent by the web browser, you use the COOKIES property of the HttpRequest object.

request.COOKIESCode language: Python (python)

To access a cookie by a key, you pass the cookie name to the request.COOKIES dictionary. For example:

request.COOKIES['cocoa']Code language: Python (python)

If the cookie 'cocoa' doesn’t exist, Django will throw an error.

To avoid the error, you can use the get() method of the dictionary to get a cookie if it exists or get a default value otherwise. For example:

request.COOKIES.get('cocoa',1)Code language: Python (python)

The code will return 1 if the cookie with the name 'cocao' doesn’t exist.

Django cookies example

We’ll use a cookie to store whether the web browser has visited the site. When the visitor visits the site for the first time, it’ll show a message:

Welcome to my website!Code language: plaintext (plaintext)

And from the second time, it’ll check the cookie and show the following message if the cookie with the name visited is available:

Welcome back!Code language: plaintext (plaintext)

First, define a new entry in the urlpatterns of the urls.py file of your app:

urlpatterns = [
    path('', views.home, name='home'),
]Code language: Python (python)

When you open the http://127.0.0.1:8000/, Django will execute the home() function in the views.py file.

Second, define the home() function in the views.py file:

def home(request):
    visited = request.COOKIES.get('visited')
    if visited:
        response = HttpResponse('Welcome back!')
    else:
        response = HttpResponse('Welcome to my website!')
        response.set_cookie('visited', True)

    return responseCode language: Python (python)

In the home() function, we read the cookie with the name visited. If the cookie with the name visited does not exist, the homepage will display the message:

Welcome to my website!Code language: plaintext (plaintext)

Otherwise, it’ll show the message:

Welcome back!Code language: plaintext (plaintext)

Also, we set the visited cookie to True.

If you view the cookie in the web browser, you’ll see the cookie with the name visited like this:

Summary

• A cookie is a piece of data that the web server sends to the web browser and the web browser may store it or not.
• The web browser sends the cookie back to the web server in the subsequent requests in the header of the HTTP request.
• Use the set_cookie() function of the HttpResponse object to set a cookie in Django.
• Use the delete_cookie() method of the HttpResponse object to delete a cookie.
• Use the request.COOKIES dictionary to read all cookies sent by the web browser.

The SQL GROUP BY clause groups the rows returned by a query into groups. Typically, you use aggregate functions like count, min, max, avg, and sum with the GROUP BY clause to return an aggregated value for each group.

Here’s the basic usage of the GROUP BY clause in a SELECT statement:

SELECT column_1, AGGREGATE(column_2)
FROM table_name
GROUP BY column1;Code language: SQL (Structured Query Language) (sql)

In Django, you can you the annotate() method with the values() to apply the aggregation on groups like this:

(Entity.objects
    .values('column_2')
    .annotate(value=AGGREGATE('column_1'))
)Code language: Python (python)

In this syntax;

• values('column_2') – pass the column that you want to group to the values() method.
• annotate(value=AGGREGATE('column_1')) – specify what to aggregate in the annotate() method.

Notice that the order of calling values() and annotates() matter. If you do not call the values() method first and annotate() second, the expression won’t produce aggregate results.

Django Group By examples

We’ll use the Employee and Department models from the HR application for the demonstration. The Emloyee and Department models map to the hr_employee and hr_department tables in the database:

1) Django Group By with Count example

The following example uses the values() and annotate() method to get the number of employees by department:

>>> (Employee.objects
...     .values('department')
...     .annotate(head_count=Count('department'))
...     .order_by('department')
...  )
SELECT "hr_employee"."department_id",
       COUNT("hr_employee"."department_id") AS "head_count"
  FROM "hr_employee"
 GROUP BY "hr_employee"."department_id"
 ORDER BY "hr_employee"."department_id" ASC
 LIMIT 21
Execution time: 0.001492s [Database: default]
<QuerySet [{'department': 1, 'head_count': 30}, {'department': 2, 'head_count': 40}, {'department': 3, 'head_count': 28}, {'department': 4, 'head_count': 29}, {'department': 5, 'head_count': 29}, {'department': 6, 'head_count': 30}, {'department': 7, 'head_count': 34}]>Code language: Python (python)

How it works.

First, group the employees by department using the values() method:

values('department')Code language: Python (python)

Second, apply the Count() to each group:

annotate(head_count=Count('department'))Code language: Python (python)

Third, sort the objects in the QuerySet by department:

order_by('department')Code language: Python (python)

Behind the scenes, Django executes the SELECT statement with the GROUP BY clause:

SELECT "hr_employee"."department_id",
       COUNT("hr_employee"."department_id") AS "head_count"
  FROM "hr_employee"
 GROUP BY "hr_employee"."department_id"
 ORDER BY "hr_employee"."department_id" ASC
 LIMIT 21
Code language: SQL (Structured Query Language) (sql)

2) Django Group By with Sum example

Similarly, you can use the Sum() aggregate to calculate the total salary of employees in each department:

>>> (Employee.objects
...     .values('department')
...     .annotate(total_salary=Sum('salary'))
...     .order_by('department')
...  )
SELECT "hr_employee"."department_id",
       SUM("hr_employee"."salary") AS "total_salary"
  FROM "hr_employee"
 GROUP BY "hr_employee"."department_id"
 ORDER BY "hr_employee"."department_id" ASC
 LIMIT 21
Execution time: 0.000927s [Database: default]
<QuerySet [{'department': 1, 'total_salary': Decimal('3615341.00')}, {'department': 2, 'total_salary': Decimal('5141611.00')}, {'department': 3, 'total_salary': Decimal('3728988.00')}, {'department': 4, 'total_salary': Decimal('3955669.00')}, {'department': 5, 'total_salary': Decimal('4385784.00')}, {'department': 6, 'total_salary': Decimal('4735927.00')}, {'department': 7, 'total_salary': Decimal('4598788.00')}]>
Code language: Python (python)

3) Django Group By with Min, Max, and Avg example

The following example applies multiple aggregate functions to groups to get the lowest, average, and highest salary of employees in each department:

>>> (Employee.objects
...     .values('department')
...     .annotate(
...         min_salary=Min('salary'),
...         max_salary=Max('salary'),
...         avg_salary=Avg('salary')
...     )
...     .order_by('department')
...  )
SELECT "hr_employee"."department_id",
       MIN("hr_employee"."salary") AS "min_salary",
       MAX("hr_employee"."salary") AS "max_salary",
       AVG("hr_employee"."salary") AS "avg_salary"
  FROM "hr_employee"
 GROUP BY "hr_employee"."department_id"
 ORDER BY "hr_employee"."department_id" ASC
 LIMIT 21
Execution time: 0.001670s [Database: default]
<QuerySet [{'department': 1, 'min_salary': Decimal('45427.00'), 'max_salary': Decimal('149830.00'), 'avg_salary': Decimal('120511.366666666667')}, {'department': 
2, 'min_salary': Decimal('46637.00'), 'max_salary': Decimal('243462.00'), 'avg_salary': Decimal('128540.275000000000')}, {'department': 3, 'min_salary': Decimal('40762.00'), 'max_salary': Decimal('248265.00'), 'avg_salary': Decimal('133178.142857142857')}, {'department': 4, 'min_salary': Decimal('43000.00'), 'max_salary': 
Decimal('238016.00'), 'avg_salary': Decimal('136402.379310344828')}, {'department': 5, 'min_salary': Decimal('42080.00'), 'max_salary': Decimal('246403.00'), 'avg_salary': Decimal('151233.931034482759')}, {'department': 6, 'min_salary': Decimal('58356.00'), 'max_salary': Decimal('248312.00'), 'avg_salary': Decimal('157864.233333333333')}, {'department': 7, 'min_salary': Decimal('40543.00'), 'max_salary': Decimal('238892.00'), 'avg_salary': Decimal('135258.470588235294')}]>Code language: Python (python)

4) Django group by with join example

The following example uses the values() and annotate() methods to get the number of employees per department:

>>> (Department.objects
...     .values('name')
...     .annotate(
...         head_count=Count('employee')
...     )
...  )
SELECT "hr_department"."name",
       COUNT("hr_employee"."id") AS "head_count"
  FROM "hr_department"
  LEFT OUTER JOIN "hr_employee"
    ON ("hr_department"."id" = "hr_employee"."department_id")
 GROUP BY "hr_department"."name"
 LIMIT 21
Execution time: 0.001953s [Database: default]
<QuerySet [{'name': 'Marketing', 'head_count': 28}, {'name': 'Finance', 'head_count': 29}, {'name': 'SCM', 'head_count': 29}, {'name': 'GA', 'head_count': 30}, {'name': 'Sales', 'head_count': 40}, {'name': 'IT', 'head_count': 30}, {'name': 'HR', 'head_count': 34}]>Code language: Python (python)

How it works.

• values('name') – groups department by name.
• annotate(headcount=Count('employee')) – counts employees in each department.

Behind the scenes, Django uses a LEFT JOIN to join the hr_department table with hr_employee table and apply the COUNT() function to each group.

Django group by with having

To apply a condition to the groups, you use the filter() method. For example, the following uses the filter() method to get the department with head counts are more than 30:

>>> (Department.objects
...     .values('name')
...     .annotate(
...         head_count=Count('employee')
...     )
...     .filter(head_count__gt=30)
...  )
SELECT "hr_department"."name",
       COUNT("hr_employee"."id") AS "head_count"
  FROM "hr_department"
  LEFT OUTER JOIN "hr_employee"
    ON ("hr_department"."id" = "hr_employee"."department_id")
 GROUP BY "hr_department"."name"
HAVING COUNT("hr_employee"."id") > 30
 LIMIT 21
Execution time: 0.002893s [Database: default]
<QuerySet [{'name': 'Sales', 'head_count': 40}, {'name': 'HR', 'head_count': 34}]>
Code language: Python (python)

Behind the scenes, Django uses the HAVING clause to filter the group based on the condition that we pass to the filter() method:

SELECT "hr_department"."name",
       COUNT("hr_employee"."id") AS "head_count"
  FROM "hr_department"
  LEFT OUTER JOIN "hr_employee"
    ON ("hr_department"."id" = "hr_employee"."department_id")
 GROUP BY "hr_department"."name"
HAVING COUNT("hr_employee"."id") > 30
Code language: Python (python)

Summary

• Use values() and annotate() method to group rows into groups.
• Use filter() to add conditions to filter groups.

We’ll use the Employee and Department models from the hr application for the demonstration. The Employee and Department models map to the hr_employee and hr_department tables:

First, add the salary field to the Employee model:

class Employee(models.Model):

    salary = models.DecimalField(max_digits=15, decimal_places=2)
    # ...Code language: Python (python)

Second, make migrations using the makemigrations command:

 python manage.py makemigrationsCode language: Python (python)

Output:

Migrations for 'hr':
  hr\migrations\0005_employee_salary.py
    - Add field salary to employeeCode language: Python (python)

Third, propagate the changes to the database by running the migrate command:

python manage.py migrateCode language: Python (python)

Output:

Operations to perform:
  Apply all migrations: admin, auth, contenttypes, hr, sessions
Running migrations:
  Applying hr.0005_employee_salary... OKCode language: Python (python)

Finally, populate values into the salary column with the data from data.json fixture:

python manage.py loaddata data.jsonCode language: Python (python)

Introduction to the Django aggregate

An aggregate function accepts a list of values and returns a single value. The commonly used aggregate functions are count, max, min, avg, and sum.

Count

The QuerySet object provides you with the count() method that returns the number of objects it contains. For example, you can use the count() method to get the number of employees:

>>> Employee.objects.count()
SELECT COUNT(*) AS "__count"
  FROM "hr_employee"        
Execution time: 0.002160s [Database: default]
220Code language: Python (python)

The count() method uses the SQL COUNT(*) function to return the number of rows in the hr_employee table.

To get the number of employees whose first names start with the letter J, you can use both filter() and count() methods of the QuerySet object like this:

>>> Employee.objects.filter(first_name__startswith='J').count()
SELECT COUNT(*) AS "__count"
  FROM "hr_employee"
 WHERE "hr_employee"."first_name"::text LIKE 'J%'
Execution time: 0.000000s [Database: default]
29Code language: Python (python)

In this case, the filter() method forms a WHERE clause while the count() method forms the COUNT() function.

Max

The Max() returns the maximum value in a set of values. It accepts a column that you want to get the highest value.

For example, the following uses the Max() to return the highest salary:

>>> Employee.objects.aggregate(Max('salary'))
SELECT MAX("hr_employee"."salary") AS "salary__max"
  FROM "hr_employee"
Execution time: 0.002001s [Database: default]
{'salary__max': Decimal('248312.00')}Code language: Python (python)

The Max() executes the SQL MAX() on the salary column of the hr_employee table and returns the highest salary.

Min

The Min() returns the minimum value in a set of values. Like the Max(), it accepts a column that you want to get the lowest value.

The following example uses the Min() to return the lowest salary of employees:

>>> Employee.objects.aggregate(Min('salary')) 
SELECT MIN("hr_employee"."salary") AS "salary__min"
  FROM "hr_employee"
Execution time: 0.002015s [Database: default]
{'salary__min': Decimal('40543.00')}Code language: Python (python)

The Min() function executes the SQL MIN() function that returns the minimum value in the salary column.

Avg

The Avg() returns the average value in a set of values. It accepts a column name and returns the average value of all the values in that column:

>>> Employee.objects.aggregate(Avg('salary')) 
SELECT AVG("hr_employee"."salary") AS "salary__avg"
  FROM "hr_employee"
Execution time: 0.005468s [Database: default]
{'salary__avg': Decimal('137100.490909090909')}Code language: Python (python)

Behind the scenes, the Avg() executes the SQL AVG() function on the salary column of the hr_employee and returns the average salary.

Sum

The Sum() returns the sum of values. For example, you can use the Sum() to calculate the total salary of the company:

>>> Employee.objects.aggregate(Sum('salary')) 
SELECT SUM("hr_employee"."salary") AS "salary__sum"
  FROM "hr_employee"
Execution time: 0.000140s [Database: default]
{'salary__sum': Decimal('30162108.00')}Code language: Python (python)

The Sum() executes the SQL SUM() function and returns the total value of all the values in the salary column of the hr_employee table.

Summary

• Use the count() method to get the number of objects of a QuerySet.
• Use the Max() to get the maximum value in a set of values.
• Use the Min() to get the minimum value in a set of values.
• Use the Avg() to get the average value in a set of values.
• Use the Sum() to get the total value of a set.

Sometimes, you want to check if a query contains any rows. To do it, you use the exists() method of the QuerySet object.

The exists() method returns True if the QuerySet contains any rows or False otherwise.

Behind the scenes, the exists() will attempt to perform the query in the fastest way possible. However, the query will be nearly identical to a regular QuerySet query.

Suppose you have a QuerySet and want to check if it has any objects. Instead of doing this:

if query_set:
   print('the queryset has at least one object')Code language: Python (python)

…you should use the exists() because it is a little bit faster:

if query_set.exists():
   print('the queryset has at least one object')Code language: Python (python)

Django 4.1 added the aexists() which is an asynchronous version of exists().

Django exists() method example

We’ll use the Employee model for the demonstration. The Employee model maps to the hr_employee table in the database:

First, run the shell_plus command:

python manage.py shell_plusCode language: plaintext (plaintext)

Second, find the employees whose first names start with the letter J:

>>> Employee.objects.filter(first_name__startswith='J').exists()
SELECT 1 AS "a"
  FROM "hr_employee"
 WHERE "hr_employee"."first_name"::text LIKE 'J%'
 LIMIT 1
Execution time: 0.000000s [Database: default]
TrueCode language: SQL (Structured Query Language) (sql)

Note that Django generated the SQL based on the PostgreSQL. If you use other databases, you may see a slightly different SQL statement.

In this example, the exists() method returns True. It selects only the first row to determine whether the QuerySet contains any row.

If you do not use the exists() method, the QuerySet will get all the rows from the hr_employee table:

>>> qs = Employee.objects.filter(first_name__startswith='J') 
>>> print(qs.query)
SELECT "hr_employee"."id",
       "hr_employee"."first_name",
       "hr_employee"."last_name",
       "hr_employee"."contact_id",
       "hr_employee"."department_id"
FROM "hr_employee"
WHERE "hr_employee"."first_name"::text LIKE J%
ORDER BY "hr_employee"."first_name" ASC,
         "hr_employee"."last_name" ASCCode language: SQL (Structured Query Language) (sql)

Summary

• Use the Django exists() method to check if a QuerySet contains any rows.

We’ll use the Employee and Contact models from the HR application for the demonstration. The Emloyee and Contact models map to the hr_employee and hr_contact tables:

In relational databases, NULL denotes missing information. For example, if you don’t know the contact of an employee, you can use NULL for the contact of the employee at the time of recording.

NULL is special because you cannot use the = operator to compare a value with it. Instead, you use the IS operator.

For example, to check if the value in the contact_id is NULL or not, you use the IS operator as follows:

contact_id IS NULLCode language: Python (python)

The IS NULL returns true if the contact_id is NULL or false otherwise.

To negate the IS operator, you can use the NOT operator like this:

contact_id IS NOT NULLCode language: Python (python)

Django uses isnull to check if a value is NUL or not:

Entity.objects.filter(field_name__isnull=True)Code language: Python (python)

The filter() method returns all instances with the field_name is NULL. To negate the isnull, you compare it with False instead:

Entity.objects.filter(field_name__isnull=False)Code language: Python (python)

In this case, the filter() returns all instances of the Entity whose field_name is not NULL.

Django isnull example

The following example uses the isnull to get employees who do not have contacts:

>>> Employee.objects.filter(contact_id__isnull=True)
SELECT "hr_employee"."id",
       "hr_employee"."first_name",
       "hr_employee"."last_name",
       "hr_employee"."contact_id",
       "hr_employee"."department_id"
  FROM "hr_employee"
 WHERE "hr_employee"."contact_id" IS NULLCode language: Python (python)

The generated query use IS NULL operator to compare the contact_id with NULL.

The following example uses the isnull to get all employees who have contacts:

>>> Employee.objects.filter(contact_id__isnull=False) 
SELECT "hr_employee"."id",
       "hr_employee"."first_name",
       "hr_employee"."last_name",
       "hr_employee"."contact_id",
       "hr_employee"."department_id"
  FROM "hr_employee"
 WHERE "hr_employee"."contact_id" IS NOT NULLCode language: Python (python)

In this case, the generated query uses the IS NOT NULL to compare the values in the contact_id column with NULL.

Summary

• Use Django isnull to check if a value is NULL or not.

In SQL, you use the BETWEEN operator to check if a value is between two values:

field_name BETWEEN low_value AND high_valueCode language: Python (python)

It’s equivalent to the following:

field_name >= low_value AND field_name <= high_valueCode language: Python (python)

The BETWEEN operator returns true if the field_name is between low_value and high_value. Otherwise, it returns False.

Using Django range with numbers

Django’s equivalent of the BETWEEN operator is range:

Entity.objects.filter(field_name__range=(low_value,high_value))Code language: Python (python)

For example, you can find employees whose id is between 1 and 5 using the range like this:

>>> Employee.objects.filter(id__range=(1,5))Code language: Python (python)

Behind the scenes, Django executes the following query:

SELECT "hr_employee"."id",
       "hr_employee"."first_name",
       "hr_employee"."last_name",
       "hr_employee"."contact_id",
       "hr_employee"."department_id"
  FROM "hr_employee"
 WHERE "hr_employee"."id" BETWEEN 1 AND 5Code language: SQL (Structured Query Language) (sql)

Using Django range with dates

Besides numbers and strings, the range also works with dates. For example, the following return all job assignments starting from January 1, 2020, to March 31, 2020:

>>> Assignment.objects.filter(begin_date__range=(start_date,end_date))      
SELECT "hr_assignment"."id",
       "hr_assignment"."employee_id",
       "hr_assignment"."job_id",
       "hr_assignment"."begin_date",
       "hr_assignment"."end_date"
  FROM "hr_assignment"
 WHERE "hr_assignment"."begin_date" BETWEEN '2020-01-01'::date AND '2020-03-31'::dateCode language: SQL (Structured Query Language) (sql)

NOT BETWEEN

The NOT operator negates the BETWEEN operator:

field_name NOT BETWEEN (low_value, high_value)

In other words, the NOT BETWEEN returns true if a value is not in a range of values. It is equivalent to the following:

field_name < low_value OR field_value > high_valueCode language: Python (python)

In Django, you can use the Q object with the range to check if a value is not in a range:

Entity.objects.filter(~Q(field_name__range=(low_value,high_value)))Code language: Python (python)

For example, you can find employees with the id are not in the range (1,5):

>>> Employee.objects.filter(~Q(id__range=(1,5)))
SELECT "hr_employee"."id",
       "hr_employee"."first_name",
       "hr_employee"."last_name",
       "hr_employee"."contact_id",
       "hr_employee"."department_id"
  FROM "hr_employee"
 WHERE NOT ("hr_employee"."id" BETWEEN 1 AND 5)Code language: SQL (Structured Query Language) (sql)

Summary

• Use the Django range to check if a value is in a range of values.

We’ll use the Employee model in the HR application for the demonstration. The Employee model maps to the hr_employee table in the database:

The SQL IN operator returns true if a value is in a set of values:

field_name IN (v1, v2, ...)Code language: Python (python)

For example, you can use the IN operator to query the rows from the hr_employee table whose department_id is in a list like this:

SELECT *
FROM hr_employee
WHERE department_id IN (1,2,3)Code language: SQL (Structured Query Language) (sql)

In Django, you use the in operator:

>>> Employee.objects.filter(department_id__in=(1,2,3)) 
SELECT "hr_employee"."id",
       "hr_employee"."first_name",
       "hr_employee"."last_name",
       "hr_employee"."contact_id",
       "hr_employee"."department_id"
  FROM "hr_employee"
 WHERE "hr_employee"."department_id" IN (1, 2, 3)Code language: SQL (Structured Query Language) (sql)

Typically, you use a subquery with the in operator rather than a list of literal values. For example, you find all employees in the Sales and Marketing departments as follows:

>>> departments = Department.objects.filter(Q(name='Sales') | Q(name='Marketing')) 
>>> Employee.objects.filter(department__in=departments)
SELECT "hr_employee"."id",
       "hr_employee"."first_name",
       "hr_employee"."last_name",
       "hr_employee"."contact_id",
       "hr_employee"."department_id"
  FROM "hr_employee"
 WHERE "hr_employee"."department_id" IN (
        SELECT U0."id"
          FROM "hr_department" U0
         WHERE (U0."name" = 'Sales' OR U0."name" = 'Marketing')
       )Code language: SQL (Structured Query Language) (sql)

How it works.

First, get the departments with the names Sales or Marketing:

departments = Department.objects.filter(Q(name='Sales') | Q(name='Marketing'))Code language: Python (python)

Second, pass the department QuerySet to the in operator:

Employee.objects.filter(department__in=departments)Code language: Python (python)

Behind the scenes, Django executes a query with the IN operator that matches the department id with a list of department id from a list:

SELECT "hr_employee"."id",
       "hr_employee"."first_name",
       "hr_employee"."last_name",
       "hr_employee"."contact_id",
       "hr_employee"."department_id"
  FROM "hr_employee"
 WHERE "hr_employee"."department_id" IN (
        SELECT U0."id"
          FROM "hr_department" U0
         WHERE (U0."name" = 'Sales' OR U0."name" = 'Marketing')
       )Code language: SQL (Structured Query Language) (sql)

NOT IN

The NOT operator negates the IN operator. The NOT IN operator returns true if a value is not in a list of values:

field_name NOT IN (v1, v2, ...)

To perform NOT IN in Django, you can use the Q object and ~ operator:

~Q(field_name__in=(v1,v2,..))Code language: Protocol Buffers (protobuf)

For example, the following finds employees whose department id is not 1, 2, or 3:

>>> Employee.objects.filter(~Q(department_id__in=(1,2,3))) 
SELECT "hr_employee"."id",
       "hr_employee"."first_name",
       "hr_employee"."last_name",
       "hr_employee"."contact_id",
       "hr_employee"."department_id"
  FROM "hr_employee"
 WHERE NOT ("hr_employee"."department_id" IN (1, 2, 3))Code language: SQL (Structured Query Language) (sql)

Alternatively, you can use the exclude() method instead of the filter() method:

>>> Employee.objects.exclude(department_id__in=(1,2,3))      
SELECT "hr_employee"."id",
       "hr_employee"."first_name",
       "hr_employee"."last_name",
       "hr_employee"."contact_id",
       "hr_employee"."department_id"
  FROM "hr_employee"
 WHERE NOT ("hr_employee"."department_id" IN (1, 2, 3))Code language: JavaScript (javascript)

Summary

• Use the Django in to check if a value is in a list of values.

startswith and istartswith

Sometimes, you want to check if a string starts with a substring. For example, you may want to find employees whose first name starts with Je.

To do that in SQL, you use the LIKE operator like this:

SELECT * 
FROM hr_employee
WHERE first_name LIKE 'Je%';Code language: Python (python)

The % is a wildcard that matches any number of characters. And the 'Je%' matches the strings that start with Je and are followed by zero or more characters.

To query data from Django using the LIKE operator, you use the startswith by appending it to a field name:

field_name__startswithCode language: Python (python)

For example, the following uses the filter() method to find employees whose first names start with Je:

>>> Employee.objects.filter(first_name__startswith='Je') 
SELECT "hr_employee"."id",        
       "hr_employee"."first_name",
       "hr_employee"."last_name", 
       "hr_employee"."contact_id",
       "hr_employee"."department_id"
  FROM "hr_employee"
 WHERE "hr_employee"."first_name"::text LIKE 'Je%'
 LIMIT 21
Execution time: 0.000998s [Database: default]
<QuerySet [<Employee: Jennifer Thompson>, <Employee: Jerry Cunningham>, <Employee: Jesus Reilly>, <Employee: Jessica Lewis>, <Employee: Jeanette Hendrix>, <Employee: Jeffrey Castro>, <Employee: Jessica Jackson>, <Employee: Jennifer Bender>, <Employee: Jennifer Moyer>]>Code language: Python (python)

If you want to find employees whose first names start with Je case-insensitively, you can use the istartswith:

>>> Employee.objects.filter(first_name__istartswith='je') 
SELECT "hr_employee"."id",
       "hr_employee"."first_name",
       "hr_employee"."last_name",
       "hr_employee"."contact_id",
       "hr_employee"."department_id"
  FROM "hr_employee"
 WHERE UPPER("hr_employee"."first_name"::text) LIKE UPPER('je%')
 LIMIT 21
Execution time: 0.001398s [Database: default]
<QuerySet [<Employee: Jennifer Thompson>, <Employee: Jerry Cunningham>, <Employee: Jesus Reilly>, <Employee: Jessica Lewis>, <Employee: Jeanette Hendrix>, <Employee: Jeffrey Castro>, <Employee: Jessica Jackson>, <Employee: Jennifer Bender>, <Employee: Jennifer Moyer>]>Code language: Python (python)

In this case, the __istartswith uses the uppercase version of the value for matching.

endswith and iendswith

The endswith and iendswith return True if a value ends with a substring. The endswith is equivalent to the following LIKE operator:

LIKE '%substring'Code language: Python (python)

For example, the following uses the endswith to find employees whose first names end with er:

>>> Employee.objects.filter(first_name__endswith='er')           
SELECT "hr_employee"."id",
       "hr_employee"."first_name",
       "hr_employee"."last_name",
       "hr_employee"."contact_id",
       "hr_employee"."department_id"
  FROM "hr_employee"
 WHERE "hr_employee"."first_name"::text LIKE '%er'
 LIMIT 21
Execution time: 0.000999s [Database: default]
<QuerySet [<Employee: Jennifer Thompson>, <Employee: Tyler Briggs>, <Employee: Spencer Riggs>, <Employee: Roger Robinson>, <Employee: Hunter Boyd>, <Employee: Amber Brown>, <Employee: Tyler Coleman>, <Employee: Jennifer Bender>, <Employee: Jennifer Moyer>]>Code language: Python (python)

It returns the employees with the first names Jennifer, Tyler, Spencer, Roger, etc.

The iendswith is the case-insensitive version of the endswith. For example:

>>> Employee.objects.filter(first_name__iendswith='ER') 
SELECT "hr_employee"."id",
       "hr_employee"."first_name",
       "hr_employee"."last_name",
       "hr_employee"."contact_id",
       "hr_employee"."department_id"
  FROM "hr_employee"
 WHERE UPPER("hr_employee"."first_name"::text) LIKE UPPER('%ER')
 LIMIT 21
Execution time: 0.000999s [Database: default]
<QuerySet [<Employee: Jennifer Thompson>, <Employee: Tyler Briggs>, <Employee: Spencer Riggs>, <Employee: Roger Robinson>, <Employee: Hunter Boyd>, <Employee: Amber Brown>, <Employee: Tyler Coleman>, <Employee: Jennifer Bender>, <Employee: Jennifer Moyer>]>Code language: Python (python)

contains and icontains

The contains allows you to check if a string contains a substring. It is equivalent to the following LIKE operator:

LIKE '%substring%'Code language: Python (python)

For example, the following finds the employees whose first name contains the substring ff:

>>> Employee.objects.filter(first_name__contains='ff')  
SELECT "hr_employee"."id",
       "hr_employee"."first_name",
       "hr_employee"."last_name",
       "hr_employee"."contact_id",
       "hr_employee"."department_id"
  FROM "hr_employee"
 WHERE "hr_employee"."first_name"::text LIKE '%ff%'
 LIMIT 21
Execution time: 0.001293s [Database: default]
<QuerySet [<Employee: Tiffany Jackson>, <Employee: Tiffany Holt>, <Employee: Jeffrey Castro>]>Code language: Python (python)

The query returns the employees with the first names Tiffany and Jeffrey.

The icontains is the case-insensitive version of the contains. So you can use the icontains to check if a string contains a substring case-insensitively:

>>> Employee.objects.filter(first_name__icontains='ff') 
SELECT "hr_employee"."id",
       "hr_employee"."first_name",
       "hr_employee"."last_name",
       "hr_employee"."contact_id",
       "hr_employee"."department_id"
  FROM "hr_employee"
 WHERE UPPER("hr_employee"."first_name"::text) LIKE UPPER('%ff%')
 LIMIT 21
Execution time: 0.002012s [Database: default]
<QuerySet [<Employee: Tiffany Jackson>, <Employee: Tiffany Holt>, <Employee: Jeffrey Castro>]>Code language: Python (python)

Summary

When defining a model, you can specify the default order of the results returned by a QuerySet by using the ordering option in the model’s Meta class.

We’ll use the Employee model for the demonstration. The Employee model maps to the hr_employee table in the database:

For example, to sort the employee by first and last names alphabetically, you can specify the first_name and last_name fields the ordering option in the Meta class as follows:

class Employee(models.Model):
    # ...

    class Meta:
        ordering = ['first_name', 'last_name']
Code language: Python (python)

To sort the values in descending order, you add the - character in front of the field name. For example, the following uses the ordering option to sort the employees by first name in descending order and last name in ascending order:

class Employee(models.Model):
    # ...

    class Meta:
        ordering = ['-first_name', 'last_name']
Code language: Python (python)

Also, you can use a query expression to make null values appear first or last in the sorted result:

from django.db.models import F

class Employee(models.Model):
    # ...

    class Meta:
        ordering = [F('first_name').asc(nulls_last=True)]
Code language: Python (python)

To override the default sort order, you use the order_by() method of a QuerySet:

order_by(*fields)Code language: Python (python)

The following example uses the order_by() to sort employees by first name in ascending order:

>>> Employee.objects.all().order_by('first_name')             
SELECT "hr_employee"."id",
       "hr_employee"."first_name",
       "hr_employee"."last_name",
       "hr_employee"."contact_id",
       "hr_employee"."department_id"
  FROM "hr_employee"
 ORDER BY "hr_employee"."first_name" ASC
 LIMIT 21
Execution time: 0.001034s [Database: default]
<QuerySet [<Employee: Aaron Pearson>, <Employee: Adam Crane>, <Employee: Adam Stewart>, <Employee: Adrienne Green>, <Employee: Alan Johnson>, <Employee: Alexa West>, <Employee: Alicia Wyatt>, <Employee: Amanda Benson>, <Employee: Amber Brown>, <Employee: Amy Lopez>, <Employee: Amy Lee>, <Employee: Andre Perez>, <Employee: 
Andrea Mcintosh>, <Employee: Andrew Guerra>, <Employee: Andrew Dixon>, <Employee: Ann Chang>, <Employee: Anne Odom>, <Employee: Anthony Welch>, <Employee: Anthony Fuentes>, <Employee: Ashley Brown>, '...(remaining elements truncated)...']>
Code language: Python (python)

Like the ordering option, you can use the order_by() method to sort the employee by the first name in descending order:

>>> Employee.objects.all().order_by('-first_name') 
SELECT "hr_employee"."id",
       "hr_employee"."first_name",
       "hr_employee"."last_name",
       "hr_employee"."contact_id",
       "hr_employee"."department_id"
  FROM "hr_employee"
 ORDER BY "hr_employee"."first_name" DESC
 LIMIT 21
Execution time: 0.000998s [Database: default]
<QuerySet [<Employee: William Wise>, <Employee: Wendy Reilly>, <Employee: Victoria Forbes>, <Employee: Victoria Schneider>, <Employee: Vicki Baker>, <Employee: Veronica Blackburn>, <Employee: Vanessa Allen>, <Employee: Valerie Nguyen>, <Employee: Tyler Coleman>, <Employee: Tyler Briggs>, <Employee: Troy Ashley>, <Employee: Travis Goodwin>, <Employee: Tony Jordan>, <Employee: Todd Evans>, <Employee: Timothy Dillon>, <Employee: Timothy Mckay>, <Employee: Timothy Williams>, <Employee: Timothy Lewis>, <Employee: Tiffany Holt>, <Employee: Tiffany Jackson>, '...(remaining elements truncated)...']>
Code language: Python (python)

The order_by() also allows you to sort the results by multiple fields. For example, the following uses the order_by() to sort employees by first and last name:

>>> Employee.objects.all().order_by('first_name','last_name')
SELECT "hr_employee"."id",
       "hr_employee"."first_name",
       "hr_employee"."last_name",
       "hr_employee"."contact_id",
       "hr_employee"."department_id"
  FROM "hr_employee"
 ORDER BY "hr_employee"."first_name" ASC,
          "hr_employee"."last_name" ASC
 LIMIT 21
Execution time: 0.000998s [Database: default]
<QuerySet [<Employee: Aaron Pearson>, <Employee: Adam Crane>, <Employee: Adam Stewart>, <Employee: Adrienne Green>, <Employee: Alan Johnson>, <Employee: Alexa West>, <Employee: Alicia Wyatt>, <Employee: Amanda Benson>, <Employee: Amber Brown>, <Employee: Amy Lee>, <Employee: Amy Lopez>, <Employee: Andre Perez>, <Employee: 
Andrea Mcintosh>, <Employee: Andrew Dixon>, <Employee: Andrew Guerra>, <Employee: Ann Chang>, <Employee: Anne Odom>, <Employee: Anthony Fuentes>, <Employee: Anthony Welch>, <Employee: Ashley Brown>, '...(remaining elements truncated)...']>
Code language: Python (python)

To order randomly, you can use a question mark ? like this:

>>> Employee.objects.all().order_by('?')                      
SELECT "hr_employee"."id",
       "hr_employee"."first_name",
       "hr_employee"."last_name",
       "hr_employee"."contact_id",
       "hr_employee"."department_id"
  FROM "hr_employee"
 ORDER BY RANDOM() ASC
 LIMIT 21
Execution time: 0.010370s [Database: default]
<QuerySet [<Employee: Daniel Meyer>, <Employee: Todd Evans>, <Employee: Roger Robinson>, <Employee: Dwayne Williams>, <Employee: Michael Murphy>, <Employee: Daniel Friedman>, <Employee: Claudia Aguilar>, <Employee: Craig Hunter>, <Employee: Amanda Benson>, <Employee: Renee Wright>, <Employee: Wendy Reilly>, <Employee: Jamie Jackson>, <Employee: Philip Jones>, <Employee: Kelly Stewart>, <Employee: Barbara Vincent>, <Employee: Drew Gonzalez>, <Employee: Derek Owens>, <Employee: Lauren Mcdonald>, <Employee: Perry Rodriguez>, <Employee: Matthew Hernandez>, '...(remaining elements truncated)...']>
Code language: Python (python)

The order_by(‘?’) maybe slow and expensive, depending on the database that you’re using. The above-generated SQL is from PostgreSQL. Other databases may have different implementations.

Ordering by related models

The order_by() allows you to order by a field in a related model. The syntax of the field is the related model, followed by double underscores and the field name:

Entity.objects.order_by(related_model__field_name)Code language: Python (python)

We’ll use the Employee and Department models for the demonstration. The Employee and Department models map to the hr_employee and hr_department tables:

The following example uses the order_by() to sort employees by their department names:

>>> Employee.objects.all().order_by('department__name')
SELECT "hr_employee"."id",
       "hr_employee"."first_name",
       "hr_employee"."last_name",
       "hr_employee"."contact_id",
       "hr_employee"."department_id"
  FROM "hr_employee"
 INNER JOIN "hr_department"
    ON ("hr_employee"."department_id" = "hr_department"."id")
 ORDER BY "hr_department"."name" ASC
 LIMIT 21
Execution time: 0.037173s [Database: default]
<QuerySet [<Employee: Brandy Morris>, <Employee: Jay Carlson>, <Employee: Jessica Lewis>, <Employee: Amanda Benson>, <Employee: Jacqueline Weaver>, <Employee: Patrick Griffith>, <Employee: Adam Stewart>, <Employee: Tiffany Holt>, <Employee: Amber Brown>, <Employee: Martin Raymond>, <Employee: Kyle Pratt>, <Employee: Cheryl Thomas>, <Employee: Linda Garcia>, <Employee: Jeanette Hendrix>, <Employee: Kimberly Gallagher>, <Employee: Kelly Stewart>, <Employee: Alan Johnson>, <Employee: 
William Wise>, <Employee: Debra Webb>, <Employee: Ryan Garcia>, '...(remaining elements truncated)...']>
Code language: Python (python)

The generated SQL shows that when ordering by a field in a related model, Django uses a join that joins the table of the current model (Employee) with the table of the related model (Department).

Summary

• Use Django order_by() to sort data by one or more fields in ascending or descending order.

Django uses the slicing syntax to limit a QuerySet to a specific number of objects. Behind the scenes, Django executes a SQL SELECT statement with LIMIT and OFFSET clauses.

Suppose you want to access the first 10 rows in a table, you can use the following slicing:

Entity.objects.all()[:10]Code language: Python (python)

Since the table stores the rows in an unspecified order, the “first 10 objects” becomes unpredictable.

Therefore, you should always sort the results before slicing the query. For example, the following gets the first 10 rows ordered by the field_name:

Entity.objects.all().order_by(field_name)[:10]Code language: Python (python)

To get the rows from 10 to 20, you can use the following slicing:

Entity.objects.all().order_by(field_name)[10:20]Code language: Python (python)

In general, the syntax for limiting results is as follows:

Entity.objects.all()[Offset:Offset+Limit]Code language: Python (python)

In this syntax, the Offset is the number of rows you want to skip and the Limit is the number of rows you want to retrieve.

Note that Django doesn’t support negative indexing like:

Entity.objects.all().order_by(field_name)[-10]Code language: Python (python)

Also, when you slice a QuerySet, Django returns a new QuerySet.

To retrieve the first or last row, you should use the first() or last() method because if the QuerySet is empty and you use a slice:

Entity.objects.all().order_by(field_name)[0]Code language: Python (python)

…then you’ll get an IndexError exception.

Django Limit Offset example

We’ll use the Employee model from the HR app for the demonstration. The Employee model maps to the hr_employee table:

The following example gets the first 10 employees ordered by their first names:

>>> Employee.objects.all().order_by('first_name')[:10] 
SELECT "hr_employee"."id",
       "hr_employee"."first_name",
       "hr_employee"."last_name",
       "hr_employee"."contact_id",
       "hr_employee"."department_id"
  FROM "hr_employee"
 ORDER BY "hr_employee"."first_name" ASC
 LIMIT 10
Execution time: 0.000000s [Database: default]
<QuerySet [<Employee: Aaron Pearson>, <Employee: Adam Crane>, <Employee: Adam Stewart>, <Employee: Adrienne Green>, <Employee: Alan Johnson>, <Employee: Alexa West>, <Employee: Alicia Wyatt>, <Employee: Amanda Benson>, <Employee: Amber Brown>, <Employee: Amy Lopez>]>Code language: Python (python)

The following example skips the first 10 rows and gets the next 10 rows from the hr_employee table:

>>> Employee.objects.order_by('first_name')[10:20] 
SELECT "hr_employee"."id",
       "hr_employee"."first_name",
       "hr_employee"."last_name",
       "hr_employee"."contact_id",
       "hr_employee"."department_id"
  FROM "hr_employee"
 ORDER BY "hr_employee"."first_name" ASC
 LIMIT 10
OFFSET 10
Execution time: 0.001001s [Database: default]
<QuerySet [<Employee: Amy Lee>, <Employee: Andre Perez>, <Employee: Andrea Mcintosh>, <Employee: Andrew Dixon>, <Employee: Andrew Guerra>, <Employee: Ann Chang>, 
<Employee: Anne Odom>, <Employee: Anthony Fuentes>, <Employee: Anthony Welch>, <Employee: Ashley Brown>]>Code language: Python (python)

Summary

• Django uses array-slicing syntax to limit the number of objects returned by a QuerySet.