Our goal is to use a URL like this: /v1/magazines?page=1&per-page=50 to generate JSON results like this:

{
    "results": [
        {
            "Programming Today",
            "publisher": "Harper Collins"
        },
        {
            "title": "Mountain Biking",
            "publisher": "Outdoors Limited"
        }
    ],
    "pagination": {
        "count": 2,
        "page": 1,
        "per_page": 50,
        "pages": 1
    }
}

Let’s get straight to the code and write a basic example.

@app.route("/magazines")
def magazines():
    # process query parameters
    page = request.args.get("page", 1, type=int)
    per_page = request.args.get("per-page", 100, type=int)

    # query
    magazines = Magazine.query.paginate(page, per_page)

    # combine results with pagination
    results = {
        "results": [{"title": m.title, "publisher": m.publisher} for m in magazines.items],
        "pagination": {
            "count": magazines.total,
            "page": page,
            "per_page": per_page,
            "pages": magazines.pages,
        },
    }
    return jsonify(results)

That’s it! The paginate method on our sqlalchemy query is giving us a lot of flexibility with very little code, returning a limited result set, displaying total records available, while calculating remaining pages based on our per_page setting.

Structuring our API this way allows us to maintain performance by returning small chunks of data at a time, while allowing systems to page through all of the records if needed.

The pagination format in JSON is a simplified version of the White House API standards. While the White House prefers offset and limit for pagination, I think page and per_page is more intuitive.

Here are some ways we can improve this example:

• Validate per_page parameter and set limits so someone cannot enter a high number (for example, above 100)
• Move our Magazine object into its own schema using marshmallow
• Add link headers which will allow some automated tools to page through results using headers

I’ve gone ahead and made these improvements. You can see a full working GitHub gist here.

The post Easy API Pagination in Flask appeared first on PyCare.

One thing that Uncle Bob said is very relevant to hiring freelancers to work on your project. He quoted Kent Beck as saying that software development is a three-step process:

First make it work, then make it right, then make it fast

The problem is that some freelance developers stop at the first step. Many organizations are only capable of grading the first step.

The Problem with Stopping at “Make it Work”

It’s possible to hammer away at the keyboard, refreshing the browser until errors go away. Rather than fix a sloppy mess the developer can say ‘Yes! Good to go.’ and move on. But with the sloppy mess that is left behind, the organization takes on technical debt. This can include:

• Vulnerability to bugs/errors
• Security flaws
• Performance issues
• Inability to add features or grow beyond current capability

Identifying Clean Code

Considering python projects such as Django and Flask, here are some key elements that signal a project has moved towards ‘make it right’:

• Project files are organized in a standard way, such that things are where you expect them to be
• Code is written in a standard such as PEP8, or uses an automatic formatter such as Black
• Tests are in place
• Very minimal duplicated code
• Functions are short
• Classes have high cohesion (share data between the various functions of the class)
• Classes, functions, variables named in a way that makes sense
• Comments where appropriate

Most important of all, clean code looks like it was written by somebody that cares. Programming in general is difficult, so while code can always be optimized, you can tell when someone gave it their best effort and tried to make their code understandable to others. You can also tell when it is sloppy and rushed.

The post The Hidden Dangers of Sloppy Python Code appeared first on PyCare.

The first thing that comes to mind is ‘something is wrong with Heroku!‘. But you check their status page and there are no reported problems.

Trust me I have been there! Here is proof:

Unable to solve the problem, you think ‘it must be gremlins’ and move on. In this post I want to show you why it’s probably not gremlins and it’s probably not Heroku.

The issue is likely that you do not have proper timeouts set within your own code. Let’s discuss the problem, do an experiment to demonstrate the issue, then offer some solutions.

Note: I will cover the standard synchronous mode in this post, and will write about asyncrhonous (gevent) situations in another article.

The Problem

Your web application depends on different services, be it a PostgreSQL database, redis cache, or an external API. At some point one or more of those services may slow down, either due to a temporary issue, or something of your own design such a slow database query.

You may also experience slowdowns due to unexpected load on your web site. This can come from bots that quickly overwhelm your app.

Without proper timeouts, these short slowdowns cascade into longer ones lasting beyond 30 seconds. If Heroku does not receive a response from your app within 30 seconds, it returns an error to the user and logs a H12 timeout error. This does not stop your web app from working on the request.

Demonstration

To demonstrate the problem, I am going to create a Flask app that simulates a very slow API service. The app is served with gunicorn and logs each visit in redis via a counter. The first 75 requests are slowed way down, taking 45 seconds to respond. The rest of the requests are processed in 600 milliseconds. Here is the script:

# Procfile
web: gunicorn app:app

# app.py
import os
import time
from flask import Flask
import redis

app = Flask(__name__)
r = redis.from_url(os.environ.get('REDIS_URL', 'redis://localhost:6379'))

@app.route('/')
def hello_world():
    r.incr('counter', 1)

    if int(r.get('counter')) < 75:
        time.sleep(45)
    else:
        time.sleep(0.6)

    return 'Hello World!'

Let’s run this app with the wonderful K6 load testing tool for 10 minutes.

Focus on the blue line, which is the overall response time. The response time immediately goes to 30 seconds and stays there for 7 minutes! This is despite the slow request portion lasting only 2 minutes. As you can see, there is a residual effect from the slow response, in that the app is trying to process requests long after the incident is over.

To make matters worse, any slowdowns that occur near the end of that 7 minutes will easily extend the time in failure. So an unstable service could make your app unusable for over an hour if the problem continues to reappear.

The Solution

Let’s run the scenario again but set a 10 second timeout in gunicorn, like this:

# Procfile
web: gunicorn app:app --timeout 10

This timeout kills and restarts workers that are silent for however many seconds are set. Here is the same scenario with a 10 second timeout:

See how fast that recovery is? With a 10 second timeout the response time gradually increases, but goes back to normal almost immediately once the problem goes away.

An Alternate Solution – Set a Timeout on the API Request

Now I know someone out there is saying ‘duh Casey, just set a timeout when you call the API!’. Well you are not wrong. If you use requests to handle your API call, you can simply set a timeout on the API request like so:

import requests
r = requests.get(url, timeout=3)

In this scenario our gunicorn timeout may not be needed, as requests will throw an error for any request that takes longer than 3 seconds. Let’s try our scenario again with this timeout in place:

The response time slowly climbs up to 5 seconds, but quickly resolves once the problem goes away.

This approach provides another benefit, in that we can check our error logs and see what caused the problem. Rather than a generic Heroku timeout error, we can see that a specific API service timed out during a particular timeframe. We can even handle the error gracefully and display a message to our users while the API is unavailable:

import requests

try:
    requests.get(url, timeout=3)
except requests.exceptions.Timeout:
    context['message'] = 'This service is not available.'

We can set timeouts on other services as well, such as Elasticsearch:

ELASTICSEARCH={
    'default': {
        'hosts': os.environ.get('ELASTIC_ENDPOINT', 'localhost:9200'),
        'http_auth': os.environ.get('ELASTIC_AUTH', ''),
        'timeout': 5,
    },
}

The goal is that the lower-level service catches the timeout situation first, but the gunicorn timeout is there to catch the broader unexpected situations.

You Need Both

Although you can catch an API timeout with requests, heavy traffic and other situations can still drive your app to need the gunicorn timeout. So it is important to implement both of these strategies.

Low Timeout Limitations

Timeouts are great and all, but sometimes setting a low timeout cuts off a critical capability of your app.

For example, if it takes 15 seconds to generate a page that is then served via cache, setting a timeout of 10 seconds will eliminate your ability to load that page. Or maybe you need to upload files and that takes 20 seconds. So you cannot set a gunicorn timeout lower than 20 seconds.

The bottom line is that your gunicorn timeout is limited to the single function that takes the longest to run.

The solution is to refactor those parts of your app. You can do this by:

• Use of background jobs to process long-running tasks
• Upload files directly to S3
• Improve slow database queries by adding an index
• Caching via background jobs

Summary – Timeouts Make Your App Stable and Resilient

So where should you add timeouts? Everywhere! We need service-level timeouts so we know when our service is having issues. We need gunicorn timeouts for unforeseen circumstances and so that our app recovers quickly after being overloaded.

Remember, lowering timeouts is an iterative process. Optimize your app, then lower your timeouts. With proper timeouts in place your app will handle slowdowns and outages well, recovering very quickly while allowing developers to identify the issues involved.

The post Timeouts on Heroku? It’s Probably You appeared first on PyCare.

For those not familiar, Performance Insights is a tool within Amazon’s Relational Database Service (RDS) that shows load on your database via an easy to use dashboard.

The Problem

I recently implemented a new feature that hides a column of data if all of the rows in that field are blank. This task usually involves small datasets of 1,000 records. But there are some outliers that consist of 400,000 records or more.

The outlier group started to cause issues with the web site, with response times occasionally spiking. When looking at the database statistics, I saw a spike in CPU usage around the same time that response times increased, so I knew something in the database was likely causing the problem.

Digging Deeper – Performance Insights

Performance Insights is enabled on the database so I decided to take a look. I changed the dropdown in the right column to ‘Slice by SQL’, then zoomed in on the timeframe with high load. This shows you the SQL queries that are putting load on the database, with the query causing the highest load at the top of the list:

Clicking on the arrow in the list below the graph shows the entire query:

Analyzing the Query

Now it’s time to analyze the query further using my local copy of the database. We do this using PostgreSQL’s built-in EXPLAIN ANALYZE function that shows us how PostgreSQL builds its query plan, and how long it takes to run. You can do this in a terminal shell, but I prefer to use Jet Brains DataGrip software. Running the query in DataGrip looks like this:

In case you cannot read the results in the image, the top portion is the query and the bottom portion is the query plan and time to execute, which is:

Limit (cost=0.00..2.56 rows=1 width=4) (actual time=11410.178..11410.179 rows=0 loops=1)
  -> Seq Scan on employee_employee (cost=0.00..503178.50 rows=196474 width=4) (actual time=11410.177..11410.177 rows=0 loops=1)
    Filter: ((ual IS NOT NULL) AND (year = 2017) AND (jurisdiction_id = 537))
    Rows Removed by Filter: 14497124
Planning time: 0.435 ms
Execution time: 11410.217 ms

As you can see, this query takes 11 seconds to run, and scans 14 million records in order to return a result!

Improving the Query

We know this query is causing problems. There are several ways to fix it:

1. Determine if the query is necessary
2. Rewrite the code that built the query
3. Add an index to the database to improve query performance

In this instance, I do feel the query is necessary and after reviewing the code, I cannot think of a better way to write it. So I’m going to try option #3, adding an index to the database.

For tips on how on how and when to add an index, I recommend reading the excellent tutorial Use the Index, Luke! When I am stuck thinking about how to specifically implement an index, I usually consult Stack Exchange’s DBA section.

Adding an Index

After reviewing those resources I decided to add an index like this:

CREATE INDEX ON employee_employee (year, jurisdiction_id) WHERE ual IS NOT NULL;

To ensure the index is ready to use, we need to run ANALYZE on the table. This builds statistics that support the query analyzer:

ANALYZE employee_employee;

Let’s run the query once again with EXPLAIN ANALYZE:

Adding the index improved the query run time from 11 seconds down to 18 milliseconds!

Limit (cost=0.29..0.81 rows=1 width=4) (actual time=0.693..0.693 rows=0 loops=1)
  -> Index Only Scan using employee_employee_year_jurisdiction_id_idx on employee_employee (cost=0.29..4.45 rows=8 width=4) (actual time=0.690..0.690 rows=0 loops=1)
    Index Cond: ((year = 2018) AND (jurisdiction_id = 537))
    Heap Fetches: 0
Planning time: 17.381 ms
Execution time: 0.784 ms

Happy with the results, I added the index to the production database. Over the next 24 hours, I saw the the problem was completely resolved.

Moving Forward

So what’s next? Continue optimizing! Go back to the top of the list and look at the next query. Can it be optimized in a similar way?

Keep in mind you do not have to wait for a problem to occur. Routinely optimizing queries is a healthy practice that will decrease load on your database, make your site faster, and ultimately improve the long-term stability of your web application.

The post Improve Django Database Performance with AWS Performance Insights appeared first on PyCare.

Besides avoiding potential risk by upgrading, there are many things you can look forward to in Python 3, such as wider iterator adoption for common APIs, format strings, the ‘nonlocal’ keyword, etc. When you’re ready to migrate your apps to Python 3 it’s important to learn the major differences between Python 2 and 3, ensure strong test coverage, and consider implementing continuous integration if you’ve not already done so.

Background

In 2008 the Python Software Foundation first announced that Python 2 would be marked EOL in 2015. Some effort was taken to get users to upgrade prior to 2015, but this was considered largely unsuccessful. When 2014 came along the Python Software Foundation extended the deadline to 2020, and as of January 1, 2020, Python 2 is officially EOL.

In Python 2, a string of the form ‘abcd’ could be interpreted as a unicode or byte string depending on the context and intention (or lack thereof) of the developer. Multiple interpretations for a single object does not adhere to the Zen of Python‘s guidance which states: “there should be one — and preferably only one — obvious way to do it”. This was the driving force behind the backwards incompatible changes introduced in the development of Python 3. Along with other enhancements to the language, Python 3 treats all string objects as unicode by default.

Risks

Neglecting to upgrade your Python 2 applications to Python 3 may seem like the path of least resistance, but be aware that Python 2 being EOL means that any major bugs or security issues found in Python 2 itself or Python 2 software are not guaranteed to be fixed by volunteers in the community. Some may still assist for a time, but the number of those willing to help is expected to quickly dwindle.

Rewards

Aside from unicode strings by default, once you upgrade your app to Python 3 you can look forward to:

• Print being a function with keyword arguments
• APIs returning views and iterators instead of lists
• Simplified ordering comparisons (i.e. <, <=, >=, >)
• Integer operations such as “1/2” returning floating point numbers
• ‘nonlocal’ statement introduced for mutating variable defined outside local scope
• Format strings
• And much more!

Migration Tips & Tricks

Migrating from Python 2 to 3 can be easier than it sounds, but does require due diligence. Some general guidance for making the transition is:

• Make sure you have good test coverage (coverage.py can help)
• Learn the differences between Python 2 & 3
• Use Futurize (or Modernize) to update your code
• Use Pylint to help make sure you don’t regress on your Python 3 support
• Use caniusepython3 to determine dependencies blocking use of Python 3
• Use continuous integration to make sure you stay compatible with Python 2 & 3 (tox can help test against multiple versions of Python)
• Consider using optional static type checking to make sure your type usage works in both Python 2 & 3 (e.g. use mypy to check your typing under both Python 2 & Python 3)

tl;dr

Now that January 1, 2020 has passed, Python 2 is officially without support and Python 3 is the default. Continued maintenance of Python 2 code comes with significant risks, mainly from the lack of support in the face of bugs and/or security issues, so proceed with caution. Risk avoidance aside, Python 3 features like wider iterator adoption for common APIs, format strings, the ‘nonlocal’ keyword, and many other enhancements are reason enough to make the switch. Of course, when you’re ready to migrate your apps to Python 3, some key tips to try to remember are: learn the major differences between Python 2 and 3, ensure strong test coverage, and implement continuous integration if you haven’t already!

The post Python 2 is End of Life – time to update your app! appeared first on PyCare.

1. Use managed services

I have a strong opinion on this one. If you do not have an in-house development team available to support your web app, you should be using managed services, such as Heroku or AWS Elastic Beanstalk.

Yes, you could save money by running your web service and database on a single raw server. But who is going to patch the operating system? Who is going to ensure backups are in place? Who is going to log in and reboot the server when something goes wrong? The great thing about managed services is when something goes wrong with that portion of the service, someone else is going to get involved quickly and get it working again.

Backups, security, and your deployment process are critical capabilities that must be managed. Why not take advantage of Heroku and AWS engineers and focus on code rather than servers?

2. Leverage caching

Caching is a very effective way to make an app fast and resilient. I recommend a layered caching strategy with:

• Static assets (images, CSS, javascript) cached through a CDN such as Cloudflare
• Expensive database queries and page fragments cached with Redis

Use Whitenoise to generate unique static asset IDs, which can then be cached forever. If you change the file then a new ID will be generated.

Caching should be applied where needed and not broadly, as it can introduce a lot of user issues if overdone.

3. Eliminate unnecessary bots and spiders

The majority of search spiders are leaches that hog resources and drag down a web site’s performance.

Many of them use your site in an abusive way. While a typical user may click through 3 to 4 pages of results in 5 seconds, a bot will quickly go from page 1 to 50 in 7 seconds. Identify bots with a logging tool such as Papertrail, then block those that provide no benefit to you via robots.txt.

# Example robots.txt
User-agent: Baiduspider
User-agent: Yandex
User-agent: Sogou web spider
User-agent: seoscanners.net
Disallow: /

A web application firewall can help with this as well. But I have seen the best results from simple robots.txt updates.

4. Monitor for errors

Utilize Django’s built-in error reporting feature, or even better, integrate Sentry. This will provide detailed insight into every error, which can then be patched to make your app more resilient.

5. Investigate performance issues continuously

Identify the root cause of performance issues and resolve them methodically.

It’s easy to brush off temporary errors or timeouts as a server glitch, hosting issue, or ‘gremlins’. But there is almost always a specific reason something went wrong. It could very likely happen again if you do not fix it.

Left unchecked, these glitches combine to cause major problems or unexpected downtime. But when fixed often, the app gets stronger and stronger.

One effective way to improve stability is by setting low timeouts across your services. For example, if you request an API service and expect the result to always take less than 3 seconds, then set a timeout for 3 seconds. Then, when that API has issues it will not impact your site.

r = requests.get(url, timeout=3)

You should set low timeouts with gunicorn as well, as it will make your app recover faster from errors and reduce the chance of cascading timeouts:

web: gunicorn my.wsgi:application --timeout 10

6. Update dependencies often

With dependencies there is a temptation to apply the old ‘if it ain’t broke, don’t fix it’ mentality. But dependency updates bring security patches, performance, and capability improvements. Letting dependencies go out of date can lead to a stale web app that requires major surgery to bring it current.

I recommend updating 2 to 3 dependencies per month at most, and patching all insecure dependencies within 48 hours.

7. Add tests

Your app will be tested one way or another. Better to test a change prior to it being live, rather than letting users see an error and hearing about it that way.

A solid test suite takes the guess work out of changes.

8. Implement features in small batches

One of my favorite books, The DevOps Handbook, covers this well. It’s much better to push small feature updates and changes than try to push a very large update. When we’re pushing small feature updates and dependency updates often, we can clearly see what causes issues and fix it quickly.

This kind of development is best supported by a quality managed service with seamless updates that can be pushed live in the middle of the day.

9. Write clean code

Readable, PEP 8 compliant code reduces complexity. One of the best ways to make an app more maintainable is to focus on Uncle Bob’s Clean Code methodologies, which include:

• Write small functions
• Name those functions so that their intention is clear
• Test everything

10. Focus on security at every level

You must constantly focus on security to keep your web app available and running well. This means:

• Utilize two-factor authentication for every backend service login
• Set strong passwords
• Properly handle web forms  to protect against Cross-Site Request Forgery
• Sanitize user input to protect against SQL injection attacks
• Update insecure dependencies quickly

The post 10 Best Practices for Keeping a Python Web App Running Flawlessly appeared first on PyCare.

SQL injection attempts occur often. You can recognize them in your logs because they look like this

/employees/?s = name&=%20UNION%20SELECT%20CHAR (45,120,49,45,81,45)

What does it mean? Basically, someone is trying to take over your web application’s database through queries. Thankfully, these attacks are easy to stop if you build some basic defense into your application.

Here is my three-layer defense that I use to stop these types of attacks.

Layer 1: Utilize an ORM

The most important step you can take is to code your database queries and commands so they are not vulnerable to SQL injection. The easiest way to do this is to simply use the Object Relational Mapper (ORM) that is built into popular web frameworks.

Django’s ORM and SQLAlchemy for Flask are great examples. If you use these tools to communicate with the database, and avoid hand-writing SQL queries, your web application will be safe from SQL injection.

Now you must be thinking, perfect! I’m doing that so what else is there to worry about? My response to that is, how are you going to feel when every day you see logs with ‘%select%all’ and ‘%drop%tables’. An attacker is out there just waiting for you to make a mistake. So let’s add another layer of defense.

Layer 2: Sanitize all input

Any input that comes from a user should be treated with extreme suspicion and sanitized.

The easiest way to do this is by using python Django’s clean method for data that is entered through a form:

name = self.cleaned_data['name']

If you are not using Django, or want an extra level of protection, you can write a simple function to sanitize input.

For example, if you have a search form that says ‘Enter a city’, we should think about what normal input looks like. Do city names contain % or * characters? No! Then let’s remove them. We can use this function to remove all special characters in a string:

def sanitize(dirty_string):
    clean_string = re.sub('\W+',' ', dirty_string)
    return clean_string

You should sanitize input that is used for filters or sorting by restricting it to the exact input you require. For example, if you are displaying a list of employees that can be sorted by name, id, or age with a query string like ‘/employees?sort=name’, then return a 404 error if it is not an expected value like so:

sort_params = ['age', 'id', 'name']
sort = request.GET.get('sort')
if sort not in sort_params:
    raise Http404('Invalid sort parameter.)

Layer 3: Use a web application firewall

So we are using an ORM. We are cleaning our queries. Nothing else to worry about, right? Well here is another scenario to consider.

Attackers don’t have time to dilly dally on your web site. They are going to come in, attack like mad looking for vulnerabilities, then leave. You can expect to receive MUCH higher traffic than normal during an attack.

Here’s the other problem. Maybe you are using some kind of caching so that the same query will become cached, reducing load on your web site and database. Well the attacks I have seen send a different query string with every request. So caching is not as effective!

The result is very high load on your database that can easily shut your site down. That’s why you need the third layer of defense, a web application firewall. The one I always recommend to clients is by Cloudflare, and is fairly affordable. It is built to cover the OWASP top threats, and is updated often.

This will detect a suspicious query string and strop it from ever reaching your web site.

In summary, putting these three layers in place will help keep your data safe, and your web application available to your users.

The post A Three-Layer Defense Against SQL Injection Attacks appeared first on PyCare.