Monday, January 30, 2017

Caching for anonymous (non-authenticated) users in Django

Hi boys and girls,

Recently I was optimizing performance on some of my django sites and needed to cache all views for anonymous users, but still render for authenticated users. Django documentation is silent on this. I've also checked out stackoverflow, but people there recommend using templates, which I don't like. So below I will give you my version of how to solve that.

Saturday, January 21, 2017

Django Haystack - how to limit number of search results

Hello friends,

Recently I needed to limit number of search results in haystack and that was a bit of a challenge. So I decided to share it with you here.

I knew how to limit search results using old style views (haystack.views.SearchView):
from haystack.views import SearchView

class MySearchView(SearchView):

    #limit maximum results
    def get_results(self):
        return self.form.search()[:100]

Friday, January 20, 2017

Fabric how to set environment variable to fix encoding

Hi there,

Today I've encountered with one weird problem and decided to share it with you.

Here is what happens. When I ssh to server manually and run python script - everything works fine. But if I try to run the same script using fabric script, which connects to the same server, then it fails. In particular, it was encoding error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xaa' in position bb: ordinal not in range(128)

Thursday, January 5, 2017

How to scrape https website with proxies

Hi all,

My last post about scraping with proxies is quite old and I decided to write a newer version of it. In particular, today I will emphasize how to scrape https website with proxies.

There are also good news about requests library. Requests has not been supporting socks proxies for quite a long time, but in 2016 there was a new release of it. So now requests fully supports both http and socks proxies.

So let's get started. Below I will show you 4 different examples of how to scrape a single https page. First, we will scrape it with requests using socks and http proxies. Second, we will do the same using urllib3 library.

Monday, December 5, 2016

How to build uwsgi from source and run as a service

As you might have already noticed, I always deploy python applications using uwsgi and nginx. But I didn't write yet how to install uwsgi. Probably there is some version in repository, but I don't really care about it. I know it's old and not what I actually need.

I need to build uwsgi with plugin support, which would allow me to work both with python 3 and 2. Then I will create service config file, so that you can use it as service. Sounds good, isn't it? Let's do it then.

1. Download uwsgi and extract it
cd ~
curl -O http://projects.unbit.it/downloads/uwsgi-2.0.12.tar.gz
tar -xvzf uwsgi-2.0.12.tar.gz

Sunday, November 27, 2016

Why Python is not good for multi-threading?

Recently I was asked this question during screening interview at Yandex (Russian search engine), and they screened fairly well from me. They said: you're cool guy, but try again after a year. You're ok for junior, but not for senior position.

Long time ago I read on some blog that multi-threading is not good idea for Python. That's the only thing came to my mind at the interview. So I only answered that's not good idea, because it will require a lot of memory. Quite silly answer.

Then the interviewer said that it's somehow related to GIL. What's GIL??? It sounded like some kind of familiar and intelligent word to me.

After that, I googled this blog which explained me why Python is not good for multi-threading. Shortly speaking, all problems come from that GIL - Global Interpreter Lock. As result Python can only execute one thread at a time. If you'd like to start many threads, all of them will be competing for a single lock (GIL). Just remember that. You can't execute multiple threads simultaneously in Python. That's one of Python disadvantages and one of popular question at interviews.

Friday, November 25, 2016

How to deploy Trac on Debian Jessie with uwsgi and nginx

Dear all,

Today I'm going to deploy one popular bug tracking tool, which is known as Trac. At first look you may ask: why is it looking so weird? Was it designed in early 90's? Well, yes, interface is very minimalistic. But on the other side it has own benefits - it's absolutely free, written in Python and open source.

Below we will deploy Trac on Debian Jessie with Postgresql, uwsgi and nginx.

As everywhere else, I'm going to stick to my scheme, which I find convenient. If you have any suggestions what can be done better, feel free to tell me in comments or by email.

Also I'm assuming that you know how to install uwsgi and nginx packages. I will only show their trac configuration files. But if you need more info, read how to build uwsgi from source.