In the field of data science, web crawlers are an important source of external data. However, under the circumstance that the existing anti-crawling mechanism is becoming more and more common and perfect, the simple violent crawler has become more and more unable to meet the actual data acquisition needs.

Specifically, one of the methods widely used in anti-crawling mechanisms is to check the IP address of the access request. If a large number of websites are accessed in a short period of time, the website’s server will trigger character verification or block the IP address. Therefore, in many cases, in order to avoid being blocked and blacklisted by the IP or manually entering the verification code, the crawler will use a proxy server to change the IP address.

For example, the ip address of passerby A’s home network is 1.1.1.1. Now he wants to know the hot topic of the crawler, so he uses the proxy server whose ip address is 2.2.2.2 to proxy him to make an access request. So in Zhihu’s view, this visit was sent from the server with ip address 2.2.2.2. By constantly replacing the proxy server, the probability of being detected as a crawler can be effectively reduced.

Of course, it is worth mentioning that the proxy server itself is largely designed for privacy. Strictly speaking, privacy is guaranteed by means of anonymity. For example, someone wants to visit some web pages, but does not want to be collected by the server of these web pages about the access content and time rules under their own IP address. So he can use proxy server for proxy access.

In the era of highly refined social division of labor, in fact, many proxy servers can be found on the Internet. Look it up on Google, there are many such sites that provide proxy server services. For example: IPHTML