Updates to proxyElite: pE-API and fresher proxies

Over the last few months, I’ve been working to refine the beast that is proxyElite to provide more functionality and better results.  One of the key ingredients for that was the pE-API, an HTTP API to our proxy databases.   This API allows for easy use of our proxy lists, including getting current proxies and querying IPs to determine if they are in our lists.  These features have varying levels, some of which require an API key and will be released and documented later on.

Another feature of the new API is to allow us to distribute the proxy scanning workload across multiple hosts.  Currently, this only involves scanning for new IP addresses, but other pieces of proxyElite will be added.  We’ve seen great results replacing our old “acquire new proxies” code with the pE-API driven proxy scanner and have added nearly 200 new proxies in the last few weeks.  These new additions are now keeping our lists well over 20 proxies, and a re-vamp of the pE-core has allowed for faster, more thorough proxy checks via the basicChecker.

If you’re keen on playing with the new features, we’ve got a bit of documentation and sample code (in python) available here.  The sample code includes basic examples of parts of the API that require no key as well as a very alpha version of the autotumbler python library for HTTP proxy tumbling, which uses the pE-API to automatically generate proxy lists.