python - Scraping web-page data with urllib with headers and proxy -
i have got web-page data, want proxy. how it?
import urllib def get_main_html(): request = urllib.request.request(url, headers=headers) doc = lh.parse(urllib.request.urlopen(request)) return doc
from documentation
urllib auto-detect proxy settings , use those. through proxyhandler, part of normal handler chain when proxy setting detected. that’s thing, there occasions when may not helpful. 1 way setup our own proxyhandler, no proxies defined. done using similar steps setting basic authentication handle.
check this, https://docs.python.org/3/howto/urllib2.html#proxies
Comments
Post a Comment