["python","http","web-scraping","http-status-code-403"]
How do I avoid HTTP error 403 when web scraping with Python?

When I try this code to scrape a web page: #import requests import urllib.request from bs4 import BeautifulSoup #from urllib import urlopen import re webpage = urllib.request.urlopen('http://www.cmegroup.com/trading/products/#sortField=oi&sortAsc=false&venues=3&page=1&cleared=1&group=1').read findrows = re.compile('<tr class="- banding(?:On|Off)>(.*?)</tr>') findlink = re.compile('<a href =">(.*)</a>') row_array = re.findall(findrows, webpage) links = re.finall(findlink, webpate) print(len(row_array)) iterator = [] I get an error like: File "C:\Python33\lib\urllib\request.py", line 160, in urlopen return opener.open(url, data, timeout) File "C:\Python33\lib\urllib\request.py", line 479, in open response = meth(req, response) File "C:\Python33\lib\urllib\request.py", line 591, in http_response 'http', request, response, code, msg, hdrs) File "C:\Python33\lib\urllib\request.py", line 517, in error return self._call_chain(*args) File "C:\Python33\lib\urllib\request.py", line 451, in _call_chain result = func(*args) File "C:\Python33\lib\urllib\request.py", line 599, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 403: Forbidden Does the website think I'm a bot? How can I fix the problem?
This is probably because of mod_security or some similar server security feature which blocks known spider/bot user agents (urllib uses something like python urllib/3.3.0, it's easily detected). Try setting a known browser user agent with: from urllib.request import Request, urlopen req = Request( url='http://www.cmegroup.com/trading/products/#sortField=oi&sortAsc=false&venues=3&page=1&cleared=1&group=1', headers={'User-Agent': 'Mozilla/5.0'} ) webpage = urlopen(req).read() This works for me. By the way, in your code you are missing the () after .read in the urlopen line, but I think that it's a typo. TIP: since this is exercise, choose a different, non restrictive site. Maybe they are blocking urllib for some reason...