["python","web-scraping","beautifulsoup"]
How to scrape a website which requires login using python and beautifulsoup?

If I want to scrape a website that requires login with password first, how can I start scraping it with python using beautifulsoup4 library? Below is what I do for websites that do not require login. from bs4 import BeautifulSoup import urllib2 url = urllib2.urlopen("http://www.python.org") content = url.read() soup = BeautifulSoup(content) How should the code be changed to accommodate login? Assume that the website I want to scrape is a forum that requires login. An example is http://forum.arduino.cc/index.php
You can use mechanize: import mechanize from bs4 import BeautifulSoup import urllib2 import cookielib ## http.cookiejar in python3 cj = cookielib.CookieJar() br = mechanize.Browser() br.set_cookiejar(cj) br.open("https://id.arduino.cc/auth/login/") br.select_form(nr=0) br.form['username'] = 'username' br.form['password'] = 'password.' br.submit() print br.response().read() Or urllib - Login to website using urllib2