python爬虫实践之模拟登录
有些网站设置了权限，只有在登录了之后才能爬取网站的内容，如何模拟登录，目前的方法主要是利用cookie模拟登录。
浏览器访问服务器的过程
在用户访问网页时，不论是通过URL输入域名或IP，还是点击链接，浏览器向WEB服务器发出了一个HTTP请求（Http Request），WEB服务器接收到客户端浏览器的请求之后，响应客户端的请求，发回相应的响应信息（Http Response），浏览器解析引擎，排版引擎分析返回的内容，呈现给用户。WEB应用程序在于服务器交互的过程中，HTTP请求和响应时发送的都是一个消息结构。
当浏览器向服务器发送请求的时候，发出http请求消息报文，服务器返回数据时，发出http响应消息报文，这两种类型的消息都是由一个起始行，消息头，一个指示消息头结束的空行和可选的消息体组成。http请求消息中，起始行包括请求方法,请求的资源， HTTP协议的版本号，消息头包含各种属性，消息体包含数据，GET请求并没有消息主体，因此在消息头后的空白行中没有其他数据。Http响应消息中，起始行包括HTTP协议版本，http状态码和状态，消息头包含各种属性，消息体包含服务器返回的数据内容。
如下图从fiddler抓取的http请求和http响应，GET请求内容为空，故消息头之后的空行和消息体都为空。
服务器发送的响应消息如下，浏览器正常接收到服务器发回的http报文
从上可以看到，cookie在http请求和http响应的头信息中，cookie是消息头的一种很重要的属性。
什么是Cookie？
当用户通过浏览器首次访问一个域名时，访问的WEB服务器会给客户端发送数据，以保持WEB服务器与客户端之间的状态保持，这些数据就是Cookie，它是 Internet 站点创建的 ,为了辨别用户身份而储存在用户本地终端上的数据，Cookie中的信息一般都是经过加密的，Cookie存在缓存中或者硬盘中，在硬盘中的是一些小文本文件,当你访问该网站时，就会读取对应网站的Cookie信息，Cookie有效地提升了我们的上网体验。一般而言，一旦将
Cookie 保存在计算机上，则只有创建该 Cookie 的网站才能读取它。
为什么需要Cookie
Http协议是一个无状态的面向连接的协议，Http协议是基于tcp/ip协议层之上的协议，当客户端与服务器建立连接之后，它们之间的TCP连接一直都是保持的，至于保持的时间是多久，是通过服务器端来设置的，当客户端再一次访问该服务器时，会继续使用上一次建立的连接，但是，由于Http协议是无状态的，WEB服务器并不知道这两个请求是否同一个客户端，这两次请求之间是独立的。 为了解决这个问题， Web程序引入了Cookie机制来维护状态.cookie可以记录用户的登录状态，通常web服务器会在用户登录成功后下发一个签名来标记session的有效性，这样免去了用户多次认证和登录网站。记录用户的访问状态。
Cookie的种类
会话Cookie(Session Cookie)：这个类型的cookie只在会话期间内有效，保存在浏览器的缓存之中，用户访问网站时，会话Cookie被创建，当关闭浏览器的时候，它会被浏览器删除。
持久Cookie(Persistent Cookie): 这个类型的cookie长期在用户会话中生效。当你设置cookie的属性Max-Age为1个月的话，那么在这个月里每个相关URL的http请求中都会带有这个cookie。所以它可以记录很多用户初始化或自定义化的信息，比如什么时候第一次登录及弱登录态等。
Secure cookie：安全cookie是在https访问下的cookie形态，以确保cookie在从客户端传递到Server的过程中始终加密的。
HttpOnly Cookie ：这个类型的cookie只能在http(https)请求上传递，对客户端脚本语言无效，从而有效避免了跨站攻击。
第三方cookie： 第一方cookie是当前访问的域名或子域名下的生成的Cookie。
第三方cookie:第三方cookie是第三方域名创建的Cookie。
Cookie的构成
Cookie是http消息头中的一种属性，包括：Cookie名字（Name）Cookie的值（Value），Cookie的过期时间（Expires / Max-Age），Cookie作用路径（Path），Cookie所在域名（Domain），使用Cookie进行安全连接（Secure）。
前两个参数是Cookie应用的必要条件，另外，还包括Cookie大小（Size，不同浏览器对Cookie个数及大小限制是有差异的）。
python模拟登录
设置一个cookie处理对象，它负责 将cookie添加到http请求中，并能从http响应中得到cookie ， 向网站登录页面发送一个请求Request, 包括登录url，POST请求的数据，Http header 利用urllib2.urlopen发送请求，接收WEB服务器的Response。
首先我们查看登陆页面源码
当我们使用urllib处理url的时候，实际上是通过urllib2.OpenerDirector实例进行工作，他会自己调用资源进行各种操作如通过协议、打开url、处理cookie等。而urlopen方法使用的是默认的opener来处理问题，基本的urlopen()函数不支持验证、cookie或其他的HTTP高级功能。要支持这些功能，必须使用build_opener()函数来创建自己的自定义Opener对象。
cookielib模块定义了自动处理HTTP cookies的类，用来访问那些需要cookie数据的网站，cookielib模块包括CookieJar，FileCookieJar，CookiePolicy，DefaultCookiePolicy，Cookie及FileCookieJar的子类MozillaCookieJar和LWPCookieJar，CookieJar对象可以管理HTTP cookies，将cookie添加到http请求中，并能从http响应中得到cookie，FileCookieJar对象主要是从文件中读取cookie或创建cookie，其中，MozillaCookieJar是为了创建与Mozilla浏览器cookies.txt兼容的FileCookieJar实例，LWPCookieJar是为了创建与libwww-perl的Set-Cookie3文件格式兼容的FileCookieJar实例，用LWPCookieJar保存的cookie文件易于人类阅读。默认的是FileCookieJar没有save函数,而MozillaCookieJar或LWPCookieJar都已经实现了。
所以可以用MozillaCookieJar或LWPCookieJar，去自动实现cookie的save。
#! /usr/bin/env python
#coding:utf-8
import sys
import urllib2
import urllib
import requests
import cookielib
## 这段代码是用于解决中文报错的问题
reload(sys)
sys.setdefaultencoding("utf8")
#####################################################
loginurl = '/PLogin.do'
logindomain = ''
class Login(object):
def __init__(self):
self.name = ''
self.passwprd = ''
self.domain = ''
self.cj = cookielib.LCookieJar()
self.opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(self.cj))
urllib2.install_opener(self.opener)
def setLoginInfo(self,username,password,domain):
'''设置用户登录信息'''
self.name = username
self.pwd = password
self.domain = domain
def login(self):
'''登录网站'''
loginparams = {'domain':self.domain,'email':self.name, 'password':self.pwd}
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.57 Safari/537.36'}
req = urllib2.Request(loginurl, urllib.urlencode(loginparams),headers=headers)
response = urllib2.urlopen(req)
self.operate = self.opener.open(req)
thePage = response.read()
if __name__ == '__main__':
userlogin = Login()
username = 'username'
password = 'password'
domain = logindomain
userlogin.setLoginInfo(username,password,domain)
userlogin.login()
您对本文章有什么意见或着疑问吗？请到您的关注和建议是我们前行的参考和动力&&
您的浏览器不支持嵌入式框架，或者当前配置为不显示嵌入式框架。requests 2.7.0
Python HTTP for Humans.
Requests is an Apache2 Licensed HTTP library, written in Python, for human
Most existing Python modules for sending HTTP requests are extremely
verbose and cumbersome. Python’s builtin urllib2 module provides most of
the HTTP capabilities you should need, but the api is thoroughly broken.
It requires an enormous amount of work (even method overrides) to
perform the simplest of tasks.
Things shouldn’t be this way. Not in Python.
&&& r = requests.get('', auth=('user', 'pass'))
&&& r.status_code
&&& r.headers['content-type']
'application/json'
&&& r.text
Requests allow you to send HTTP/1.1 requests. You can add headers, form data,
multipart files, and parameters with simple Python dictionaries, and access the
response data in the same way. It’s powered by httplib and , but it does all the hard work and crazy
hacks for you.
International Domains and URLs
Keep-Alive & Connection Pooling
Sessions with Cookie Persistence
Browser-style SSL Verification
Basic/Digest Authentication
Elegant Key/Value Cookies
Automatic Decompression
Unicode Response Bodies
Multipart File Uploads
Connection Timeouts
Thread-safety
HTTP(S) proxy support
Installation
To install Requests, simply:
$ pip install requests
Documentation
Documentation is available at .
Contribute
Check for open issues or open a fresh issue to start a discussion around a feature idea or a bug. There is a
tag for issues that should be ideal for people who are not very familiar with the codebase yet.
If you feel uncomfortable or uncertain about an issue or your changes, feel free to email @sigmavirus24 and he will happily help you via email, Skype, remote pairing or whatever you are comfortable with.
on GitHub to start making your changes to the master branch (or branch off of it).
Write a test which shows that the bug was fixed or that the feature works as expected.
Send a pull request and bug the maintainer until it gets merged and published. :) Make sure to add yourself to .
Release History
This is the first release that follows our new release process. For more, see
[our documentation]().
Updated urllib3 to 1.10.4, resolving several bugs involving chunked transfer
encoding and response framing.
Fix regression where compressed data that was sent as chunked data was not
properly decompressed. (#2561)
Remove VendorAlias import machinery introduced in v2.5.2.
Simplify the PreparedRequest.prepare API: We no longer require the user to
pass an empty list to the hooks keyword argument. (c.f. #2552)
Resolve redirects now receives and forwards all of the original arguments to
the adapter. (#2503)
Handle UnicodeDecodeErrors when trying to deal with a unicode URL that
cannot be encoded in ASCII. (#2540)
Populate the parsed path of the URI field when performing Digest
Authentication. (#2426)
Copy a PreparedRequest’s CookieJar more reliably when it is not an instance
of RequestsCookieJar. (#2527)
CVE-: Fix handling of cookies on redirect. Previously a cookie
without a host value set would use the hostname for the redirected URL
exposing requests users to session fixation attacks and potentially cookie
stealing. This was disclosed privately by Matthew Daley of
. This affects all versions of requests from
v2.1.0 to v2.5.3 (inclusive on both ends).
Fix error when requests is an install_requires dependency and python
setup.py test is run. (#2462)
Fix error when urllib3 is unbundled and requests continues to use the
vendored import location.
Include fixes to urllib3’s header handling.
Requests’ handling of unvendored dependencies is now more restrictive.
Features and Improvements
Support bytearrays when passed as parameters in the files argument.
Avoid data duplication when creating a request with str, bytes, or
bytearray input to the files argument.
Revert changes to our vendored certificate bundle. For more context see
(#2455, #2456, and )
Features and Improvements
Add sha256 fingerprint support. ()
Improve the performance of headers. ()
Copy pip’s import machinery. When downstream redistributors remove
requests.packages.urllib3 the import machinery will continue to let those
same symbols work. Example usage in requests’ documentation and 3rd-party
libraries relying on the vendored copies of urllib3 will work without having
to fallback to the system urllib3.
Attempt to quote parts of the URL on redirect if unquoting and then quoting
fails. (#2356)
Fix filename type check for multipart form-data uploads. (#2411)
Properly handle the case where a server issuing digest authentication
challenges provides both auth and auth-int qop-values. (#2408)
Fix a socket leak. ()
Fix multiple Set-Cookie headers properly. ()
Disable the built-in hostname verification. ()
Fix the behaviour of decoding an exhausted stream. ()
Pulled in an updated cacert.pem.
Drop RC4 from the default cipher list. ()
Behavioural Changes
Only catch HTTPErrors in raise_for_status (#2382)
Handle LocationParseError from urllib3 (#2344)
Handle file-like object filenames that are not strings (#2379)
Unbreak HTTPDigestAuth handler. Allow new nonces to be negotiated (#2389)
Improvements
Allow usage of urllib3’s Retry object with HTTPAdapters (#2216)
The iter_lines method on a response now accepts a delimiter with which
to split the content (#2295)
Behavioural Changes
Add deprecation warnings to functions in requests.utils that will be removed
in 3.0 (#2309)
Sessions used by the functional API are always closed (#2326)
Restrict requests to HTTP/1.1 and HTTP/1.0 (stop accepting HTTP/0.9) (#2323)
Only parse the URL once (#2353)
Allow Content-Length header to always be overriden (#2332)
Properly handle files in HTTPDigestAuth (#2333)
Cap redirect_cache size to prevent memory abuse (#2299)
Fix HTTPDigestAuth handling of redirects after authenticating successfully
Fix crash with custom method parameter to Session.request (#2317)
Fix how Link headers are parsed using the regular expression library (#2271)
Documentation
Add more references for interlinking (#2348)
Update CSS for theme (#2290)
Update width of buttons and sidebar (#2289)
Replace references of Gittip with Gratipay (#2282)
Add link to changelog in sidebar (#2273)
Unicode URL improvements for Python 2.
Re-order JSON param for backwards compat.
Automatically defrag authentication schemes from host/pass URIs. ()
Improvements
FINALLY! Add json parameter for uploads! ()
Support for bytestring URLs on Python 3.x ()
Avoid getting stuck in a loop ()
Multiple calls to iter* fail with unhelpful error. (, )
Documentation
Correct redirection introduction ()
Added example of how to send multiple files in one request. ()
Clarify how to pass a custom set of CAs ()
Now has a “security” package extras set, $ pip install requests[security]
Requests will now use Certifi if it is available.
Capture and re-raise urllib3 ProtocolError
Bugfix for responses that attempt to redirect to themselves forever (wtf?).
Behavioral Changes
Connection: keep-alive header is now sent automatically.
Improvements
Support for connect timeouts! Timeout now accepts a tuple (connect, read) which is used to set individual connect and read timeouts.
Allow copying of PreparedRequests without headers/cookies.
Updated bundled urllib3 version.
Refactored settings loading from environment – new Session.merge_environment_settings.
Handle socket errors in iter_content.
API Changes
New Response property is_redirect, which is true when the
library could have processed this response as a redirection (whether
or not it actually did).
The timeout parameter now affects requests with both stream=True and
stream=False equally.
The change in v2.0.0 to mandate explicit proxy schemes has been reverted.
Proxy schemes now default to http://.
The CaseInsensitiveDict used for HTTP headers now behaves like a normal
dictionary when references as string or viewed in the interpreter.
No longer expose Authorization or Proxy-Authorization headers on redirect.
Fix CVE- and CVE- respectively.
Authorization is re-evaluated each redirect.
On redirect, pass url as native strings.
Fall-back to autodetected encoding for JSON when Unicode detection fails.
Headers set to None on the Session are now correctly not sent.
Correctly honor decode_unicode even if it wasn’t used earlier in the same
Stop advertising compress as a supported Content-Encoding.
The Response.history parameter is now always a list.
Many, many urllib3 bugfixes.
Fixes incorrect parsing of proxy credentials that contain a literal or encoded ‘#’ character.
Assorted urllib3 fixes.
API Changes
New exception: ContentDecodingError. Raised instead of urllib3
DecodeError exceptions.
Avoid many many exceptions from the buggy implementation of proxy_bypass on OS X in Python 2.6.
Avoid crashing when attempting to get authentication credentials from ~/.netrc when running as a user without a home directory.
Use the correct pool size for pools of connections to proxies.
Fix iteration of CookieJar objects.
Ensure that cookies are persisted over redirect.
Switch back to using chardet, since it has merged with charade.
Updated CA Bundle, of course.
Cookies set on individual Requests through a Session (e.g. via Session.get()) are no longer persisted to the Session.
Clean up connections when we hit problems during chunked upload, rather than leaking them.
Return connections to the pool when a chunked upload is successful, rather than leaking it.
Match the HTTPbis recommendation for HTTP 301 redirects.
Prevent hanging when using streaming uploads and Digest Auth when a 401 is received.
Values of headers set by Requests are now always the native string type.
Fix previously broken SNI support.
Fix accessing HTTP proxies using proxy authentication.
Unencode HTTP Basic usernames and passwords extracted from URLs.
Support for IP address ranges for no_proxy environment variable
Parse headers correctly when users override the default Host: header.
Avoid munging the URL in case of case-sensitive servers.
Looser URL handling for non-HTTP/HTTPS urls.
Accept unicode methods in Python 2.6 and 2.7.
More resilient cookie handling.
Make Response objects pickleable.
Actually added MD5-sess to Digest Auth instead of pretending to like last time.
Updated internal urllib3.
Fixed @Lukasa’s lack of taste.
Updated included CA Bundle with new mistrusts and automated process for the future
Added MD5-sess to Digest Auth
Accept per-file headers in multipart file POST messages.
Fixed: Don’t send the full URL on CONNECT messages.
Fixed: Correctly lowercase a redirect scheme.
Fixed: Cookies not persisted when set via functional API.
Fixed: Translate urllib3 ProxyError into a requests ProxyError derived from ConnectionError.
Updated internal urllib3 and chardet.
API Changes:
Keys in the Headers dictionary are now native strings on all Python versions,
i.e. bytestrings on Python 2, unicode on Python 3.
Proxy URLs now must have an explicit scheme. A MissingSchema exception
will be raised if they don’t.
Timeouts now apply to read time if Stream=False.
RequestException is now a subclass of IOError, not RuntimeError.
Added new method to PreparedRequest objects: PreparedRequest.copy().
Added new method to Session objects: Session.update_request(). This
method updates a Request object with the data (e.g. cookies) stored on
the Session.
Added new method to Session objects: Session.prepare_request(). This
method updates and prepares a Request object, and returns the
corresponding PreparedRequest object.
Added new method to HTTPAdapter objects: HTTPAdapter.proxy_headers().
This should not be called directly, but improves the subclass interface.
httplib.IncompleteRead exceptions caused by incorrect chunked encoding
will now raise a Requests ChunkedEncodingError instead.
Invalid percent-escape sequences now cause a Requests InvalidURL
exception to be raised.
HTTP 208 no longer uses reason phrase "im_used". Correctly uses
"already_reported".
HTTP 226 reason added ("im_used").
Vastly improved proxy support, including the CONNECT verb. Special thanks to
the many contributors who worked towards this improvement.
Cookies are now properly managed when 401 authentication responses are
Chunked encoding fixes.
Support for mixed case schemes.
Better handling of streaming downloads.
Retrieve environment proxies from more locations.
Minor cookies fixes.
Improved redirect behaviour.
Improved streaming behaviour, particularly for compressed data.
Miscellaneous small Python 3 text encoding bugs.
.netrc no longer overrides explicit auth.
Cookies set by hooks are now correctly persisted on Sessions.
Fix problem with cookies that specify port numbers in their host field.
BytesIO can be used to perform streaming uploads.
More generous parsing of the no_proxy environment variable.
Non-string objects can be passed in data values alongside files.
Simple packaging fix
Simple packaging fix
301 and 302 redirects now change the verb to GET for all verbs, not just
POST, improving browser compatibility.
Python 3.3.2 compatibility
Always percent-encode location headers
Fix connection adapter matching to be most-specific first
new argument to the default connection adapter for passing a block argument
prevent a KeyError when there’s no link headers
Fixed cookies on sessions and on requests
Significantly change how hooks are dispatched - hooks now receive all the
arguments specified by the user when making a request so hooks can make a
secondary request with the same parameters. This is especially necessary for
authentication handler authors
certifi support was removed
Fixed bug where using OAuth 1 with body signature_type sent no data
Major proxy work thanks to @Lukasa including parsing of proxy authentication
from the proxy url
Fix DigestAuth handling too many 401s
Update vendored urllib3 to include SSL bug fixes
Allow keyword arguments to be passed to json.loads() via the
Response.json() method
Don’t send Content-Length header by default on GET or HEAD
Add elapsed attribute to Response objects to time how long a request
Fix RequestsCookieJar
Sessions and Adapters are now picklable, i.e., can be used with the
multiprocessing library
Update charade to version 1.0.3
The change in how hooks are dispatched will likely cause a great deal of
CHUNKED REQUESTS
Support for iterable response bodies
Assume servers persist redirect params
Allow explicit content types to be specified for file data
Make merge_kwargs case-insensitive when looking up keys
Fix file upload encoding bug
Fix cookie behavior
Proxy fix for HTTPAdapter.
Cert verification exception bug.
Proxy fix for HTTPAdapter.
Massive Refactor and Simplification
Switch to Apache 2.0 license
Swappable Connection Adapters
Mountable Connection Adapters
Mutable ProcessedRequest chain
/s/prefetch/stream
Removal of all configuration
Standard library logging
Make Response.json() callable, not property.
Usage of new charade project, which provides python 2 and 3 simultaneous chardet.
Removal of all hooks except ‘response’
Removal of all authentication helpers (OAuth, Kerberos)
This is not a backwards compatible change.
Improved mime-compatible JSON handling
Proxy fixes
Path hack fixes
Case-Insensistive Content-Encoding headers
Support for CJK parameters in form posts
Python 3.3 Compatibility
Simply default accept-encoding
No more iter_content errors if already downloaded.
Fix for OAuth + POSTs
Remove exception eating from dispatch_hook
General bugfixes
Incredible Link header support :)
Support for (key, value) lists everywhere.
Digest Authentication improvements.
Ensure proxy exclusions work properly.
Clearer UnicodeError exceptions.
Automatic casting of URLs to tsrings (fURL and such)
Long awaited fix for hanging connections!
Packaging fix
GSSAPI/Kerberos authentication!
App Engine 2.7 Fixes!
Fix leaking connections (from urllib3 update)
OAuthlib path hack fix
OAuthlib URL parameters fix.
Use simplejson if available.
Do not hide SSLErrors behind Timeouts.
Fixed param handling with urls containing fragments.
Significantly improved information in User Agent.
client certificates are ignored when verify=False
Zero dependencies (once again)!
New: Response.reason
Sign querystring parameters in OAuth 1.0
Client certificates no longer ignored when verify=False
Add openSUSE certificate support
Allow passing a file or file-like object as data.
Allow hooks to return responses that indicate errors.
Fix Response.text and Response.json for body-less responses.
Removal of Requests.async in favor of
Allow disabling of cookie persistiance.
New implimentation of safe_mode
cookies.get now supports default argument
Session cookies not saved when Session.request is called with return_response=False
Env: no_proxy support.
RequestsCookieJar improvements.
Various bug fixes.
New Response.json property.
Ability to add string file uploads.
Fix out-of-range issue with iter_lines.
Fix iter_content default size.
Fix POST redirects containing files.
EXPERIMENTAL OAUTH SUPPORT!
Proper CookieJar-backed cookies interface with awesome dict-like interface.
Speed fix for non-iterated content chunks.
Move pre_request to a more usable place.
New pre_send hook.
Lazily encode data, params, files.
Load system Certificate Bundle if certify isn’t available.
Cleanups, fixes.
Attempt to use the OS’s certificate bundle if certifi isn’t available.
Infinite digest auth redirect fix.
Multi-part file upload improvements.
Fix decoding of invalid %encodings in URLs.
If there is no content in a response don’t throw an error the second time that content is attempted to be read.
Upload data on redirects.
POST redirects now break RFC to do what browsers do: Follow up with a GET.
New strict_mode configuration to disable new redirect behavior.
Private SSL Certificate support
Remove select.poll from Gevent monkeypatching
Remove redundant generator for chunked transfer encoding
Fix: Response.ok raises Timeout Exception in safe_mode
Generate chunked ValueError fix
Proxy configuration by environment variables
Simplification of iter_lines.
New trust_env configuration for disabling system/environment hints.
Suppress cookie errors.
encode_uri = False
Allow ‘=’ in cookies.
Response body with 0 content-length fix.
New async.imap.
Don’t fail on netrc.
Honor netrc.
HEAD requests don’t follow redirects anymore.
raise_for_status() doesn’t raise for 3xx anymore.
Make Session objects picklable.
ValueError for invalid schema URLs.
Vastly improved URL quoting.
Additional allowed cookie key values.
Attempted fix for “Too many open files” Error
Replace unicode errors on first pass, no need for second pass.
Append ‘/’ to bare-domain urls before query insertion.
Exceptions now inherit from RuntimeError.
Binary uploads + auth fix.
PYTHON 3 SUPPORT!
Dropped 2.5 Support. (Backwards Incompatible)
Response.content is now bytes-only. (Backwards Incompatible)
New Response.text is unicode-only.
If no Response.encoding is specified and chardet is available, Respoonse.text will guess an encoding.
Default to ISO-8859-1 (Western) encoding for “text” subtypes.
Removal of decode_unicode. (Backwards Incompatible)
New multiple-hooks system.
New Response.register_hook for registering hooks within the pipeline.
Response.url is now Unicode.
SSL verify=False bugfix (apparent on windows machines).
Asynchronous async.send method.
Support for proper chunk streams with boundaries.
session argument for Session classes.
Print entire hook tracebacks, not just exception instance.
Fix response.iter_lines from pending next line.
Fix but in HTTP-digest auth w/ URI having query strings.
Fix in Event Hooks section.
Urllib3 update.
danger_mode for automatic Response.raise_for_status()
Response.iter_lines refactor
verify ssl is default.
Packaging fix.
SSL CERT VERIFICATION!
Release of Cerifi: Mozilla’s cert list.
New ‘verify’ argument for SSL requests.
Urllib3 update.
iter_lines last-line truncation fix
Force safe_mode for async requests
Handle safe_mode exceptions more consistently
Fix iteration on null responses in safe_mode
Socket timeout fixes.
Proxy Authorization support.
Response.iter_lines!
Prefetch bugfix.
Added license to installed version.
Converted auth system to use simpler callable objects.
New session parameter to API methods.
Display full URL while logging.
New Unicode decoding system, based on over-ridable Response.encoding.
Proper URL slash-quote handling.
Cookies with [, ], and _ allowed.
URL Request path fix
Proxy fix.
Timeouts fix.
Keep-alive support!
Complete removal of Urllib2
Complete removal of Poster
Complete removal of CookieJars
New ConnectionError raising
Safe_mode for error catching
prefetch parameter for request methods
OPTION method
Async pool size throttling
File uploads send real names
Vendored in urllib3
Digest authentication bugfix (attach query data to path)
Response.content = None if there was an invalid repsonse.
Redirection auth handling.
Session Hooks fix.
Digest Auth fix.
PATCH Fix.
Move away from urllib2 authentication handling.
Fully Remove AuthManager, AuthObject, &c.
New tuple-based auth system with handler callbacks.
Sessions are now the primary interface.
Deprecated InvalidMethodException.
PATCH fix.
New config system (no more global settings).
Session parameter bugfix (params merging).
Offline (fast) test suite.
Session dictionary argument merging.
Automatic decoding of unicode, based on HTTP Headers.
New decode_unicode setting.
Removal of r.read/close methods.
New r.faw interface for advanced response usage.*
Automatic expansion of parameterized headers.
Beautiful requests.async module, for making async requests w/ gevent.
GET/HEAD obeys allow_redirects=False.
Enhanced status codes experience \o/
Set a maximum number of redirects (settings.max_redirects)
Full Unicode URL support
Support for protocol-less redirects.
Allow for arbitrary request types.
New callback hook system
New persistient sessions object and context manager
Transparent Dict-cookie handling
Status code reference object
Removed Response.cached
Added Response.request
All args are kwargs
Relative redirect support
HTTPError handling improvements
Improved https testing
International Domain Name Support!
Access headers without fetching entire body (read())
Use lists as dicts for parameters
Add Forced Basic Authentication
Forced Basic is default authentication type
python-requests.org default User-Agent header
CaseInsensitiveDict lower-case caching
Response.history bugfix
PATCH Support
Support for Proxies
HTTPBin Test Suite
Redirect Fixes
settings.verbose stream writing
Querystrings for all methods
URLErrors (Connection Refused, Timeout, Invalid URLs) are treated as explicity raised
r.requests.get('hwe://blah'); r.raise_for_status()
Improved Redirection Handling
New ‘allow_redirects’ param for following non-GET/HEAD Redirects
Settings module refactoring
Response.history: list of redirected responses
Case-Insensitive Header Dictionaries!
Unicode URLs
Urllib2 HTTPAuthentication Recursion fix (Basic/Digest)
Internal Refactor
Bytes data upload Bugfix
Request timeouts
Unicode url-encoded data
Settings context manager and module
Automatic Decompression of GZip Encoded Content
AutoAuth Support for Tupled HTTP Auth
Cookie Changes
Response.read()
Poster fix
Automatic Authentication API Change
Smarter Query URL Parameterization
Allow file uploads and POST data together
New Authentication Manager System
Simpler Basic HTTP System
Supports all build-in urllib2 Auths
Allows for custom Auth Handlers
Python 2.5 Support
PyPy-c v1.4 Support
Auto-Authentication tests
Improved Request object constructor
New HTTPHandling Methods
Response.__nonzero__ (false if bad HTTP Status)
Response.ok (True if expected HTTP Status)
Response.error (Logged HTTPError if bad HTTP Status)
Response.raise_for_status() (Raises stored HTTPError)
Still handles request in the event of an HTTPError. (Issue #2)
Eventlet and Gevent Monkeypatch support.
Cookie Support (Issue #1)
Added file attribute to POST and PUT requests for multipart-encode file uploads.
Added Request.url attribute for context and redirects
Frustration
Conception
Py Version
Uploaded on
Python Wheel
Downloads (All Versions):
118455 downloads in the last day
1092248 downloads in the last week
4262932 downloads in the last month
Kenneth Reitz
Home Page:
Apache 2.0
Categories
Package Index Owner:
kennethreitz, graffatcolmingov, Lukasa
Package Index Maintainer:
graffatcolmingov, Lukasa
Copyright (C) ,}