发表更新后端开发5 分钟读完 (大约818个字)

Flask如何使用logging.FileHandler将日志保存到文件

需求

将日志尽可能往文件中输，自带的默认只输出到屏幕上。

代码

获取文件名

1
2
3
4
5
6
7
8
9
10
11
def get_custom_file_name():
    def make_dir(make_dir_path):
        path = make_dir_path.strip()
        if not os.path.exists(path):
            os.makedirs(path)
        return path
    log_dir = "ac_logs"
    file_name = 'logger-' + time.strftime('%Y-%m-%d', time.localtime(time.time())) + '.log'
    file_folder = os.path.abspath(os.path.dirname(__file__)) + os.sep + log_dir
    make_dir(file_folder)
    return file_folder + os.sep + file_name

配置logging

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
dictConfig({
    'version': 1,
    'formatters': {'default': {
        'format': '%(asctime)s - %(levelname)s - %(filename)s - %(funcName)s - %(lineno)s - %(message)s',
    }},
    'handlers': {
        'default': {
            'class': 'logging.StreamHandler',
            'stream': 'ext://flask.logging.wsgi_errors_stream',
            'formatter': 'default'
        },
        'custom': {
            'class' : 'logging.FileHandler',
            'formatter': 'default',
            'filename' : get_custom_file_name(),
            'encoding' : 'utf-8'
        },
    },
    'root': {
        'level': 'INFO',
        'handlers': ['custom']
    }
})

代码分析

在官方文档中，有一个默认的handler，当我添加一个自定义的handler，名叫custom的时候，读取配置失败，程序中断，也就无法继续执行下去，提示说，少一个叫做filename的参数。

loggin.FileHandler的构造函数中，有一个必填的参数，叫做filename。如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
class FileHandler(StreamHandler):
    """
    A handler class which writes formatted logging records to disk files.
    """
    def __init__(self, filename, mode='a', encoding=None, delay=False):
        """
        Open the specified file and use it as the stream for logging.
        """
        # Issue #27493: add support for Path objects to be passed in
        filename = os.fspath(filename)
        #keep the absolute path, otherwise derived classes which use this
        #may come a cropper when the current directory changes
        self.baseFilename = os.path.abspath(filename)
        self.mode = mode
        self.encoding = encoding
        self.delay = delay

如何才能传入参数filename?

①进入dictConfig()

1
2
3
def dictConfig(config):
    """Configure logging using a dictionary."""
    dictConfigClass(config).configure()

②进入configure()，找到对handler的处理逻辑，如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# 前面的代码省略
# 获取handlers
handlers = config.get('handlers', EMPTY_DICT)
deferred = []
# 遍历其中的每一个handler
for name in sorted(handlers):
    try:
    	# 具体处理每一个handler
        handler = self.configure_handler(handlers[name])
        handler.name = name
        handlers[name] = handler
    except Exception as e:
        if 'target not configured yet' in str(e.__cause__):
            deferred.append(name)
        else:
            raise ValueError('Unable to configure handler '
                             '%r' % name) from e
# 后面的代码省略

③进入具体处理handler的逻辑，self.configure_handler()：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
def configure_handler(self, config):
    # 省略代码若干行 ...
    if '()' in config:
        c = config.pop('()')
        if not callable(c):
            c = self.resolve(c)
        factory = c
    else:
        cname = config.pop('class')
        klass = self.resolve(cname)
        # 省略代码若干行 ...
        # 此处的kclass就是配置的class名所对应的类
        factory = klass
    props = config.pop('.', None)
    # 读取完类名，获取到该类、获取了formatter之后，接着读取conf里面的数据
    # 在这里将剩下所定义的参数，弄成dict类型的数据
    kwargs = {k: config[k] for k in config if valid_ident(k)}
    try:
    	# 直接将dict类型的数据作为函数入参，实例化出一个FileHandler
        result = factory(**kwargs)
    except TypeError as te:
        if "'stream'" not in str(te):
            raise

其中的调试数据截图如下：
在这里插入图片描述
如果没有配置filename的信息，那么实例化类的时候就报错也是理所应当，因此还可以尝试性地配置encoding参数到其中，工作正常。

总结

虽然对这一块的整体了解还不够，但是对于能够参照官方文档，参考源代码实现，完成自己的想法，做到的那一刻还是有点成就感。

【参考】
官方文档：http://flask.pocoo.org/docs/dev/logging

发表更新大数据16 分钟读完 (大约2422个字)

Python爬虫常用库的使用

builtwith的使用

这里写图片描述

Requests库的使用

这里写图片描述

实例引入

1
2
3
4
5
6
7
8
import requests

response = requests.get('https://www.baidu.com/')
print(type(response))
print(response.status_code)
print(type(response.text))
print(response.text)
print(response.cookies)

各种请求方式

1
2
3
4
5
6
import requests
requests.post('http://httpbin.org/post')
requests.put('http://httpbin.org/put')
requests.delete('http://httpbin.org/delete')
requests.head('http://httpbin.org/get')
requests.options('http://httpbin.org/get')

基本GET请求

基本写法

1
2
3
4
import requests

response = requests.get('http://httpbin.org/get')
print(response.text)

带参数GET请求

1
2
3
import requests
response = requests.get("http://httpbin.org/get?name=germey&age=22")
print(response.text)
1
2
3
4
5
6
7
8
import requests

data = {
    'name': 'germey',
    'age': 22
}
response = requests.get("http://httpbin.org/get", params=data)
print(response.text)

解析json

1
2
3
4
5
6
7
8
import requests
import json

response = requests.get("http://httpbin.org/get")
print(type(response.text))
print(response.json())
print(json.loads(response.text))
print(type(response.json()))

获取二进制数据

1
2
3
4
5
6
import requests

response = requests.get("https://github.com/favicon.ico")
print(type(response.text), type(response.content))
print(response.text)
print(response.content)
1
2
3
4
5
6
import requests

response = requests.get("https://github.com/favicon.ico")
with open('favicon.ico', 'wb') as f:
    f.write(response.content)
    f.close()

添加headers

1
2
3
4
import requests

response = requests.get("https://www.zhihu.com/explore")
print(response.text)
1
2
3
4
5
6
7
import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'
}
response = requests.get("https://www.zhihu.com/explore", headers=headers)
print(response.text)

基本POST请求

1
2
3
4
5
import requests

data = {'name': 'germey', 'age': '22'}
response = requests.post("http://httpbin.org/post", data=data)
print(response.text)
1
2
3
4
5
6
7
8
import requests

data = {'name': 'germey', 'age': '22'}
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'
}
response = requests.post("http://httpbin.org/post", data=data, headers=headers)
print(response.json())

reponse属性

1
2
3
4
5
6
7
8
import requests

response = requests.get('http://www.jianshu.com')
print(type(response.status_code), response.status_code)
print(type(response.headers), response.headers)
print(type(response.cookies), response.cookies)
print(type(response.url), response.url)
print(type(response.history), response.history)

状态码判断

1
2
3
4
import requests

response = requests.get('http://www.jianshu.com/hello.html')
exit() if not response.status_code == requests.codes.not_found else print('404 Not Found')
1
2
3
4
import requests

response = requests.get('http://www.jianshu.com')
exit() if not response.status_code == 200 else print('Request Successfully')
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
100: ('continue',),
101: ('switching_protocols',),
102: ('processing',),
103: ('checkpoint',),
122: ('uri_too_long', 'request_uri_too_long'),
200: ('ok', 'okay', 'all_ok', 'all_okay', 'all_good', '\\o/', '✓'),
201: ('created',),
202: ('accepted',),
203: ('non_authoritative_info', 'non_authoritative_information'),
204: ('no_content',),
205: ('reset_content', 'reset'),
206: ('partial_content', 'partial'),
207: ('multi_status', 'multiple_status', 'multi_stati', 'multiple_stati'),
208: ('already_reported',),
226: ('im_used',),

# Redirection.
300: ('multiple_choices',),
301: ('moved_permanently', 'moved', '\\o-'),
302: ('found',),
303: ('see_other', 'other'),
304: ('not_modified',),
305: ('use_proxy',),
306: ('switch_proxy',),
307: ('temporary_redirect', 'temporary_moved', 'temporary'),
308: ('permanent_redirect',
      'resume_incomplete', 'resume',), # These 2 to be removed in 3.0

# Client Error.
400: ('bad_request', 'bad'),
401: ('unauthorized',),
402: ('payment_required', 'payment'),
403: ('forbidden',),
404: ('not_found', '-o-'),
405: ('method_not_allowed', 'not_allowed'),
406: ('not_acceptable',),
407: ('proxy_authentication_required', 'proxy_auth', 'proxy_authentication'),
408: ('request_timeout', 'timeout'),
409: ('conflict',),
410: ('gone',),
411: ('length_required',),
412: ('precondition_failed', 'precondition'),
413: ('request_entity_too_large',),
414: ('request_uri_too_large',),
415: ('unsupported_media_type', 'unsupported_media', 'media_type'),
416: ('requested_range_not_satisfiable', 'requested_range', 'range_not_satisfiable'),
417: ('expectation_failed',),
418: ('im_a_teapot', 'teapot', 'i_am_a_teapot'),
421: ('misdirected_request',),
422: ('unprocessable_entity', 'unprocessable'),
423: ('locked',),
424: ('failed_dependency', 'dependency'),
425: ('unordered_collection', 'unordered'),
426: ('upgrade_required', 'upgrade'),
428: ('precondition_required', 'precondition'),
429: ('too_many_requests', 'too_many'),
431: ('header_fields_too_large', 'fields_too_large'),
444: ('no_response', 'none'),
449: ('retry_with', 'retry'),
450: ('blocked_by_windows_parental_controls', 'parental_controls'),
451: ('unavailable_for_legal_reasons', 'legal_reasons'),
499: ('client_closed_request',),

# Server Error.
500: ('internal_server_error', 'server_error', '/o\\', '✗'),
501: ('not_implemented',),
502: ('bad_gateway',),
503: ('service_unavailable', 'unavailable'),
504: ('gateway_timeout',),
505: ('http_version_not_supported', 'http_version'),
506: ('variant_also_negotiates',),
507: ('insufficient_storage',),
509: ('bandwidth_limit_exceeded', 'bandwidth'),
510: ('not_extended',),
511: ('network_authentication_required', 'network_auth', 'network_authentication'),

文件上传

1
2
3
4
5
import requests

files = {'file': open('favicon.ico', 'rb')}
response = requests.post("http://httpbin.org/post", files=files)
print(response.text)

获取cookie

1
2
3
4
5
6
import requests

response = requests.get("https://www.baidu.com")
print(response.cookies)
for key, value in response.cookies.items():
    print(key + '=' + value)

会话维持

模拟登录

1
2
3
4
5
import requests

requests.get('http://httpbin.org/cookies/set/number/123456789')
response = requests.get('http://httpbin.org/cookies')
print(response.text)
1
2
3
4
5
6
import requests

s = requests.Session()
s.get('http://httpbin.org/cookies/set/number/123456789')
response = s.get('http://httpbin.org/cookies')
print(response.text)

证书验证

1
2
3
4
import requests

response = requests.get('https://www.12306.cn')
print(response.status_code)
1
2
3
4
5
import requests
from requests.packages import urllib3
urllib3.disable_warnings()
response = requests.get('https://www.12306.cn', verify=False)
print(response.status_code)
1
2
3
4
import requests

response = requests.get('https://www.12306.cn', cert=('/path/server.crt', '/path/key'))
print(response.status_code)

代理设置

1
2
3
4
5
6
7
8
9
import requests

proxies = {
  "http": "http://127.0.0.1:9743",
  "https": "https://127.0.0.1:9743",
}

response = requests.get("https://www.taobao.com", proxies=proxies)
print(response.status_code)
1
2
3
4
5
6
7
import requests

proxies = {
    "http": "http://user:[email protected]:9743/",
}
response = requests.get("https://www.taobao.com", proxies=proxies)
print(response.status_code)
1
pip3 install 'requests[socks]'
1
2
3
4
5
6
7
8
import requests

proxies = {
    'http': 'socks5://127.0.0.1:9742',
    'https': 'socks5://127.0.0.1:9742'
}
response = requests.get("https://www.taobao.com", proxies=proxies)
print(response.status_code)

超时设置

1
2
3
4
5
6
7
import requests
from requests.exceptions import ReadTimeout
try:
    response = requests.get("http://httpbin.org/get", timeout = 0.5)
    print(response.status_code)
except ReadTimeout:
    print('Timeout')

认证设置

1
2
3
4
5
import requests
from requests.auth import HTTPBasicAuth

r = requests.get('http://120.27.34.24:9001', auth=HTTPBasicAuth('user', '123'))
print(r.status_code)
1
2
3
4
import requests

r = requests.get('http://120.27.34.24:9001', auth=('user', '123'))
print(r.status_code)

异常处理

1
2
3
4
5
6
7
8
9
10
11
import requests
from requests.exceptions import ReadTimeout, ConnectionError, RequestException
try:
    response = requests.get("http://httpbin.org/get", timeout = 0.5)
    print(response.status_code)
except ReadTimeout:
    print('Timeout')
except ConnectionError:
    print('Connection error')
except RequestException:
    print('Error')
Connection error

Urllib库的使用

这里写图片描述

urllib.urlopen

get请求

1
2
3
import urllib.request
response = urllib.request.urlopen('http://asahii.cn')
print(response.read().decode('utf-8'))
<html>
    <script>
        window.location.href="http://blog.csdn.net/asahinokawa"
    </script>
</html>

post请求

1
2
3
4
5
6
import urllib.parse
import urllib.request

data = bytes(urllib.parse.urlencode({'word':'hello'}), encoding='utf8')
response = urllib.request.urlopen('http://httpbin.org/post', data=data)
print(response.read().decode('utf-8'))
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "word": "hello"
  }, 
  "headers": {
    "Accept-Encoding": "identity", 
    "Connection": "close", 
    "Content-Length": "10", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "Python-urllib/3.6"
  }, 
  "json": null, 
  "origin": "14.154.29.216", 
  "url": "http://httpbin.org/post"
}

超时时限设置。超过此值将抛出异常。

1
2
3
4
import urllib.request

response = urllib.request.urlopen('http://httpbin.org/get?data=haha', timeout=1)
print(response.read().decode('utf-8'))
{
  "args": {
    "data": "haha"
  }, 
  "headers": {
    "Accept-Encoding": "identity", 
    "Connection": "close", 
    "Host": "httpbin.org", 
    "User-Agent": "Python-urllib/3.6"
  }, 
  "origin": "14.154.29.216", 
  "url": "http://httpbin.org/get?data=haha"
}

捕捉异常

1
2
3
4
5
6
7
8
9
import socket
import urllib.request
import urllib.error

try:
    response = urllib.request.urlopen('http://httpbin.org/get', timeout=0.1)
except urllib.error.URLError as e:
    if isinstance(e.reason, socket.timeout):
        print('Time Out')
Time Out

响应

响应类型

1
2
3
4
5
6
7
8
9
10
11
import urllib.request

resp = urllib.request.urlopen('http://asahii.cn')
print(type(resp))

#resp = urllib.request.urlopen('https://www.python.org')
#print(type(resp))

print(resp.status)
print(resp.getheaders())
print(resp.getheader('Server'))
<class 'http.client.HTTPResponse'>
200
[('Server', 'GitHub.com'), ('Content-Type', 'text/html; charset=utf-8'), ('Last-Modified', 'Sat, 20 Jan 2018 04:10:07 GMT'), ('Access-Control-Allow-Origin', '*'), ('Expires', 'Mon, 22 Jan 2018 13:32:43 GMT'), ('Cache-Control', 'max-age=600'), ('X-GitHub-Request-Id', '71A8:11832:91183A:9A71F6:5A65E5A3'), ('Content-Length', '106'), ('Accept-Ranges', 'bytes'), ('Date', 'Mon, 22 Jan 2018 14:26:04 GMT'), ('Via', '1.1 varnish'), ('Age', '480'), ('Connection', 'close'), ('X-Served-By', 'cache-hnd18748-HND'), ('X-Cache', 'HIT'), ('X-Cache-Hits', '1'), ('X-Timer', 'S1516631164.467386,VS0,VE0'), ('Vary', 'Accept-Encoding'), ('X-Fastly-Request-ID', 'f3adc99bea78d082bc4447a4a04f427731a40dad')]
GitHub.com

Request

构造请求，包括增添一些请求头部信息等。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
from urllib import request, parse

req = urllib.request.Request('http://asahii.cn')
resp = urllib.request.urlopen(req)
print(resp.read().decode('utf-8'))

##########################   分隔线   ##########################
url = 'http://httpbin.org/post'
headers = {
    'User-Agent':'Mozilla/4.0(compatible;MSIE 5.5;Windows NT)',
    'Host':'httpbin.org'
}
dict = {
    'name':'Germey'
}
data = bytes(parse.urlencode(dict), encoding='utf8')
req = request.Request(url=url, data=data, headers=headers, method='POST')
resp = request.urlopen(req)
print(resp.read().decode('utf-8'))
<html>
    <script>
        window.location.href="http://blog.csdn.net/asahinokawa"
    </script>
</html>

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "name": "Germey"
  }, 
  "headers": {
    "Accept-Encoding": "identity", 
    "Connection": "close", 
    "Content-Length": "11", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "Mozilla/4.0(compatible;MSIE 5.5;Windows NT)"
  }, 
  "json": null, 
  "origin": "14.154.29.0", 
  "url": "http://httpbin.org/post"
}

1
2
3
4
5
6
7
8
9
10
11
from urllib import request,parse

url = 'http://httpbin.org/post'
dict = {
    'name':'China'
}
data = bytes(parse.urlencode(dict), encoding='utf8')
req = request.Request(url=url, data=data, method='POST')
req.add_header('User-Agent', 'Mozilla/4.0(compatible;MSIE 5.5;Windows NT)')
resp = request.urlopen(req)
print(resp.read().decode('utf-8'))
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "name": "China"
  }, 
  "headers": {
    "Accept-Encoding": "identity", 
    "Connection": "close", 
    "Content-Length": "10", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "Mozilla/4.0(compatible;MSIE 5.5;Windows NT)"
  }, 
  "json": null, 
  "origin": "14.154.28.26", 
  "url": "http://httpbin.org/post"
}

Handler

设置代理。可以通过来回切换代理，以防止服务器禁用我们的IP。

1
2
3
4
5
6
7
8
9
10
import urllib.request

proxy_handler = urllib.request.ProxyHandler({
    'http':'http://127.0.0.1:9743',
    'https':'https://127.0.0.1:9743'
})

opener = urllib.request.build_opener(proxy_handler)
resp = opener.open('http://www.baidu.com')
print(resp.read())

Cookie相关操作

用来维持登录状态。可以在遇到需要登录的网站使用。

1
2
3
4
5
6
7
8
# 获取cookie
import http.cookiejar, urllib.request
cookie = http.cookiejar.CookieJar()
handler = urllib.request.HTTPCookieProcessor(cookie)
opener = urllib.request.build_opener(handler)
resp = opener.open('http://www.baidu.com')
for item in cookie:
    print(item.name+"="+item.value)
BAIDUID=417382BEB774A45EA7FC2C374C846E04:FG=1
BIDUPSID=417382BEB774A45EA7FC2C374C846E04
H_PS_PSSID=25639_1446_21107_20930
PSTM=1516632429
BDSVRTM=0
BD_HOME=0

保存cookie到本地文件中

1
2
3
4
5
6
7
import http.cookiejar, urllib.request
filename = "cookie.txt"
cookie = http.cookiejar.MozillaCookieJar(filename)
handler = urllib.request.HTTPCookieProcessor(cookie)
opener = urllib.request.build_opener(handler)
resp = opener.open('http://www.baidu.com')
cookie.save(ignore_discard=True, ignore_expires=True)

另一种cookie保存格式

1
2
3
4
5
6
7
import http.cookiejar, urllib.request
filename = "cookie.txt"
cookie = http.cookiejar.LWPCookieJar(filename)
handler = urllib.request.HTTPCookieProcessor(cookie)
opener = urllib.request.build_opener(handler)
resp = opener.open('http://www.baidu.com')
cookie.save(ignore_discard=True, ignore_expires=True)

加载本地cookie到请求中去。用什么样的方式存cookie，就用什么样的方式去读。

1
2
3
4
5
6
7
import http.cookiejar, urllib.request
cookie = http.cookiejar.LWPCookieJar()
cookie.load('cookie.txt', ignore_discard=True, ignore_expires=True)
handler = urllib.request.HTTPCookieProcessor(cookie)
opener = urllib.request.build_opener(handler)
resp = opener.open('http://httpbin.org/get')
print(resp.read().decode('utf8'))
{
  "args": {}, 
  "headers": {
    "Accept-Encoding": "identity", 
    "Connection": "close", 
    "Host": "httpbin.org", 
    "User-Agent": "Python-urllib/3.6"
  }, 
  "origin": "14.154.29.0", 
  "url": "http://httpbin.org/get"
}

页面请求时的异常处理

1
2
3
4
5
from urllib import request, error
try:
    resp = request.urlopen('http://asahii.cn/index.htm')
except error.URLError as e:
    print(e.reason)#可考虑进行重拾
Not Found

详细的错误类型

1
2
3
4
5
6
7
from urllib import request, error
try:
    resp = request.urlopen('http://asahii.cn/index.htm')
except error.HTTPError as ee:
    print(ee.reason, ee.code, ee.headers, sep='\n')
except error.URLError as e:
    print(e.reason)
Not Found
404
Server: GitHub.com
Content-Type: text/html; charset=utf-8
ETag: "5a62c123-215e"
Access-Control-Allow-Origin: *
X-GitHub-Request-Id: F758:1D9CC:916D4F:9B5FD5:5A65FE22
Content-Length: 8542
Accept-Ranges: bytes
Date: Mon, 22 Jan 2018 15:09:43 GMT
Via: 1.1 varnish
Age: 148
Connection: close
X-Served-By: cache-hnd18733-HND
X-Cache: HIT
X-Cache-Hits: 1
X-Timer: S1516633784.646920,VS0,VE0
Vary: Accept-Encoding
X-Fastly-Request-ID: fb6b42a4706a17c3de8bd318b76be3513b68c49b

加上原因判断

1
2
3
4
5
6
7
8
import socket
from urllib import request, error
try:
    resp = request.urlopen('http://asahii.cn/index.html', timeout=0.01)
except error.URLError as e:
    print(type(e.reason))
    if isinstance(e.reason, socket.timeout):
        print('TIME OUT')
<class 'socket.timeout'>
TIME OUT

网页解析

解析URL

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
from urllib.parse import urlparse
result = urlparse('http://www.baidu.com/index.html;user?id=5#comment')
print(type(result), result, sep='\n')

result = urlparse('www.baidu.com/index.html;user?id=5#comment', scheme='https')
print(result)

result = urlparse('http://www.baidu.com/index.html;user?id=5#comment', scheme='https')
print(result)

result = urlparse('http://www.baidu.com/index.html;user?id=5#comment', allow_fragments=False)
print(result)

result = urlparse('http://www.baidu.com/index.html#comment', allow_fragments=False)
print(result)
<class 'urllib.parse.ParseResult'>
ParseResult(scheme='http', netloc='www.baidu.com', path='/index.html', params='user', query='id=5', fragment='comment')
ParseResult(scheme='https', netloc='', path='www.baidu.com/index.html', params='user', query='id=5', fragment='comment')
ParseResult(scheme='http', netloc='www.baidu.com', path='/index.html', params='user', query='id=5', fragment='comment')
ParseResult(scheme='http', netloc='www.baidu.com', path='/index.html', params='user', query='id=5#comment', fragment='')
ParseResult(scheme='http', netloc='www.baidu.com', path='/index.html#comment', params='', query='', fragment='')

URL拼装

1
2
3
from urllib.parse import urlunparse
data=['http', 'www.baidu.com', 'index.html', 'user', 'a=6', 'comment']
print(urlunparse(data))
http://www.baidu.com/index.html;user?a=6#comment

1
2
3
4
5
6
7
8
from urllib.parse import urlencode
params = {
    'name':'germey',
    'age':22
}
base_url = 'http://www.baidu.com?'
url = base_url+urlencode(params)
print(url)
http://www.baidu.com?name=germey&age=22
发表更新大数据5 分钟读完 (大约769个字)

Python爬虫环境搭建（Mac）

这是一篇对此教程视频的笔记。看视频太磨叽了，安装都是分成了win、linux、mac三种，再看视频的话怕是没那个耐心看。Homebrew与AnacondaHomebrew充当的角色是mac下的apt-get，是一种包管理工具。先把Homebrew安装到mac上。然后用它安装python3，最后验证p
阅读更多
发表更新编程语言13 分钟读完 (大约1951个字)

Python语法简要概览

很久没用过Python了，熟悉一下用法准备ms。

输入和输出

1
2
3
4
>>> var=input('input:')
input:sjdf asdkjf 123 adsf ;dfa--..
>>> print(var)
sjdf asdkjf 123 adsf ;dfa--..

暂时理解input()读入一行数据，且可以加入提示信息。
读入一个整数：

1
2
s = input('birth: ')
birth = int(s)

基本注意事项

# 注释某行中其后的内容。
缩进代替C系列语言中的大括号。
大小写敏感。
字符串可用''或""包裹, \可用于转义。\n\t等
r’’表示’’内部的字符串默认不转义
'''表示多行输入 

1
2
3
4
5
6
7
8
9
10
11
>>> print(r'\\\t\\')
\\\t\\
>>> print('''hi
... hi
... hi
... hello,py
... ''')
hi
hi
hi
hello,py

空值是一个特殊的值，用None表示。None不能理解为0，因为0是有意义的，而None是一个特殊的空值。
用全部大写的变量名表示常量。
三种除法，//地板除，/整除，得到浮点数，%取余。
在计算机内存中，统一使用Unicode编码，当需要保存到硬盘或者需要传输的时候，就转换为UTF-8编码。
ord()函数获取字符的整数表示，chr()函数把编码转换为对应的字符。
格式化：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
>>> 'Hi, %s, you have $%d.' % ('Michael', 1000000)
'Hi, Michael, you have $1000000.'
```  
## 有意思的数据类型
#### list
```python
>>> fruits = ['apple', 'banana', 'orange']
>>> fruits
['apple', 'banana', 'orange']
>>> len(fruits)
3
>>> fruits[1]
'banana'
>>> fruits[6]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: list index out of range
  • 倒数
    1
    2
    3
    4
    5
    6
    7
    8
    >>> fruits[-1]
    'orange'
    >>> fruits[-2]
    'banana'
    >>> fruits[-6]
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    IndexError: list index out of range
    append()
    insert(1, ‘hi’)
    pop()
    list中的元素类型可以不同

tuple

不可更改的list，声明用()
当你定义一个tuple时，在定义的时候，tuple的元素就必须被确定下来。
t = (1)定义的是自然数1，要定义成tuple需要加‘,’，规则。
t = (‘a’, ‘b’, [‘A’, ‘B’])其中的list是可变的。

dict

1
2
3
4
5
6
>>> d = {'Michael': 95, 'Bob': 75, 'Tracy': 85}
>>> d['Michael']
95
>>> d['Adam'] = 67
>>> d['Adam']
67
  • 判断是否存在dict中的两个方法
    一是通过in判断key是否存在；
    二是通过dict提供的get()方法，如果key不存在，可以返回None，或者自己指定的value。
    1
    2
    3
    4
    5
    >>> 'Thomas' in d
    False
    >>> d.get('Thomas')
    >>> d.get('Thomas', -1)
    -1
    删除用pop(‘Bob’)。

set

1
2
3
s = set([1, 2, 3])
s.add(4)
s.remove(4)

可以对集合进行&和|操作。

判断&循环

if

1
2
3
4
5
6
7
8
9
10
11
12
13
14
age = 3
if age >= 18:
    print('adult')
elif age >= 6:
    print('teenager')
else:
    print('kid')

s = input('birth: ')
birth = int(s)
if birth < 2000:
    print('00前')
else:
    print('00后')

for…in

可打印list、tuple中的数据。 

1
2
3
4
5
6
7
8
names = ['Michael', 'Bob', 'Tracy']
for name in names:
    print(name)

sum = 0
for x in range(101):
    sum = sum + x
print(sum)

while

1
2
3
4
5
6
sum = 0
n = 99
while n > 0:
    sum = sum + n
    n = n - 2
print(sum)

break&continue

同C系列语言

函数

内置函数

  • abs()
  • int()
  • max()
  • hex()
  • isinstance()
    函数名其实就是指向一个函数对象的引用，完全可以把函数名赋给一个变量，相当于给这个函数起了一个“别名”：
    1
    2
    3
    >>> a = abs # 变量a指向abs函数
    >>> a(-1) # 所以也可以通过a调用abs函数
    1

自定义函数

1
2
3
4
5
def my_abs(x):
    if x >= 0:
        return x
    else:
        return -x
  • pass
    什么都不做,作为占位符
  • 返回多个值
    在语法上，返回一个tuple可以省略括号，而多个变量可以同时接收一个tuple，按位置赋给对应的值，所以，Python的函数返回多值其实就是返回一个tuple

函数参数

  • 默认参数

    1
    2
    3
    4
    5
    6
    def power(x, n=2):
        s = 1
        while n > 0:
            n = n - 1
            s = s * x
        return s

  • 可变参数

    1
    2
    3
    4
    5
    6
    7
    8
    def calc(*numbers):
        sum = 0
        for n in numbers:
            sum = sum + n * n
        return sum
    >>> nums = [1, 2, 3]
    >>> calc(*nums)
    14

    *nums表示把nums这个list的所有元素作为可变参数传进去。

  • 关键字参数 

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    def person(name, age, **kw):
        print('name:', name, 'age:', age, 'other:', kw)
    >>> person('Michael', 30)
    name: Michael age: 30 other: {}
    >>> person('Bob', 35, city='Beijing')
    name: Bob age: 35 other: {'city': 'Beijing'}
    >>> person('Adam', 45, gender='M', job='Engineer')
    name: Adam age: 45 other: {'gender': 'M', 'job': 'Engineer'}
    >>> extra = {'city': 'Beijing', 'job': 'Engineer'}
    >>> person('Jack', 24, city=extra['city'], job=extra['job'])
    name: Jack age: 24 other: {'city': 'Beijing', 'job': 'Engineer'}
    >>> extra = {'city': 'Beijing', 'job': 'Engineer'}
    >>> person('Jack', 24, **extra)
    name: Jack age: 24 other: {'city': 'Beijing', 'job': 'Engineer'}

    **extra表示把extra这个dict的所有key-value用关键字参数传入到函数的**kw参数，kw将获得一个dict，注意kw获得的dict是extra的一份拷贝，对kw的改动不会影响到函数外的extra

  • 命名关键字参数

    • 命名关键字参数需要一个特殊分隔符*，*后面的参数被视为命名关键字参数。
    • 如果函数定义中已经有了一个可变参数，后面跟着的命名关键字参数就不再需要一个特殊分隔符*了。
    • 命名关键字参数必须传入参数名，这和位置参数不同。如果没有传入参数名，调用将报错 
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      def person(name, age, *, city, job):
          print(name, age, city, job)
      >>> person('Jack', 24, city='Beijing', job='Engineer')
      Jack 24 Beijing Engineer
      def person(name, age, *args, city, job):
          print(name, age, args, city, job)
      def person(name, age, *, city='Beijing', job):
          print(name, age, city, job)
      >>> person('Jack', 24, job='Engineer')
      Jack 24 Beijing Engineer
      限定了kw中的关键字只能为city,job

  • 参数组合
    参数定义的顺序必须是：必选参数、默认参数、可变参数、命名关键字参数和关键字参数。

  • 支持递归

高级特性

切片

1
2
3
4
5
6
7
8
L = ['Michael', 'Sarah', 'Tracy', 'Bob', 'Jack']
L[0:3]
L[:3]
L[-2:]
L[-2:-1]
L[:10:2]
L[::5]
L[:]

迭代

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
>>> d = {'a': 1, 'b': 2, 'c': 3}
>>> for key in d:
...     print(key)
...
a
b
c
>>> for k, v in d.items():
...     print(k,v)
...
a 1
b 2
c 3
>>> for value in d.values():
...     print(value)
...
1
2
3
>>> for i, value in enumerate(['A', 'B', 'C']):
...     print(i, value)
...
0 A
1 B
2 C

列表生成式

1
2
3
4
5
6
7
8
9
10
11
12
13
>>> [x * x for x in range(1, 11)]
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
>>> [m + n for m in 'ABC' for n in 'XYZ']
['AX', 'AY', 'AZ', 'BX', 'BY', 'BZ', 'CX', 'CY', 'CZ']
>>> import os # 导入os模块，模块的概念后面讲到
>>> [d for d in os.listdir('.')] # os.listdir可以列出文件和目录
['.emacs.d', '.ssh', '.Trash', 'Adlm', 'Applications', 'Desktop', 'Documents', 'Downloads', 'Library', 'Movies', 'Music', 'Pictures', 'Public', 'VirtualBox VMs', 'Workspace', 'XCode']
>>> d = {'x': 'A', 'y': 'B', 'z': 'C' }
>>> [k + '=' + v for k, v in d.items()]
['y=B', 'x=A', 'z=C']
>>> L = ['Hello', 'World', 'IBM', 'Apple']
>>> [s.lower() for s in L]
['hello', 'world', 'ibm', 'apple']

生成器

1
2
3
4
5
6
7
def fib(max):
    n, a, b = 0, 0, 1
    while n < max:
        yield b
        a, b = b, a + b
        n = n + 1
    return 'done'

函数是顺序执行，遇到return语句或者最后一行函数语句就返回。而变成generator的函数，在每次调用next()的时候执行，遇到yield语句返回，再次执行时从上次返回的yield语句处继续执行。

小结

生成器都是Iterator对象，但list、dict、str虽然是Iterable，却不是Iterator。

把list、dict、str等Iterable变成Iterator可以使用iter()函数：

凡是可作用于for循环的对象都是Iterable类型；

凡是可作用于next()函数的对象都是Iterator类型，它们表示一个惰性计算的序列；

集合数据类型如list、dict、str等是Iterable但不是Iterator，不过可以通过iter()函数获得一个Iterator对象。

Python的for循环本质上就是通过不断调用next()函数实现的，例如：

1
2
3
4
5
6
7
8
9
10
11
12
for x in [1, 2, 3, 4, 5]:
    pass
# 首先获得Iterator对象:
it = iter([1, 2, 3, 4, 5])
# 循环:
while True:
    try:
        # 获得下一个值:
        x = next(it)
    except StopIteration:
        # 遇到StopIteration就退出循环
        break

函数式编程

高阶函数

  • 传入函数
    1
    2
    def add(x, y, f):
        return f(x) + f(y)

map/reduce

  • map
    1
    2
    3
    4
    5
    6
    >>> def f(x):
    ...     return x * x
    ...
    >>> r = map(f, [1, 2, 3, 4, 5, 6, 7, 8, 9])
    >>> list(r)
    [1, 4, 9, 16, 25, 36, 49, 64, 81]
  • reduce
    reduce(f, [x1, x2, x3, x4]) = f(f(f(x1, x2), x3), x4)
  • 两者综合 
    1
    2
    3
    4
    5
    6
    7
    8
    from functools import reduce
    DIGITS = {'0': 0, '1': 1, '2': 2, '3': 3, '4': 4, '5': 5, '6': 6, '7': 7, '8': 8, '9': 9}
    def str2int(s):
        def fn(x, y):
            return x * 10 + y
        def char2num(s):
            return DIGITS[s]
        return reduce(fn, map(char2num, s))
  • filter
    1
    2
    3
    4
    5
    6
    7
    8
    def is_odd(n):
        return n % 2 == 1
    list(filter(is_odd, [1, 2, 4, 5, 6, 9, 10, 15]))
    # 结果: [1, 5, 9, 15]
    def not_empty(s):
        return s and s.strip()
    list(filter(not_empty, ['A', '', 'B', None, 'C', '  ']))
    # 结果: ['A', 'B', 'C']
  • sorted
    1
    2
    3
    4
    5
    6
    >>> sorted([36, 5, -12, 9, -21], key=abs)
    [5, 9, -12, -21, 36]
    >>> sorted(['bob', 'about', 'Zoo', 'Credit'], key=str.lower)
    ['about', 'bob', 'Credit', 'Zoo']
    >>> sorted(['bob', 'about', 'Zoo', 'Credit'], key=str.lower, reverse=True)
    ['Zoo', 'Credit', 'bob', 'about']

归档

最新文章

链接

×