[問題] 爬蟲相關問題（BeautifulSoup）

看板Python作者TZULIU (消費券收購商)時間6年前 (2017/09/02 16:20)推噓1(1推 0噓 14→)

留言15則, 4人參與討論串1/1

小弟目前正在撰寫計算從網頁A到其他網站內連結的level，採用的方式就是在各網頁內直接抓取'href'，當我將最高level設定在2時，沒有任何問題，但當我將其設定為3時，會跑出錯誤訊息如下： UnboundLocalError: local variable 'soup' referenced before assignment 請問高手可以幫忙看看問題出在哪裡嗎？謝謝！程式碼如下： import lxml.html import urllib.request from bs4 import BeautifulSoup foundUrls = {} for rootUrl in rootUrls: foundUrls.update({rootUrl : 0}) def getProtocolAndDomainName(url): protocolAndOther = url.split('://') # splitting url by '://' and retrun a list ptorocol = protocolAndOther[0] domainName = protocolAndOther[1].split('/')[0] # this will only return 'https://xxxxx.com' return ptorocol + '://' + domainName stopLevel = 3 ## 此處若改為2時不會有任何問題 rootUrls = ['http://ps.ucdavis.edu/'] foundUrls = {} for rootUrl in rootUrls: foundUrls.update({rootUrl : 0}) def getProtocolAndDomainName(url): protocolAndOther = url.split('://') ptorocol = protocolAndOther[0] domainName = protocolAndOther[1].split('/')[0] return ptorocol + '://' + domainName def crawl(urls, stopLevel = 5, level=1): nextUrls = [] if (level <= stopLevel): for url in urls: # need to handle urls (e.g., https) that cannot be read try: openedUrl = urllib.request.urlopen(url) soup = BeautifulSoup(openedUrl, 'lxml') except: print('cannot read for :' + url) for a in soup.find_all('a', href=True): href = a['href'] if href is not None: # for the case of a link is relative path if '://' not in href: href = getProtocolAndDomainName(url) + href # check url has been already visited or not if href not in foundUrls: foundUrls.update({href : level}) nextUrls.append(href) # recursive call crawl(nextUrls, stopLevel, level + 1) crawl(rootUrls, stopLevel) print(foundUrls) -- ※ 發信站: 批踢踢實業坊(ptt.cc), 來自: 36.225.30.106 ※ 文章網址: https://www.ptt.cc/bbs/Python/M.1504340414.A.5F1.html

→

stucode

09/02 22:05, , 1^F

09/02 22:05, 1^F

→

stucode

09/02 22:07, , 2^F

09/02 22:07, 2^F

推

Sunal

09/02 22:59, , 3^F

09/02 22:59, 3^F