U
.eWF @ s d Z ddlZddlZddlZddlZddlZddlmZ ddlm Z m
Z
ddl m Z ddl
mZmZmZ ddlmZ ddlmZ dd lmZ dd
lmZ dd lmZ dd l m Z dd
l m Z m Z ddl!m"Z"m#Z# e rDddl$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z- ddl.Z/ddl0m1Z1 ddl2m3Z3 ddl4m5Z5 e/j6j7j8Z9e)e:e:f Z;edd Z?dd Z@G dd deAZBdd ZCG dd d eAZDd d ZEd d ZFd!d" ZGd#d$ ZHd%d& ZId'd( ZJd)d* ZKG d+d, d,eLZMdd6d7ZRG d8d9 d9eLZSG d:d; d;eLZTdS )?zM
The main purpose of this module is to expose LinkCollector.collect_links().
N) OrderedDict)html5librequests)unescape) HTTPError
RetryErrorSSLError)parse)requestLink)ARCHIVE_EXTENSIONS)redact_auth_from_url)MYPY_CHECK_RUNNING) path_to_url url_to_path)is_urlvcs) CallableDictIterableListMutableMappingOptionalSequenceTupleUnion)Response) SearchScope)
PipSessionc C s6 t jD ]*}| |r| t| dkr| S qdS )zgLook for VCS schemes in the URL.
Returns the matched VCS scheme, or None if there's no match.
z+:N)r Zschemeslower
startswithlen)urlscheme r% ;/usr/lib/python3.8/site-packages/pip/_internal/collector.py_match_vcs_scheme/ s
r' c C s( t | j}tD ]}||r dS qdS )z2Return whether the URL looks like an archive.
TF)r filenamer
endswith)r# r( Zbad_extr% r% r& _is_url_like_archive; s
r* c s e Zd Z fddZ ZS )_NotHTMLc s" t t| || || _|| _d S N)superr+ __init__ content_type request_desc)selfr/ r0 __class__r% r& r. G s z_NotHTML.__init__)__name__
__module__ __qualname__r.
__classcell__r% r% r2 r& r+ F s r+ c C s. | j dd}| ds*t|| jjdS )zCheck the Content-Type header to ensure the response contains HTML.
Raises `_NotHTML` if the content type is not text/html.
Content-Type text/htmlN)headersgetr r! r+ r
method)responser/ r% r% r& _ensure_html_headerN s r? c @ s e Zd ZdS )_NotHTTPN)r4 r5 r6 r% r% r% r& r@ Y s r@ c C sD t | \}}}}}|dkr"t |j| dd}| t| dS )zSend a HEAD request to the URL, and ensure the response contains HTML.
Raises `_NotHTTP` if the URL is not available for a HEAD request, or
`_NotHTML` if the content type is not text/html.
> httphttpsT)Zallow_redirectsN) urllib_parseZurlsplitr@ headraise_for_statusr? )r# sessionr$ netlocpathZqueryZfragmentrespr% r% r& _ensure_html_response] s rJ c C sL t | rt| |d tdt| |j| dddd}| t| |S )a Access an HTML page with GET, and return the response.
This consists of three parts:
1. If the URL looks suspiciously like an archive, send a HEAD first to
check the Content-Type is HTML, to avoid downloading a large file.
Raise `_NotHTTP` if the content type cannot be determined, or
`_NotHTML` if it is not HTML.
2. Actually perform the request. Raise HTTP exceptions on network failures.
3. Check the Content-Type header to make sure we got HTML, and raise
`_NotHTML` otherwise.
rF zGetting page %sr: z max-age=0)ZAcceptz
Cache-Control)r; )r* rJ loggerdebugr r< rE r? )r# rF rI r% r% r& _get_html_responsen s rN c C s2 | r.d| kr.t | d \}}d|kr.|d S dS )zBDetermine if we have any encoding information in our headers.
r8 charsetN)cgiZ parse_header)r; r/ Zparamsr% r% r& _get_encoding_from_headers s
rQ c C s. | dD ] }|d}|dk r
| S q
|S )a Determine the HTML document's base URL.
This looks for a ```` tag in the HTML document. If present, its href
attribute denotes the base URL of anchor tags in the document. If there is
no such tag (or if it does not have a valid href attribute), the HTML
file's URL is used as the base URL.
:param document: An HTML document representation. The current
implementation expects the result of ``html5lib.parse()``.
:param page_url: The URL of the HTML document.
z.//basehrefN)findallr< )documentpage_urlbaserR r% r% r& _determine_base_url s
rW c C sP t | }|jdkr(tt|j}nt jt |jdd}t |j
|dS )zMakes sure a link is fully encoded. That is, if a ' ' shows up in
the link, it will be rewritten to %20 (while not over-quoting
% or other characters).r9 z/@)Zsafe)rH ) rC urlparserG urllib_requestZ pathname2url url2pathnamerH ZquoteZunquoteZ
urlunparse_replace)r# resultrH r% r% r& _clean_link s
r] c C sf | d}|sdS tt||}| d}|r8t|nd}| d}|rRt|}t||||d}|S )zJ
Convert an anchor element in a simple repository page to a Link.
rR Nzdata-requires-pythonz data-yanked)Z
comes_fromZrequires_python
yanked_reason)r< r] rC urljoinr r )anchorrU base_urlrR r# Z pyrequirer^ linkr% r% r& _create_link_from_element s
rc c c sV t j| j| jdd}| j}t||}|dD ]"}t|||d}|dkrJq.|V q.dS )zP
Parse an HTML document, and yield its anchor elements as Link objects.
F)Ztransport_encodingZnamespaceHTMLElementsz.//a)rU ra N)r r contentencodingr# rW rS rc )pagerT r# ra r` rb r% r% r& parse_links s
rg c @ s e Zd ZdZdd Zdd ZdS )HTMLPagez'Represents one page, along with its URLc C s || _ || _|| _dS )z
:param encoding: the encoding to decode the given content.
:param url: the URL from which the HTML was downloaded.
N)rd re r# )r1 rd re r# r% r% r& r. s zHTMLPage.__init__c C s
t | jS r, )r r# r1 r% r% r& __str__$ s zHTMLPage.__str__N)r4 r5 r6 __doc__r. rj r% r% r% r& rh s rh c C s |d krt j}|d| | d S )Nz%Could not fetch URL %s: %s - skipping)rL rM )rb reasonmethr% r% r& _handle_get_page_fail( s rn c C s t | j}t| j|| jdS )N)re r# )rQ r; rh rd r# )r> re r% r% r& _make_html_page3 s
ro c
C s |d krt d| jddd }t|}|r@td|| d S t|\}}}}}}|dkrtj
t |r|
ds|d7 }t|d}td | zt||d
}W nD tk
r td | Y n, tk
r } ztd | |j|j W 5 d }~X Y n tk
r0 } zt| | W 5 d }~X Y n tk
r\ } zt| | W 5 d }~X Y n tk
r } z$d
} | t|7 } t| | tjd W 5 d }~X Y n\ tjk
r } zt| d| W 5 d }~X Y n* tj k
r t| d Y n
X t |S d S )Nz?_get_html_page() missing 1 required keyword argument: 'session'# r zCannot look at %s URL %sfile/z
index.htmlz# file: URL is directory, getting %srK zQSkipping page %s because it looks like an archive, and cannot be checked by HEAD.z.sort_pathzfile:z)Path '{0}' is ignored: it is a directory.z:Url '%s' is ignored: it is neither a file nor a directory.zQUrl '%s' is ignored. It is either a non-existing path or lacks a specific scheme.)rv rH existsr! r rw realpathlistdirjoinr rL Zwarningformatisfiler ) locations
expand_dirr r# Z
is_local_pathZ is_file_urlrH itemr% r r& group_locationsx sF
r c @ s e Zd ZdZdd ZdS )CollectedLinksa
Encapsulates all the Link objects collected by a call to
LinkCollector.collect_links(), stored separately as--
(1) links from the configured file locations,
(2) links from the configured find_links, and
(3) a dict mapping HTML page url to links from that page.
c C s || _ || _|| _dS )z
:param files: Links from file locations.
:param find_links: Links from find_links.
:param pages: A dict mapping HTML page url to links from that page.
Nr
find_linkspages)r1 r r r r% r% r& r. s zCollectedLinks.__init__N)r4 r5 r6 rk r. r% r% r% r& r s r c @ s4 e Zd ZdZdd Zedd Zdd Zdd Zd
S )
LinkCollectorz
Responsible for collecting Link objects from all configured locations,
making network requests as needed.
The class's main method is its collect_links() method.
c C s || _ || _d S r, ) search_scoperF )r1 rF r r% r% r& r. s zLinkCollector.__init__c C s | j jS r, )r r ri r% r% r& r s zLinkCollector.find_linksc c s, |D ]"}t || jd}|dkr q|V qdS )zp
Yields (page, page_url) from the given locations, skipping
locations that have errors.
rK N)r} rF )r1 r locationrf r% r% r&
_get_pages s
zLinkCollector._get_pagesc s j }||}t|\}}t jdd\}}dd t||D }dd jD } fddtdd |D d d |D D }
t|
}
d
t|
|g} |
D ]} | d | qt
d | i }
|
D ]}tt||
|j< qt|| |
d
S )zFind all available links for the given project name.
:return: All the Link objects (unfiltered), as a CollectedLinks object.
T)r c S s g | ] }t |qS r% r .0r# r% r% r&
s z/LinkCollector.collect_links..c S s g | ]}t |d qS )z-fr r r% r% r& r s c s g | ]} j |r|qS r% )rF Zis_secure_origin)r rb ri r% r& r
s c s s | ]}t |V qd S r, r r r% r% r& s z.LinkCollector.collect_links..c s s | ]}t |V qd S r, r r r% r% r& r
s z,{} location(s) to search for versions of {}:z* {}
r )r Zget_index_urls_locationsr r itertoolschainr r r" r rL rM r r r~ rg r# r )r1 Z project_namer Zindex_locationsZindex_file_locZ
index_url_locZ fl_file_locZ
fl_url_locZ
file_linksZfind_link_linksZ
url_locationslinesrb Z pages_linksrf r% ri r&
collect_links sD
zLinkCollector.collect_linksN) r4 r5 r6 rk r. propertyr r r r% r% r% r& r s
r )N)N)F)Urk rP r Zloggingr rv collectionsr Z pip._vendorr r Zpip._vendor.distlib.compatr Z pip._vendor.requests.exceptionsr r r Z pip._vendor.six.moves.urllibr rC r
rY Zpip._internal.models.linkr Z pip._internal.utils.filetypesr
Zpip._internal.utils.miscr Zpip._internal.utils.typingr Zpip._internal.utils.urlsr r Zpip._internal.vcsr r typingr r r r r r r r r Zxml.etree.ElementTreeZxmlZpip._vendor.requestsr Z!pip._internal.models.search_scoper Z pip._internal.network.sessionr ZetreeZ ElementTreeZElementZ HTMLElementrx ZResponseHeadersZ getLoggerr4 rL r' r* Exceptionr+ r? r@ rJ rN rQ rW r] rc rg objectrh rn ro r} r r r r r% r% r% r& s^ ,
3
6
;