U 84iH_@sxdZddlZddlZddlmZddlmZmZmZm Z ddl m Z ddl m Z mZe GdddZGd d d ZdS) z:Website detection and configuration generation for Scrapy.N)Path)DictOptionalListAny)urlparse) dataclassfieldc@steZdZUdZeed<eed<eeefed<eed<eed<eeefed<eed<e e d Z eee fed <d S) WebsiteConfigz&Suggested configuration for a website.nametype selectors rate_limit requires_jsheadersnotes)default_factoryextraN) __name__ __module__ __qualname____doc__str__annotations__rfloatboolr dictrrrr"./scrapy_project/utils/detector.pyr s r c@seZdZdZddZdddddd d d d d ddddddddddddddddd dddd ddd!ddd"dd#d$d%d&d'd(d d)ddd d*d+ddd,dd-d.d/d0d1d2d dddd d*d3ddd4dd5d6d7d8d9d:d d;ddd dd?d@dAdBdCdDdEd dddd d*dFddd>dGdHdIdJdKdLdMd dddd d*dNdddOZeeedPdQdRZ dSdTZ eeedPdUdVZ ee dPdWdXZ eeedYdZd[Zd`eeeeeeed]d^d_Zd\S)aWebsiteDetectorz?Detect website type and suggest optimal crawling configuration.cCs d|_dS)N)last_generated_slug)selfrrr__init__szWebsiteDetector.__init__z!shopee\.(vn|sg|my|th|ph|id|tw|br)Z marketplaceZShopeez[data-testid="product-item"]z[data-testid="product-title"]z[data-testid="product-price"]z img[data-testid="product-image"]za[data-testid="product-link"]product_containertitlepriceimagelink?Tsz*WebsiteDetector.detect..r-r.r r r rrrrr r r rrrrr) rnetloclower_detect_etsy_shopWEBSITE_PATTERNSitemsresearchr _detect_generic_ecommerce) r!r6parseddomainZ etsy_configZspecific_patternsZsite_keyZ site_infoZ config_datarrrdetects*    zWebsiteDetector.detectc Csdd|jpddD}t|dkr|ddkr|d}|jpFd }|jpPd }|d |d |}|d |d |d }|d |||dd}td|dddddddddddddddd |d!Sd"S)#z/Detect Etsy shop URLs and return configuration.cSsg|] }|r|qSrr)r8partrrr sz5WebsiteDetector._detect_etsy_shop../rZshophttpsz www.etsy.com://z/shop/z/rss) shop_nameZ shop_pathshop_urlrss_url max_pagesz Etsy Shop () etsy_shopzdiv.js-merch-stash-check-listing.v2-listing-card[data-listing-id], div[data-listing-id].v2-listing-card, div.js-merch-stash-check-listing[data-listing-id]zOh3.v2-listing-card__title, a.listing-link::attr(title), .v2-listing-card__titlezjdiv.n-listing-card__price span.currency-value, .n-listing-card__price .currency-value, span.currency-valuezydiv.v2-listing-card__img img, a.listing-link img[src*="etsystatic.com"], .v2-listing-card__img img[src*="etsystatic.com"]zJa.listing-link[href*="/listing/"], a[href*="/listing/"][data-listing-link]za.listing-link::attr(title)zdiv.js-merch-stash-check-listing::attr(data-listing-id), a.listing-link::attr(data-listing-id), [data-listing-id]::attr(data-listing-id))r$r%r&r'r( descriptionskur/FzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36z?text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8r1zUDetected Etsy shop. Crawling from shop page HTML with pagination to get all products.r<N)pathsplitlenr>schemer=r ) r!Z parsed_url path_partsrQr\r=rRrSrrrrr?sB    ,z!WebsiteDetector._detect_etsy_shopc Cs*tddddddddd d d d d didS)z$Detect generic e-commerce platforms.zGeneric E-commercegenericz+.product, .product-item, [class*="product"]zh2, h3, .title, .product-titlez.price, [class*="price"]Zimgar#r)Fr*r0r1zEGeneric detection. You may need to customize selectors for this site.r<)r )r!r6rrrrD.s"z)WebsiteDetector._detect_generic_ecommercec CsF||}|sdddSd|j|j||j|j|j|j|j||d S)z Suggest complete configuration for a website. Args: url: The website URL Returns: Dictionary with suggested configuration zCould not detect website typez#Please provide manual configuration)errorZ suggestionT) ZdetectedZ website_nameZ website_typer6r Zrate_limit_secondsrequires_javascriptZsuggested_headersrrecommendations) rGr r r rrrr_get_recommendations)r!r6r.rrrsuggest_configDs zWebsiteDetector.suggest_config)r.r7cCsfg}|jr|d|jdkr(|dd|jksDd|jkrN|d|d|d|S) z.Get recommendations based on detected website.z>Consider using Selenium or Playwright for JavaScript renderingr)zDThis site has strict rate limiting. Use proxies if crawling at scaler4r5z>Check if this site has a public API - it may be more efficientz5Always respect robots.txt and implement proper delaysz0Test with a small sample first before full crawl)rappendrr r>)r!r.rbrrrrcbs      z$WebsiteDetector._get_recommendationsN)r6 start_urls output_pathr7c Cs||}|std|t|}|jd|j}|jp>i}|jdkr|d}|dp`|} |d} |dd} |j||jd || | | d d | gd d | ddd|j |j |j dd|j |j ggddgdd|j d } n|j||jd |p|gd dddd d|j |j dd|j dd|j dd|j d d!d"|j |j dd|j |j ggddgdd|j d#} |s|j} | d$r| d%d&} d'd(lm}|| }|j}|jdkr |dr td)d*|d}|d*|}td+|d,}n t|j}t|}|jjd d d-i}|rzt|d.d/d0}t|pTi}W5QRXd1|kr|d1i}| d1i}||}d2D]}||kr||||<q|| d1<d3|krd4|d3kr|d3d4}|r|d5kr|| d3d4<d6|kr:d7|d6kr:| d6i|d6d7| d6d7<|d6d7}WnHtk r}z(d'd&l}|t }|!d8|d9W5d&}~XYnX| d6i|| d6d7<||_"t|d:d/d0}tj#| |dd dd;W5QRXt$|S)rparentmkdirexistsopenyaml safe_load setdefault ExceptionloggingZ getLoggerrZwarningr dumpr)r!r6rfrgr.rErhrrQrRrSrT yaml_configrFr{rZ shop_slugZexisting_configfZexisting_crawlingZ new_crawlingZmerged_crawlingkeyZ existing_typeerZloggerrrrgenerate_yaml_configts       ,     *            $  z$WebsiteDetector.generate_yaml_config)NN)rrrrr"r@rrr rGr?rDrrdrrcrrrrrrrs,&> r)rrBrpathlibrtypingrrrr urllib.parserZ dataclassesrr r rrrrrs