09-25-2018, 03:01 AM
(09-24-2018, 11:55 AM)Vuluts Wrote: @tryp4vps Woop! Not aware of this Wayback Machine Scraper this is what I really needed of. Have you or does this scrapper can able to download data file like 500mb above? as I be needed to download some old installers from an old website.
I think this paragraph below explains your case?
Quote:The command-line utility is highly configurable in terms of what it scrapes but it only saves the unparsed content of the pages on the site. If you're interested in parsing data from the pages that are crawled then you might want to check out scrapy-wayback-machine instead. It's a downloader middleware that handles all of the tricky parts and passes normal response objects to your Scrapy spiders with archive timestamp information attached. The middleware is very unobtrusive and should work seamlessly with existing Scrapy middlewares, extensions, and spiders. It's what wayback-machine-scraper uses behind the scenes and it offers more flexibility for advanced use cases.
It may be able to download your content from that old website since it only has access to the scraped part of the site?
No one knows what the future holds, that's why its potential is infinite