Since a few weeks.
e.g. curl https://gallica.bnf.fr//iiif/ark:/12148 ... 0542322t/f[14-14]/full/full/0/native.jpg -o q#1.jpg
(only one page to make things simpler)
Note that https://gallica.bnf.fr//iiif/ark:/12148 ... native.jpg works well if you copy-paste it into the banner of your browser.
curl with gallica now gives tiny images.
Moderator: kcleung
Re: curl with gallica now gives tiny images.
I suspect this is a server-side change to help prevent scraping from AI bots... For a while, I've not been able to download Gallica's IIIF manifests with curl, due to the captcha, so I was downloading the manifests through the web interface. But at that time, curl was still able to download the full-size images themselves. Apparently that's now changed... I have not downloaded from Gallica in a while, so not sure when the behavior changed (again), but I now see 403 error responses when attempting to download the images with curl. And similar to what you report, if I copy/paste the image URLs in a web browser, they download successfully.
-
coulonnus
- active poster
- Posts: 1682
- Joined: Thu Jul 12, 2007 8:53 am
- notabot: 42
- notabot2: Human
- Location: Nice, France
- Contact:
Re: curl with gallica now gives tiny images.
Thanks for this contribution. If you know where the executable of your browser is, you can edit a command file containing e.g. start " " "C:\Program Files\Google\Chrome\Application\chrome.exe" https://gallica.bnf.fr/iiif/ark:/12148/ ... native.jpg
Then if you start if from the command prompt window you obtain this wanted image on the screen and you can save it. But how to automate the process? Or is there another web browser that you can control with à la DOS commands? In the past I used Lynx but it does'nt support https pages.
And mycommand>mypage.jpg gives no result.
Then if you start if from the command prompt window you obtain this wanted image on the screen and you can save it. But how to automate the process? Or is there another web browser that you can control with à la DOS commands? In the past I used Lynx but it does'nt support https pages.
And mycommand>mypage.jpg gives no result.
Re: curl with gallica now gives tiny images.
I'm not sure what text-based browser would provide the easiest (or best) way to automate. The difficulty with using text browsers in this case (without taking additional steps) is that they do not support Javascript (as far as I know), and Javascript is generally required to solve the anti-bot measures that websites often use.
With difficulty, I managed to download a full-res image from Gallica using elinks (https://github.com/rkd77/elinks). First, I had to visit Gallica in a graphical browser, then export cookies, and also save that browser's user-agent string. Then I had elinks use the cookies I saved, as well as the user-agent string, before navigating to a Gallica IIIF URL. Elinks prompted to save to a local file, and it successfully downloaded (though didn't notify me when the download finished). Also to note, because one of the required cookies expires after 30 minutes, you have to go back to the graphical browser, refresh the page, re-export cookies, etc. All of this extra work just because the text browser doesn't support Javascript...
There are graphical browsers that are more keyboard-driven, but I've not tried any of them yet. One example is Luakit (https://luakit.github.io/).
Alternatively, if you want to take a completely different approach, maybe a system like AutoHotkey (https://www.autohotkey.com/) would be useful. When looping through each image you want to download, you could first open the IIIF URL a graphical browser as you describe, then launch an AutoHotKey macro to perform the necessary actions to download the image from the browser.
With difficulty, I managed to download a full-res image from Gallica using elinks (https://github.com/rkd77/elinks). First, I had to visit Gallica in a graphical browser, then export cookies, and also save that browser's user-agent string. Then I had elinks use the cookies I saved, as well as the user-agent string, before navigating to a Gallica IIIF URL. Elinks prompted to save to a local file, and it successfully downloaded (though didn't notify me when the download finished). Also to note, because one of the required cookies expires after 30 minutes, you have to go back to the graphical browser, refresh the page, re-export cookies, etc. All of this extra work just because the text browser doesn't support Javascript...
There are graphical browsers that are more keyboard-driven, but I've not tried any of them yet. One example is Luakit (https://luakit.github.io/).
Alternatively, if you want to take a completely different approach, maybe a system like AutoHotkey (https://www.autohotkey.com/) would be useful. When looping through each image you want to download, you could first open the IIIF URL a graphical browser as you describe, then launch an AutoHotKey macro to perform the necessary actions to download the image from the browser.