Personal Video Database
English => Development => PVD Python Scripts => Topic started by: afrocuban on March 13, 2025, 11:44:06 pm
-
IF YOU DON'T READ THIS POST CAREFULLY AND FOLLOW EVERYTHING WRITTEN HERE, BUT JUST DOWNLOAD FILES, I COULD BET IT WILL NOT WORK FOR YOU AND YOU WILL COME BACK HERE ASKING QUESTIONS ALREADY ANSWERED IN THIS POST.
Almost 4 months after for the first time ever I heard the word "Selenium" knowing nothing about programming, I am finally bringing practically new PVD MOD considering amount of files and programs brought. It consists of the scripts and program as described here (https://www.videodb.info/forum_en/index.php/topic,4377.msg22782.html#msg22782).
You now need only one script for IMDb movies, one for IMDb people and one for FilmAffinity movies for everything: search and download. Selenium scripts in the background are doing all "external" work, so in your PVD you have clean situation: 2 scripts and configurator for movies (plus .batch file for these 2 at your will), and one script and configurator for the people. Check the screenshots below.
I strongly suggest to rename your current "Scripts" folder to, for instance, "Scripts-Original", and to put this Scripts.7z in your PVD folder and extract it there. It will create "Scripts" folder with all the scripts and files needed for the PVD to work (as a bonus, I'm contributing source code for the Scripts Configurator program, as well as updated and polished UDL file for PVD scripts in Notepad++ that is just to be imported to Notepad++). If you want to, after testing you can merge two folders, Selenium and non-Selenium scripts and files into "Scripts" folder.
Before that....
As stated here (https://www.videodb.info/forum_en/index.php/topic,4364.msg22706.html#msg22706)ensure that:
A. You installed python
B. You installed selenium via cmd, with
pip install selenium
B. You installed requests via cmd, with
pip install requests
C. You have your Chrome bin on a PATH (to test this, open cmd and simply type "chrome" and check if Chrome opens).
D. You have Python folder on your PATH (to test this, open cmd and simply type "python --version" and check if got the proper feedback, for instance:
C:\Users\user>python --version
Python 3.12.6
E. pythonw.exe is not missing, or it's containing folder is on the PATH (to test this, open cmd and simply type "pythonw" and check if got the proper feedback, for instance:
C:\Users\user>pythonw
C:\Users\user> (empty output)
These scripts:
1. Use Chrome browser instead Firefox
2. Use chromedriver.exe instead geckodriver
3. Start chromedriver.exe silently
4. Silently invoke browser in a headless mode (no pop-up windows of browser)
5. Scrape .htm pages of a given urls
6. No path is needed to set manually inside the script - it is set to be relative to the path of selenium script!
You just use your PVD as ever, just be sure to extract as instructed above.
For using relative path, ensure:
6B. You put appropirate chromedriver.exe to the "Script" folder, too. There is no installation for chromedriver, just extract it from the .zip file into your "Scripts" folder described above. IMPORTANT!!!! You need to download chromedriver.exe of the same version as your Chrome browser. At the moment of this post, stable version is v134. You can find Crome browser download and appropriate chromedriver here (https://googlechromelabs.github.io/chrome-for-testing/). For example, for v134, Stable links are:
Chrome browser:
1. chrome win32 https://storage.googleapis.com/chrome-for-testing-public/134.0.6998.88/win32/chrome-win32.zip (https://storage.googleapis.com/chrome-for-testing-public/134.0.6998.88/win32/chrome-win32.zip)
2. chrome win64 https://storage.googleapis.com/chrome-for-testing-public/134.0.6998.88/win64/chrome-win64.zip (https://storage.googleapis.com/chrome-for-testing-public/134.0.6998.88/win64/chrome-win64.zip)
Chromedriver:
1. chromedriver win32 https://storage.googleapis.com/chrome-for-testing-public/134.0.6998.88/win32/chromedriver-win32.zip (https://storage.googleapis.com/chrome-for-testing-public/134.0.6998.88/win32/chromedriver-win32.zip)
2. chromedriver win64 https://storage.googleapis.com/chrome-for-testing-public/134.0.6998.88/win64/chromedriver-win64.zip (https://storage.googleapis.com/chrome-for-testing-public/134.0.6998.88/win64/chromedriver-win64.zip)
From this point on, everything is automated and headless, silent as never before.
Amount of data imported is huge! I have included dozens of new custom fields. IF YOU ARE DATA HUNGRY AS I AM, THE MOST DATA CAN BE COLLECTED IF YOU CHECK AND UNCHECK OPTIONS AS IN THE SCRIPTS CONFIGURATOR SCREENSHOTS BELOW. The updated table with all the fields in these 3 scripts can be found here (https://www.videodb.info/forum_en/index.php/topic,4291.msg22824.html#msg22824). To see them all, you have to use Classic skin, or you have to add custom fields to your PVD and from there to create your own custom skin, or to use one of my skins from here (https://www.videodb.info/forum_en/index.php/topic,4356.msg22648.html#msg22648), once I complete them and adjust them for the final Selenium v4 scripts.
Examine the table. That is the only way you will learn what fields comes from which movie/person page and if you want them or not. The less you want, the faster PVD will be.
Please feel free to test the scripts and give me a feedback if something doesn't work. When I say "it doesn't work", everything works, the only issue that can happen is that sites have changed html layout and again not all fileds are available, or you updated your Chrome automatically to a higher version and you didn't download and extract corespondent chromedriver version. The best indicator for this is that no .log file is created in "\Scripts\Tmp\" folder. I will update whatever you report in a month at most from the first report, to give us all the time to collect and report as many as possible issues.
What I have learned
On this long journey, what I have learned was how hard work coding is. Also, I had to learn pretty deep about Pascal/Delphi, about python, and most frustratingly (https://www.thefreedictionary.com/frustratingly) - ahk! I had to revise all the scripts from the scratch several times. It was either because of the concepts i was developing along the way, or IMDb and FilmAffinity were changing their layout. For example, just yesterday new Chrome version was brought, and my chromedriver didn't work anymore, so I had to download new version of it too. Also, just 2 days ago I learned that FilmAffinity introduced their AKA for some movies, so I had to update FA script again. And so on and on for 4 months. Thus, I learned to appreciate it. The most important, now I even more appreciate EasyVVV's, and especially Ivek's work for more than a decade (!!!) to provide us with PVD alive!
So, humbly, I dedicate this hard work to EasyVVV, but most, and before anyone else I dedicate this to IVEK and to memory to his late mother! I HOPE IVEK CAN IMAGINE HOW GRATEFUL TO HIM I PERSONALLY AM, AFTER I REALIZED HOW HARD THIS ALL WAS! GOD BLESS YOU IVEK!
Last version in this message on this topic
2026-01-06: https://www.videodb.info/forum_en/index.php/topic,4379.msg23024.html#msg23024
-
As i wrote, I completely rewrote Sripts configurator from the scratch practically in order to be able to bring all the options to automate scripts behaviour. It now is resizable and has scrollbars, so we can put there as much as we wish options from the scripts.
Also, the most of the data can be imported if you check and uncheck options as in the screenshots below. Feel free to test though. Don't be afraid! You can't mess your database, because backup files are created anyway!
READ THE MESSAGES UPON CLICKING "SAVE" BUTTON IN CONFIGURATOR. THOSE MESSAGES ARE VERY INFORMATIVE, HELPFUL AND ESSENTIAL TO UNDERSTAND WHAT IS GOING ON.
-
Due to the post limit, in this message, Scripts Configurator window shrinked as a proof of concept, with horizontal and vertical scrollbars. I haven't had no idea how huge challenge was to get scrollbars to this. AHK is pretty hard to get such feature and I lost at least 2 weeks to get it proper, so maybe in the future I will build this with python. I already tested it.
-
IMPORTANT!!!
A few hours ago, IMDb completely changed /fullcredits page html layout, so that page doesn't work any more. Soon, /reference page will be changed too. I know because I got popups offering me to peek to a new "Reference" page. So, until that happen, I will not update scripts, because both pages will share the same code again, and it will be easier to change. For now I made a quick fix everything to work if you check the options in Configurator as I suggested earlier. In addition you have to check "Download the Cast or Credit (text only) provider page to retrieve the full information. Or else, only the info from the main movie page will be downloaded." option and to download fullcredits page too!!! This should work until /reference page changes, or any other page changes meanwhile.
And it happened just when I finished ''all-in one script" while successfully doing final tests. Here's the pack.
So, with one IMDb Script you get all movies, Series, episode list, and then you apply the same script for episodes.
Also, new search window introduced, with different types of search and countdown of 10 seconds defaulted to "general" search.
It took only 600 additional lines comparing to Movie script, including a lot of commented out lines, and one simple python script to get all of this.
Extract and overwrite existing scripts with this pack.
I will soon start to re-birth AllMovie and RottentTomatoes scripts. I will not revive any other script.
-
Hello! I'm trying to make it work with this new system, but when I go to scripts.7z Windows flags SeleniumPVDbScriptsConfig-v4.exe as a threat and won't let me unzip
-
Hello! I'm trying to make it work with this new system, but when I go to scripts.7z Windows flags SeleniumPVDbScriptsConfig-v4.exe as a threat and won't let me unzip
You probably mean the anti-virus program. It's a known issue, from my experiences with the NOD32 Antivirus program it was the same. The solution was when the antivirus program quarantined the SeleniumPVDbScriptsConfig-v4.exe file, I put it back out of quarantine and excluded it from being scanned again by the antivirus program. If necessary, I always repeated the described procedure and thus prevented it from being quarantined again by the antivirus program.
If it still won't be possible to unpack the file, let me know and I'll help you do it another way.
And this notice.
You need the SeleniumPVDbScriptsConfig-v4.exe file to be able to configure the script for transferring information.
It is very likely that the IMDb scripts will not work.
-
Soon, /reference page will be changed too. I know because I got popups offering me to peek to a new "Reference" page. So, until that happen, I will not update scripts, because both pages will share the same code again, and it will be easier to change.
Hello to all. As I already said in March, now we all know that this happened. I was busy meanwhile and had no time to deal with it. There are changes across many imdb pages, but reference page is the culprit. I will need couple of months to fix it, since you all know that I am not a programer, and I have to remind my self about everything, especially about special cases. For now, you can pull up significant portion of data without reference page (unfortunately, not the whole cast). To do that and if you are using my scripts, open Script configurator, uncheck to download Reference page and PVD will restart. Then, in "Set Overwrite options...", check all the data you would want to download (Studio, etc..) and restart PVD. Then run the script and this would be what you can get at the moment.
After March, I had some minor adjustments to the scripts and started to work on a Reference page, so I'm uploading them so you could get the same amount data as me. Backup your existing scripts, then overwrite them with these, all to "Scripts" folder. Reminder: these scripts are just starting point to fix them, but they should get you more data comparing to March scripts could get you now, so please do not ask for the support for these scripts. I know they don't fully work.
What should be promising for you is that I'm not planning to abandon using PVD, so I will for sure fix the scripts at some point, so please be patient: if long time no see me, that just means I also haven't fixed the scripts, and I'm working on them. Meanwhile, just add your movies to PVD and later you will update with full data. That is exactly what I've been doing recently.
Best regards
-
IMDB_Movie_[EN][Selenium]-v4.psf
Between lines 138 and 139 is this:
GET_FULL_CONNECTIONS = False ; //To call Function ParsePage_IMDBMovieCONNECTIONS. Not used for parsing, since it is enough to choose both SIMPLE and COMPLEX connections. --> to "True".
Between lines 7992 and 7993 is this:
If GET_FULL_CONNECTIONS Then
Begin
However, when you run the ** SeleniumPVDbScriptsConfig v4 ** script in PVD and uncheck all Connections settings and then restart PVD, the following happens.
IMDB_Movie_[EN][Selenium]-v4.psf does not work.
The following parts of the code are missing.
Between lines 138 and 139 is missing:
GET_FULL_CONNECTIONS = False ; //To call Function ParsePage_IMDBMovieCONNECTIONS. Not used for parsing, since it is enough to choose both SIMPLE and COMPLEX connections. --> to "True".
However, between lines 7992 and 7993, this
[code] If GET_FULL_CONNECTIONS Then
Begin[/code]
changes to this:
If GET_FULL_CONNECTIONS = False ;
Add the following code to the script
//(*
function CustomStringReplace(const Source: string; const OldPattern: array of string; const NewPattern: array of string): string;
var
i: Integer;
ResultString: string;
begin
ResultString := Source;
for i := Low(OldPattern) to High(OldPattern) do
begin
ResultString := StringReplace(ResultString, OldPattern[i], NewPattern[i], True, False, True);
end;
Result := ResultString;
end;
//*)
and then you can add the following code to the Metascore code.
ItemValue := CustomStringReplace(ItemValue, ['0</', '1</', '2</', '3</', '4</', '5</', '6</', '7</', '8</', '9</'], [',0', ',1', ',2', ',3', ',4', ',5', ',6', ',7', ',8', ',9']);
//ItemValue := StringReplace(ItemValue, '0</', '.0', True, False, True);
//ItemValue := StringReplace(ItemValue, '1</', '.1', True, False, True);
//ItemValue := StringReplace(ItemValue, '2</', '.2', True, False, True);
//ItemValue := StringReplace(ItemValue, '3</', '.3', True, False, True);
//ItemValue := StringReplace(ItemValue, '4</', '.4', True, False, True);
//ItemValue := StringReplace(ItemValue, '5</', '.5', True, False, True);
//ItemValue := StringReplace(ItemValue, '6</', '.6', True, False, True);
//ItemValue := StringReplace(ItemValue, '7</', '.7', True, False, True);
//ItemValue := StringReplace(ItemValue, '8</', '.8', True, False, True);
//ItemValue := StringReplace(ItemValue, '9</', '.9', True, False, True);
The same can be done with this.
//Date ~Updated~ (choose simple or verbose version)
Date := DateToStr(CurrentDateTime);
//AddFieldValueXML('viewdate', Date); //Only date, don't admit time-. Set Seen value at the same time.
//AddFieldValueXML('moddate', Date + ' ' + TimeToStr(CurrentDateTime)); //Block the dB saving
ExplodeString(Date, DateParts, '-');
Date:=DateParts[2]+'.'+ DateParts[1]+'.'+DateParts[0];
Date := CustomStringReplace(Date, ['01.', '02.', '03.', '04.', '05.', '06.', '07.', '08.', '09.'], ['1.', '2.', '3.', '4.', '5.', '6.', '7.', '8.', '9.']);
// Date := StringReplace(Date, '01.', '1.', True, True, False);
//Date := StringReplace(Date, '02.', '2.', True, True, False);
//Date := StringReplace(Date, '03.', '3.', True, True, False);
//Date := StringReplace(Date, '04.', '4.', True, True, False);
//Date := StringReplace(Date, '05.', '5.', True, True, False);
//Date := StringReplace(Date, '06.', '6.', True, True, False);
//Date := StringReplace(Date, '07.', '7.', True, True, False);
//Date := StringReplace(Date, '08.', '8.', True, True, False);
//Date := StringReplace(Date, '09.', '9.', True, True, False);
//AddCustomFieldValueByName('Updated', Date); // (Left for FA Script)
//AddCustomFieldValueByName('Updated0', Date + ' at ' + TimeToStr(CurrentDateTime)); // Saved for RottenTomatoes
AddCustomFieldValueByName('IUpdated', Date + ' at ' + TimeToStr(CurrentDateTime) + ' • ' + SCRIPT_FILE_NAME + ' ' + SCRIPT_VERSION); // Annoying
LogMessage('Function ParsePage - Provider data info retreived Ok in ' + DateToStr(CurrentDateTime) + ' ' + TimeToStr(CurrentDateTime) + '| (~Updated~)');
Mode := smFinished;
LogMessage('Function ParsePage smNormal END====================== |');
Exit;
End;
-
You need the latest chromedriver.exe, otherwise it won't work.
And this tip.
For the Episode List page, it would be necessary and even better if a script was made specifically for it.
-
For the Episode List page, it would be necessary and even better if a script was made specifically for it.
Episode list works totally fine for me, even now. It would be probably impossible for me to create new script. We can look at it like this: Episode list has it's own script: it's Selenium script "Selenium_Chrome_IMDB_Episode_List_page_v4.py" which produces txt file with all episodes, and .psf script just scrapes that txt file, so I am not sure what we could achieve with additional psf script whose purpose would be only to scrape txt file.
Anyway, I have corrected FilmAffinity script, and made some improvements in IMDb Movie scripts (for example, storyline section dynamic load scraping is fixed). I have improved FilmAffinity script speed enormously! In order this script to work properly and fast you need:
1. To install python 3.12+
2. in a cmd to install psutil with
pip install psutil
in order to hopefully prevent selenium hangups when html elements aren't found on the page.
3. To download and overwrite scripts I'm uploading in this post.
4. In a script configurator to deselect Reference page. PVD will restart. Then select "Studio" and "Description" and all others you want and do not restart PVD. Try to import data, and now you should get director, cast, tagline and some other original and related custom fields scraped from the Main page. If you don't get cast, tagline and director, restart PVD manually and try again to import. I am not sure about this second restart, so try both. One will work for sure.
I will not provide support for these scripts until I finish, because I know they still don't fully work. I just want to share with you same amount of data I'm getting at the moment when some significant improvement is done. In a pictures you can get the sense of what i'm getting now with IMDB script.
Once I finish IMDb script we will test and correct it together.
-
For the Episode List page, it would be necessary and even better if a script was made specifically for it.
Episode list works totally fine for me, even now. It would be probably impossible for me to create new script. We can look at it like this: Episode list has it's own script: it's Selenium script "Selenium_Chrome_IMDB_Episode_List_page_v4.py" which produces txt file with all episodes, and .psf script just scrapes that txt file, so I am not sure what we could achieve with additional psf script whose purpose would be only to scrape txt file.
We'll see how it works now. Before these fixes, there was a problem with the Episode List code for the movie title and the PVD froze, but if the code in the script was blocked and the episode field was unchecked, it worked without any problems.
-
I have optimized Selenium_Chrome_Base_page_v4.py script so now downloading of the IMDb main page should be dramatically faster - around 18 seconds on my computer.
-
I have optimized Selenium_Chrome_Base_page_v4.py script so now downloading of the IMDb main page should be dramatically faster - around 18 seconds on my computer.
The Selenium_Chrome_Base_page_v4.py script works fine. However, it doesn't work at all with Firefox and Geckodriver options.
Changed the path for Geckodriver in Selenium_Chrome_Base_page_v4.py for Firefox script in 7.z file.
-
Great. I have fixed Movie Connections, and full AKA. Also fixed Selenium_Chrome_Movie_Additional_pages_v4.py so now it clicks on "See more" and similar objects, again. I will not upload next iterations because I want to fix whole Base page function first, and for me it looks that IMDb makes it harder and harder to scrape data, since for now, I'm often seeing errors IMDb killing connections to host when loading additional pages with selenium, so I have to investigate selenium and python options to mimic human browsing as best as possible.
If you still want to have current versions of my scripts, let me know so I could upload them with the manual how to get data I'm getting at the moment.
-
I have optimized Selenium_Chrome_Base_page_v4.py script so now downloading of the IMDb main page should be dramatically faster - around 18 seconds on my computer.
The Selenium_Chrome_Base_page_v4.py script works fine. However, it doesn't work at all with Firefox and Geckodriver options.
The Selenium_Chrome_Base_page_v4.py script, which was before the latest update, works on Firefox and Geckodriver options.
-
Oh, sorry to hear. I even didn't know it worked earlier and how it could work since I never published geckodriver version??? Probably I don't understand the context of your message.
-
Oh, sorry to hear. I even didn't know it worked earlier and how it could work since I never published geckodriver version??? Probably I don't understand the context of your message.
I changed all parts of the code where there is a Chrome and Chromedriver options record to Firefox and Geckodriver options record and the Selenium_Chrome_Base_page_v4.py script also worked using the Firefox browser.
As I mentioned before, with the latest Selenium_Chrome_Base_page_v4.py script update and the Chrome and Chromedriver options record, I changed it to Firefox and Geckodriver options record, but the Selenium_Chrome_Base_page_v4.py script no longer works at all using the Firefox browser.
We already talked about this a while ago in another PVD Python Scripts topic.
-
Maybe Selenium_Chrome_Movie_Additional_pages_v4.py will be loaded to see how it works.
I have optimized Selenium_Chrome_Base_page_v4.py script so now downloading of the IMDb main page should be dramatically faster - around 18 seconds on my computer.
The Selenium_Chrome_Base_page_v4.py script works fine. However, it doesn't work at all with Firefox and Geckodriver options.
I managed to edit the Selenium_Chrome_Base_page_v4.py script with Firefox and Geckodriver options using AI and it works now.
I would then do the same in the Selenium_Chrome_Movie_Additional_pages_v4.py script.
-
Great. I couldn't tell what it could be, but if I had to guess I'd say it was that new chrome.options still weren't implemented in geckodriver.
Where am I at the point?
Now I'm left with /fullcredits and /reference page. But... After reviewing new /reference design it became totaly pointless to download and parse it. It now simply doesn't have anything more than other pages already being downloaded, except full "Production Companies" and "Distributors". So, I will probably implement CompanyCredits page which is tiny and much faster to download in order to get full "Production Companies" and "Distributors". That will also drastically simplify the code.
Another important change is that I introduced new error MessageBox as seen in the last screenshot, because IMDB makes it harder and harder to fetch data in a non-human way, with Selenium or so. Read the message on it to know what it means to press any button.
More on everything in the next message.
-
Meanwhile, I have made huge fixes and improvements on all other files/scripts. Especially Main page, so now a lot of new data is pulled from the Main page which is now downloaded much faster. I encourage you to try it, by selecting fields in Script Configurator as in the screenshots, waiting PVD to restart, and then to select to overwrite all fields like in the other screenshot, and immediately start to download.
One of the biggest changes is that now I have created a procedure that enables poster to be downloaded from any page. For example, if you choose only "AKA" page to download (to update only aka's for example) you can download poster with that function too, now. Just be sure to select "Download Posters" in the Script Configurator.
I have updated and recompiled a Script Configurator with minor changes too. You need to replace all the files with the given in the attachment, and they all go only to /Scripts folder as usual.
Here's from the CHANGE LOGs of the IMDB and FilmAffinity scripts what I did:
IMDb Script
---------------------------------------------
CHANGE LOG :
V 4.1.0.1 (5/11/2025) afrocuban
- Procedure EnsurePosterDownloaded is introduced in order to be able to download poster from any page.
- Fixed pages due to change layout.
- Improved ParsePage_IMDBMovieBASE function.
- Introduced revised Messagebox now includes Cancel, Retry, or Continue (Ignore). NOTE: IF YOU PRESS IGNORE YOU WILL NOT GET DATA FROM THAT PAGE, SO CONSIDER TO RETRY OR TO CANCEL AND START DOWNLOAD AGAIN! IMDb really makes it harder and harder to get the data.
- Script Configurator decriptions adjusted to reflect actual processes.
- In corresponding Selenium scripts significantly improved downloading pages speed.
- In corresponding Selenium scripts fixed searching titles.
FilmAffinity Script
CHANGE LOG :
V 4.1.0.1-afrocuban (11/5/2025) afrocuban:
- Backup CHEAT_PREFIX_URLs introduced, since httpbin.org now almost always CREATES "503 Service Temporarily Unavailable" PROBLEMS FOR FILMAFFINITY RECENTLY. At the moment, the one that works is 'http://httpbingo.org/response-headers?key= (http://httpbingo.org/response-headers?key=)'
(*@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
// IF NON OF THESE WORK THEN THERE ARE NO OTHER SERVICES AVAILABLE AND THE ONLY OPTION LEFT IS TO INSTALL LOCAL HTTPBIN LIKE THIS
Install and run httpbin locally like this:
```bash
pip install httpbin
python -m httpbin.core
```
It will start a local server on port `5000`, and you can absolutely use:
```
http://localhost:5000/response-headers?key=https://www.filmaffinity.com/en/film699169.html/ (http://localhost:5000/response-headers?key=https://www.filmaffinity.com/en/film699169.html/)
```
Just like you would with the public httpbin.org — and it will behave the same way: echoing back the `key` header in the response.
---
### ? What to expect
When you visit that URL in your browser or send a request via code, you’ll get a JSON response like:
```json
{
"key": "https://www.filmaffinity.com/en/film699169.html/ (https://www.filmaffinity.com/en/film699169.html/)"
}
{
```
No HTTPS redirection, no 503 errors, and full control — because it’s running locally.
---
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
- Fixed "FA Critics" field due to layout changes.
- Fixed "FAMovieTrailers" field to now properly include source name (youtube, dailymotion, etc).
- Other layout changes fixed.
- In corresponding Selenium scripts significantly improved downloading pages speed, especially Trailers page..
- In corresponding Selenium scripts fixed searching titles due to FilmAffinity Search page layout changed. (Movie title now has to be retrieved from the section for mobile devices).
- Other minor layout changes updated, and the script cleaned additionaly.
So with these scripts, you can get all possible data for now except:
- Full Cast (for now you can get only cast found on the main page)
- Producers
- Composers
- Full Production Companies
- Distributors
- All Directors for Series (only one director for now if you download /fullcredits page, because I have just started to fix it).
I will not provide support for these scripts until I finish all because I know IMDB script still doesn't fully work. I am interested in you to test border case titles which I'd fix upon finishing. For that, please provide the link and the log for specific field (or what you get in the field and what you expect). I will be able to fix things only if I can reproduce them and for that you need to provide me with data above.
-
I forgot to upload ahk for the Script Configurator .exe
SeleniumPVDbScriptsConfig-v4.exe is compiled with ahk2exe for AutoHotkey v1.1.37.02 option "U32 (default) bin", without compression.
-
I have finished CompanyCredits function and now no need for Reference page at all. Consequently, I have updated, redesigned and compiled Script Configurator.
Now I need to update FullCredits function, meaning to get full cast & crew, directors for series, producers and composers and everything will be done. I will not clean Reference page from the code. I will only comment it out, so it could be possibly used in the future IMDb page changes.
Most probably I will not post again until finishing everything.
-
I have completed FullCredits function, so now all the data can be imported again to PVD. Before I publish it I want to check 2 things:
1. What to do with Reference page code, and should I exclude its options from Script Configurator, since now it is completely not needed.
2. To check People script and if I can fix it quickly, then I will publish all the scripts and files again in one, final package for this IMDb html layout change.
After that please check the scripts and let me know what doesn't work
-
I have completed FullCredits function, so now all the data can be imported again to PVD. Before I publish it I want to check 2 things:
1. What to do with Reference page code, and should I exclude its options from Script Configurator, since now it is completely not needed.
2. To check People script and if I can fix it quickly, then I will publish all the scripts and files again in one, final package for this IMDb html layout change.
After that please check the scripts and let me know what doesn't work
First of all, Function ParsePage_IMDBMovieREFERENCE should be moved to the end of the script before Function ParsePage.
Secondly, the entire part of the reference page code should be left and, as mentioned above, before Function ParsePage, or even better, it should be moved to the very end of the script, where there is already a History of changes, so that it can be re-included in the script if necessary in the future or completely blocked. All its options should also be excluded from both the script code and the script configurator, because now, as mentioned, it is no longer needed.
As for the People script, I don't know what works or doesn't work, because I don't use it.
-
Yes, I just commented out reference code in the script, and I already intended to leave it there and everywhere it is mentioned, to preserve the logic for the future. I was actually talking to comment it out in the Script Configurator too, to avoid inexperienced users to click on those options expecting to get data from /reference page. I will move the function before ParsePage function, though, that is a nice tip.
Now, meanwhile, I introduced another function:
function WaitForPageFile(const FileName, PageLabel: string;
InitialWait: Integer; StabilizeMs: Integer): string; //BlockOpen
var
i, currentWait: Integer;
tryResult: string;
begin
Result := '';
i := 0;
// provide defaults manually
if InitialWait = 0 then
InitialWait := 2000;
if StabilizeMs = 0 then
StabilizeMs := 1000;
currentWait := InitialWait;
while not FileExists(FileName) do
begin
LogMessage('Function DownloadPage - Waiting ' + IntToStr(currentWait div 1000) +
's for the presence of: ' + FileName + '|');
Wait(currentWait);
// escalate wait times
case i of
0: currentWait := 5000;
1: currentWait := 15000;
2: currentWait := 10000;
3: currentWait := 10000;
4: currentWait := 15000;
5: currentWait := 15000;
end;
Inc(i);
if i = INTERNET_TEST_ITERATIONS then
begin
case MessageBox('IMDb Movie Function DownloadPage - Too many faulty attempts to internet connection for ' + PageLabel +
'. Cancel, Retry, or Continue (Ignore)? NOTE: IF YOU PRESS IGNORE YOU WILL NOT GET DATA FROM THAT PAGE, SO CONSIDER TO RETRY OR TO CANCEL AND START DOWNLOAD AGAIN! IMDb really makes it harder and harder to get the data.',
SCRIPT_FILE_NAME, 2) of
3: // Cancel
begin
LogMessage('Function DownloadPage for ' + PageLabel +
' ended with NO INTERNET connection ===============|');
Result := '';
Exit;
end;
4: // Retry
begin
i := 0;
currentWait := InitialWait;
end;
5: // Ignore
begin
LogMessage('Function DownloadPage - Creating dummy ' + PageLabel +
' HTML file due to Ignore selection|');
with TStringList.Create do
try
Add('<html><body>Dummy ' + PageLabel + ' due to user Ignore.</body></html>');
SaveToFile(FileName);
finally
Free;
end;
Break;
end;
end;
end;
end;
// stabilization wait
LogMessage('Function DownloadPage - ' + PageLabel +
' file detected, waiting extra ' + IntToStr(StabilizeMs) + 'ms to stabilize...');
Wait(StabilizeMs);
// manual error handling instead of try/except
if not FileExists(FileName) then
begin
LogMessage(ProcException('FileError', 'NOT_FOUND: ' + FileName));
Exit;
end;
tryResult := FileToString(FileName); // if your environment throws, replace with safe read
if tryResult = '' then
LogMessage(ProcException('FileError', 'FAILED to read ' + FileName))
else
begin
Result := ConvertEncoding(tryResult, 65001);
LogMessage(ProcException('FileRead', 'SUCCESS: ' + FileName));
end;
end; //BlockClose
so, now instead this snippet for every page:
// Initialize currentWait for the FullCredits file
currentWait := 5000; // Start with 5 seconds
//Wait for the FullCredits file to finish downloading
If Not(((USE_SAVED_PVDCONFIG) And ((ConfigOptions[8] = '0') And (ConfigOptions[10] = '0'))) Or
(GET_FULL_CREDIT_FROM_REFERENCE And ((Pos('Series', MediaType) = 0) And (Pos('Series', GetFieldValueXML('category')) = 0)))) Then Begin // Also Known As (FullCredits)
i := 0;
currentWait := 2000; // Initialize wait time
while not FileExists(FilePath + FileTitleFullCredits) do begin
LogMessage('Function DownloadPage - Waiting ' + IntToStr(currentWait div 1000) + 's (because of the people like Alfonso Cuaron with 268 wins and 208 nominations at the moment of writing this script) for the presence of: ' + FilePath + FileTitleFullCredits);
wait(currentWait);
// Increment the wait time for the next iteration
case i of
0: currentWait := 5000; // 5 seconds
1: currentWait := 15000; // 15 seconds
2: currentWait := 10000; // 10 seconds
3: currentWait := 10000; // 10 seconds
4: currentWait := 15000; // 15 seconds
5: currentWait := 15000; // 15 seconds
end;
i := i + 1;
if i = INTERNET_TEST_ITERATIONS then
begin
case MessageBox('IMDb Movie Function DownloadPage - Too many faulty attempts to internet connection for FullCredits. Cancel, Retry, or Continue (Ignore)? NOTE: IF YOU PRESS IGNORE YOU WILL NOT GET DATA FROM THAT PAGE, SO CONSIDER TO RETRY OR TO CANCEL AND START DOWNLOAD AGAIN! IMDb really makes it harder and harder to get the data.', SCRIPT_FILE_NAME, 2) of
3: // Abort -> treat as Cancel
begin
LogMessage('Function DownloadPage for FullCredits END with NO INTERNET connection =============== |');
Result := '';
Exit;
end;
4: // Retry
begin
i := 0;
currentWait := 2000; // Reset wait time
end;
5: // Ignore->create dummy file
begin
LogMessage('Creating dummy FullCredits HTML file due to Ignore selection...');
with TStringList.Create do
try
Add('<html><body>Dummy FullCredits due to user Ignore.</body></html>');
SaveToFile(FilePath + FileTitleFullCredits);
finally
Free;
end;
Result := FilePath + FileTitleFullCredits;
end;
end;
end;
end;
// Add a short stabilization wait after the file is recognized
LogMessage('Function DownloadPage - CHANGETHISWITHPROPERFILENAME file detected, waiting extra 1s to stabilize...');
Wait(2000); // wait 2 second (adjust as needed)
WebText := FileToString(FilePath + FileTitleFullCredits);
WebText := ConvertEncoding(WebText, 65001); // UTF-8
FullCreditsPageDownloaded := True;
LogMessage('Function DownloadPage - FullCredits file found: ' + FilePath + FileTitleFullCredits);
LogMessage('Value of FullCreditsPageDownloaded: ' + BoolToStr(FullCreditsPageDownloaded));
end;
we will have only this
if not (USE_SAVED_PVDCONFIG and (ConfigOptions[15] = '0')) then
begin
WebText := WaitForPageFile(FilePath + FileTitleParentalGuide, 'ParentalGuide', 5000, 2000);
ParentalGuidePageDownloaded := WebText <> '';
LogMessage('Function DownloadPage - Value of ParentalGuidePageDownloaded: ' +
BoolToStr(ParentalGuidePageDownloaded));
end;
That way we will speed up the script and have hundreds if not thousands of lines less in the script.
I will do that and test for all repetitive tasks.
Also, I'm exploring ways IMDb not to refuse connections with selenium/python and testing at the moment creating fake userData folders, rotating user agents, and it goes good for now.
-
As of now, per title, I'm getting these timings:
Main Page downloading: ~18-22sec
Other 9 pages Page downloading: ~48-55 sec
Image downloading: ~6-8 sec
PVD script processing: ~5-10 sec
So in total for now, to get biggest possible amount of data per title, especially those with many awards, connections, etc... it is needed ~77-95 sec which is pretty acceptable for the amount of data.
-
Oh, and please remind me to upload purge_tmp_files.vbs if I forget, because I changed it by adding to delete fake UserData folders now too
-
Oh, and please remind me to upload purge_tmp_files.vbs if I forget, because I changed it by adding to delete fake UserData folders now too
Ok.
-
Here are all scripts and files fully updated, fixed and polished in a less than a month I started to fix all 16 of them, and I was so happy I got back into it easily and quickly. I have tested all scripts and files against many border case titles and persons and for me everything worked more than smooth and satisfying.
They are now faster and more stable and I am not facing anymore internet interruptions, because I heavily redesigned the most problematic python selenium scripts.
If especially Selenium_Chrome_Movie_Additional_pages_v4.py script is demanding for your CPU when downloading movies and you experience lags of any kind, open the file in Notepad++ and in the line 375:
with ThreadPoolExecutor(max_workers=4) as executor:
reduce number 4 to 3, 2 or 1, just test it. Whenever you lessen the number, the process of downloading files will be longer, so find your balance. If you have good CPU and a lot of RAM, then you can even increase the number above 4.
I'd be happy to further fine tuning and fix it, so please let me know about each case details so I could reproduce it too and then being able to fix it. If you have any further suggestion, I'd be happy to hear it as well while I didn't forget it again, but please explain why and how by giving specific examples, because I am not a programmer, but just using common sense and AI, and that is the only way I can understand the problem.
My plan is versions to stay on v4.2 for a long time unless something significant in their design changes.
Enjoy! ;) :)
-
My next goal is to include new switch in the Script Configurator - UPDATE_DYNAMIC_VALUES_ONLY, by adding few dozens of lines into movie selenium script that would call only main page and update only dynamic values like: Rating, Top 250, Bottom 100, Number of votes. And for the Awards summary when the movie is not older than 2 years than current date catching fresh wins for recent releases.
-
My next goal is to include new switch in the Script Configurator - UPDATE_DYNAMIC_VALUES_ONLY, by adding few dozens of lines into movie selenium script that would call only main page and update only dynamic values like: Rating, Top 250, Bottom 100, Number of votes. And for the Awards summary when the movie is not older than 2 years than current date catching fresh wins for recent releases.
Well, I finished it earlier than expected, for IMDb Movie script. Also with a lot of chalenges I have redesigned Script Configurator once again, bringing new functionalities:
//Retreive Data Config
USE_SAVED_PVDCONFIG = True ; // ***PVDCONFIG*** - Turn this ON to unlock and change the options below (from pvdconf.ini). Settings are applied when you click "Save All Script Configurations (Personal Video Database will automatically restart)" button below. Use carefully!
//############################################
//# All options below require USE_SAVED_PVDCONFIG
//# to be enabled so the Script Configurator
//# can apply your settings correctly.
//############################################
UPDATE_DYNAMIC_VALUES_ONLY = True ; //Update only dynamic values such as: Rating, Top 250, Metascore, Number of votes. Also update the Awards summary when the movie is less than 2 years old, to capture fresh wins for recent releases. Deselect to enable the options
//################################################
//# All options below require UPDATE_DYNAMIC_VALUES_ONLY
//# to be enabled so the Script Configurator
//# can apply your settings correctly.
//###############################################
So you can see in the screenshots that now checking specific boxes disables or enables other options. Which means that....
My plan is versions to stay on v4.2 for a long time unless something significant in their design changes.
my plan will not last long since thee changes are huge for users to make them easy navigating and choosing proper options withoout to much contemplating, so with this I will soon go to 4.3.
But I still will not publish anything, because I want to finish People and FA scripts in terms of UPDATE_DYNAMIC_VALUES_ONLY, and I also want to further tweak Script Configurator GUI. I just hate "Save button" will not autosize to the last option in each tab, so for example in a People and FA tab we have to scroll all the way down. That is not just visual thing, but I rather want to implement "Apply" button that will be applied to each tab independently, while we will have overall "Cancel" and "Save & Restart PVD" button. That is tremendous challenge for ahk, that made me last year even to start creating GUI with python, but at the moment it looked even more difficult with python, so I abandoned it then. Now it looks the time to try it again is spot on.
-
New v4.3 Files
Most comprehensive and stable I have done so far. Two main things:
1. New PVD Scripts Configurator built from scratch in Python Tkinter. Much better GUI than AHK.
Nice quirk - I have introduced dark/light theme for it as in the first photo. Reordered tabs, so default tab is IMDb Movie tab. You can still use both configurators (second photo - new Configurator has prefix "py"), all updated to v4.3, same options and functionalities. I'm giving them all now, for continuity, but in the future, most probably I will discontinue AHK. In any case, I'm giving .ahk file so anyone can maintain it in the future. I will stick to Python TKinter GUI.
2. As anounced earlier, new **UPDATE DYNAMIC VALUES ONLY** switch for a fast update of only certain dynamic fields.
From the Change logs:
IMDb Movie Script
CHANGE LOG :
V 4.3.0.1 (11/15/2025) afrocuban: Script Configurator Enhancements
-------------------------------------------------------------------------------
- Built from the scratch new PythonTkinter Script Conigurator:
• It has all the functionalities as AHK.
• Plus litght/dark theme developed
• Unlike AHK can be used as a standalone application with the same effect as when invoked in PVD.
- Added new feature in Script Configurator to enable and manage saved settings from `pvdconf.ini`:
• **USE SAVED PVDCONFIG** now needs to be enabled to unlock configuration options below.
• This allows users to apply settings that require a restart of Personal Video Database upon saving.
• **Use this setting carefully!** Any changes will take effect only after clicking "Save All Script Configurations" (which will restart the application).
- **UPDATE DYNAMIC VALUES_ONLY**:
• Allows users to update only **dynamic values** like Rating, Top 250, Metascore, and Number of votes.
• Updates the **Awards summary** for movies released within the last two years, capturing recent wins for fresh releases.
• When disabled, additional configuration options become available for comprehensive updates.
• Now poster can be downloaded from any page - separate procedure provided for it in Script Configurator,
• Single Instance in Script Configurator. No more flooding with multiple instance by mistake.
• Redesigned whole script to now accept UPDATE DYNAMIC VALUES ONLY switch properly
FilmaFfinity ScriptCHANGE LOG :
V 4.3.0.1 (11/27/2025) afrocuban: Script Configurator Enhancements
-------------------------------------------------------------------------------
- Built from the scratch new PythonTkinter Script Conigurator:
• It has all the functionalities as AHK.
• Plus litght/dark theme developed
• Unlike AHK can be used as a standalone application with the same effect as when invoked in PVD.
- Added new feature in Script Configurator to enable and manage saved settings from `pvdconf.ini`:
• **USE SAVED PVDCONFIG** now needs to be enabled to unlock configuration options below.
• This allows users to apply settings that require a restart of Personal Video Database upon saving.
• **Use this setting carefully!** Any changes will take effect only after clicking "Save All Script Configurations" (which will restart the application).
- **UPDATE DYNAMIC VALUES_ONLY**:
• Allows users to update only **dynamic values** like Rating, Number of votes and Awards for movies made in last 2 years.
• When disabled, additional configuration options become available for comprehensive updates.
IMDb People Script
CHANGE LOG :
V 4.3.0.1 (11/27/2025) afrocuban: Script Configurator Enhancements
-------------------------------------------------------------------------------
- Built from the scratch new PythonTkinter Script Conigurator:
• It has all the functionalities as AHK.
• Plus litght/dark theme developed
• Unlike AHK can be used as a standalone application with the same effect as when invoked in PVD.
- Added new feature in Script Configurator to enable and manage saved settings from `pvdconf.ini`:
• **USE SAVED PVDCONFIG** now needs to be enabled to unlock configuration options below.
• This allows users to apply settings that require a restart of Personal Video Database upon saving.
• **Use this setting carefully!** Any changes will take effect only after clicking "Save" (which will restart the application).
- **UPDATE DYNAMIC VALUES_ONLY**:
• Allows users to update only **dynamic values** for persons that were alive at the moment of adding them to PVD, or at their last update. Updating only from the Main page.
• When disabled, additional configuration options become available for comprehensive updates.
-
Here are v4.3 scripts.
1. Unpack the first 7z in "PersonalVideoDB" folder (3 files, overwrite existing, but backup them first if you want).
2. Unpack the second 7z in your "Scripts" folder. It is safe to move everything from it before extracting. These are all you need to safely run PVD with python.
You need all of these in order PVD to run as intended. Especially People script is complex, since I have integrated options to make it easier to dynamically update them and not to wait at all for deceased or the people that have only name and url. Test it.
If something desn't work, first check:
1. That you are running same version of Chrome and chromedriver.
2. That you installed whatever is needed for Python to work as described so far on this topic.
If that doesn't help, please publish screenshots and logs, so I could reproduce the issue too and being able to fix it.
Please test and let me know if everything work or not.
Enjoy!
-
I tested it, IMDB_Movie_[EN][Selenium]-v4.psf and IMDB_People_[EN][Selenium]-v4.psf work fine, with some errors in certain information, where it will be necessary to fix parts of the code in the scripts. The errors have been there for a long time and have not been fixed. The code fixes will be described and added below.
As for FilmAffinity_Movie_[EN][Selenium]-v4.psf, I have not tested it.
-
IMDB_Movie_[EN][Selenium]-v4.psf
Fixed part of the code in Function ParsePage_IMDBMovieMPAA
Previous
// Get (CF~IMDbmpaaSummary~)
curPos := Pos('<select id="jump-to"', HTML);
If curPos > 0 Then
Begin
// Extract the relevant section for categories
curPos := PosFrom('<option', HTML, curPos);
endPos := PosFrom('</select>', HTML, curPos);
mpaaSection := Copy(HTML, curPos, endPos - curPos);
LogMessage('Function ParsePage_IMDBMovieMPAA - Extracted Category section for (CF~IMDbmpaaSummary~): ' + mpaaSection);
// Parse the options and category names
curPos := 1;
mpaaContent := '';
While PosFrom('<option', mpaaSection, curPos) > 0 Do
Begin
curPos := PosFrom('<option', mpaaSection, curPos) + Length('<option');
optionValue := TextBetween(mpaaSection, 'value="', '">', False, curPos);
categoryName := TextBetween(mpaaSection, '">', '</option>', False, curPos);
// Format the category link
mpaaContent := mpaaContent + '<link url="' + MovieURL + optionValue + '">' + categoryName + ' • </link>';
// Move to the next position
curPos := PosFrom('</option>', mpaaSection, curPos) + Length('</option>');
End;
// Remove the trailing " • " if it exists and keep the last </link> intact
If Copy(mpaaContent, Length(mpaaContent) - 13, 7) = ' • ' Then
Begin
mpaaContent := Copy(mpaaContent, 1, Length(mpaaContent) - 14) + '</link>';
End;
// Combine and format the final result
mpaaContent := '<link url="' + MovieURL + '#contentRating' + '">Content Ratings Summary: </link>' + mpaaContent;
// Store the result in the custom field
AddCustomFieldValueByName('IMDbmpaaSummary', mpaaContent);
LogMessage('Function ParsePage_IMDBMovieMPAA - Stored result for (CF~IMDbmpaaSummary~)');
End
Else
Begin
LogMessage('Function ParsePage_IMDBMovieMPAA - Content Rating section for (CF~IMDbmpaaSummary~) not found');
//Result := prError; // Set to error if content rating section is not found
End;
Fixed
// Get (CF~IMDbmpaaSummary~)
curPos := Pos('<select id="jump-to"', HTML);
If curPos > 0 Then
Begin
// Extract the relevant section for categories
curPos := PosFrom('<option', HTML, curPos);
endPos := PosFrom('</select>', HTML, curPos);
mpaaSection := Copy(HTML, curPos, endPos - curPos);
LogMessage('Function ParsePage_IMDBMovieMPAA - Extracted Category section for (CF~IMDbmpaaSummary~): ' + mpaaSection);
// Parse the options and category names
curPos := 1;
mpaaContent := '';
While PosFrom('<option', mpaaSection, curPos) > 0 Do
Begin
curPos := PosFrom('<option', mpaaSection, curPos) + Length('<option');
optionValue := TextBetween(mpaaSection, 'value="', '">', False, curPos);
categoryName := TextBetween(mpaaSection, '">', '</option>', False, curPos);
// Format the category link
// mpaaContent := mpaaContent + '<link url="' + MovieURL + optionValue + '">' + categoryName + ' • </link>';
mpaaContent := mpaaContent + '<link url="' + MovieURL + optionValue + '">' + categoryName + '</link> • ';
// Move to the next position
curPos := PosFrom('</option>', mpaaSection, curPos) + Length('</option>');
End;
// Remove the trailing " • " if it exists and keep the last </link> intact
If Copy(mpaaContent, Length(mpaaContent) - 13, 7) = ' • ' Then
Begin
mpaaContent := Copy(mpaaContent, 1, Length(mpaaContent) - 14) + '</link>';
End;
// Combine and format the final result
//mpaaContent := '<link url="' + MovieURL + '#contentRating' + '">Content Ratings Summary: </link>' + mpaaContent;
mpaaContent := '<link url="' + MovieURL + '#contentRating' + '">Content Ratings Summary:</link> ' + mpaaContent;
// Store the result in the custom field
AddCustomFieldValueByName('IMDbmpaaSummary', mpaaContent);
LogMessage('Function ParsePage_IMDBMovieMPAA - Stored result for (CF~IMDbmpaaSummary~)');
End
Else
Begin
LogMessage('Function ParsePage_IMDBMovieMPAA - Content Rating section for (CF~IMDbmpaaSummary~) not found');
//Result := prError; // Set to error if content rating section is not found
End;
-
IMDB_Movie_[EN][Selenium]-v4.psf
Fixed part of the code in Function ParsePage_IMDBMovieCONNECTIONS
Previous
// Get (CF~Connections~) info
If Pos('>Connections<', HTML) > 0 Then
Begin
LogMessage('===Function ParsePage_IMDBMovieCONNECTIONS - Starting Connections Parsing ===');
curPos := Pos('>Connections<', HTML);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - curPos for "Connections" section found at: ' + IntToStr(curPos));
If Pos('It looks like we don'+Chr(39)+'t have any Connections for this title yet.', HTML) > 0 Then
Begin
Category1 := 'No Connections';
Category3 := '';
AddCustomFieldValueByName('Connections', Category1);
AddCustomFieldValueByName('Connect', Category3);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - No Connections found for this title');
End Else Begin
// Extract the relevant section for categories
If PosFrom('<span class="ipc-simple-select__container" data-testid="jumpTo">', HTML, curPos) > 0 Then Begin
curPos := PosFrom('<span class="ipc-simple-select__container" data-testid="jumpTo">', HTML, curPos);
endPos := PosFrom('</select></span></span>', HTML, curPos);
Category2 := Copy(HTML, curPos, endPos - curPos);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Extracted Category2 section: ' + Category2);
// Parse the options and category names
curPos := 1;
Category1 := '';
While PosFrom('<option', Category2, curPos) > 0 Do
Begin
curPos := PosFrom('<option', Category2, curPos) + Length('<option');
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - curPos at option tag: ' + IntToStr(curPos));
optionValue := TextBetween(Category2, 'value="', '">', False, curPos);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Extracted optionValue: ' + optionValue);
categoryName := TextBetween(Category2, '">', '</option>', False, curPos);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Extracted categoryName: ' + categoryName);
// Format the category link
Category1 := Category1 + '<link url="' + MovieURL + optionValue + '">' + categoryName + ' • </link>';
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Formatted and added category link: ' + '<link url="' + MovieURL + optionValue + '">' + categoryName + ' • </link>');
// Move to the next position
curPos := PosFrom('</option>', Category2, curPos) + Length('</option>');
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Moved curPos to next option tag: ' + IntToStr(curPos));
End;
// Remove the trailing " • " if it exists and keep the last </link> intact
If Copy(Category1, Length(Category1) - 13, 7) = ' • ' Then
Begin
Category1 := Copy(Category1, 1, Length(Category1) - 14) + '</link>';
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Removed trailing " • " and kept last </link> in Category1');
End;
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Final Category1 before combining: ' + Category1);
// Combine and format the final result
Category1 := '<link url="' + MovieURL + '">Connections: </link>' + Category1;
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Combined and formatted final Category1: ' + Category1);
// Store the result in the custom field
AddCustomFieldValueByName('Connections', Category1);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Stored result in (CF~Connections~)');
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Final Category1 stored: ' + Category1);
LogMessage('===Function ParsePage_IMDBMovieCONNECTIONS - Finished (CF~Connections~) Parsing ===');
End;
Fixed
// Get (CF~Connections~) info
If Pos('>Connections<', HTML) > 0 Then
Begin
LogMessage('===Function ParsePage_IMDBMovieCONNECTIONS - Starting Connections Parsing ===');
curPos := Pos('>Connections<', HTML);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - curPos for "Connections" section found at: ' + IntToStr(curPos));
If Pos('It looks like we don'+Chr(39)+'t have any Connections for this title yet.', HTML) > 0 Then
Begin
Category1 := 'No Connections';
Category3 := '';
AddCustomFieldValueByName('Connections', Category1);
AddCustomFieldValueByName('Connect', Category3);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - No Connections found for this title');
End Else Begin
// Extract the relevant section for categories
If PosFrom('<span class="ipc-simple-select__container" data-testid="jumpTo">', HTML, curPos) > 0 Then Begin
curPos := PosFrom('<span class="ipc-simple-select__container" data-testid="jumpTo">', HTML, curPos);
endPos := PosFrom('</select></span></span>', HTML, curPos);
Category2 := Copy(HTML, curPos, endPos - curPos);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Extracted Category2 section: ' + Category2);
// Parse the options and category names
curPos := 1;
Category1 := '';
While PosFrom('<option', Category2, curPos) > 0 Do
Begin
curPos := PosFrom('<option', Category2, curPos) + Length('<option');
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - curPos at option tag: ' + IntToStr(curPos));
optionValue := TextBetween(Category2, 'value="', '">', False, curPos);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Extracted optionValue: ' + optionValue);
categoryName := TextBetween(Category2, '">', '</option>', False, curPos);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Extracted categoryName: ' + categoryName);
// Format the category link
//Category1 := Category1 + '<link url="' + MovieURL + optionValue + '">' + categoryName + ' • </link>';
// Category1 := Category1 + '<link url="' + MovieURL + optionValue + '">' + categoryName + '</link> • ';
Category1 := Category1 + '<link url="' + MovieURL + optionValue + '">' + categoryName + '</link> ';
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Formatted and added category link: ' + '<link url="' + MovieURL + optionValue + '">' + categoryName + ' • </link>');
// Move to the next position
curPos := PosFrom('</option>', Category2, curPos) + Length('</option>');
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Moved curPos to next option tag: ' + IntToStr(curPos));
End;
// Remove the trailing " • " if it exists and keep the last </link> intact
If Copy(Category1, Length(Category1) - 13, 7) = ' • ' Then
Begin
Category1 := Copy(Category1, 1, Length(Category1) - 14) + '</link>';
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Removed trailing " • " and kept last </link> in Category1');
End;
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Final Category1 before combining: ' + Category1);
// Combine and format the final result
//Category1 := '<link url="' + MovieURL + '">Connections: </link>' + Category1;
if Category1 <> '' then
Category1 := '<link url="' + MovieURL + '">Connections:</link> • ' + Category1
Else
Category1 := '<link url="' + MovieURL + '">Connections:</link> ' + Category1;
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Combined and formatted final Category1: ' + Category1);
// Store the result in the custom field
AddCustomFieldValueByName('Connections', Category1);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Stored result in (CF~Connections~)');
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Final Category1 stored: ' + Category1);
LogMessage('===Function ParsePage_IMDBMovieCONNECTIONS - Finished (CF~Connections~) Parsing ===');
End;
-
Hey, Ivek. Thanks. Can you please post examples or imdb links where those matters and my code doesn't work? I just can't grasp just by looking at the code. Thanks.
-
IMDB_Movie_[EN][Selenium]-v4.psf
Fixed part of the code in Function ParsePage
Remove
AddCustomFieldValueByName('LDDbSearch', '<link url="https://www.lddb.com/search/IMDb/' + MovieID1 + '">LaserDiscDbSearch</link> ' + '<link url="http://www.soundtrack.net/search/?q=' + GetFieldValue(2) + '">Soundtrack.NetSearch</link> ' + '<link url="https://store.intrada.com/s.nl?sc=16&category=&search=' + GetFieldValue(2) + '">Intrada</link> ' + '<link url="https://www.aveleyman.com/?Film.aspx/' + GetFieldValue(2) + '">Aveleyman</link> ' + '<link url="https://www.bing.com/Search?q=' + GetFieldValue(2) + '%20site%3Awww.aveleyman.com/">BingAvSearch</link> ' + '<link url="https://www.google.com/search?q=' + GetFieldValue(2) + '%20site%3Awww.aveleyman.com/">GoogleAvSearch</link> ' + '<link url="https://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Dmovies-tv&field-keywords=' + GetFieldValue(2) + '">AmazonSearch</link> ' + '<link url="http://www.impawards.com/cgi-bin/htsearch?method=or&words=' + GetFieldValue(2) + '">IMPawards/cgi-bin</link> ' + '<link url="http://www.impawards.com/search.php/' + GetFieldValue(2) + '">impawards.com</link>');
Previous
//Get ~IMDb Movie Url~ (CF~IMDb.com~) and (CF~WaybackArchive IMDb URLs~)
Movie_URL := StringReplace(DownloadURL, BASE_URL_PRE_TRUE, BASE_URL_PRE, True, False, False);
AddCustomFieldValueByName('IMDb.com', '<link url="' + Movie_URL + '">IMDb.com</link>');
AddCustomFieldValueByName('WaybackArchive IMDb URLs', '<link url="https://web.archive.org/web/*/' + Movie_URL + '*">*IMDb.com*</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'business">Busines</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'combined">Combined</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'fullcredits">Full Cast&Crew</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'awards">Awards</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'releaseinfo">ReleaseDates</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'technical">TechSpecs</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'parentalguide">ParentsGuide</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'keywords">PlotKeywords</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'plotsummary">PlotSummary</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'taglines">Taglines</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'movieconnections">Connections</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'soundtrack">Soundtracks</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'externalsites">MiscSites</link>');
AddCustomFieldValueByName('WaybackArchive IMDb URLs', '<link url="https://web.archive.org/web/*/' + Movie_URL + '*">*IMDb.com*</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'business">Busines</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'combined">Combined</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'technical">TechSpecs</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'dvd">DVD</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'laserdisc">Laserdisc</link>');
Fixed
//Get ~IMDb Movie Url~ (CF~IMDb.com~) and (CF~WaybackArchive IMDb URLs~)
Movie_URL := StringReplace(DownloadURL, BASE_URL_PRE_TRUE, BASE_URL_PRE, True, False, False);
AddCustomFieldValueByName('IMDb.com', '<link url="' + Movie_URL + '">IMDb.com</link>');
AddCustomFieldValueByName('WaybackArchive IMDb URLs', '<link url="https://web.archive.org/web/*/' + Movie_URL + '*">*IMDb.com*</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'business">Busines</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'combined">Combined</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'fullcredits">Full Cast&Crew</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'awards">Awards</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'releaseinfo">ReleaseDates</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'technical">TechSpecs</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'parentalguide">ParentsGuide</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'keywords">PlotKeywords</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'plotsummary">PlotSummary</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'taglines">Taglines</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'movieconnections">Connections</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'soundtrack">Soundtracks</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'externalsites">MiscSites</link>');
Fixed
Fullinfo := '';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + '">MainPage</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'reference">Reference</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'fullcredits">Full Cast&Crew</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'awards">Awards</link> ';
//Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'releaseinfo">ReleaseDates</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'technical">TechSpecs</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'parentalguide">ParentsGuide</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'keywords">PlotKeywords</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'plotsummary">PlotSummary</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'locations">FilmLocations</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'taglines">Taglines</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'companycredits">CompanyCredits</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'movieconnections">Connections</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'soundtrack">Soundtracks</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'externalsites">MiscSites</link> ';
//Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'externalsites#photos">MiscPhoto</link> ';
Fullinfo:=Fullinfo+'<link url="'+Movie_URL+'externalsites/#misc">MiscSites</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'externalsites#photo">MiscPhoto</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'mediaindex">PhotoGallery</link> ';
Fullinfo := Fullinfo + '<link url="http://www.imdb.com/chart/top">IMDb Top 250</link> ';
Fullinfo := Fullinfo + '<link url="http://www.imdb.com/chart/bottom">Lowest Rated</link> ';
Fullinfo := Fullinfo + '<link url="http://www.imdb.com/chart/toptv">Top 250 TV</link>';
If Fullinfo <> '' then IMDB_URL := Fullinfo;
If IMDB_URL <> '' then AddCustomFieldValueByName('IMDb Url', IMDB_URL);
If IMDB_URL <> '' then AddCustomFieldValueByName('IMDb Movie Url', IMDB_URL);
LogMessage('Function ParsePage - Get result Fullinfo-IMDB_URL (CF~IMDb Movie Url~ ): ' + #13 + IMDB_URL + '| |');
Remove
AddCustomFieldValueByName('IMDbExternalSitesUrl', '<link url="' + Movie_URL + 'releaseinfo">ReleaseDates</link> <link url="' + Movie_URL + 'plotsummary">PlotSummary</link> <link url="' + Movie_URL + 'companycredits">CompanyCredits</link> <link url="' + Movie_URL + 'movieconnections">Connections</link> <link url="' + Movie_URL + 'externalsites/#misc">MiscSites</link> <link url="' + Movie_URL + 'externalsites#photos">MiscPhoto</link>');
Fixed
Fullinfo1 := '';
Fullinfo1:=Fullinfo1+'<link url="http://www.boxofficemojo.com/title/'+MovieID+'">BoxOfficeMojo</link> ';
Fullinfo1:=Fullinfo1+'<link url="http://en.wikipedia.org/w/index.php?search='+GetFieldValue(2)+'">Wikipedia</link> ';
Fullinfo1:=Fullinfo1+'<link url="https://www.movieposterdb.com/search?category=title&q='+MovieID+'">MoviePosterDB Info</link> ';
Fullinfo1:=Fullinfo1+'<link url="https://www.dvdsreleasedates.com/search/?searchStr='+GetFieldValue(2)+'">DVDs ReleaseDates</link> ';
Fullinfo1:=Fullinfo1+'<link url="https://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Dmovies-tv&field-keywords='+GetFieldValue(2)+'">AmazonSearch</link> ';
Fullinfo1:=Fullinfo1+'<link url="https://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Dmovies-tv&field-keywords='+GetFieldValue(2)+'%20'+GetFieldValue(5)+'">AmazonSearch1</link> ';
Fullinfo1:=Fullinfo1+'<link url="https://www.allmovie.com/search/movies/'+GetFieldValue(2)+'">AllMovieSearch</link> ';
Fullinfo1:=Fullinfo1+'<link url="https://www.themoviedb.org/search?query=' +GetFieldValue(2)+'">TMDBSearch</link> ';
Fullinfo1:=Fullinfo1+'<link url="https://www.rottentomatoes.com/search/?search='+GetFieldValue(2)+'">RottenTomatoesSearch</link> ';
Fullinfo1:=Fullinfo1+'<link url="https://www.thetvdb.com/search?query='+GetFieldValue(2)+'">TVDB Search</link> ';
Fullinfo1:=Fullinfo1+'<link url="https://www.soundtrackcollector.com/catalog/search.php?searchon=all&searchtext='+GetFieldValue(2)+'">SoundCollSearch</link> ';
Fullinfo1:=Fullinfo1+'<link url="https://www.blu-ray.com/search/?quicksearch=1&quicksearch_country=all&quicksearch_keyword='+MovieID+'§ion=theatrical">BR.comSearch</link> ';
Fullinfo1:=Fullinfo1+'<link url="https://www.lddb.com/search/IMDb/'+MovieID1+'">LaserDiscDb Search</link> ';
Fullinfo1:=Fullinfo1+'<link url="http://cse.google.com/cse?cx=004917987473580823572:eonwdtnjfi8&cof=forid%3a9&q='+GetFieldValue(2)+'">IMPawards-cse</link> ';
Fullinfo1:=Fullinfo1+'<link url="http:///www.impawards.com/cgi-bin/htsearch?method=or&words='+GetFieldValue(2)+'">impawards.com/cgi-bin</link> ';
If Fullinfo1 <> '' then Fullinfo1 := Fullinfo1;
If Fullinfo1 <> '' then AddCustomFieldValueByName('IMDbMovieLinksInfo', Fullinfo1);
-
Hey, Ivek. Thanks. Can you please post examples or imdb links where those matters and my code doesn't work? I just can't grasp just by looking at the code. Thanks.
To be clear, these parts work correctly, my fixes to some of the code are only cosmetic in nature. Below are examples of what I had in mind.
Before
<link url="https://www.imdb.com/title/tt0147800/parentalguide/#contentRating">Content Ratings Summary: </link><link url="https://www.imdb.com/title/tt0147800/parentalguide/#contentRating">Content rating (5) • </link><link url="https://www.imdb.com/title/tt0147800/parentalguide/#nudity">Sex & Nudity (10) • </link><link url="https://www.imdb.com/title/tt0147800/parentalguide/#violence">Violence & Gore (7) • </link><link url="https://www.imdb.com/title/tt0147800/parentalguide/#profanity">Profanity (8) • </link><link url="https://www.imdb.com/title/tt0147800/parentalguide/#alcohol">Alcohol, Drugs & Smoking (5) • </link><link url="https://www.imdb.com/title/tt0147800/parentalguide/#frightening">Frightening & Intense Scenes (1) • </link><link url="https://www.imdb.com/title/tt0147800/parentalguide/#certificates">Certifications (49)</link>
and after
<link url="https://www.imdb.com/title/tt0147800/movieconnections/">Connections:</link> • <link url="https://www.imdb.com/title/tt0147800/movieconnections/#featured_in">Featured in (134)</link> <link url="https://www.imdb.com/title/tt0147800/movieconnections/#features">Features (1)</link> <link url="https://www.imdb.com/title/tt0147800/movieconnections/#followed_by">Followed by (2)</link> <link url="https://www.imdb.com/title/tt0147800/movieconnections/#referenced_in">Referenced in (59)</link> <link url="https://www.imdb.com/title/tt0147800/movieconnections/#references">References (8)</link> <link url="https://www.imdb.com/title/tt0147800/movieconnections/#spin_off">Spin-off (1)</link> <link url="https://www.imdb.com/title/tt0147800/movieconnections/#spoofed_in">Spoofed in (2)</link> <link url="https://www.imdb.com/title/tt0147800/movieconnections/#spoofs">Spoofs (1)</link> <link url="https://www.imdb.com/title/tt0147800/movieconnections/#version_of">Version of (58)</link>
Before
<link url="https://www.imdb.com/title/tt0147800/movieconnections/">Connections: </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#featured_in">Featured in (134) • </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#features">Features (1) • </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#followed_by">Followed by (2) • </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#referenced_in">Referenced in (59) • </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#references">References (8) • </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#spin_off">Spin-off (1) • </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#spoofed_in">Spoofed in (2) • </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#spoofs">Spoofs (1) • </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#version_of">Version of (58)</link>
and after
<link url="https://www.imdb.com/title/tt0147800/parentalguide/#contentRating">Content Ratings Summary:</link> <link url="https://www.imdb.com/title/tt0147800/parentalguide/#contentRating">Content rating (5)</link> • <link url="https://www.imdb.com/title/tt0147800/parentalguide/#nudity">Sex & Nudity (10)</link> • <link url="https://www.imdb.com/title/tt0147800/parentalguide/#violence">Violence & Gore (7)</link> • <link url="https://www.imdb.com/title/tt0147800/parentalguide/#profanity">Profanity (8)</link> • <link url="https://www.imdb.com/title/tt0147800/parentalguide/#alcohol">Alcohol, Drugs & Smoking (5)</link> • <link url="https://www.imdb.com/title/tt0147800/parentalguide/#frightening">Frightening & Intense Scenes (1)</link> • <link url="https://www.imdb.com/title/tt0147800/parentalguide/#certificates">Certifications (49)</link> •
-
To be clear, these parts work correctly, my fixes to some of the code are only cosmetic in nature. Below are examples of what I had in mind.
Oh, ok then. Unfortunately, the cosmetics is very important to my custom skin design to visually separate fields and sections (screenshot below), so it would be huge overload for me to keep two versions when updating.
Regarding cleanning FullInfo, it is very important section for many reasons, and I admit it was always too clummsy for me to clean so I was primarily focused on it to work, and I will clean it at next update release.
Thanks for reviewing though!
-
Merry Christmas and a Happy New Year to everyone.
I am announcing definitive v4.3 scripts. Only description and screenshots in this message because of attachments limit.
Tons of improvements, bugs fixing, stabilizing and other things.
New Search window, with 30 seconds to choose now.
Separated python scripts for IMDb People script.
Fully stabilized and normalized code, now finally easy to navigate through, with as much as possible comments left in the scripts.
New AllMovie and Rottentomatoes scripts as promised to finish in a year:
WISHFUL THINKING:
- Bringing back Allmovie and Rottentomatoes scripts too.
Tons of custom fields for AllMovie and RottenTomatoes.
Also, Rottentomatoes all-in-one script for movies, series and episodes.
Search window for Rottentomatoes to choose Movies or TV Shows to search for.
-
In this message I'm attaching udl files for Notepad++, which now is perfectly fit for PVD scripting.
Most important - folding and unfolding is now seamless as in the screenshot.
As usual, replace stylers.xml with the given one and import PVD v4.3_2026-01-06.xml and it should look as in the screenshot.
-
Finally, here are all the scripts.
Never forget to read first message in the topic. All the answers and solutions are there, scripts and PVD to work flawlessly.
As usual, backup and empty Scripts folder and extract Scripts_2026-01-06.7z there. Extract the other file into PVD root folder.
If you want to use the scripts with my skin, you can download it with the list of custom fields here:
https://www.videodb.info/forum_en/index.php/topic,4388.msg23025.html#msg23025 (https://www.videodb.info/forum_en/index.php/topic,4388.msg23025.html#msg23025)
Important note: Since I didn't see even "thanks", or any kind of feedback (except from Ivek, and I haven't seem him recently either) for a more than a year of hard work, I guess there is no interest for these, so I will not update scripts anymore. But anyway, given files are firm base someone else to take over and continue where I left. If I could do it with AI, anyone can.
Best regards.
-
Thank you for all the effort put into creating all the scripts. As for me, I currently have quite a few health problems, so I am currently less present on the forum and currently because of this I am using the PVD program less or very little and testing scripts and the like.
My wish is that you would still help update all the scripts.
As for other users, I assume that some people find it difficult to use or install the python program on their computer, because they may also be less skilled in using such programs.
To clarify, I myself do not know many things about programming, because I am self-taught and have never had courses in using Windows and programming.
-
It is important to note that you must have the latest version of chromedriver.exe for this to work. Chromedriver always needs to be updated to the latest version, this is a prerequisite for all scripts to work.
-
It is important to note that you must have the latest version of chromedriver.exe for this to work. Chromedriver always needs to be updated to the latest version, this is a prerequisite for all scripts to work.
Somehow, despite my health problems, I managed to check how the latest IMDb Movie, Allmovie and Rottentomatoes v4 Scripts work. They work fine (with some cosmetic errors when transferring information) provided that you use the latest chromedriver.exe program in what I mentioned a little higher up.
I didn't check the other scripts.
-
Thanks Ivek! Wish you a great health!
-
Thanks Ivek! Wish you a great health!
Thanks.
-
Finally, here are all the scripts.
Never forget to read first message in the topic. All the answers and solutions are there, scripts and PVD to work flawlessly.
As usual, backup and empty Scripts folder and extract Scripts_2026-01-06.7z there. Extract the other file into PVD root folder.
If you want to use the scripts with my skin, you can download it with the list of custom fields here:
https://www.videodb.info/forum_en/index.php/topic,4388.msg23025.html#msg23025 (https://www.videodb.info/forum_en/index.php/topic,4388.msg23025.html#msg23025)
Important note: Since I didn't see even "thanks", or any kind of feedback (except from Ivek, and I haven't seem him recently either) for a more than a year of hard work, I guess there is no interest for these, so I will not update scripts anymore. But anyway, given files are firm base someone else to take over and continue where I left. If I could do it with AI, anyone can.
Best regards.
Thank you for your support of the PVD. But I'm having trouble working with Selenium. I updated ChromeDriver (144.0.7559.59), updated Python (3.14.2). And still, I can't get information from the IMDB. The log file keeps showing no connection.
-
144.0.7559.31 not 144.0.7559.59
And also, you don't need external sites. Nothing is parsed so far from external sites so far. It was placed there for possible use in the future.
-
Hello,
thank you for your epic work on the keeping the scripts and PVD alive.
After working very well for 3-4 days, today 22.01.2026 I keep getting on keywords, reviews pages download this:
<html lang="en"><head>
"context":"
};
</script>
<script src="https://1c5c1ecf7303.8b78215a.eu-north-1.token.awswaf.com/1c5c1ecf7303/e231f0619a5e/0319a8d4ae69/challenge.js"></script>
</head>
<body>
<div id="challenge-container"></div>
<script type="text/javascript">
AwsWafIntegration.saveReferrer();
AwsWafIntegration.checkForceRefresh().then((forceRefresh) => {
if (forceRefresh) {
AwsWafIntegration.forceRefreshToken().then(() => {
window.location.reload(true);
});
} else {
AwsWafIntegration.getToken().then(() => {
window.location.reload(true);
});
}
});
</script>
<noscript>
<h1>JavaScript is disabled</h1>
In order to continue, we need to verify that you're not a robot.
This requires JavaScript. Enable JavaScript and then reload the page.
</noscript>
</body></html>
After some searching i got this from chatgpt:
What the error actually is the file you’re saving is not the keywords page. It’s an AWS WAF (Web Application Firewall) challenge page returned by IMDb
Key signs from the HTML:
challenge.js
AwsWafIntegration
“verify that you're not a robot”
JavaScript-based token refresh
This means:IMDb detected automation and served a bot-check page instead of real content
Just in case other people get this to fix it in Selenium_Chrome_Movie_Additional_pages_v4:
after driver.get(download_url)
i added:
time.sleep(random.uniform(8, 12))
-
Just in case other people get this to fix it in Selenium_Chrome_Movie_Additional_pages_v4:
after driver.get(download_url)
i added:
time.sleep(random.uniform(8, 12))
This change does not work because it blocks the download of Additional pages.
-
This will work most probably, but I think it is fragile too... I am sure soon it will be not possible again, but maybe something new will come up...
This will work most probably, but I think it is fragile too... I am sure soon it will be not possible again, but maybe something new will come up...
Yes — what you currently have is a classic Selenium workflow:
It spins up a new ChromeDriver instance for each URL.
It applies stealth tweaks, sets cookies, navigates, clicks “See more” buttons, saves HTML, and then quits.
You’re running this in parallel threads (ThreadPoolExecutor(max_workers=4)), so four fresh Chrome sessions at a time.
That’s exactly the pattern IMDb’s AWS WAF is now blocking: fresh, headless, parallel, automation‑fingerprinted sessions.
Why this fails against IMDb now
Headless mode is fingerprinted. Even with stealth, AWS WAF detects it.
Fresh profiles per run (no browsing history, no persistent cookies) scream “bot.”
Parallel sessions look like automation, not human browsing.
Adding fake cookies doesn’t help — WAF requires valid tokens generated by a real browser session.
How to adapt this code to “attach to Chrome”
Instead of creating a new webdriver.Chrome(...) each time, you’d:
Start Chrome manually with debugging enabled
chrome.exe --remote-debugging-port=9222 --user-data-dir="C:\ChromeIMDb"
This opens Chrome with a persistent profile (C:\ChromeIMDb).
You log in once, build up cookies/history naturally.
Change your Selenium init code
Replace:
service = Service(chrome_path)
chrome_options = build_chrome_options(headed=False)
driver = webdriver.Chrome(service=service, options=chrome_options)
With:
service = Service(chrome_path)
chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("debuggerAddress", "127.0.0.1:9222")
driver = webdriver.Chrome(service=service, options=chrome_options)
Now Selenium attaches to the Chrome you already opened.
It inherits your cookies, extensions, and fingerprint.
Run serially, not in parallel
IMDb WAF is sensitive to multiple simultaneous sessions.
Change ThreadPoolExecutor(max_workers=4) → max_workers=1.
Important adjustments
Don’t quit the browser (driver.quit()) after each run — that would kill your attached Chrome. Instead, just close tabs (driver.close()) or reuse the same driver.
Remove fake cookie injection — you don’t need it if you’re using your real Chrome profile.
Headed mode only — you’ll see the browser window, but that’s what passes WAF.
In short: your current script is fine for FilmAffinity, but IMDb now requires either:
Attach to Chrome (reuse your real session), or
Switch to IMDb datasets / APIs for long‑term stability.
-
Just in case other people get this to fix it in Selenium_Chrome_Movie_Additional_pages_v4:
after driver.get(download_url)
i added:
time.sleep(random.uniform(8, 12))
This change does not work because it blocks the download of Additional pages.
Hello,
without the modification i was getting AWS pages on the additional pages. I attached one example renamed to txt.
-
This just sleeps between 8 and 12 seconds and it could be very fragile. Also, it makes whole process longer 1-2 minutes per title?
-
This just sleeps between 8 and 12 seconds and it could be very fragile. Also, it makes whole process longer 1-2 minutes per title?
I use Moviedb to get the picture and IMDB Selenium to get the data
Tests:
Witches' Well 2024 https://www.imdb.com/title/tt29793692/ - it took 1 min 55 sec
The Matrix 1999 https://www.imdb.com/title/tt0133093/ - it took 2 min 10 sec
I limited the tags to 300 as above 500 it crashed the database and i had to manually edited it with DBeaver, rest is in the pictures attached.
I run PVD in a win10 VM as in win11 i can't get it download any data.
When i first got the AWS pages instead of the data ones i thought i got ip banned by imdb so i tried to proxy and VPN my connection with no success. I even copied the vm to my computer at work to test and same result.
Then i looked into why i get the pages and the results pointed to the fact i appeared as a bot getting page after page with no "human" pause between them so i added the sleep.
I found other solutions but not tested them:
change: chrome_options = build_chrome_options(headed=False)
to this: headed_mode = "keywords" in download_url
chrome_options = build_chrome_options(headed=headed_mode)
also this but seemed longer:
add this after page load:
if "challenge.js" in driver.page_source or "AwsWafIntegration" in driver.page_source:
logging.warning("AWS WAF detected — retrying with longer delay")
time.sleep(15)
driver.refresh()
time.sleep(8 )
Regards
-
Great. Do you mind to try with my suggestion and to measure timings, because it makes sense to measure only in the same environment?
Also, if you want you could upload your scripts too.
And yes - too many keywords crashes PVD.
-
I have tried in my environment with your
time.sleep(random.uniform(8, 12))
and got 02:56.68
And tried with another approach by starting 4 instances of chrome manually and with:
# Function to download a page and handle "See more" clicks for specific pages
def download_page(download_url, output_path, port, retries=3):
time.sleep(random.uniform(1.0, 3.0))
logging.debug(f"Starting download for URL: {download_url} on port {port}")
logging.debug(f"Output path: {output_path}")
marker_file_path = os.path.splitext(output_path)[0] + "_status.txt" # Define marker file path
attempt = 0
success = False
try:
while attempt < retries:
attempt += 1
logging.debug(f"Attempt {attempt} for URL: {download_url}")
try:
# Attach to existing Chrome instance on given port
service = Service(chrome_path)
chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("debuggerAddress", f"127.0.0.1:{port}")
driver = webdriver.Chrome(service=service, options=chrome_options)
logging.info(f"Attached to Chrome on port {port}")
try:
# Navigate to the target page
driver.get(download_url)
logging.info(f"Page {download_url} loaded successfully.")
# Wait for the page to load
WebDriverWait(driver, 3).until(
EC.presence_of_element_located((By.TAG_NAME, "body"))
)
# Click all "See more" buttons only for specific pages
if any(keyword in download_url for keyword in [
'fullcredits', 'awards', 'keywords', 'releaseinfo', 'plotsummary',
'reviews', 'companycredits', 'locations', 'technical',
'externalsites', 'movieconnections'
]):
def click_all_or_more_buttons():
"""
Clicks every 'See all' or 'X more' button on the page,
starting from the bottom to handle pages like Movie Connections.
Waits for content to load after each click.
"""
while True:
try:
# Find all current see-more buttons
buttons = driver.find_elements(
By.XPATH, "//button[contains(@class, 'ipc-see-more__button')]"
)
if not buttons:
break
# Reverse order: start from bottom-most button
buttons = list(reversed(buttons))
clicked_any = False
for button in buttons:
try:
text = button.text.strip().lower()
if "all" in text or "more" in text:
logging.info(f"Clicking button with text: {text}")
driver.execute_script("arguments[0].scrollIntoView(true);", button)
time.sleep(0.5)
driver.execute_script("arguments[0].click();", button)
# Wait until the button becomes stale (DOM updated)
WebDriverWait(driver, 10).until(EC.staleness_of(button))
clicked_any = True
break # re-find buttons after DOM update
except Exception as e:
logging.warning(f"Could not click button: {e}")
if not clicked_any:
break
except Exception:
break
click_all_or_more_buttons()
# Retrieve the full source HTML of the page after all "More" buttons are clicked
html_source = driver.page_source
logging.debug(f"HTML source length: {len(html_source)}")
# Detect AWS WAF challenge page before saving
if "challenge-container" in html_source or "awswaf.com" in html_source:
logging.warning(f"IMDb WAF challenge detected for {download_url}. Skipping normal save.")
# Append the skipped URL to imdb_skipped_urls.log (one per line)
with open("imdb_skipped_urls.log", "a", encoding="utf-8") as skip_log:
skip_log.write(download_url + "\n")
success = False # mark as failed so marker file shows FAILED
else:
# Save the HTML using helper (sets success via try/except)
try:
save_artifacts(driver, output_path)
success = True
logging.info(f"HTML saved to file: {output_path}")
except Exception as e:
logging.error(f"Failed to save artifacts for {download_url}: {e}")
success = False
# If successful, set the success flag and break
if success:
break
except WebDriverException as e:
logging.error(f"An error occurred while processing {download_url}: {e}")
except Exception as e:
logging.error(f"An error occurred: {e}")
finally:
try:
# Reset the window instead of closing it
driver.get("about:blank")
logging.info("Reset Chrome window to blank, still running.")
except Exception as e:
logging.warning(f"Could not reset window: {e}")
finally:
# Always create the marker file, even if everything failed
with open(marker_file_path, "w") as marker_file:
if success and os.path.exists(output_path):
marker_file.write("SUCCESS")
logging.info(f"Marker file created: {marker_file_path} with status SUCCESS")
elif not success:
marker_file.write("FAILED")
logging.info(f"Marker file created: {marker_file_path} with status FAILED")
else:
marker_file.write("NOT_FOUND")
logging.info(f"Marker file created: {marker_file_path} with status NOT_FOUND")
from concurrent.futures import ThreadPoolExecutor, as_completed
# Define the ports for the Chrome instances you started manually
ports = [9222, 9223, 9224, 9225]
# Main script execution - use ThreadPoolExecutor for clean concurrency
try:
# Adjust max_workers to control how many browser sessions run in parallel
with ThreadPoolExecutor(max_workers=4) as executor:
# Submit all download tasks, assigning ports round-robin
futures = {
executor.submit(download_page, url, save_path, ports[i % len(ports)]): (url, save_path)[/i]
for i, (url, save_path) in enumerate(URLS_AND_PATHS.items())
}
# Wait for all tasks to complete
for future in as_completed(futures):
url, save_path = futures[future]
try:
future.result() # raises exception if download_page failed
logging.info(f"Download task finished for {url}")
except Exception as e:
logging.error(f"Download task failed for {url}: {e}")
# Build list of marker files for all processed URLs
marker_files = [os.path.splitext(save_path)[0] + "_status.txt"
for save_path in URLS_AND_PATHS.values()]
# Check markers to decide final status
if all(os.path.exists(path) and "SUCCESS" in open(path).read()
for path in marker_files):
logging.info("All pages have been saved successfully.")
else:
logging.warning("Some pages failed. Check marker files for details.")
except Exception as e:
logging.error(f"An error occurred during threading: {e}")
sys.exit(1) # Exit the script with an error code
# Exit the script successfully
sys.exit(0) # Exit successfully
I got 02:03.27 for the same movie Carrie, so basically my assumption that it would least a minute or so longer is basically correct, at least in my environment. If you could try too, it would be good to compare.
-
Hello,
there are my scripts. will try the modification on the weekend .
-
Hello,
i replaced in Selenium_Chrome_Movie_Additional_pages_v4.py from the "Function to download a page and handle "See more" clicks for specific pages" till the end with the script in quote but i get the all errors.
maybe i got the script wrong or maybe i'm getting port blocked.
Regards.
-
Did you start 4 chrome instances manually prior to run the PVD?
start chrome.exe --remote-debugging-port=9222 --user-data-dir="C:\PVD\1"
start chrome.exe --remote-debugging-port=9223 --user-data-dir="C:\PVD\2"
start chrome.exe --remote-debugging-port=9224 --user-data-dir="C:\PVD\3"
start chrome.exe --remote-debugging-port=9225 --user-data-dir="C:\PVD\4"
-
Hello, everybody!
Thanks for all the effort and the updates!!
Can I get a little assistance? I just can't make the Selenium scripts to work for me.
When I use any of them I get an "open" window, If I pick a .jpg file, it just loads it as a poster and stops. If I cancel the "open" window, the PVD gives an error message. Any idea what seems to be the problem?
Right now I'm downloading movie info with TheMovieDB[EN][API] script, which is pretty reliable
-
Hello, everybody!
Thanks for all the effort and the updates!!
Can I get a little assistance? I just can't make the Selenium scripts to work for me.
When I use any of them I get an "open" window, If I pick a .jpg file, it just loads it as a poster and stops. If I cancel the "open" window, the PVD gives an error message. Any idea what seems to be the problem?
Right now I'm downloading movie info with TheMovieDB[EN][API] script, which is pretty reliable
This can happen when searching for a movie and you don't have .png files in /Scripts folder. For other cases, I have no idea without logs.