Personal Video Database
English => Development => PVD Python Scripts => Topic started by: afrocuban on March 13, 2025, 11:44:06 pm
-
IF YOU DON'T READ THIS POST CAREFULLY AND FOLLOW EVERYTHING WRITTEN HERE, BUT JUST DOWNLOAD FILES, I COULD BET IT WILL NOT WORK FOR YOU AND YOU WILL COME BACK HERE ASKING QUESTIONS ALREADY ANSWERED IN THIS POST.
Almost 4 months after for the first time ever I heard the word "Selenium" knowing nothing about programming, I am finally bringing practically new PVD MOD considering amount of files and programs brought. It consists of the scripts and program as described here (https://www.videodb.info/forum_en/index.php/topic,4377.msg22782.html#msg22782).
You now need only one script for IMDb movies, one for IMDb people and one for FilmAffinity movies for everything: search and download. Selenium scripts in the background are doing all "external" work, so in your PVD you have clean situation: 2 scripts and configurator for movies (plus .batch file for these 2 at your will), and one script and configurator for the people. Check the screenshots below.
I strongly suggest to rename your current "Scripts" folder to, for instance, "Scripts-Original", and to put this Scripts.7z in your PVD folder and extract it there. It will create "Scripts" folder with all the scripts and files needed for the PVD to work (as a bonus, I'm contributing source code for the Scripts Configurator program, as well as updated and polished UDL file for PVD scripts in Notepad++ that is just to be imported to Notepad++). If you want to, after testing you can merge two folders, Selenium and non-Selenium scripts and files into "Scripts" folder.
Before that....
As stated here (https://www.videodb.info/forum_en/index.php/topic,4364.msg22706.html#msg22706)ensure that:
A. You installed python
B. You installed selenium via cmd, with
pip install selenium
B. You installed requests via cmd, with
pip install requests
C. You have your Chrome bin on a PATH (to test this, open cmd and simply type "chrome" and check if Chrome opens).
D. You have Python folder on your PATH (to test this, open cmd and simply type "python --version" and check if got the proper feedback, for instance:
C:\Users\user>python --version
Python 3.12.6
E. pythonw.exe is not missing, or it's containing folder is on the PATH (to test this, open cmd and simply type "pythonw" and check if got the proper feedback, for instance:
C:\Users\user>pythonw
C:\Users\user> (empty output)
These scripts:
1. Use Chrome browser instead Firefox
2. Use chromedriver.exe instead geckodriver
3. Start chromedriver.exe silently
4. Silently invoke browser in a headless mode (no pop-up windows of browser)
5. Scrape .htm pages of a given urls
6. No path is needed to set manually inside the script - it is set to be relative to the path of selenium script!
You just use your PVD as ever, just be sure to extract as instructed above.
For using relative path, ensure:
6B. You put appropirate chromedriver.exe to the "Script" folder, too. There is no installation for chromedriver, just extract it from the .zip file into your "Scripts" folder described above. IMPORTANT!!!! You need to download chromedriver.exe of the same version as your Chrome browser. At the moment of this post, stable version is v134. You can find Crome browser download and appropriate chromedriver here (https://googlechromelabs.github.io/chrome-for-testing/). For example, for v134, Stable links are:
Chrome browser:
1. chrome win32 https://storage.googleapis.com/chrome-for-testing-public/134.0.6998.88/win32/chrome-win32.zip (https://storage.googleapis.com/chrome-for-testing-public/134.0.6998.88/win32/chrome-win32.zip)
2. chrome win64 https://storage.googleapis.com/chrome-for-testing-public/134.0.6998.88/win64/chrome-win64.zip (https://storage.googleapis.com/chrome-for-testing-public/134.0.6998.88/win64/chrome-win64.zip)
Chromedriver:
1. chromedriver win32 https://storage.googleapis.com/chrome-for-testing-public/134.0.6998.88/win32/chromedriver-win32.zip (https://storage.googleapis.com/chrome-for-testing-public/134.0.6998.88/win32/chromedriver-win32.zip)
2. chromedriver win64 https://storage.googleapis.com/chrome-for-testing-public/134.0.6998.88/win64/chromedriver-win64.zip (https://storage.googleapis.com/chrome-for-testing-public/134.0.6998.88/win64/chromedriver-win64.zip)
From this point on, everything is automated and headless, silent as never before.
Amount of data imported is huge! I have included dozens of new custom fields. IF YOU ARE DATA HUNGRY AS I AM, THE MOST DATA CAN BE COLLECTED IF YOU CHECK AND UNCHECK OPTIONS AS IN THE SCRIPTS CONFIGURATOR SCREENSHOTS BELOW. The updated table with all the fields in these 3 scripts can be found here (https://www.videodb.info/forum_en/index.php/topic,4291.msg22824.html#msg22824). To see them all, you have to use Classic skin, or you have to add custom fields to your PVD and from there to create your own custom skin, or to use one of my skins from here (https://www.videodb.info/forum_en/index.php/topic,4356.msg22648.html#msg22648), once I complete them and adjust them for the final Selenium v4 scripts.
Examine the table. That is the only way you will learn what fields comes from which movie/person page and if you want them or not. The less you want, the faster PVD will be.
Please feel free to test the scripts and give me a feedback if something doesn't work. When I say "it doesn't work", everything works, the only issue that can happen is that sites have changed html layout and again not all fileds are available, or you updated your Chrome automatically to a higher version and you didn't download and extract corespondent chromedriver version. The best indicator for this is that no .log file is created in "\Scripts\Tmp\" folder. I will update whatever you report in a month at most from the first report, to give us all the time to collect and report as many as possible issues.
What I have learned
On this long journey, what I have learned was how hard work coding is. Also, I had to learn pretty deep about Pascal/Delphi, about python, and most frustratingly (https://www.thefreedictionary.com/frustratingly) - ahk! I had to revise all the scripts from the scratch several times. It was either because of the concepts i was developing along the way, or IMDb and FilmAffinity were changing their layout. For example, just yesterday new Chrome version was brought, and my chromedriver didn't work anymore, so I had to download new version of it too. Also, just 2 days ago I learned that FilmAffinity introduced their AKA for some movies, so I had to update FA script again. And so on and on for 4 months. Thus, I learned to appreciate it. The most important, now I even more appreciate EasyVVV's, and especially Ivek's work for more than a decade (!!!) to provide us with PVD alive!
So, humbly, I dedicate this hard work to EasyVVV, but most, and before anyone else I dedicate this to IVEK and to memory to his late mother! I HOPE IVEK CAN IMAGINE HOW GRATEFUL TO HIM I PERSONALLY AM, AFTER I REALIZED HOW HARD THIS ALL WAS! GOD BLESS YOU IVEK!
Last version in this message on this topic
2025-11-13: https://www.videodb.info/forum_en/index.php/topic,4379.msg23006.html#msg23006 (https://www.videodb.info/forum_en/index.php/topic,4379.msg23006.html#msg23006)
-
As i wrote, I completely rewrote Sripts configurator from the scratch practically in order to be able to bring all the options to automate scripts behaviour. It now is resizable and has scrollbars, so we can put there as much as we wish options from the scripts.
Also, the most of the data can be imported if you check and uncheck options as in the screenshots below. Feel free to test though. Don't be afraid! You can't mess your database, because backup files are created anyway!
READ THE MESSAGES UPON CLICKING "SAVE" BUTTON IN CONFIGURATOR. THOSE MESSAGES ARE VERY INFORMATIVE, HELPFUL AND ESSENTIAL TO UNDERSTAND WHAT IS GOING ON.
-
Due to the post limit, in this message, Scripts Configurator window shrinked as a proof of concept, with horizontal and vertical scrollbars. I haven't had no idea how huge challenge was to get scrollbars to this. AHK is pretty hard to get such feature and I lost at least 2 weeks to get it proper, so maybe in the future I will build this with python. I already tested it.
-
IMPORTANT!!!
A few hours ago, IMDb completely changed /fullcredits page html layout, so that page doesn't work any more. Soon, /reference page will be changed too. I know because I got popups offering me to peek to a new "Reference" page. So, until that happen, I will not update scripts, because both pages will share the same code again, and it will be easier to change. For now I made a quick fix everything to work if you check the options in Configurator as I suggested earlier. In addition you have to check "Download the Cast or Credit (text only) provider page to retrieve the full information. Or else, only the info from the main movie page will be downloaded." option and to download fullcredits page too!!! This should work until /reference page changes, or any other page changes meanwhile.
And it happened just when I finished ''all-in one script" while successfully doing final tests. Here's the pack.
So, with one IMDb Script you get all movies, Series, episode list, and then you apply the same script for episodes.
Also, new search window introduced, with different types of search and countdown of 10 seconds defaulted to "general" search.
It took only 600 additional lines comparing to Movie script, including a lot of commented out lines, and one simple python script to get all of this.
Extract and overwrite existing scripts with this pack.
I will soon start to re-birth AllMovie and RottentTomatoes scripts. I will not revive any other script.
-
Hello! I'm trying to make it work with this new system, but when I go to scripts.7z Windows flags SeleniumPVDbScriptsConfig-v4.exe as a threat and won't let me unzip
-
Hello! I'm trying to make it work with this new system, but when I go to scripts.7z Windows flags SeleniumPVDbScriptsConfig-v4.exe as a threat and won't let me unzip
You probably mean the anti-virus program. It's a known issue, from my experiences with the NOD32 Antivirus program it was the same. The solution was when the antivirus program quarantined the SeleniumPVDbScriptsConfig-v4.exe file, I put it back out of quarantine and excluded it from being scanned again by the antivirus program. If necessary, I always repeated the described procedure and thus prevented it from being quarantined again by the antivirus program.
If it still won't be possible to unpack the file, let me know and I'll help you do it another way.
And this notice.
You need the SeleniumPVDbScriptsConfig-v4.exe file to be able to configure the script for transferring information.
It is very likely that the IMDb scripts will not work.
-
Soon, /reference page will be changed too. I know because I got popups offering me to peek to a new "Reference" page. So, until that happen, I will not update scripts, because both pages will share the same code again, and it will be easier to change.
Hello to all. As I already said in March, now we all know that this happened. I was busy meanwhile and had no time to deal with it. There are changes across many imdb pages, but reference page is the culprit. I will need couple of months to fix it, since you all know that I am not a programer, and I have to remind my self about everything, especially about special cases. For now, you can pull up significant portion of data without reference page (unfortunately, not the whole cast). To do that and if you are using my scripts, open Script configurator, uncheck to download Reference page and PVD will restart. Then, in "Set Overwrite options...", check all the data you would want to download (Studio, etc..) and restart PVD. Then run the script and this would be what you can get at the moment.
After March, I had some minor adjustments to the scripts and started to work on a Reference page, so I'm uploading them so you could get the same amount data as me. Backup your existing scripts, then overwrite them with these, all to "Scripts" folder. Reminder: these scripts are just starting point to fix them, but they should get you more data comparing to March scripts could get you now, so please do not ask for the support for these scripts. I know they don't fully work.
What should be promising for you is that I'm not planning to abandon using PVD, so I will for sure fix the scripts at some point, so please be patient: if long time no see me, that just means I also haven't fixed the scripts, and I'm working on them. Meanwhile, just add your movies to PVD and later you will update with full data. That is exactly what I've been doing recently.
Best regards
-
IMDB_Movie_[EN][Selenium]-v4.psf
Between lines 138 and 139 is this:
GET_FULL_CONNECTIONS = False ; //To call Function ParsePage_IMDBMovieCONNECTIONS. Not used for parsing, since it is enough to choose both SIMPLE and COMPLEX connections. --> to "True".
Between lines 7992 and 7993 is this:
If GET_FULL_CONNECTIONS Then
Begin
However, when you run the ** SeleniumPVDbScriptsConfig v4 ** script in PVD and uncheck all Connections settings and then restart PVD, the following happens.
IMDB_Movie_[EN][Selenium]-v4.psf does not work.
The following parts of the code are missing.
Between lines 138 and 139 is missing:
GET_FULL_CONNECTIONS = False ; //To call Function ParsePage_IMDBMovieCONNECTIONS. Not used for parsing, since it is enough to choose both SIMPLE and COMPLEX connections. --> to "True".
However, between lines 7992 and 7993, this
[code] If GET_FULL_CONNECTIONS Then
Begin[/code]
changes to this:
If GET_FULL_CONNECTIONS = False ;
Add the following code to the script
//(*
function CustomStringReplace(const Source: string; const OldPattern: array of string; const NewPattern: array of string): string;
var
i: Integer;
ResultString: string;
begin
ResultString := Source;
for i := Low(OldPattern) to High(OldPattern) do
begin
ResultString := StringReplace(ResultString, OldPattern[i], NewPattern[i], True, False, True);
end;
Result := ResultString;
end;
//*)
and then you can add the following code to the Metascore code.
ItemValue := CustomStringReplace(ItemValue, ['0</', '1</', '2</', '3</', '4</', '5</', '6</', '7</', '8</', '9</'], [',0', ',1', ',2', ',3', ',4', ',5', ',6', ',7', ',8', ',9']);
//ItemValue := StringReplace(ItemValue, '0</', '.0', True, False, True);
//ItemValue := StringReplace(ItemValue, '1</', '.1', True, False, True);
//ItemValue := StringReplace(ItemValue, '2</', '.2', True, False, True);
//ItemValue := StringReplace(ItemValue, '3</', '.3', True, False, True);
//ItemValue := StringReplace(ItemValue, '4</', '.4', True, False, True);
//ItemValue := StringReplace(ItemValue, '5</', '.5', True, False, True);
//ItemValue := StringReplace(ItemValue, '6</', '.6', True, False, True);
//ItemValue := StringReplace(ItemValue, '7</', '.7', True, False, True);
//ItemValue := StringReplace(ItemValue, '8</', '.8', True, False, True);
//ItemValue := StringReplace(ItemValue, '9</', '.9', True, False, True);
The same can be done with this.
//Date ~Updated~ (choose simple or verbose version)
Date := DateToStr(CurrentDateTime);
//AddFieldValueXML('viewdate', Date); //Only date, don't admit time-. Set Seen value at the same time.
//AddFieldValueXML('moddate', Date + ' ' + TimeToStr(CurrentDateTime)); //Block the dB saving
ExplodeString(Date, DateParts, '-');
Date:=DateParts[2]+'.'+ DateParts[1]+'.'+DateParts[0];
Date := CustomStringReplace(Date, ['01.', '02.', '03.', '04.', '05.', '06.', '07.', '08.', '09.'], ['1.', '2.', '3.', '4.', '5.', '6.', '7.', '8.', '9.']);
// Date := StringReplace(Date, '01.', '1.', True, True, False);
//Date := StringReplace(Date, '02.', '2.', True, True, False);
//Date := StringReplace(Date, '03.', '3.', True, True, False);
//Date := StringReplace(Date, '04.', '4.', True, True, False);
//Date := StringReplace(Date, '05.', '5.', True, True, False);
//Date := StringReplace(Date, '06.', '6.', True, True, False);
//Date := StringReplace(Date, '07.', '7.', True, True, False);
//Date := StringReplace(Date, '08.', '8.', True, True, False);
//Date := StringReplace(Date, '09.', '9.', True, True, False);
//AddCustomFieldValueByName('Updated', Date); // (Left for FA Script)
//AddCustomFieldValueByName('Updated0', Date + ' at ' + TimeToStr(CurrentDateTime)); // Saved for RottenTomatoes
AddCustomFieldValueByName('IUpdated', Date + ' at ' + TimeToStr(CurrentDateTime) + ' • ' + SCRIPT_FILE_NAME + ' ' + SCRIPT_VERSION); // Annoying
LogMessage('Function ParsePage - Provider data info retreived Ok in ' + DateToStr(CurrentDateTime) + ' ' + TimeToStr(CurrentDateTime) + '| (~Updated~)');
Mode := smFinished;
LogMessage('Function ParsePage smNormal END====================== |');
Exit;
End;
-
You need the latest chromedriver.exe, otherwise it won't work.
And this tip.
For the Episode List page, it would be necessary and even better if a script was made specifically for it.
-
For the Episode List page, it would be necessary and even better if a script was made specifically for it.
Episode list works totally fine for me, even now. It would be probably impossible for me to create new script. We can look at it like this: Episode list has it's own script: it's Selenium script "Selenium_Chrome_IMDB_Episode_List_page_v4.py" which produces txt file with all episodes, and .psf script just scrapes that txt file, so I am not sure what we could achieve with additional psf script whose purpose would be only to scrape txt file.
Anyway, I have corrected FilmAffinity script, and made some improvements in IMDb Movie scripts (for example, storyline section dynamic load scraping is fixed). I have improved FilmAffinity script speed enormously! In order this script to work properly and fast you need:
1. To install python 3.12+
2. in a cmd to install psutil with
pip install psutil
in order to hopefully prevent selenium hangups when html elements aren't found on the page.
3. To download and overwrite scripts I'm uploading in this post.
4. In a script configurator to deselect Reference page. PVD will restart. Then select "Studio" and "Description" and all others you want and do not restart PVD. Try to import data, and now you should get director, cast, tagline and some other original and related custom fields scraped from the Main page. If you don't get cast, tagline and director, restart PVD manually and try again to import. I am not sure about this second restart, so try both. One will work for sure.
I will not provide support for these scripts until I finish, because I know they still don't fully work. I just want to share with you same amount of data I'm getting at the moment when some significant improvement is done. In a pictures you can get the sense of what i'm getting now with IMDB script.
Once I finish IMDb script we will test and correct it together.
-
For the Episode List page, it would be necessary and even better if a script was made specifically for it.
Episode list works totally fine for me, even now. It would be probably impossible for me to create new script. We can look at it like this: Episode list has it's own script: it's Selenium script "Selenium_Chrome_IMDB_Episode_List_page_v4.py" which produces txt file with all episodes, and .psf script just scrapes that txt file, so I am not sure what we could achieve with additional psf script whose purpose would be only to scrape txt file.
We'll see how it works now. Before these fixes, there was a problem with the Episode List code for the movie title and the PVD froze, but if the code in the script was blocked and the episode field was unchecked, it worked without any problems.
-
I have optimized Selenium_Chrome_Base_page_v4.py script so now downloading of the IMDb main page should be dramatically faster - around 18 seconds on my computer.
-
I have optimized Selenium_Chrome_Base_page_v4.py script so now downloading of the IMDb main page should be dramatically faster - around 18 seconds on my computer.
The Selenium_Chrome_Base_page_v4.py script works fine. However, it doesn't work at all with Firefox and Geckodriver options.
Changed the path for Geckodriver in Selenium_Chrome_Base_page_v4.py for Firefox script in 7.z file.
-
Great. I have fixed Movie Connections, and full AKA. Also fixed Selenium_Chrome_Movie_Additional_pages_v4.py so now it clicks on "See more" and similar objects, again. I will not upload next iterations because I want to fix whole Base page function first, and for me it looks that IMDb makes it harder and harder to scrape data, since for now, I'm often seeing errors IMDb killing connections to host when loading additional pages with selenium, so I have to investigate selenium and python options to mimic human browsing as best as possible.
If you still want to have current versions of my scripts, let me know so I could upload them with the manual how to get data I'm getting at the moment.
-
I have optimized Selenium_Chrome_Base_page_v4.py script so now downloading of the IMDb main page should be dramatically faster - around 18 seconds on my computer.
The Selenium_Chrome_Base_page_v4.py script works fine. However, it doesn't work at all with Firefox and Geckodriver options.
The Selenium_Chrome_Base_page_v4.py script, which was before the latest update, works on Firefox and Geckodriver options.
-
Oh, sorry to hear. I even didn't know it worked earlier and how it could work since I never published geckodriver version??? Probably I don't understand the context of your message.
-
Oh, sorry to hear. I even didn't know it worked earlier and how it could work since I never published geckodriver version??? Probably I don't understand the context of your message.
I changed all parts of the code where there is a Chrome and Chromedriver options record to Firefox and Geckodriver options record and the Selenium_Chrome_Base_page_v4.py script also worked using the Firefox browser.
As I mentioned before, with the latest Selenium_Chrome_Base_page_v4.py script update and the Chrome and Chromedriver options record, I changed it to Firefox and Geckodriver options record, but the Selenium_Chrome_Base_page_v4.py script no longer works at all using the Firefox browser.
We already talked about this a while ago in another PVD Python Scripts topic.
-
Maybe Selenium_Chrome_Movie_Additional_pages_v4.py will be loaded to see how it works.
I have optimized Selenium_Chrome_Base_page_v4.py script so now downloading of the IMDb main page should be dramatically faster - around 18 seconds on my computer.
The Selenium_Chrome_Base_page_v4.py script works fine. However, it doesn't work at all with Firefox and Geckodriver options.
I managed to edit the Selenium_Chrome_Base_page_v4.py script with Firefox and Geckodriver options using AI and it works now.
I would then do the same in the Selenium_Chrome_Movie_Additional_pages_v4.py script.
-
Great. I couldn't tell what it could be, but if I had to guess I'd say it was that new chrome.options still weren't implemented in geckodriver.
Where am I at the point?
Now I'm left with /fullcredits and /reference page. But... After reviewing new /reference design it became totaly pointless to download and parse it. It now simply doesn't have anything more than other pages already being downloaded, except full "Production Companies" and "Distributors". So, I will probably implement CompanyCredits page which is tiny and much faster to download in order to get full "Production Companies" and "Distributors". That will also drastically simplify the code.
Another important change is that I introduced new error MessageBox as seen in the last screenshot, because IMDB makes it harder and harder to fetch data in a non-human way, with Selenium or so. Read the message on it to know what it means to press any button.
More on everything in the next message.
-
Meanwhile, I have made huge fixes and improvements on all other files/scripts. Especially Main page, so now a lot of new data is pulled from the Main page which is now downloaded much faster. I encourage you to try it, by selecting fields in Script Configurator as in the screenshots, waiting PVD to restart, and then to select to overwrite all fields like in the other screenshot, and immediately start to download.
One of the biggest changes is that now I have created a procedure that enables poster to be downloaded from any page. For example, if you choose only "AKA" page to download (to update only aka's for example) you can download poster with that function too, now. Just be sure to select "Download Posters" in the Script Configurator.
I have updated and recompiled a Script Configurator with minor changes too. You need to replace all the files with the given in the attachment, and they all go only to /Scripts folder as usual.
Here's from the CHANGE LOGs of the IMDB and FilmAffinity scripts what I did:
IMDb Script
---------------------------------------------
CHANGE LOG :
V 4.1.0.1 (5/11/2025) afrocuban
- Procedure EnsurePosterDownloaded is introduced in order to be able to download poster from any page.
- Fixed pages due to change layout.
- Improved ParsePage_IMDBMovieBASE function.
- Introduced revised Messagebox now includes Cancel, Retry, or Continue (Ignore). NOTE: IF YOU PRESS IGNORE YOU WILL NOT GET DATA FROM THAT PAGE, SO CONSIDER TO RETRY OR TO CANCEL AND START DOWNLOAD AGAIN! IMDb really makes it harder and harder to get the data.
- Script Configurator decriptions adjusted to reflect actual processes.
- In corresponding Selenium scripts significantly improved downloading pages speed.
- In corresponding Selenium scripts fixed searching titles.
FilmAffinity Script
CHANGE LOG :
V 4.1.0.1-afrocuban (11/5/2025) afrocuban:
- Backup CHEAT_PREFIX_URLs introduced, since httpbin.org now almost always CREATES "503 Service Temporarily Unavailable" PROBLEMS FOR FILMAFFINITY RECENTLY. At the moment, the one that works is 'http://httpbingo.org/response-headers?key= (http://httpbingo.org/response-headers?key=)'
(*@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
// IF NON OF THESE WORK THEN THERE ARE NO OTHER SERVICES AVAILABLE AND THE ONLY OPTION LEFT IS TO INSTALL LOCAL HTTPBIN LIKE THIS
Install and run httpbin locally like this:
```bash
pip install httpbin
python -m httpbin.core
```
It will start a local server on port `5000`, and you can absolutely use:
```
http://localhost:5000/response-headers?key=https://www.filmaffinity.com/en/film699169.html/ (http://localhost:5000/response-headers?key=https://www.filmaffinity.com/en/film699169.html/)
```
Just like you would with the public httpbin.org — and it will behave the same way: echoing back the `key` header in the response.
---
### ? What to expect
When you visit that URL in your browser or send a request via code, you’ll get a JSON response like:
```json
{
"key": "https://www.filmaffinity.com/en/film699169.html/ (https://www.filmaffinity.com/en/film699169.html/)"
}
{
```
No HTTPS redirection, no 503 errors, and full control — because it’s running locally.
---
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
- Fixed "FA Critics" field due to layout changes.
- Fixed "FAMovieTrailers" field to now properly include source name (youtube, dailymotion, etc).
- Other layout changes fixed.
- In corresponding Selenium scripts significantly improved downloading pages speed, especially Trailers page..
- In corresponding Selenium scripts fixed searching titles due to FilmAffinity Search page layout changed. (Movie title now has to be retrieved from the section for mobile devices).
- Other minor layout changes updated, and the script cleaned additionaly.
So with these scripts, you can get all possible data for now except:
- Full Cast (for now you can get only cast found on the main page)
- Producers
- Composers
- Full Production Companies
- Distributors
- All Directors for Series (only one director for now if you download /fullcredits page, because I have just started to fix it).
I will not provide support for these scripts until I finish all because I know IMDB script still doesn't fully work. I am interested in you to test border case titles which I'd fix upon finishing. For that, please provide the link and the log for specific field (or what you get in the field and what you expect). I will be able to fix things only if I can reproduce them and for that you need to provide me with data above.
-
I forgot to upload ahk for the Script Configurator .exe
SeleniumPVDbScriptsConfig-v4.exe is compiled with ahk2exe for AutoHotkey v1.1.37.02 option "U32 (default) bin", without compression.
-
I have finished CompanyCredits function and now no need for Reference page at all. Consequently, I have updated, redesigned and compiled Script Configurator.
Now I need to update FullCredits function, meaning to get full cast & crew, directors for series, producers and composers and everything will be done. I will not clean Reference page from the code. I will only comment it out, so it could be possibly used in the future IMDb page changes.
Most probably I will not post again until finishing everything.
-
I have completed FullCredits function, so now all the data can be imported again to PVD. Before I publish it I want to check 2 things:
1. What to do with Reference page code, and should I exclude its options from Script Configurator, since now it is completely not needed.
2. To check People script and if I can fix it quickly, then I will publish all the scripts and files again in one, final package for this IMDb html layout change.
After that please check the scripts and let me know what doesn't work
-
I have completed FullCredits function, so now all the data can be imported again to PVD. Before I publish it I want to check 2 things:
1. What to do with Reference page code, and should I exclude its options from Script Configurator, since now it is completely not needed.
2. To check People script and if I can fix it quickly, then I will publish all the scripts and files again in one, final package for this IMDb html layout change.
After that please check the scripts and let me know what doesn't work
First of all, Function ParsePage_IMDBMovieREFERENCE should be moved to the end of the script before Function ParsePage.
Secondly, the entire part of the reference page code should be left and, as mentioned above, before Function ParsePage, or even better, it should be moved to the very end of the script, where there is already a History of changes, so that it can be re-included in the script if necessary in the future or completely blocked. All its options should also be excluded from both the script code and the script configurator, because now, as mentioned, it is no longer needed.
As for the People script, I don't know what works or doesn't work, because I don't use it.
-
Yes, I just commented out reference code in the script, and I already intended to leave it there and everywhere it is mentioned, to preserve the logic for the future. I was actually talking to comment it out in the Script Configurator too, to avoid inexperienced users to click on those options expecting to get data from /reference page. I will move the function before ParsePage function, though, that is a nice tip.
Now, meanwhile, I introduced another function:
function WaitForPageFile(const FileName, PageLabel: string;
InitialWait: Integer; StabilizeMs: Integer): string; //BlockOpen
var
i, currentWait: Integer;
tryResult: string;
begin
Result := '';
i := 0;
// provide defaults manually
if InitialWait = 0 then
InitialWait := 2000;
if StabilizeMs = 0 then
StabilizeMs := 1000;
currentWait := InitialWait;
while not FileExists(FileName) do
begin
LogMessage('Function DownloadPage - Waiting ' + IntToStr(currentWait div 1000) +
's for the presence of: ' + FileName + '|');
Wait(currentWait);
// escalate wait times
case i of
0: currentWait := 5000;
1: currentWait := 15000;
2: currentWait := 10000;
3: currentWait := 10000;
4: currentWait := 15000;
5: currentWait := 15000;
end;
Inc(i);
if i = INTERNET_TEST_ITERATIONS then
begin
case MessageBox('IMDb Movie Function DownloadPage - Too many faulty attempts to internet connection for ' + PageLabel +
'. Cancel, Retry, or Continue (Ignore)? NOTE: IF YOU PRESS IGNORE YOU WILL NOT GET DATA FROM THAT PAGE, SO CONSIDER TO RETRY OR TO CANCEL AND START DOWNLOAD AGAIN! IMDb really makes it harder and harder to get the data.',
SCRIPT_FILE_NAME, 2) of
3: // Cancel
begin
LogMessage('Function DownloadPage for ' + PageLabel +
' ended with NO INTERNET connection ===============|');
Result := '';
Exit;
end;
4: // Retry
begin
i := 0;
currentWait := InitialWait;
end;
5: // Ignore
begin
LogMessage('Function DownloadPage - Creating dummy ' + PageLabel +
' HTML file due to Ignore selection|');
with TStringList.Create do
try
Add('<html><body>Dummy ' + PageLabel + ' due to user Ignore.</body></html>');
SaveToFile(FileName);
finally
Free;
end;
Break;
end;
end;
end;
end;
// stabilization wait
LogMessage('Function DownloadPage - ' + PageLabel +
' file detected, waiting extra ' + IntToStr(StabilizeMs) + 'ms to stabilize...');
Wait(StabilizeMs);
// manual error handling instead of try/except
if not FileExists(FileName) then
begin
LogMessage(ProcException('FileError', 'NOT_FOUND: ' + FileName));
Exit;
end;
tryResult := FileToString(FileName); // if your environment throws, replace with safe read
if tryResult = '' then
LogMessage(ProcException('FileError', 'FAILED to read ' + FileName))
else
begin
Result := ConvertEncoding(tryResult, 65001);
LogMessage(ProcException('FileRead', 'SUCCESS: ' + FileName));
end;
end; //BlockClose
so, now instead this snippet for every page:
// Initialize currentWait for the FullCredits file
currentWait := 5000; // Start with 5 seconds
//Wait for the FullCredits file to finish downloading
If Not(((USE_SAVED_PVDCONFIG) And ((ConfigOptions[8] = '0') And (ConfigOptions[10] = '0'))) Or
(GET_FULL_CREDIT_FROM_REFERENCE And ((Pos('Series', MediaType) = 0) And (Pos('Series', GetFieldValueXML('category')) = 0)))) Then Begin // Also Known As (FullCredits)
i := 0;
currentWait := 2000; // Initialize wait time
while not FileExists(FilePath + FileTitleFullCredits) do begin
LogMessage('Function DownloadPage - Waiting ' + IntToStr(currentWait div 1000) + 's (because of the people like Alfonso Cuaron with 268 wins and 208 nominations at the moment of writing this script) for the presence of: ' + FilePath + FileTitleFullCredits);
wait(currentWait);
// Increment the wait time for the next iteration
case i of
0: currentWait := 5000; // 5 seconds
1: currentWait := 15000; // 15 seconds
2: currentWait := 10000; // 10 seconds
3: currentWait := 10000; // 10 seconds
4: currentWait := 15000; // 15 seconds
5: currentWait := 15000; // 15 seconds
end;
i := i + 1;
if i = INTERNET_TEST_ITERATIONS then
begin
case MessageBox('IMDb Movie Function DownloadPage - Too many faulty attempts to internet connection for FullCredits. Cancel, Retry, or Continue (Ignore)? NOTE: IF YOU PRESS IGNORE YOU WILL NOT GET DATA FROM THAT PAGE, SO CONSIDER TO RETRY OR TO CANCEL AND START DOWNLOAD AGAIN! IMDb really makes it harder and harder to get the data.', SCRIPT_FILE_NAME, 2) of
3: // Abort -> treat as Cancel
begin
LogMessage('Function DownloadPage for FullCredits END with NO INTERNET connection =============== |');
Result := '';
Exit;
end;
4: // Retry
begin
i := 0;
currentWait := 2000; // Reset wait time
end;
5: // Ignore->create dummy file
begin
LogMessage('Creating dummy FullCredits HTML file due to Ignore selection...');
with TStringList.Create do
try
Add('<html><body>Dummy FullCredits due to user Ignore.</body></html>');
SaveToFile(FilePath + FileTitleFullCredits);
finally
Free;
end;
Result := FilePath + FileTitleFullCredits;
end;
end;
end;
end;
// Add a short stabilization wait after the file is recognized
LogMessage('Function DownloadPage - CHANGETHISWITHPROPERFILENAME file detected, waiting extra 1s to stabilize...');
Wait(2000); // wait 2 second (adjust as needed)
WebText := FileToString(FilePath + FileTitleFullCredits);
WebText := ConvertEncoding(WebText, 65001); // UTF-8
FullCreditsPageDownloaded := True;
LogMessage('Function DownloadPage - FullCredits file found: ' + FilePath + FileTitleFullCredits);
LogMessage('Value of FullCreditsPageDownloaded: ' + BoolToStr(FullCreditsPageDownloaded));
end;
we will have only this
if not (USE_SAVED_PVDCONFIG and (ConfigOptions[15] = '0')) then
begin
WebText := WaitForPageFile(FilePath + FileTitleParentalGuide, 'ParentalGuide', 5000, 2000);
ParentalGuidePageDownloaded := WebText <> '';
LogMessage('Function DownloadPage - Value of ParentalGuidePageDownloaded: ' +
BoolToStr(ParentalGuidePageDownloaded));
end;
That way we will speed up the script and have hundreds if not thousands of lines less in the script.
I will do that and test for all repetitive tasks.
Also, I'm exploring ways IMDb not to refuse connections with selenium/python and testing at the moment creating fake userData folders, rotating user agents, and it goes good for now.
-
As of now, per title, I'm getting these timings:
Main Page downloading: ~18-22sec
Other 9 pages Page downloading: ~48-55 sec
Image downloading: ~6-8 sec
PVD script processing: ~5-10 sec
So in total for now, to get biggest possible amount of data per title, especially those with many awards, connections, etc... it is needed ~77-95 sec which is pretty acceptable for the amount of data.
-
Oh, and please remind me to upload purge_tmp_files.vbs if I forget, because I changed it by adding to delete fake UserData folders now too
-
Oh, and please remind me to upload purge_tmp_files.vbs if I forget, because I changed it by adding to delete fake UserData folders now too
Ok.
-
Here are all scripts and files fully updated, fixed and polished in a less than a month I started to fix all 16 of them, and I was so happy I got back into it easily and quickly. I have tested all scripts and files against many border case titles and persons and for me everything worked more than smooth and satisfying.
They are now faster and more stable and I am not facing anymore internet interruptions, because I heavily redesigned the most problematic python selenium scripts.
If especially Selenium_Chrome_Movie_Additional_pages_v4.py script is demanding for your CPU when downloading movies and you experience lags of any kind, open the file in Notepad++ and in the line 375:
with ThreadPoolExecutor(max_workers=4) as executor:
reduce number 4 to 3, 2 or 1, just test it. Whenever you lessen the number, the process of downloading files will be longer, so find your balance. If you have good CPU and a lot of RAM, then you can even increase the number above 4.
I'd be happy to further fine tuning and fix it, so please let me know about each case details so I could reproduce it too and then being able to fix it. If you have any further suggestion, I'd be happy to hear it as well while I didn't forget it again, but please explain why and how by giving specific examples, because I am not a programmer, but just using common sense and AI, and that is the only way I can understand the problem.
My plan is versions to stay on v4.2 for a long time unless something significant in their design changes.
Enjoy! ;) :)
-
My next goal is to include new switch in the Script Configurator - UPDATE_DYNAMIC_VALUES_ONLY, by adding few dozens of lines into movie selenium script that would call only main page and update only dynamic values like: Rating, Top 250, Bottom 100, Number of votes. And for the Awards summary when the movie is not older than 2 years than current date catching fresh wins for recent releases.
-
My next goal is to include new switch in the Script Configurator - UPDATE_DYNAMIC_VALUES_ONLY, by adding few dozens of lines into movie selenium script that would call only main page and update only dynamic values like: Rating, Top 250, Bottom 100, Number of votes. And for the Awards summary when the movie is not older than 2 years than current date catching fresh wins for recent releases.
Well, I finished it earlier than expected, for IMDb Movie script. Also with a lot of chalenges I have redesigned Script Configurator once again, bringing new functionalities:
//Retreive Data Config
USE_SAVED_PVDCONFIG = True ; // ***PVDCONFIG*** - Turn this ON to unlock and change the options below (from pvdconf.ini). Settings are applied when you click "Save All Script Configurations (Personal Video Database will automatically restart)" button below. Use carefully!
//############################################
//# All options below require USE_SAVED_PVDCONFIG
//# to be enabled so the Script Configurator
//# can apply your settings correctly.
//############################################
UPDATE_DYNAMIC_VALUES_ONLY = True ; //Update only dynamic values such as: Rating, Top 250, Metascore, Number of votes. Also update the Awards summary when the movie is less than 2 years old, to capture fresh wins for recent releases. Deselect to enable the options
//################################################
//# All options below require UPDATE_DYNAMIC_VALUES_ONLY
//# to be enabled so the Script Configurator
//# can apply your settings correctly.
//###############################################
So you can see in the screenshots that now checking specific boxes disables or enables other options. Which means that....
My plan is versions to stay on v4.2 for a long time unless something significant in their design changes.
my plan will not last long since thee changes are huge for users to make them easy navigating and choosing proper options withoout to much contemplating, so with this I will soon go to 4.3.
But I still will not publish anything, because I want to finish People and FA scripts in terms of UPDATE_DYNAMIC_VALUES_ONLY, and I also want to further tweak Script Configurator GUI. I just hate "Save button" will not autosize to the last option in each tab, so for example in a People and FA tab we have to scroll all the way down. That is not just visual thing, but I rather want to implement "Apply" button that will be applied to each tab independently, while we will have overall "Cancel" and "Save & Restart PVD" button. That is tremendous challenge for ahk, that made me last year even to start creating GUI with python, but at the moment it looked even more difficult with python, so I abandoned it then. Now it looks the time to try it again is spot on.
-
New v4.3 Files
Most comprehensive and stable I have done so far. Two main things:
1. New PVD Scripts Configurator built from scratch in Python Tkinter. Much better GUI than AHK.
Nice quirk - I have introduced dark/light theme for it as in the first photo. Reordered tabs, so default tab is IMDb Movie tab. You can still use both configurators (second photo - new Configurator has prefix "py"), all updated to v4.3, same options and functionalities. I'm giving them all now, for continuity, but in the future, most probably I will discontinue AHK. In any case, I'm giving .ahk file so anyone can maintain it in the future. I will stick to Python TKinter GUI.
2. As anounced earlier, new **UPDATE DYNAMIC VALUES ONLY** switch for a fast update of only certain dynamic fields.
From the Change logs:
IMDb Movie Script
CHANGE LOG :
V 4.3.0.1 (11/15/2025) afrocuban: Script Configurator Enhancements
-------------------------------------------------------------------------------
- Built from the scratch new PythonTkinter Script Conigurator:
• It has all the functionalities as AHK.
• Plus litght/dark theme developed
• Unlike AHK can be used as a standalone application with the same effect as when invoked in PVD.
- Added new feature in Script Configurator to enable and manage saved settings from `pvdconf.ini`:
• **USE SAVED PVDCONFIG** now needs to be enabled to unlock configuration options below.
• This allows users to apply settings that require a restart of Personal Video Database upon saving.
• **Use this setting carefully!** Any changes will take effect only after clicking "Save All Script Configurations" (which will restart the application).
- **UPDATE DYNAMIC VALUES_ONLY**:
• Allows users to update only **dynamic values** like Rating, Top 250, Metascore, and Number of votes.
• Updates the **Awards summary** for movies released within the last two years, capturing recent wins for fresh releases.
• When disabled, additional configuration options become available for comprehensive updates.
• Now poster can be downloaded from any page - separate procedure provided for it in Script Configurator,
• Single Instance in Script Configurator. No more flooding with multiple instance by mistake.
• Redesigned whole script to now accept UPDATE DYNAMIC VALUES ONLY switch properly
FilmaFfinity ScriptCHANGE LOG :
V 4.3.0.1 (11/27/2025) afrocuban: Script Configurator Enhancements
-------------------------------------------------------------------------------
- Built from the scratch new PythonTkinter Script Conigurator:
• It has all the functionalities as AHK.
• Plus litght/dark theme developed
• Unlike AHK can be used as a standalone application with the same effect as when invoked in PVD.
- Added new feature in Script Configurator to enable and manage saved settings from `pvdconf.ini`:
• **USE SAVED PVDCONFIG** now needs to be enabled to unlock configuration options below.
• This allows users to apply settings that require a restart of Personal Video Database upon saving.
• **Use this setting carefully!** Any changes will take effect only after clicking "Save All Script Configurations" (which will restart the application).
- **UPDATE DYNAMIC VALUES_ONLY**:
• Allows users to update only **dynamic values** like Rating, Number of votes and Awards for movies made in last 2 years.
• When disabled, additional configuration options become available for comprehensive updates.
IMDb People Script
CHANGE LOG :
V 4.3.0.1 (11/27/2025) afrocuban: Script Configurator Enhancements
-------------------------------------------------------------------------------
- Built from the scratch new PythonTkinter Script Conigurator:
• It has all the functionalities as AHK.
• Plus litght/dark theme developed
• Unlike AHK can be used as a standalone application with the same effect as when invoked in PVD.
- Added new feature in Script Configurator to enable and manage saved settings from `pvdconf.ini`:
• **USE SAVED PVDCONFIG** now needs to be enabled to unlock configuration options below.
• This allows users to apply settings that require a restart of Personal Video Database upon saving.
• **Use this setting carefully!** Any changes will take effect only after clicking "Save" (which will restart the application).
- **UPDATE DYNAMIC VALUES_ONLY**:
• Allows users to update only **dynamic values** for persons that were alive at the moment of adding them to PVD, or at their last update. Updating only from the Main page.
• When disabled, additional configuration options become available for comprehensive updates.
-
Here are v4.3 scripts.
1. Unpack the first 7z in "PersonalVideoDB" folder (3 files, overwrite existing, but backup them first if you want).
2. Unpack the second 7z in your "Scripts" folder. It is safe to move everything from it before extracting. These are all you need to safely run PVD with python.
You need all of these in order PVD to run as intended. Especially People script is complex, since I have integrated options to make it easier to dynamically update them and not to wait at all for deceased or the people that have only name and url. Test it.
If something desn't work, first check:
1. That you are running same version of Chrome and chromedriver.
2. That you installed whatever is needed for Python to work as described so far on this topic.
If that doesn't help, please publish screenshots and logs, so I could reproduce the issue too and being able to fix it.
Please test and let me know if everything work or not.
Enjoy!
-
I tested it, IMDB_Movie_[EN][Selenium]-v4.psf and IMDB_People_[EN][Selenium]-v4.psf work fine, with some errors in certain information, where it will be necessary to fix parts of the code in the scripts. The errors have been there for a long time and have not been fixed. The code fixes will be described and added below.
As for FilmAffinity_Movie_[EN][Selenium]-v4.psf, I have not tested it.
-
IMDB_Movie_[EN][Selenium]-v4.psf
Fixed part of the code in Function ParsePage_IMDBMovieMPAA
Previous
// Get (CF~IMDbmpaaSummary~)
curPos := Pos('<select id="jump-to"', HTML);
If curPos > 0 Then
Begin
// Extract the relevant section for categories
curPos := PosFrom('<option', HTML, curPos);
endPos := PosFrom('</select>', HTML, curPos);
mpaaSection := Copy(HTML, curPos, endPos - curPos);
LogMessage('Function ParsePage_IMDBMovieMPAA - Extracted Category section for (CF~IMDbmpaaSummary~): ' + mpaaSection);
// Parse the options and category names
curPos := 1;
mpaaContent := '';
While PosFrom('<option', mpaaSection, curPos) > 0 Do
Begin
curPos := PosFrom('<option', mpaaSection, curPos) + Length('<option');
optionValue := TextBetween(mpaaSection, 'value="', '">', False, curPos);
categoryName := TextBetween(mpaaSection, '">', '</option>', False, curPos);
// Format the category link
mpaaContent := mpaaContent + '<link url="' + MovieURL + optionValue + '">' + categoryName + ' • </link>';
// Move to the next position
curPos := PosFrom('</option>', mpaaSection, curPos) + Length('</option>');
End;
// Remove the trailing " • " if it exists and keep the last </link> intact
If Copy(mpaaContent, Length(mpaaContent) - 13, 7) = ' • ' Then
Begin
mpaaContent := Copy(mpaaContent, 1, Length(mpaaContent) - 14) + '</link>';
End;
// Combine and format the final result
mpaaContent := '<link url="' + MovieURL + '#contentRating' + '">Content Ratings Summary: </link>' + mpaaContent;
// Store the result in the custom field
AddCustomFieldValueByName('IMDbmpaaSummary', mpaaContent);
LogMessage('Function ParsePage_IMDBMovieMPAA - Stored result for (CF~IMDbmpaaSummary~)');
End
Else
Begin
LogMessage('Function ParsePage_IMDBMovieMPAA - Content Rating section for (CF~IMDbmpaaSummary~) not found');
//Result := prError; // Set to error if content rating section is not found
End;
Fixed
// Get (CF~IMDbmpaaSummary~)
curPos := Pos('<select id="jump-to"', HTML);
If curPos > 0 Then
Begin
// Extract the relevant section for categories
curPos := PosFrom('<option', HTML, curPos);
endPos := PosFrom('</select>', HTML, curPos);
mpaaSection := Copy(HTML, curPos, endPos - curPos);
LogMessage('Function ParsePage_IMDBMovieMPAA - Extracted Category section for (CF~IMDbmpaaSummary~): ' + mpaaSection);
// Parse the options and category names
curPos := 1;
mpaaContent := '';
While PosFrom('<option', mpaaSection, curPos) > 0 Do
Begin
curPos := PosFrom('<option', mpaaSection, curPos) + Length('<option');
optionValue := TextBetween(mpaaSection, 'value="', '">', False, curPos);
categoryName := TextBetween(mpaaSection, '">', '</option>', False, curPos);
// Format the category link
// mpaaContent := mpaaContent + '<link url="' + MovieURL + optionValue + '">' + categoryName + ' • </link>';
mpaaContent := mpaaContent + '<link url="' + MovieURL + optionValue + '">' + categoryName + '</link> • ';
// Move to the next position
curPos := PosFrom('</option>', mpaaSection, curPos) + Length('</option>');
End;
// Remove the trailing " • " if it exists and keep the last </link> intact
If Copy(mpaaContent, Length(mpaaContent) - 13, 7) = ' • ' Then
Begin
mpaaContent := Copy(mpaaContent, 1, Length(mpaaContent) - 14) + '</link>';
End;
// Combine and format the final result
//mpaaContent := '<link url="' + MovieURL + '#contentRating' + '">Content Ratings Summary: </link>' + mpaaContent;
mpaaContent := '<link url="' + MovieURL + '#contentRating' + '">Content Ratings Summary:</link> ' + mpaaContent;
// Store the result in the custom field
AddCustomFieldValueByName('IMDbmpaaSummary', mpaaContent);
LogMessage('Function ParsePage_IMDBMovieMPAA - Stored result for (CF~IMDbmpaaSummary~)');
End
Else
Begin
LogMessage('Function ParsePage_IMDBMovieMPAA - Content Rating section for (CF~IMDbmpaaSummary~) not found');
//Result := prError; // Set to error if content rating section is not found
End;
-
IMDB_Movie_[EN][Selenium]-v4.psf
Fixed part of the code in Function ParsePage_IMDBMovieCONNECTIONS
Previous
// Get (CF~Connections~) info
If Pos('>Connections<', HTML) > 0 Then
Begin
LogMessage('===Function ParsePage_IMDBMovieCONNECTIONS - Starting Connections Parsing ===');
curPos := Pos('>Connections<', HTML);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - curPos for "Connections" section found at: ' + IntToStr(curPos));
If Pos('It looks like we don'+Chr(39)+'t have any Connections for this title yet.', HTML) > 0 Then
Begin
Category1 := 'No Connections';
Category3 := '';
AddCustomFieldValueByName('Connections', Category1);
AddCustomFieldValueByName('Connect', Category3);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - No Connections found for this title');
End Else Begin
// Extract the relevant section for categories
If PosFrom('<span class="ipc-simple-select__container" data-testid="jumpTo">', HTML, curPos) > 0 Then Begin
curPos := PosFrom('<span class="ipc-simple-select__container" data-testid="jumpTo">', HTML, curPos);
endPos := PosFrom('</select></span></span>', HTML, curPos);
Category2 := Copy(HTML, curPos, endPos - curPos);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Extracted Category2 section: ' + Category2);
// Parse the options and category names
curPos := 1;
Category1 := '';
While PosFrom('<option', Category2, curPos) > 0 Do
Begin
curPos := PosFrom('<option', Category2, curPos) + Length('<option');
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - curPos at option tag: ' + IntToStr(curPos));
optionValue := TextBetween(Category2, 'value="', '">', False, curPos);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Extracted optionValue: ' + optionValue);
categoryName := TextBetween(Category2, '">', '</option>', False, curPos);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Extracted categoryName: ' + categoryName);
// Format the category link
Category1 := Category1 + '<link url="' + MovieURL + optionValue + '">' + categoryName + ' • </link>';
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Formatted and added category link: ' + '<link url="' + MovieURL + optionValue + '">' + categoryName + ' • </link>');
// Move to the next position
curPos := PosFrom('</option>', Category2, curPos) + Length('</option>');
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Moved curPos to next option tag: ' + IntToStr(curPos));
End;
// Remove the trailing " • " if it exists and keep the last </link> intact
If Copy(Category1, Length(Category1) - 13, 7) = ' • ' Then
Begin
Category1 := Copy(Category1, 1, Length(Category1) - 14) + '</link>';
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Removed trailing " • " and kept last </link> in Category1');
End;
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Final Category1 before combining: ' + Category1);
// Combine and format the final result
Category1 := '<link url="' + MovieURL + '">Connections: </link>' + Category1;
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Combined and formatted final Category1: ' + Category1);
// Store the result in the custom field
AddCustomFieldValueByName('Connections', Category1);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Stored result in (CF~Connections~)');
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Final Category1 stored: ' + Category1);
LogMessage('===Function ParsePage_IMDBMovieCONNECTIONS - Finished (CF~Connections~) Parsing ===');
End;
Fixed
// Get (CF~Connections~) info
If Pos('>Connections<', HTML) > 0 Then
Begin
LogMessage('===Function ParsePage_IMDBMovieCONNECTIONS - Starting Connections Parsing ===');
curPos := Pos('>Connections<', HTML);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - curPos for "Connections" section found at: ' + IntToStr(curPos));
If Pos('It looks like we don'+Chr(39)+'t have any Connections for this title yet.', HTML) > 0 Then
Begin
Category1 := 'No Connections';
Category3 := '';
AddCustomFieldValueByName('Connections', Category1);
AddCustomFieldValueByName('Connect', Category3);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - No Connections found for this title');
End Else Begin
// Extract the relevant section for categories
If PosFrom('<span class="ipc-simple-select__container" data-testid="jumpTo">', HTML, curPos) > 0 Then Begin
curPos := PosFrom('<span class="ipc-simple-select__container" data-testid="jumpTo">', HTML, curPos);
endPos := PosFrom('</select></span></span>', HTML, curPos);
Category2 := Copy(HTML, curPos, endPos - curPos);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Extracted Category2 section: ' + Category2);
// Parse the options and category names
curPos := 1;
Category1 := '';
While PosFrom('<option', Category2, curPos) > 0 Do
Begin
curPos := PosFrom('<option', Category2, curPos) + Length('<option');
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - curPos at option tag: ' + IntToStr(curPos));
optionValue := TextBetween(Category2, 'value="', '">', False, curPos);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Extracted optionValue: ' + optionValue);
categoryName := TextBetween(Category2, '">', '</option>', False, curPos);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Extracted categoryName: ' + categoryName);
// Format the category link
//Category1 := Category1 + '<link url="' + MovieURL + optionValue + '">' + categoryName + ' • </link>';
// Category1 := Category1 + '<link url="' + MovieURL + optionValue + '">' + categoryName + '</link> • ';
Category1 := Category1 + '<link url="' + MovieURL + optionValue + '">' + categoryName + '</link> ';
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Formatted and added category link: ' + '<link url="' + MovieURL + optionValue + '">' + categoryName + ' • </link>');
// Move to the next position
curPos := PosFrom('</option>', Category2, curPos) + Length('</option>');
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Moved curPos to next option tag: ' + IntToStr(curPos));
End;
// Remove the trailing " • " if it exists and keep the last </link> intact
If Copy(Category1, Length(Category1) - 13, 7) = ' • ' Then
Begin
Category1 := Copy(Category1, 1, Length(Category1) - 14) + '</link>';
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Removed trailing " • " and kept last </link> in Category1');
End;
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Final Category1 before combining: ' + Category1);
// Combine and format the final result
//Category1 := '<link url="' + MovieURL + '">Connections: </link>' + Category1;
if Category1 <> '' then
Category1 := '<link url="' + MovieURL + '">Connections:</link> • ' + Category1
Else
Category1 := '<link url="' + MovieURL + '">Connections:</link> ' + Category1;
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Combined and formatted final Category1: ' + Category1);
// Store the result in the custom field
AddCustomFieldValueByName('Connections', Category1);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Stored result in (CF~Connections~)');
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Final Category1 stored: ' + Category1);
LogMessage('===Function ParsePage_IMDBMovieCONNECTIONS - Finished (CF~Connections~) Parsing ===');
End;
-
Hey, Ivek. Thanks. Can you please post examples or imdb links where those matters and my code doesn't work? I just can't grasp just by looking at the code. Thanks.
-
IMDB_Movie_[EN][Selenium]-v4.psf
Fixed part of the code in Function ParsePage
Remove
AddCustomFieldValueByName('LDDbSearch', '<link url="https://www.lddb.com/search/IMDb/' + MovieID1 + '">LaserDiscDbSearch</link> ' + '<link url="http://www.soundtrack.net/search/?q=' + GetFieldValue(2) + '">Soundtrack.NetSearch</link> ' + '<link url="https://store.intrada.com/s.nl?sc=16&category=&search=' + GetFieldValue(2) + '">Intrada</link> ' + '<link url="https://www.aveleyman.com/?Film.aspx/' + GetFieldValue(2) + '">Aveleyman</link> ' + '<link url="https://www.bing.com/Search?q=' + GetFieldValue(2) + '%20site%3Awww.aveleyman.com/">BingAvSearch</link> ' + '<link url="https://www.google.com/search?q=' + GetFieldValue(2) + '%20site%3Awww.aveleyman.com/">GoogleAvSearch</link> ' + '<link url="https://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Dmovies-tv&field-keywords=' + GetFieldValue(2) + '">AmazonSearch</link> ' + '<link url="http://www.impawards.com/cgi-bin/htsearch?method=or&words=' + GetFieldValue(2) + '">IMPawards/cgi-bin</link> ' + '<link url="http://www.impawards.com/search.php/' + GetFieldValue(2) + '">impawards.com</link>');
Previous
//Get ~IMDb Movie Url~ (CF~IMDb.com~) and (CF~WaybackArchive IMDb URLs~)
Movie_URL := StringReplace(DownloadURL, BASE_URL_PRE_TRUE, BASE_URL_PRE, True, False, False);
AddCustomFieldValueByName('IMDb.com', '<link url="' + Movie_URL + '">IMDb.com</link>');
AddCustomFieldValueByName('WaybackArchive IMDb URLs', '<link url="https://web.archive.org/web/*/' + Movie_URL + '*">*IMDb.com*</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'business">Busines</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'combined">Combined</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'fullcredits">Full Cast&Crew</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'awards">Awards</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'releaseinfo">ReleaseDates</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'technical">TechSpecs</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'parentalguide">ParentsGuide</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'keywords">PlotKeywords</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'plotsummary">PlotSummary</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'taglines">Taglines</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'movieconnections">Connections</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'soundtrack">Soundtracks</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'externalsites">MiscSites</link>');
AddCustomFieldValueByName('WaybackArchive IMDb URLs', '<link url="https://web.archive.org/web/*/' + Movie_URL + '*">*IMDb.com*</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'business">Busines</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'combined">Combined</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'technical">TechSpecs</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'dvd">DVD</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'laserdisc">Laserdisc</link>');
Fixed
//Get ~IMDb Movie Url~ (CF~IMDb.com~) and (CF~WaybackArchive IMDb URLs~)
Movie_URL := StringReplace(DownloadURL, BASE_URL_PRE_TRUE, BASE_URL_PRE, True, False, False);
AddCustomFieldValueByName('IMDb.com', '<link url="' + Movie_URL + '">IMDb.com</link>');
AddCustomFieldValueByName('WaybackArchive IMDb URLs', '<link url="https://web.archive.org/web/*/' + Movie_URL + '*">*IMDb.com*</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'business">Busines</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'combined">Combined</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'fullcredits">Full Cast&Crew</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'awards">Awards</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'releaseinfo">ReleaseDates</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'technical">TechSpecs</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'parentalguide">ParentsGuide</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'keywords">PlotKeywords</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'plotsummary">PlotSummary</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'taglines">Taglines</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'movieconnections">Connections</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'soundtrack">Soundtracks</link> <link url="https://web.archive.org/web/*/' + Movie_URL + 'externalsites">MiscSites</link>');
Fixed
Fullinfo := '';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + '">MainPage</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'reference">Reference</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'fullcredits">Full Cast&Crew</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'awards">Awards</link> ';
//Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'releaseinfo">ReleaseDates</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'technical">TechSpecs</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'parentalguide">ParentsGuide</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'keywords">PlotKeywords</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'plotsummary">PlotSummary</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'locations">FilmLocations</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'taglines">Taglines</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'companycredits">CompanyCredits</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'movieconnections">Connections</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'soundtrack">Soundtracks</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'externalsites">MiscSites</link> ';
//Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'externalsites#photos">MiscPhoto</link> ';
Fullinfo:=Fullinfo+'<link url="'+Movie_URL+'externalsites/#misc">MiscSites</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'externalsites#photo">MiscPhoto</link> ';
Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'mediaindex">PhotoGallery</link> ';
Fullinfo := Fullinfo + '<link url="http://www.imdb.com/chart/top">IMDb Top 250</link> ';
Fullinfo := Fullinfo + '<link url="http://www.imdb.com/chart/bottom">Lowest Rated</link> ';
Fullinfo := Fullinfo + '<link url="http://www.imdb.com/chart/toptv">Top 250 TV</link>';
If Fullinfo <> '' then IMDB_URL := Fullinfo;
If IMDB_URL <> '' then AddCustomFieldValueByName('IMDb Url', IMDB_URL);
If IMDB_URL <> '' then AddCustomFieldValueByName('IMDb Movie Url', IMDB_URL);
LogMessage('Function ParsePage - Get result Fullinfo-IMDB_URL (CF~IMDb Movie Url~ ): ' + #13 + IMDB_URL + '| |');
Remove
AddCustomFieldValueByName('IMDbExternalSitesUrl', '<link url="' + Movie_URL + 'releaseinfo">ReleaseDates</link> <link url="' + Movie_URL + 'plotsummary">PlotSummary</link> <link url="' + Movie_URL + 'companycredits">CompanyCredits</link> <link url="' + Movie_URL + 'movieconnections">Connections</link> <link url="' + Movie_URL + 'externalsites/#misc">MiscSites</link> <link url="' + Movie_URL + 'externalsites#photos">MiscPhoto</link>');
Fixed
Fullinfo1 := '';
Fullinfo1:=Fullinfo1+'<link url="http://www.boxofficemojo.com/title/'+MovieID+'">BoxOfficeMojo</link> ';
Fullinfo1:=Fullinfo1+'<link url="http://en.wikipedia.org/w/index.php?search='+GetFieldValue(2)+'">Wikipedia</link> ';
Fullinfo1:=Fullinfo1+'<link url="https://www.movieposterdb.com/search?category=title&q='+MovieID+'">MoviePosterDB Info</link> ';
Fullinfo1:=Fullinfo1+'<link url="https://www.dvdsreleasedates.com/search/?searchStr='+GetFieldValue(2)+'">DVDs ReleaseDates</link> ';
Fullinfo1:=Fullinfo1+'<link url="https://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Dmovies-tv&field-keywords='+GetFieldValue(2)+'">AmazonSearch</link> ';
Fullinfo1:=Fullinfo1+'<link url="https://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Dmovies-tv&field-keywords='+GetFieldValue(2)+'%20'+GetFieldValue(5)+'">AmazonSearch1</link> ';
Fullinfo1:=Fullinfo1+'<link url="https://www.allmovie.com/search/movies/'+GetFieldValue(2)+'">AllMovieSearch</link> ';
Fullinfo1:=Fullinfo1+'<link url="https://www.themoviedb.org/search?query=' +GetFieldValue(2)+'">TMDBSearch</link> ';
Fullinfo1:=Fullinfo1+'<link url="https://www.rottentomatoes.com/search/?search='+GetFieldValue(2)+'">RottenTomatoesSearch</link> ';
Fullinfo1:=Fullinfo1+'<link url="https://www.thetvdb.com/search?query='+GetFieldValue(2)+'">TVDB Search</link> ';
Fullinfo1:=Fullinfo1+'<link url="https://www.soundtrackcollector.com/catalog/search.php?searchon=all&searchtext='+GetFieldValue(2)+'">SoundCollSearch</link> ';
Fullinfo1:=Fullinfo1+'<link url="https://www.blu-ray.com/search/?quicksearch=1&quicksearch_country=all&quicksearch_keyword='+MovieID+'§ion=theatrical">BR.comSearch</link> ';
Fullinfo1:=Fullinfo1+'<link url="https://www.lddb.com/search/IMDb/'+MovieID1+'">LaserDiscDb Search</link> ';
Fullinfo1:=Fullinfo1+'<link url="http://cse.google.com/cse?cx=004917987473580823572:eonwdtnjfi8&cof=forid%3a9&q='+GetFieldValue(2)+'">IMPawards-cse</link> ';
Fullinfo1:=Fullinfo1+'<link url="http:///www.impawards.com/cgi-bin/htsearch?method=or&words='+GetFieldValue(2)+'">impawards.com/cgi-bin</link> ';
If Fullinfo1 <> '' then Fullinfo1 := Fullinfo1;
If Fullinfo1 <> '' then AddCustomFieldValueByName('IMDbMovieLinksInfo', Fullinfo1);
-
Hey, Ivek. Thanks. Can you please post examples or imdb links where those matters and my code doesn't work? I just can't grasp just by looking at the code. Thanks.
To be clear, these parts work correctly, my fixes to some of the code are only cosmetic in nature. Below are examples of what I had in mind.
Before
<link url="https://www.imdb.com/title/tt0147800/parentalguide/#contentRating">Content Ratings Summary: </link><link url="https://www.imdb.com/title/tt0147800/parentalguide/#contentRating">Content rating (5) • </link><link url="https://www.imdb.com/title/tt0147800/parentalguide/#nudity">Sex & Nudity (10) • </link><link url="https://www.imdb.com/title/tt0147800/parentalguide/#violence">Violence & Gore (7) • </link><link url="https://www.imdb.com/title/tt0147800/parentalguide/#profanity">Profanity (8) • </link><link url="https://www.imdb.com/title/tt0147800/parentalguide/#alcohol">Alcohol, Drugs & Smoking (5) • </link><link url="https://www.imdb.com/title/tt0147800/parentalguide/#frightening">Frightening & Intense Scenes (1) • </link><link url="https://www.imdb.com/title/tt0147800/parentalguide/#certificates">Certifications (49)</link>
and after
<link url="https://www.imdb.com/title/tt0147800/movieconnections/">Connections:</link> • <link url="https://www.imdb.com/title/tt0147800/movieconnections/#featured_in">Featured in (134)</link> <link url="https://www.imdb.com/title/tt0147800/movieconnections/#features">Features (1)</link> <link url="https://www.imdb.com/title/tt0147800/movieconnections/#followed_by">Followed by (2)</link> <link url="https://www.imdb.com/title/tt0147800/movieconnections/#referenced_in">Referenced in (59)</link> <link url="https://www.imdb.com/title/tt0147800/movieconnections/#references">References (8)</link> <link url="https://www.imdb.com/title/tt0147800/movieconnections/#spin_off">Spin-off (1)</link> <link url="https://www.imdb.com/title/tt0147800/movieconnections/#spoofed_in">Spoofed in (2)</link> <link url="https://www.imdb.com/title/tt0147800/movieconnections/#spoofs">Spoofs (1)</link> <link url="https://www.imdb.com/title/tt0147800/movieconnections/#version_of">Version of (58)</link>
Before
<link url="https://www.imdb.com/title/tt0147800/movieconnections/">Connections: </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#featured_in">Featured in (134) • </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#features">Features (1) • </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#followed_by">Followed by (2) • </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#referenced_in">Referenced in (59) • </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#references">References (8) • </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#spin_off">Spin-off (1) • </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#spoofed_in">Spoofed in (2) • </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#spoofs">Spoofs (1) • </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#version_of">Version of (58)</link>
and after
<link url="https://www.imdb.com/title/tt0147800/parentalguide/#contentRating">Content Ratings Summary:</link> <link url="https://www.imdb.com/title/tt0147800/parentalguide/#contentRating">Content rating (5)</link> • <link url="https://www.imdb.com/title/tt0147800/parentalguide/#nudity">Sex & Nudity (10)</link> • <link url="https://www.imdb.com/title/tt0147800/parentalguide/#violence">Violence & Gore (7)</link> • <link url="https://www.imdb.com/title/tt0147800/parentalguide/#profanity">Profanity (8)</link> • <link url="https://www.imdb.com/title/tt0147800/parentalguide/#alcohol">Alcohol, Drugs & Smoking (5)</link> • <link url="https://www.imdb.com/title/tt0147800/parentalguide/#frightening">Frightening & Intense Scenes (1)</link> • <link url="https://www.imdb.com/title/tt0147800/parentalguide/#certificates">Certifications (49)</link> •
-
To be clear, these parts work correctly, my fixes to some of the code are only cosmetic in nature. Below are examples of what I had in mind.
Oh, ok then. Unfortunately, the cosmetics is very important to my custom skin design to visually separate fields and sections (screenshot below), so it would be huge overload for me to keep two versions when updating.
Regarding cleanning FullInfo, it is very important section for many reasons, and I admit it was always too clummsy for me to clean so I was primarily focused on it to work, and I will clean it at next update release.
Thanks for reviewing though!