PVD Selenium MOD v4 IMDb Movie, People and FilmAffinity Scripts

[afrocuban]

I forgot to upload ahk for the Script Configurator .exe


SeleniumPVDbScriptsConfig-v4.exe is compiled with ahk2exe for AutoHotkey v1.1.37.02 option "U32 (default) bin", without compression.

[afrocuban]

I have finished CompanyCredits function and now no need for Reference page at all. Consequently, I have updated, redesigned and compiled Script Configurator.

Now I need to update FullCredits function, meaning to get full cast & crew, directors for series, producers and composers and everything will be done. I will not clean Reference page from the code. I will only comment it out, so it could be possibly used in the future IMDb page changes.

Most probably I will not post again until finishing everything.

[afrocuban]

I have completed FullCredits function, so now all the data can be imported again to PVD. Before I publish it I want to check 2 things:
1. What to do with Reference page code, and should I exclude its options from Script Configurator, since now it is completely not needed.
2. To check People script and if I can fix it quickly, then I will publish all the scripts and files again in one, final package for this IMDb html layout change.

After that please check the scripts and let me know what doesn't work

[Ivek23]

I have completed FullCredits function, so now all the data can be imported again to PVD. Before I publish it I want to check 2 things:
1. What to do with Reference page code, and should I exclude its options from Script Configurator, since now it is completely not needed.
2. To check People script and if I can fix it quickly, then I will publish all the scripts and files again in one, final package for this IMDb html layout change.

After that please check the scripts and let me know what doesn't work

First of all, Function ParsePage_IMDBMovieREFERENCE should be moved to the end of the script before Function ParsePage.

Secondly, the entire part of the reference page code should be left and, as mentioned above, before Function ParsePage, or even better, it should be moved to the very end of the script, where there is already a History of changes, so that it can be re-included in the script if necessary in the future or completely blocked. All its options should also be excluded from both the script code and the script configurator, because now, as mentioned, it is no longer needed.

As for the People script, I don't know what works or doesn't work, because I don't use it.

[afrocuban]

Yes, I just commented out reference code in the script, and I already intended to leave it there and everywhere it is mentioned, to preserve the logic for the future. I was actually talking to comment it out in the Script Configurator too, to avoid inexperienced users to click on those options expecting to get data from /reference page. I will move the function before ParsePage function, though, that is a nice tip.

Now, meanwhile, I introduced another function:

function WaitForPageFile(const FileName, PageLabel: string;
  InitialWait: Integer; StabilizeMs: Integer): string; //BlockOpen
var
  i, currentWait: Integer;
  tryResult: string;
begin
  Result := '';
  i := 0;

  // provide defaults manually
  if InitialWait = 0 then
    InitialWait := 2000;
  if StabilizeMs = 0 then
    StabilizeMs := 1000;

  currentWait := InitialWait;

  while not FileExists(FileName) do
  begin
    LogMessage('Function DownloadPage - Waiting ' + IntToStr(currentWait div 1000) +
               's for the presence of: ' + FileName + '|');
    Wait(currentWait);

    // escalate wait times
    case i of
      0: currentWait := 5000;
      1: currentWait := 15000;
      2: currentWait := 10000;
      3: currentWait := 10000;
      4: currentWait := 15000;
      5: currentWait := 15000;
    end;

    Inc(i);
    if i = INTERNET_TEST_ITERATIONS then
    begin
      case MessageBox('IMDb Movie Function DownloadPage - Too many faulty attempts to internet connection for ' + PageLabel +
                      '. Cancel, Retry, or Continue (Ignore)? NOTE: IF YOU PRESS IGNORE YOU WILL NOT GET DATA FROM THAT PAGE, SO CONSIDER TO RETRY OR TO CANCEL AND START DOWNLOAD AGAIN! IMDb really makes it harder and harder to get the data.',
                      SCRIPT_FILE_NAME, 2) of
        3: // Cancel
        begin
          LogMessage('Function DownloadPage for ' + PageLabel +
                     ' ended with NO INTERNET connection ===============|');
          Result := '';
          Exit;
        end;
        4: // Retry
        begin
          i := 0;
          currentWait := InitialWait;
        end;
        5: // Ignore
        begin
          LogMessage('Function DownloadPage - Creating dummy ' + PageLabel +
                     ' HTML file due to Ignore selection|');
          with TStringList.Create do
          try
            Add('<html><body>Dummy ' + PageLabel + ' due to user Ignore.</body></html>');
            SaveToFile(FileName);
          finally
            Free;
          end;
          Break;
        end;
      end;
    end;
  end;

  // stabilization wait
  LogMessage('Function DownloadPage - ' + PageLabel +
             ' file detected, waiting extra ' + IntToStr(StabilizeMs) + 'ms to stabilize...');
  Wait(StabilizeMs);

  // manual error handling instead of try/except
  if not FileExists(FileName) then
  begin
    LogMessage(ProcException('FileError', 'NOT_FOUND: ' + FileName));
    Exit;
  end;

  tryResult := FileToString(FileName); // if your environment throws, replace with safe read
  if tryResult = '' then
    LogMessage(ProcException('FileError', 'FAILED to read ' + FileName))
  else
  begin
    Result := ConvertEncoding(tryResult, 65001);
    LogMessage(ProcException('FileRead', 'SUCCESS: ' + FileName));
  end;
end; //BlockClose

so, now instead this snippet for every page:

     
   // Initialize currentWait for the FullCredits file
      currentWait := 5000;  // Start with 5 seconds
      //Wait for the FullCredits file to finish downloading
      If Not(((USE_SAVED_PVDCONFIG) And ((ConfigOptions[8] = '0') And (ConfigOptions[10] = '0'))) Or
         (GET_FULL_CREDIT_FROM_REFERENCE And ((Pos('Series', MediaType) = 0) And (Pos('Series', GetFieldValueXML('category')) = 0)))) Then Begin // Also Known As (FullCredits)
      i := 0;
      currentWait := 2000;  // Initialize wait time
      while not FileExists(FilePath + FileTitleFullCredits) do begin
         LogMessage('Function DownloadPage - Waiting ' + IntToStr(currentWait div 1000) + 's (because of the people like Alfonso Cuaron with 268 wins and 208 nominations at the moment of writing this script) for the presence of: ' + FilePath + FileTitleFullCredits);
         wait(currentWait);
          // Increment the wait time for the next iteration
         case i of
            0: currentWait := 5000;  // 5 seconds
            1: currentWait := 15000;  // 15 seconds
            2: currentWait := 10000;  // 10 seconds
            3: currentWait := 10000;  // 10 seconds
            4: currentWait := 15000;  // 15 seconds
            5: currentWait := 15000;  // 15 seconds
         end;
         i := i + 1;
         if i = INTERNET_TEST_ITERATIONS then
         begin
            case MessageBox('IMDb Movie Function DownloadPage - Too many faulty attempts to internet connection for FullCredits. Cancel, Retry, or Continue (Ignore)? NOTE: IF YOU PRESS IGNORE YOU WILL NOT GET DATA FROM THAT PAGE, SO CONSIDER TO RETRY OR TO CANCEL AND START DOWNLOAD AGAIN! IMDb really makes it harder and harder to get the data.', SCRIPT_FILE_NAME, 2) of
               3: // Abort -> treat as Cancel
               begin
               LogMessage('Function DownloadPage for FullCredits END with NO INTERNET connection =============== |');
                  Result := '';
                  Exit;
               end;
               4: // Retry
               begin
                  i := 0;
                  currentWait := 2000;  // Reset wait time
               end;
               5: // Ignore->create dummy file
               begin
                  LogMessage('Creating dummy FullCredits HTML file due to Ignore selection...');
                  with TStringList.Create do
                  try
                     Add('<html><body>Dummy FullCredits due to user Ignore.</body></html>');
                     SaveToFile(FilePath + FileTitleFullCredits);
                  finally
                     Free;
                  end;
                  Result := FilePath + FileTitleFullCredits;
               end;
            end;
         end;
      end;


    // Add a short stabilization wait after the file is recognized
    LogMessage('Function DownloadPage - CHANGETHISWITHPROPERFILENAME file detected, waiting extra 1s to stabilize...');
    Wait(2000); // wait 2 second (adjust as needed)


      WebText := FileToString(FilePath + FileTitleFullCredits);
      WebText := ConvertEncoding(WebText, 65001); // UTF-8
      FullCreditsPageDownloaded := True;
      LogMessage('Function DownloadPage - FullCredits file found: ' + FilePath + FileTitleFullCredits);
      LogMessage('Value of FullCreditsPageDownloaded: ' + BoolToStr(FullCreditsPageDownloaded));
   end;

we will have only this

if not (USE_SAVED_PVDCONFIG and (ConfigOptions[15] = '0')) then
begin
  WebText := WaitForPageFile(FilePath + FileTitleParentalGuide, 'ParentalGuide', 5000, 2000);
  ParentalGuidePageDownloaded := WebText <> '';
  LogMessage('Function DownloadPage - Value of ParentalGuidePageDownloaded: ' +
             BoolToStr(ParentalGuidePageDownloaded));
end;

That way we will speed up the script and have hundreds if not thousands of lines less in the script.


I will do that and test for all repetitive tasks.

Also, I'm exploring ways IMDb not to refuse connections with selenium/python and testing at the moment creating fake userData folders, rotating user agents, and it goes good for now.

[afrocuban]

As of now, per title, I'm getting these timings:
Main Page downloading: ~18-22sec
Other 9 pages Page downloading: ~48-55 sec
Image downloading: ~6-8 sec
PVD script processing: ~5-10 sec

So in total for now, to get biggest possible amount of data per title, especially those with many awards, connections, etc... it is needed ~77-95 sec which is pretty acceptable for the amount of data.

[afrocuban]

Oh, and please remind me to upload purge_tmp_files.vbs if I forget, because I changed it by adding to delete fake UserData folders now too

[Ivek23]

Oh, and please remind me to upload purge_tmp_files.vbs if I forget, because I changed it by adding to delete fake UserData folders now too

Ok.

[afrocuban]

Here are all scripts and files fully updated, fixed and polished in a less than a month I started to fix all 16 of them, and I was so happy I got back into it easily and quickly. I have tested all scripts and files against many border case titles and persons and for me everything worked more than smooth and satisfying.

They are now faster and more stable and I am not facing anymore internet interruptions, because I heavily redesigned the most problematic python selenium scripts.

If especially Selenium_Chrome_Movie_Additional_pages_v4.py script is demanding for your CPU when downloading movies and you experience lags of any kind, open the file in Notepad++ and in the line 375:

with ThreadPoolExecutor(max_workers=4) as executor:

reduce number 4 to 3, 2 or 1, just test it. Whenever you lessen the number, the process of downloading files will be longer, so find your balance. If you have good CPU and a lot of RAM, then you can even increase the number above 4.

I'd be happy to further fine tuning and fix it, so please let me know about each case details so I could reproduce it too and then being able to fix it. If you have any further suggestion, I'd be happy to hear it as well while I didn't forget it again, but please explain why and how by giving specific examples, because I am not a programmer, but just using common sense and AI, and that is the only way I can understand the problem.

My plan is versions to stay on v4.2 for a long time unless something significant in their design changes.

Enjoy!

[afrocuban]

My next goal is to include new switch in the Script Configurator - UPDATE_DYNAMIC_VALUES_ONLY, by adding few dozens of lines into movie selenium script that would call only main page and update only dynamic values like: Rating, Top 250, Bottom 100, Number of votes.  And for the Awards summary when the movie is not older than 2 years than current date catching fresh wins for recent releases.

[afrocuban]

My next goal is to include new switch in the Script Configurator - UPDATE_DYNAMIC_VALUES_ONLY, by adding few dozens of lines into movie selenium script that would call only main page and update only dynamic values like: Rating, Top 250, Bottom 100, Number of votes.  And for the Awards summary when the movie is not older than 2 years than current date catching fresh wins for recent releases.

Well, I finished it earlier than expected, for IMDb Movie script. Also with a lot of chalenges I have redesigned Script Configurator once again, bringing new functionalities:

      //Retreive Data Config
  USE_SAVED_PVDCONFIG  = True ; // ***PVDCONFIG*** - Turn this ON to unlock and change the options below (from pvdconf.ini). Settings are applied when you click "Save All Script Configurations (Personal Video Database will automatically restart)" button below. Use carefully!


//############################################
//#  All options below require USE_SAVED_PVDCONFIG
//#  to be enabled so the Script Configurator
//#  can apply your settings correctly.
//############################################

  UPDATE_DYNAMIC_VALUES_ONLY  = True ;   //Update only dynamic values such as: Rating, Top 250, Metascore, Number of votes. Also update the Awards summary when the movie is less than 2 years old, to capture fresh wins for recent releases. Deselect to enable the options

//################################################
//#  All options below require UPDATE_DYNAMIC_VALUES_ONLY
//#  to be enabled so the Script Configurator
//#  can apply your settings correctly.
//###############################################

So you can see in the screenshots that now checking specific boxes disables or enables other options. Which means that....

My plan is versions to stay on v4.2 for a long time unless something significant in their design changes.

my plan will not last long since thee changes are huge for users to make them easy navigating and choosing proper options withoout to much contemplating, so with this I will soon go to 4.3.

But I still will not publish anything, because I want to finish People and FA scripts in terms of UPDATE_DYNAMIC_VALUES_ONLY, and I also want to further tweak Script Configurator GUI. I just hate "Save button" will not autosize to the last option in each tab, so for example in a People and FA tab we have to scroll all the way down. That is not just visual thing, but I rather want to implement "Apply" button that will be applied to each tab independently, while we will have overall "Cancel" and "Save & Restart PVD" button. That is tremendous challenge for ahk, that made me last year even to start creating GUI with python, but at the moment it looked even more difficult with python, so I abandoned it then. Now it looks the time to try it again is spot on.

[afrocuban]

New v4.3 Files

Most comprehensive and stable I have done so far. Two main things:

1. New PVD Scripts Configurator built from scratch in Python Tkinter. Much better GUI than AHK.
    Nice quirk - I have introduced dark/light theme for it as in the first photo. Reordered tabs, so default tab is IMDb Movie tab. You can still use both configurators (second photo - new Configurator has prefix "py"), all updated to v4.3, same options and functionalities. I'm giving them all now, for continuity, but in the future, most probably I will discontinue AHK. In any case, I'm giving .ahk file so anyone can maintain it in the future. I will stick to Python TKinter GUI.


2. As anounced earlier, new  **UPDATE DYNAMIC VALUES ONLY** switch for a fast update of only certain dynamic fields.

From the Change logs:

IMDb Movie Script

CHANGE LOG :
V 4.3.0.1 (11/15/2025) afrocuban: Script Configurator Enhancements
-------------------------------------------------------------------------------
- Built from the scratch new  PythonTkinter Script Conigurator:
    • It has all the functionalities as AHK. 
    • Plus litght/dark theme developed
    • Unlike AHK can be used as a standalone application with the same effect as when invoked in PVD.
   
- Added new feature in Script Configurator to enable and manage saved settings from `pvdconf.ini`:
    • **USE SAVED PVDCONFIG** now needs to be enabled to unlock configuration options below. 
    • This allows users to apply settings that require a restart of Personal Video Database upon saving. 
    • **Use this setting carefully!** Any changes will take effect only after clicking "Save All Script Configurations" (which will restart the application). 
- **UPDATE DYNAMIC VALUES_ONLY**: 
   • Allows users to update only **dynamic values** like Rating, Top 250, Metascore, and Number of votes. 
    • Updates the **Awards summary** for movies released within the last two years, capturing recent wins for fresh releases.
    • When disabled, additional configuration options become available for comprehensive updates.
    • Now poster can be downloaded from any page - separate procedure provided for it in Script Configurator,
    • Single Instance in Script Configurator. No more flooding with multiple instance by mistake.
    • Redesigned whole script to now accept UPDATE DYNAMIC VALUES ONLY switch properly

FilmaFfinity Script

CHANGE LOG :

V 4.3.0.1 (11/27/2025) afrocuban: Script Configurator Enhancements
-------------------------------------------------------------------------------
- Built from the scratch new  PythonTkinter Script Conigurator:
    • It has all the functionalities as AHK. 
    • Plus litght/dark theme developed
    • Unlike AHK can be used as a standalone application with the same effect as when invoked in PVD.


- Added new feature in Script Configurator to enable and manage saved settings from `pvdconf.ini`:
    • **USE SAVED PVDCONFIG** now needs to be enabled to unlock configuration options below. 
    • This allows users to apply settings that require a restart of Personal Video Database upon saving. 
    • **Use this setting carefully!** Any changes will take effect only after clicking "Save All Script Configurations" (which will restart the application). 
- **UPDATE DYNAMIC VALUES_ONLY**: 
    • Allows users to update only **dynamic values** like Rating, Number of votes and Awards for movies made in last 2 years.
    • When disabled, additional configuration options become available for comprehensive updates.

IMDb People Script

CHANGE LOG :
V 4.3.0.1 (11/27/2025) afrocuban: Script Configurator Enhancements
-------------------------------------------------------------------------------
- Built from the scratch new  PythonTkinter Script Conigurator:
    • It has all the functionalities as AHK. 
    • Plus litght/dark theme developed
    • Unlike AHK can be used as a standalone application with the same effect as when invoked in PVD.
   
- Added new feature in Script Configurator to enable and manage saved settings from `pvdconf.ini`:
    • **USE SAVED PVDCONFIG** now needs to be enabled to unlock configuration options below. 
    • This allows users to apply settings that require a restart of Personal Video Database upon saving. 
    • **Use this setting carefully!** Any changes will take effect only after clicking "Save" (which will restart the application). 
- **UPDATE DYNAMIC VALUES_ONLY**:
    • Allows users to update only **dynamic values** for persons that were alive at the moment of adding them to PVD, or at their last update. Updating only from the Main page.
    • When disabled, additional configuration options become available for comprehensive updates.

[afrocuban]

Here are v4.3 scripts.

1. Unpack the first 7z in "PersonalVideoDB" folder (3 files, overwrite existing, but backup them first if you want).
2. Unpack the second 7z in your "Scripts" folder. It is safe to move everything from it before extracting. These are all you need to safely run PVD with python.


You need all of these in order PVD to run as intended. Especially People script is complex, since I have integrated options to make it easier to dynamically update them and not to wait at all for deceased or the people that have only name and url. Test it.

If something desn't work, first check:
1. That you are running same version of Chrome and chromedriver.
2. That you installed whatever is needed for Python to work as described so far on this topic.

If that doesn't help, please publish screenshots and logs, so I could reproduce the issue too and being able to fix it.

Please test and let me know if everything work or not.


Enjoy!

[Ivek23]

I tested it, IMDB_Movie_[EN][Selenium]-v4.psf and IMDB_People_[EN][Selenium]-v4.psf work fine, with some errors in certain information, where it will be necessary to fix parts of the code in the scripts. The errors have been there for a long time and have not been fixed. The code fixes will be described and added below.

As for FilmAffinity_Movie_[EN][Selenium]-v4.psf, I have not tested it.

[Ivek23]

IMDB_Movie_[EN][Selenium]-v4.psf

Fixed part of the code in Function ParsePage_IMDBMovieMPAA

Previous

Code: [Select]

// Get (CF~IMDbmpaaSummary~)
curPos := Pos('<select id="jump-to"', HTML);
If curPos > 0 Then
Begin
  // Extract the relevant section for categories
  curPos := PosFrom('<option', HTML, curPos);
  endPos := PosFrom('</select>', HTML, curPos);
  mpaaSection := Copy(HTML, curPos, endPos - curPos);
  LogMessage('Function ParsePage_IMDBMovieMPAA - Extracted Category section for (CF~IMDbmpaaSummary~): ' + mpaaSection);

  // Parse the options and category names
  curPos := 1;
  mpaaContent := '';
  While PosFrom('<option', mpaaSection, curPos) > 0 Do
  Begin
curPos := PosFrom('<option', mpaaSection, curPos) + Length('<option');
optionValue := TextBetween(mpaaSection, 'value="', '">', False, curPos);
categoryName := TextBetween(mpaaSection, '">', '</option>', False, curPos);

// Format the category link
mpaaContent := mpaaContent + '<link url="' + MovieURL + optionValue + '">' + categoryName + '   •   </link>';

// Move to the next position
curPos := PosFrom('</option>', mpaaSection, curPos) + Length('</option>');
  End;

  // Remove the trailing "   •   " if it exists and keep the last </link> intact
  If Copy(mpaaContent, Length(mpaaContent) - 13, 7) = '   •   ' Then
  Begin
mpaaContent := Copy(mpaaContent, 1, Length(mpaaContent) - 14) + '</link>';
  End;

  // Combine and format the final result
  mpaaContent := '<link url="' + MovieURL + '#contentRating' + '">Content Ratings Summary: </link>' + mpaaContent;
 
  // Store the result in the custom field
  AddCustomFieldValueByName('IMDbmpaaSummary', mpaaContent);
  LogMessage('Function ParsePage_IMDBMovieMPAA - Stored result for (CF~IMDbmpaaSummary~)');
End
Else
Begin
  LogMessage('Function ParsePage_IMDBMovieMPAA - Content Rating section for (CF~IMDbmpaaSummary~) not found');
  //Result := prError; // Set to error if content rating section is not found
End;
Fixed

Code: [Select]

// Get (CF~IMDbmpaaSummary~)
curPos := Pos('<select id="jump-to"', HTML);
If curPos > 0 Then
Begin
  // Extract the relevant section for categories
  curPos := PosFrom('<option', HTML, curPos);
  endPos := PosFrom('</select>', HTML, curPos);
  mpaaSection := Copy(HTML, curPos, endPos - curPos);
  LogMessage('Function ParsePage_IMDBMovieMPAA - Extracted Category section for (CF~IMDbmpaaSummary~): ' + mpaaSection);

  // Parse the options and category names
  curPos := 1;
  mpaaContent := '';
  While PosFrom('<option', mpaaSection, curPos) > 0 Do
  Begin
curPos := PosFrom('<option', mpaaSection, curPos) + Length('<option');
optionValue := TextBetween(mpaaSection, 'value="', '">', False, curPos);
categoryName := TextBetween(mpaaSection, '">', '</option>', False, curPos);

// Format the category link
// mpaaContent := mpaaContent + '<link url="' + MovieURL + optionValue + '">' + categoryName + '   •   </link>';
mpaaContent := mpaaContent + '<link url="' + MovieURL + optionValue + '">' + categoryName + '</link>   •   ';

// Move to the next position
curPos := PosFrom('</option>', mpaaSection, curPos) + Length('</option>');
  End;

  // Remove the trailing "   •   " if it exists and keep the last </link> intact
  If Copy(mpaaContent, Length(mpaaContent) - 13, 7) = '   •   ' Then
  Begin
mpaaContent := Copy(mpaaContent, 1, Length(mpaaContent) - 14) + '</link>';
  End;

  // Combine and format the final result
  //mpaaContent := '<link url="' + MovieURL + '#contentRating' + '">Content Ratings Summary: </link>' + mpaaContent;
  mpaaContent := '<link url="' + MovieURL + '#contentRating' + '">Content Ratings Summary:</link> ' + mpaaContent;  
 
  // Store the result in the custom field
  AddCustomFieldValueByName('IMDbmpaaSummary', mpaaContent);
  LogMessage('Function ParsePage_IMDBMovieMPAA - Stored result for (CF~IMDbmpaaSummary~)');
End
Else
Begin
  LogMessage('Function ParsePage_IMDBMovieMPAA - Content Rating section for (CF~IMDbmpaaSummary~) not found');
  //Result := prError; // Set to error if content rating section is not found
End;

[Ivek23]

IMDB_Movie_[EN][Selenium]-v4.psf

Fixed part of the code in Function ParsePage_IMDBMovieCONNECTIONS

Previous

Code: [Select]

// Get (CF~Connections~) info   
If Pos('>Connections<', HTML) > 0 Then
Begin
LogMessage('===Function ParsePage_IMDBMovieCONNECTIONS - Starting Connections Parsing ===');

curPos := Pos('>Connections<', HTML);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - curPos for "Connections" section found at: ' + IntToStr(curPos));
If Pos('It looks like we don'+Chr(39)+'t have any Connections for this title yet.', HTML) > 0 Then
Begin
Category1 := 'No Connections';
Category3 := '';
AddCustomFieldValueByName('Connections', Category1);
AddCustomFieldValueByName('Connect', Category3);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - No Connections found for this title');
End Else Begin
// Extract the relevant section for categories
If PosFrom('<span class="ipc-simple-select__container" data-testid="jumpTo">', HTML, curPos) > 0 Then Begin
curPos := PosFrom('<span class="ipc-simple-select__container" data-testid="jumpTo">', HTML, curPos);
endPos := PosFrom('</select></span></span>', HTML, curPos);
Category2 := Copy(HTML, curPos, endPos - curPos);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Extracted Category2 section: ' + Category2);

// Parse the options and category names
curPos := 1;
Category1 := '';
While PosFrom('<option', Category2, curPos) > 0 Do
Begin
curPos := PosFrom('<option', Category2, curPos) + Length('<option');
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - curPos at option tag: ' + IntToStr(curPos));

optionValue := TextBetween(Category2, 'value="', '">', False, curPos);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Extracted optionValue: ' + optionValue);

categoryName := TextBetween(Category2, '">', '</option>', False, curPos);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Extracted categoryName: ' + categoryName);

// Format the category link
Category1 := Category1 + '<link url="' + MovieURL + optionValue + '">' + categoryName + '   •   </link>';
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Formatted and added category link: ' + '<link url="' + MovieURL + optionValue + '">' + categoryName + '   •   </link>');

// Move to the next position
curPos := PosFrom('</option>', Category2, curPos) + Length('</option>');
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Moved curPos to next option tag: ' + IntToStr(curPos));
End;

// Remove the trailing "   •   " if it exists and keep the last </link> intact
If Copy(Category1, Length(Category1) - 13, 7) = '   •   ' Then
Begin
Category1 := Copy(Category1, 1, Length(Category1) - 14) + '</link>';
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Removed trailing "   •   " and kept last </link> in Category1');
End;

LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Final Category1 before combining: ' + Category1);

// Combine and format the final result
Category1 := '<link url="' + MovieURL + '">Connections:                               </link>' + Category1;
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Combined and formatted final Category1: ' + Category1);

// Store the result in the custom field
AddCustomFieldValueByName('Connections', Category1);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Stored result in (CF~Connections~)');

LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Final Category1 stored: ' + Category1);
LogMessage('===Function ParsePage_IMDBMovieCONNECTIONS - Finished (CF~Connections~) Parsing ===');
End;
Fixed

Code: [Select]

// Get (CF~Connections~) info   
If Pos('>Connections<', HTML) > 0 Then
Begin
LogMessage('===Function ParsePage_IMDBMovieCONNECTIONS - Starting Connections Parsing ===');

curPos := Pos('>Connections<', HTML);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - curPos for "Connections" section found at: ' + IntToStr(curPos));
If Pos('It looks like we don'+Chr(39)+'t have any Connections for this title yet.', HTML) > 0 Then
Begin
Category1 := 'No Connections';
Category3 := '';
AddCustomFieldValueByName('Connections', Category1);
AddCustomFieldValueByName('Connect', Category3);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - No Connections found for this title');
End Else Begin
// Extract the relevant section for categories
If PosFrom('<span class="ipc-simple-select__container" data-testid="jumpTo">', HTML, curPos) > 0 Then Begin
curPos := PosFrom('<span class="ipc-simple-select__container" data-testid="jumpTo">', HTML, curPos);
endPos := PosFrom('</select></span></span>', HTML, curPos);
Category2 := Copy(HTML, curPos, endPos - curPos);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Extracted Category2 section: ' + Category2);

// Parse the options and category names
curPos := 1;
Category1 := '';
While PosFrom('<option', Category2, curPos) > 0 Do
Begin
curPos := PosFrom('<option', Category2, curPos) + Length('<option');
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - curPos at option tag: ' + IntToStr(curPos));

optionValue := TextBetween(Category2, 'value="', '">', False, curPos);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Extracted optionValue: ' + optionValue);

categoryName := TextBetween(Category2, '">', '</option>', False, curPos);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Extracted categoryName: ' + categoryName);

// Format the category link
//Category1 := Category1 + '<link url="' + MovieURL + optionValue + '">' + categoryName + '   •   </link>';
// Category1 := Category1 + '<link url="' + MovieURL + optionValue + '">' + categoryName + '</link>   •   ';
Category1 := Category1 + '<link url="' + MovieURL + optionValue + '">' + categoryName + '</link>    ';
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Formatted and added category link: ' + '<link url="' + MovieURL + optionValue + '">' + categoryName + '   •   </link>');

// Move to the next position
curPos := PosFrom('</option>', Category2, curPos) + Length('</option>');
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Moved curPos to next option tag: ' + IntToStr(curPos));
End;

// Remove the trailing "   •   " if it exists and keep the last </link> intact
If Copy(Category1, Length(Category1) - 13, 7) = '   •   ' Then
Begin
Category1 := Copy(Category1, 1, Length(Category1) - 14) + '</link>';
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Removed trailing "   •   " and kept last </link> in Category1');
End;

LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Final Category1 before combining: ' + Category1);

// Combine and format the final result
//Category1 := '<link url="' + MovieURL + '">Connections:    </link>' + Category1;
if Category1 <> '' then
Category1 := '<link url="' + MovieURL + '">Connections:</link>     •     ' + Category1
Else
Category1 := '<link url="' + MovieURL + '">Connections:</link>          ' + Category1;
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Combined and formatted final Category1: ' + Category1);

// Store the result in the custom field
AddCustomFieldValueByName('Connections', Category1);
LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Stored result in (CF~Connections~)');

LogMessage('Function ParsePage_IMDBMovieCONNECTIONS - Final Category1 stored: ' + Category1);
LogMessage('===Function ParsePage_IMDBMovieCONNECTIONS - Finished (CF~Connections~) Parsing ===');
End;

[afrocuban]

Hey, Ivek. Thanks. Can you please post examples or imdb links where those matters and my code doesn't work? I just can't grasp just by looking at the code. Thanks.

[Ivek23]

IMDB_Movie_[EN][Selenium]-v4.psf

Fixed part of the code in Function ParsePage

Remove

Code: [Select]

AddCustomFieldValueByName('LDDbSearch', '<link url="https://www.lddb.com/search/IMDb/' + MovieID1 + '">LaserDiscDbSearch</link>  ' + '<link url="http://www.soundtrack.net/search/?q=' + GetFieldValue(2) + '">Soundtrack.NetSearch</link>  ' + '<link url="https://store.intrada.com/s.nl?sc=16&category=&search=' + GetFieldValue(2) + '">Intrada</link>  ' + '<link url="https://www.aveleyman.com/?Film.aspx/' + GetFieldValue(2) + '">Aveleyman</link>  ' + '<link url="https://www.bing.com/Search?q=' + GetFieldValue(2) + '%20site%3Awww.aveleyman.com/">BingAvSearch</link>  ' + '<link url="https://www.google.com/search?q=' + GetFieldValue(2) + '%20site%3Awww.aveleyman.com/">GoogleAvSearch</link>  ' + '<link url="https://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Dmovies-tv&field-keywords=' + GetFieldValue(2) + '">AmazonSearch</link>  ' + '<link url="http://www.impawards.com/cgi-bin/htsearch?method=or&words=' + GetFieldValue(2) + '">IMPawards/cgi-bin</link>  ' + '<link url="http://www.impawards.com/search.php/' + GetFieldValue(2) + '">impawards.com</link>');Previous

Code: [Select]

        //Get ~IMDb Movie Url~ (CF~IMDb.com~) and (CF~WaybackArchive IMDb URLs~)
        Movie_URL := StringReplace(DownloadURL, BASE_URL_PRE_TRUE, BASE_URL_PRE, True, False, False);
AddCustomFieldValueByName('IMDb.com', '<link url="' + Movie_URL + '">IMDb.com</link>');
AddCustomFieldValueByName('WaybackArchive IMDb URLs', '<link url="https://web.archive.org/web/*/' + Movie_URL + '*">*IMDb.com*</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'business">Busines</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'combined">Combined</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'fullcredits">Full Cast&Crew</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'awards">Awards</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'releaseinfo">ReleaseDates</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'technical">TechSpecs</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'parentalguide">ParentsGuide</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'keywords">PlotKeywords</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'plotsummary">PlotSummary</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'taglines">Taglines</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'movieconnections">Connections</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'soundtrack">Soundtracks</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'externalsites">MiscSites</link>');
AddCustomFieldValueByName('WaybackArchive IMDb URLs', '<link url="https://web.archive.org/web/*/' + Movie_URL + '*">*IMDb.com*</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'business">Busines</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'combined">Combined</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'technical">TechSpecs</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'dvd">DVD</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'laserdisc">Laserdisc</link>');
Fixed

Code: [Select]

        //Get ~IMDb Movie Url~ (CF~IMDb.com~) and (CF~WaybackArchive IMDb URLs~)
        Movie_URL := StringReplace(DownloadURL, BASE_URL_PRE_TRUE, BASE_URL_PRE, True, False, False);
AddCustomFieldValueByName('IMDb.com', '<link url="' + Movie_URL + '">IMDb.com</link>');
AddCustomFieldValueByName('WaybackArchive IMDb URLs', '<link url="https://web.archive.org/web/*/' + Movie_URL + '*">*IMDb.com*</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'business">Busines</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'combined">Combined</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'fullcredits">Full Cast&Crew</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'awards">Awards</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'releaseinfo">ReleaseDates</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'technical">TechSpecs</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'parentalguide">ParentsGuide</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'keywords">PlotKeywords</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'plotsummary">PlotSummary</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'taglines">Taglines</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'movieconnections">Connections</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'soundtrack">Soundtracks</link>  <link url="https://web.archive.org/web/*/' + Movie_URL + 'externalsites">MiscSites</link>');Fixed

Code: [Select]

Fullinfo := '';
        Fullinfo := Fullinfo + '<link url="' + Movie_URL + '">MainPage</link>  ';
        Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'reference">Reference</link>  ';
        Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'fullcredits">Full Cast&Crew</link>  ';
        Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'awards">Awards</link>  ';
        //Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'releaseinfo">ReleaseDates</link>  ';
        Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'technical">TechSpecs</link>  ';
        Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'parentalguide">ParentsGuide</link>  ';
        Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'keywords">PlotKeywords</link>  ';
        Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'plotsummary">PlotSummary</link>  ';
        Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'locations">FilmLocations</link>  ';
        Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'taglines">Taglines</link>  ';
        Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'companycredits">CompanyCredits</link>  ';
        Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'movieconnections">Connections</link>  ';
        Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'soundtrack">Soundtracks</link>  ';
        Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'externalsites">MiscSites</link>  ';
        //Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'externalsites#photos">MiscPhoto</link>  ';
        Fullinfo:=Fullinfo+'<link url="'+Movie_URL+'externalsites/#misc">MiscSites</link>  ';
        Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'externalsites#photo">MiscPhoto</link>  ';
        Fullinfo := Fullinfo + '<link url="' + Movie_URL + 'mediaindex">PhotoGallery</link>  ';
Fullinfo := Fullinfo + '<link url="http://www.imdb.com/chart/top">IMDb Top 250</link>  ';
Fullinfo := Fullinfo + '<link url="http://www.imdb.com/chart/bottom">Lowest Rated</link>  ';
Fullinfo := Fullinfo + '<link url="http://www.imdb.com/chart/toptv">Top 250 TV</link>';
If Fullinfo <> '' then IMDB_URL := Fullinfo;
        If IMDB_URL <> '' then AddCustomFieldValueByName('IMDb Url', IMDB_URL);
        If IMDB_URL <> '' then AddCustomFieldValueByName('IMDb Movie Url', IMDB_URL);
        LogMessage('Function ParsePage -      Get result Fullinfo-IMDB_URL (CF~IMDb Movie Url~ ): ' + #13 + IMDB_URL + '| |');Remove

Code: [Select]

AddCustomFieldValueByName('IMDbExternalSitesUrl', '<link url="' + Movie_URL + 'releaseinfo">ReleaseDates</link>  <link url="' + Movie_URL + 'plotsummary">PlotSummary</link>  <link url="' + Movie_URL + 'companycredits">CompanyCredits</link>  <link url="' + Movie_URL + 'movieconnections">Connections</link>  <link url="' + Movie_URL + 'externalsites/#misc">MiscSites</link>  <link url="' + Movie_URL + 'externalsites#photos">MiscPhoto</link>');Fixed

Code: [Select]

Fullinfo1 := '';
Fullinfo1:=Fullinfo1+'<link url="http://www.boxofficemojo.com/title/'+MovieID+'">BoxOfficeMojo</link>  ';
Fullinfo1:=Fullinfo1+'<link url="http://en.wikipedia.org/w/index.php?search='+GetFieldValue(2)+'">Wikipedia</link>  ';
Fullinfo1:=Fullinfo1+'<link url="https://www.movieposterdb.com/search?category=title&q='+MovieID+'">MoviePosterDB Info</link>  ';
Fullinfo1:=Fullinfo1+'<link url="https://www.dvdsreleasedates.com/search/?searchStr='+GetFieldValue(2)+'">DVDs ReleaseDates</link>  ';
Fullinfo1:=Fullinfo1+'<link url="https://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Dmovies-tv&field-keywords='+GetFieldValue(2)+'">AmazonSearch</link>  ';
Fullinfo1:=Fullinfo1+'<link url="https://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Dmovies-tv&field-keywords='+GetFieldValue(2)+'%20'+GetFieldValue(5)+'">AmazonSearch1</link>  ';
Fullinfo1:=Fullinfo1+'<link url="https://www.allmovie.com/search/movies/'+GetFieldValue(2)+'">AllMovieSearch</link>  ';
Fullinfo1:=Fullinfo1+'<link url="https://www.themoviedb.org/search?query=' +GetFieldValue(2)+'">TMDBSearch</link>  ';
Fullinfo1:=Fullinfo1+'<link url="https://www.rottentomatoes.com/search/?search='+GetFieldValue(2)+'">RottenTomatoesSearch</link>  ';
Fullinfo1:=Fullinfo1+'<link url="https://www.thetvdb.com/search?query='+GetFieldValue(2)+'">TVDB Search</link>  ';
Fullinfo1:=Fullinfo1+'<link url="https://www.soundtrackcollector.com/catalog/search.php?searchon=all&searchtext='+GetFieldValue(2)+'">SoundCollSearch</link>  ';
Fullinfo1:=Fullinfo1+'<link url="https://www.blu-ray.com/search/?quicksearch=1&quicksearch_country=all&quicksearch_keyword='+MovieID+'&section=theatrical">BR.comSearch</link>  ';
Fullinfo1:=Fullinfo1+'<link url="https://www.lddb.com/search/IMDb/'+MovieID1+'">LaserDiscDb Search</link>  ';
Fullinfo1:=Fullinfo1+'<link url="http://cse.google.com/cse?cx=004917987473580823572:eonwdtnjfi8&cof=forid%3a9&q='+GetFieldValue(2)+'">IMPawards-cse</link>  ';
Fullinfo1:=Fullinfo1+'<link url="http:///www.impawards.com/cgi-bin/htsearch?method=or&words='+GetFieldValue(2)+'">impawards.com/cgi-bin</link>  ';
If Fullinfo1 <> '' then Fullinfo1 := Fullinfo1;
If Fullinfo1 <> '' then AddCustomFieldValueByName('IMDbMovieLinksInfo', Fullinfo1);

[Ivek23]

Hey, Ivek. Thanks. Can you please post examples or imdb links where those matters and my code doesn't work? I just can't grasp just by looking at the code. Thanks.

To be clear, these parts work correctly, my fixes to some of the code are only cosmetic in nature. Below are examples of what I had in mind.

Before

<link url="https://www.imdb.com/title/tt0147800/parentalguide/#contentRating">Content Ratings Summary:          </link><link url="https://www.imdb.com/title/tt0147800/parentalguide/#contentRating">Content rating (5)   •   </link><link url="https://www.imdb.com/title/tt0147800/parentalguide/#nudity">Sex & Nudity (10)   •   </link><link url="https://www.imdb.com/title/tt0147800/parentalguide/#violence">Violence & Gore (7)   •   </link><link url="https://www.imdb.com/title/tt0147800/parentalguide/#profanity">Profanity (   •   </link><link url="https://www.imdb.com/title/tt0147800/parentalguide/#alcohol">Alcohol, Drugs & Smoking (5)   •   </link><link url="https://www.imdb.com/title/tt0147800/parentalguide/#frightening">Frightening & Intense Scenes (1)   •   </link><link url="https://www.imdb.com/title/tt0147800/parentalguide/#certificates">Certifications (49)</link>

and after

<link url="https://www.imdb.com/title/tt0147800/movieconnections/">Connections:</link>     •     <link url="https://www.imdb.com/title/tt0147800/movieconnections/#featured_in">Featured in (134)</link>    <link url="https://www.imdb.com/title/tt0147800/movieconnections/#features">Features (1)</link>    <link url="https://www.imdb.com/title/tt0147800/movieconnections/#followed_by">Followed by (2)</link>    <link url="https://www.imdb.com/title/tt0147800/movieconnections/#referenced_in">Referenced in (59)</link>    <link url="https://www.imdb.com/title/tt0147800/movieconnections/#references">References (</link>    <link url="https://www.imdb.com/title/tt0147800/movieconnections/#spin_off">Spin-off (1)</link>    <link url="https://www.imdb.com/title/tt0147800/movieconnections/#spoofed_in">Spoofed in (2)</link>    <link url="https://www.imdb.com/title/tt0147800/movieconnections/#spoofs">Spoofs (1)</link>    <link url="https://www.imdb.com/title/tt0147800/movieconnections/#version_of">Version of (58)</link>

Before

<link url="https://www.imdb.com/title/tt0147800/movieconnections/">Connections:                               </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#featured_in">Featured in (134)   •   </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#features">Features (1)   •   </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#followed_by">Followed by (2)   •   </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#referenced_in">Referenced in (59)   •   </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#references">References (   •   </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#spin_off">Spin-off (1)   •   </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#spoofed_in">Spoofed in (2)   •   </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#spoofs">Spoofs (1)   •   </link><link url="https://www.imdb.com/title/tt0147800/movieconnections/#version_of">Version of (58)</link>

and after

<link url="https://www.imdb.com/title/tt0147800/parentalguide/#contentRating">Content Ratings Summary:</link>          <link url="https://www.imdb.com/title/tt0147800/parentalguide/#contentRating">Content rating (5)</link>   •   <link url="https://www.imdb.com/title/tt0147800/parentalguide/#nudity">Sex & Nudity (10)</link>   •   <link url="https://www.imdb.com/title/tt0147800/parentalguide/#violence">Violence & Gore (7)</link>   •   <link url="https://www.imdb.com/title/tt0147800/parentalguide/#profanity">Profanity (</link>   •   <link url="https://www.imdb.com/title/tt0147800/parentalguide/#alcohol">Alcohol, Drugs & Smoking (5)</link>   •   <link url="https://www.imdb.com/title/tt0147800/parentalguide/#frightening">Frightening & Intense Scenes (1)</link>   •   <link url="https://www.imdb.com/title/tt0147800/parentalguide/#certificates">Certifications (49)</link>   •

[afrocuban]

To be clear, these parts work correctly, my fixes to some of the code are only cosmetic in nature. Below are examples of what I had in mind.

Oh, ok then. Unfortunately, the cosmetics is very important to my custom skin design to visually separate fields and sections (screenshot below), so it would be huge overload for me to keep two versions when updating.

Regarding cleanning FullInfo, it is very important section for many reasons, and I admit it was always too clummsy for me to clean so I was primarily focused on it to work, and I will clean it at next update release.


Thanks for reviewing though!