Personal Video Database

Quote from: Ivek23 on March 16, 2025, 07:47:18 am

I must have made a mistake, I just didn't notice it (I still have a lot of things to sort out for my mother's passing, so some details are missing and I don't notice them). I'll fix that and see if it works.

I fixed it now and it works perfectly.

Great! Enjoy it!

May I ask you a question? Can you describe the flow of adding series, seasons and episode links? Which script does which task? I'm almost done integrating series script into movie script, but I am stuck at the moment what generates seasons and what generates  episodes in each season, what provides links to the episodes and so on. Thank you in advance.

Share your tips here...

Hello Ivek.

Do you have an idea how could I force "BioList" from this function to populate to the field from the snippet in the file attached, while ShouldParseBio is set to 0 or is set to "Set If Empty" but bio field is not empty, on the last Exit from PVD?

BioList properly parses but it won't populate. The only trick I can do to populate BioList is to set "Bio" differently to "Overwrite" regardles of the state of the bio field, or to set it to "Set If Empty" if it was empty upon starting PVD with opBio=0 in the pvdconfig.ini (from the last PVD exit).

I have brought back search function to Movie script too, using Selenium, and will publish it altogether with People v4.0 script and correpsondent Selenium scripts, after I finish full transition to Selenium.

All People scripts are ready, but I need to finish Movie scripts.


It'll probably take me a month.

Thanks Ivek! Maybe we should agree to add to the script's names some suffix like "Chrome" and "FF" or similar, so people could easier recognize which ones they use. Or, we can agree that I will put "Chrome" to all of my scripts and the default is Firefox.

To all users: there is no obstacle to put both gecko and cromedriver to proper places, when you have both Chrome and Firefox installed (on the PATH), and to use both scripts concurently!

Completely new IMDb People script with integrated Python+Selenium script. Put selenium script to the Scripts folder of your PVD too.


Everything works now, except (for me at least) populating photo to database. Please check and report if it works for you.


From the Change log -

CHANGE LOG :
            V 3.0.0.1 (04/01/2025) afrocuban (THANKS TO IVEK'S HUGE HELP):
         - Selenium integration to PVD introduced. Check http://www.videodb.info/forum_en/index.php/topic,4368.0.html and
         http://www.videodb.info/forum_en/index.php?topic=4367.0 for more
         - Awards properly parsed now (HUGE THANKS TO IVEK HERE!!!).
         - Birthplace, Filmography Bio and Genre field fixed. 
         - DownloadPage and ParsePage functions modified to split downpage-UTF8_NO_BOM.htm into downloading different file for each function:                       Awards, Bio, Credit and Genre. Main (Prncipal) page still downloaded with PVdBDownPage.exe to downpage-UTF8_NO_BOM.htm
         - New HandlePhoto function to separately parse and add photo to record.

This is fork and upgrade of Ivek's selenium script found here:

http://www.videodb.info/forum_en/index.php/topic,4362.msg22691.html#msg22691

For this to work ensure that:

A. You installed python
B. You installed selenium by

Quote

pip install selenium

C. You have your Chrome bin on a PATH
D. You have Python folder on your PATH
E. pythonw.exe is not missing, or it's containing folder is on the PATH

This script:

1. Uses Chrome browser instead Firefox
2. Uses chromedriver.exe instead geckodriver
3. Starts chromedriver.exe silently
4. Silently invokes browser in a headless mode (no pop-up windows of browser)
5. Scrapes .htm page of a given url
6. No path is needed to set manually inside the script - it is set to be relative to the path of selenium script!

For using relative path, ensure:

6A. You put this script into "Scripts" folder of your PVD instance.
6B. You put appropirate chromedriver.exe to the "Script" folder, too.

To silently invoke selenium script itself by PVD's .psf script (no pop-up windows of selenium script's cmd window), be sure to use pythonw.exe instead of python.exe, like this for example:

FileExecute('pythonw.exe', '"' + ScriptPath + 'selenium_script-Chrome.py" "' + URL + '" "' + ScriptPath + BASE_DOWNLOAD_FILE_NO_BOM + '"');

Now, the last one will probably be ensured by those who maintain corresponding scripts if interested in, and for now, those are Ivek and me, but be sure to check if it's there anyway.

From this point on, everything is automated and headless.

Here's my completely new FilmAffinity_[EN][HTTPS_Poster]_v6.1.psf script.


It includes major changes and additions.


CHANGE LOG:

           V 6.1.0.1-afrocuban (12/15/2024) afrocuban: MAJOR CHANGES INTRODUCED.

   Custom field types need to be strictly followed in order to be presented properly.

   Following custom/original fields added/changed respectively:

   • ~Studio~ section completely re-written:
1. Original "producer (human, category 4)" field now properly imports only when there is a Producer explicitly stated on FA.
2. Original "studio" (AField, value 8 ), populated now with studios and distributors concatenated. Co-productions (countries) and producer persons not included.
3. "FA Co-productions" Multiselect list custom field introduced for the first time.
4. "FA Producers"  Multiselect list custom field introduced for the first time.
5. "FA Studio" Multiselect list custom field introduced for the first time. This field inludes only studios, not Co-production (countires), producer person, nor Distributors.
6. "FA Distributors" Multiselect list custom field introduced for the first time.

   • ~Writers~ section completely re-written:
1. Original "writers (human, category 2)" field now properly imports only when there is a screenwriter explicitely stated on FA.
2. "FA Writers" Multiselect list custom field added for the same value as under 1.
3. "FA Script" Memo custom field introduced for the first time filled with everything else found in the Writers block ("novel:...", "book:...", etc).

   • New custom Memo field "FA Related". Related movies to the current one.
   • New custom Memo field "FA SimMov". Similar movies with % of similiraty.
   • New custom Memo field "FA Awards". Years linked to FA pages with all awards for that competition for that year.
   • New custom Memo field "FA Misc URLs". Trailers, Image gallery and Pro reviews links with the counts present for each three, for the current movie.
   • New custom Multiselect list field "FA Cinematography".
   • New custom Memo field "FA Ranking Position", underlinked to external top lists.
   • New custom Memo field "FA Ranking Lists Position", underlinked to external users' lists.
   • New custom Memo field "FA Critics" with Review, Authors and Magazines, each underlinked to external sources, when there is one.
   • New custom Memo field "FA Released By" added.
   • New custom Memo field "FA Release Date" added.
   • New custom Memo field "FA Lists" added.
   • New custom Multiselect list field "FA Genre" added.
   • New custom Multiselect list field "FA Category" added.
   • New custom Memo field "FA OrigTitle" added.
   • New custom Multiselect listfield "FA Year" added.
   • New custom Long Text field "FA length" added.
   • New custom Long Text field "FA features" added.
   • New custom Multiselect List field "FA Country" added.
   • New custom Multiselect List field "FA Directors" added.
   • New custom Multiselect List field "FA Writers" added.
   • New custom Multiselect List field "FA Composers" added.
   • New custom Memo field "FA Description" added.
   • New custom Memo field "FA Actors" added.


   • Two new functions introduced in the script:
1. "DownloadAndParseTrailerPage" Function added which constructs the URL for the trailer page, uses the DownloadPage Function to download it, and then calls a parsing function to process it.
2. "ParseTrailerPage" Function added to parse the downloaded trailer page HTML content.
3. "DownloadPage" Function modified to accept an output file name, by adding a parameter to specify the output file name, so that we can use this function to download both the main movie page and the trailer page. This modified function now accepts OutFile as a parameter to specify the output file name.
4. "ParsePage" Main Function modified to include the call to "DownloadAndParseTrailerPage" function.
Additional parsing page "downpage_trailer-UTF8_NO_BOM_FA.htm" is downloaded beside "downpage-UTF8_NO_BOM_FA.htm", but nothing still parsed since it contains only dynamic content which needs Selenium integration, because PVdBDownPage.exe can download only static content of html.

TO DO:
- Selenium integration to PVD.
- Logging to movie site throught the script.

           V 5.0.0.1-afrocuban (12/05/2024) afrocuban: Poster now available via HTTPS.

You can view how imported data looks in my new skins on this topic, and what is imported actually.

You can look for most recent custom fields from this topic's message and later, and use it for easier tracking which custom fields to add to your PVD

Sorry that I'm in the conversation uninvited and not knowing programming. But I learned a bit along the way trying to locally download FA page with trailers. I succeeded to download it as downpage_trailer-UTF8_NO_BOM_FA.htm beside downpage-UTF8_NO_BOM.htm in order to try to parse them both, but no good news. I mean, I can parse them both, but dynamic content isn't downloaded,

I don't think we can do it with PVdBDownPage.exe:

// Download the URL page.
  LogMessage(Chr(9) + Chr(9) + '      Download with PVdBDownPage in file:|' + ScriptPath + OutFile + ' the information of:|' + URL + '||');
  FileExecute(ScriptPath + 'PVdBDownPage.exe', '"' + URL + '" "' + ScriptPath + OutFile + '"');

because it cannot download dynamic content since it

is likely due to the JavaScript that loads the video elements dynamically on the original page. When you download the page using DownloadPage, the JavaScript doesn’t execute, so the dynamic content (like embedded trailers) isn’t included in the downloaded HTML file.
To work around this, we have a couple of options:

Use a Web Browser Automation Tool: Tools like Selenium can control a web browser to visit the page and let all JavaScript execute before saving the fully rendered page’s HTML. This way, the dynamic content will be included.

Find the Video URL Manually: Check the original HTML of the page in your browser (right-click and “View Page Source” or inspect elements with Developer Tools). Look for the actual URLs of the video files. You can then use those URLs directly in your script.

Using a web browser automation tool like Selenium is a bit more involved and requires additional setup, but it is a more robust solution for handling dynamic content. If you prefer to go this route, I can guide you through the setup process.

Alternatively, manually inspecting the page’s source for video URLs is quicker but less convenient for repeated tasks.

Here's the rest of the conversation:

Invoking Selenium directly from within a PVD script (written in Pascal) is not straightforward because Selenium typically operates via Python, Java, or another high-level language. However, you can use a workaround by invoking an external Python script from your PVD Pascal script.

Here’s how you can integrate Selenium into your workflow:

Create a Python Script Using Selenium:

This script will navigate to the URL, wait for the content to load, and save the fully rendered HTML.

python

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Setup the WebDriver
driver = webdriver.Chrome(executable_path='path_to_chromedriver')

# Navigate to the URL
driver.get("https://www.filmaffinity.com/en/evideos.php?movie_id=xxxx")

# Wait for the trailers to load (adjust the waiting condition as needed)
WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CLASS_NAME, "divvid"))
)

# Get the fully rendered HTML
html = driver.page_source

# Save the HTML to a file
with open('downpage_trailer-UTF8_NO_BOM_FA.htm', 'w', encoding='utf-8') as file:
    file.write(html)

# Close the browser

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Setup the WebDriver
driver = webdriver.Chrome(executable_path='path_to_chromedriver')

# Navigate to the URL
driver.get("https://www.filmaffinity.com/en/evideos.php?movie_id=xxxx")

# Wait for the trailers to load (adjust the waiting condition as needed)
WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CLASS_NAME, "divvid"))
)

# Get the fully rendered HTML
html = driver.page_source

# Save the HTML to a file
with open('downpage_trailer-UTF8_NO_BOM_FA.htm', 'w', encoding='utf-8') as file:
    file.write(html)

# Close the browser
driver.quit()

Invoke the Python Script from Your PVD Script:

Use the FileExecute method to call the Python script from within your Pascal script.

pascal

// Log the action
LogMessage(Chr(9) + Chr(9) + '      Download with Selenium in file:|' + ScriptPath + OutFile + ' the information of:|' + URL + '||');

// Construct the command to run the Python script
FileExecute('python.exe', ScriptPath + 'selenium_script.py');

In this setup:

The Python script handles the browsing and downloading of the fully rendered HTML using Selenium.

Your PVD script uses the FileExecute method to call the Python script, passing any necessary parameters if needed.

After the Python script finishes and the HTML is saved, your PVD script can continue processing the saved HTML as usual.

Example Complete Workflow
1. Python Script (selenium_script.py)

python

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import sys

# Get the URL from the command line arguments
url = sys.argv[1]
output_file = sys.argv[2]

# Setup the WebDriver
driver = webdriver.Chrome(executable_path='path_to_chromedriver')

# Navigate to the URL
driver.get(url)

# Wait for the trailers to load (adjust the waiting condition as needed)
WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CLASS_NAME, "divvid"))
)

# Get the fully rendered HTML
html = driver.page_source

# Save the HTML to a file
with open(output_file, 'w', encoding='utf-8') as file:
    file.write(html)

# Close the browser
driver.quit()

2. Pascal Script (Invoke Selenium Script)

pascal

// Log the action
LogMessage(Chr(9) + Chr(9) + '      Download with Selenium in file:|' + ScriptPath + OutFile + ' the information of:|' + URL + '||');

// Construct the command to run the Python script
FileExecute('python.exe', '"' + ScriptPath + 'selenium_script.py" "' + URL + '" "' + ScriptPath + OutFile + '"');

By following this workflow, you can effectively integrate Selenium into your PVD script to handle pages with dynamic content. This way, you can ensure that all necessary content, like embedded trailers, is included in the HTML that your PVD script processes.

How would selenium know which url to open?
Great question. You can pass the URL to Selenium through command line arguments when you invoke the Python script. This way, your PVD script can dynamically supply the URL to Selenium. Here's how you can do it:

Step-by-Step Implementation
Modify Your Python Script to Accept Command Line Arguments:

You’ll adjust the Python script to accept the URL as an argument. This ensures that Selenium knows which URL to open.

Example Python Script (selenium_script.py):

python

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import sys

# Get the URL and output file from the command line arguments
url = sys.argv[1]
output_file = sys.argv[2]

# Setup the WebDriver
driver = webdriver.Chrome(executable_path='path_to_chromedriver')

# Navigate to the URL
driver.get(url)

# Wait for the trailers to load (adjust the waiting condition as needed)
WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CLASS_NAME, "divvid"))
)

# Get the fully rendered HTML
html = driver.page_source

# Save the HTML to a file
with open(output_file, 'w', encoding='utf-8') as file:
    file.write(html)

# Close the browser
driver.quit()

Invoke the Python Script from Your PVD Script:

Use the FileExecute method to call the Python script, passing the URL and the output file path as arguments.

Example Pascal Script:

pascal

// Log the action
LogMessage(Chr(9) + Chr(9) + '      Download with Selenium in file:|' + ScriptPath + OutFile + ' the information of:|' + URL + '||');

// Construct the command to run the Python script
FileExecute('python.exe', '"' + ScriptPath + 'selenium_script.py" "' + URL + '" "' + ScriptPath + OutFile + '"');

How It Works
Python Script: The Python script (selenium_script.py) receives the URL and the output file path as command line arguments. It then uses Selenium to open the URL, waits for the trailers to load, and saves the fully rendered HTML to the specified output file.

Pascal Script: Your PVD script constructs the command to run the Python script with the necessary arguments (the URL and the output file path) and then executes it. This ensures that the correct URL is passed to Selenium dynamically.

This approach allows you to integrate Selenium seamlessly into your PVD script, enabling it to handle pages with dynamic content effectively. If you need further assistance with any part of the setup, feel free to ask!

How slow is this process?
The process of using Selenium to render and download the HTML can be slower than simply downloading a static page because Selenium has to:

Open a Web Browser: Launching a browser instance (e.g., Chrome or Firefox) takes a few seconds.

Navigate to the URL: Just like a human user, Selenium will navigate to the specified URL, which involves network latency.

Wait for JavaScript Execution: Selenium waits for the page to fully load, including the execution of any JavaScript that dynamically loads content. This can vary in time depending on the complexity of the page and the speed of your internet connection.

Save the HTML: Once the page is fully loaded, Selenium will extract and save the HTML content, which is relatively quick.

Estimated Time
Browser Launch: 1-3 seconds

Page Navigation: 1-5 seconds (depends on network speed)

JavaScript Execution & Content Load: 3-10 seconds (depends on page complexity)

Saving HTML: <1 second

In total, it can take anywhere from 5 to 20 seconds or more per page, depending on various factors.

Performance Considerations
Network Speed: Faster internet connections will reduce the navigation and loading time.

Page Complexity: Pages with heavy JavaScript and multiple dynamic elements will take longer to fully load.

Browser & System Performance: Using a faster browser and a system with more resources (CPU, RAM) will speed up the process.

While this is slower compared to a direct HTML download using PVdBDownPage.exe, it ensures that all dynamic content is fully loaded and accessible. This trade-off can be worthwhile if you need to scrape content that is loaded dynamically by JavaScript.

If speed becomes a critical issue, you might look into optimizing the Selenium script further, such as by reducing the wait times where possible or running multiple instances in parallel if applicable.

For creating skins I'm using Serna Free xml editor. It's obsolete, it's code is open sourced on GitHub, and I couldn't make anything without it. It's WYSIWYG, so you don't deal with any notepad-like coding. You can move elements (PVD fields and other elements) through the tree and in PVD to refresh the skin to immediately check results.
Serna Free is 12 years obsolete, but still working properly, but it's not easy to find it today, so you probably have to use some torrent downloader if you want to find, download and use it. I didn't find any other xml editor that is near Serna, just like there is no movie database program near like PVD.
Once you try it, you will never use anything else for at least skinning PVD.