English > Talk
curl - solution for https
Ivek23:
--- Quote ---2) For touching the code:
I see that the structure is a little bit confusing ¿do you want that I arrange it?
I see that the script use AddFieldValue that is deprecated. It better AddFieldValueXML ¿do you want that I arrange it?
I think that the ParseSearchResults don't work ¿do you want that I programme it?
--- End quote ---
Yes please, because the script with additional score in custom fields and search results will also be of use for other users of this forum.
For me personally it should be added some additional information in a common custom field. What and how I had arranged you can see in the below attached RottenTomatoes (full) script. This should be combined into a single script with the possibility for publication may then remain only just additional score in the custom fields, if possible to regulate.
Thank you for your work and effort already in advance.
Btw:
How to edit or connect now all files from curl_try to Rottentomatoes_ [HTTPS] in one executable file.
Will now required for each movie in curl_try file manually enter the URL address of the movie, which is of course more movies lengthy work, or it will be automatically regulated.
Ivek23:
--- Quote ---I think that the ParseSearchResults don't work ¿do you want that I programme it?
--- End quote ---
Yes, it really does not work.
When I changed the code in the procedure ParseSearchResults,
--- Code: ---procedure ParseSearchResults(HTML : String);
var
curPos, endPos : Integer;
Title, Year, URL, Preview : String;
begin
LogMessage('Function ParseSearchResults|');
LogMessage('Global Var-Mode|'+IntToStr(Mode)+'|');
LogMessage('Global Var-DownloadURL|'+DownloadURL+'|');
curPos := Pos('<h1>Search Results for : ', HTML);
if curPos < 1 then
Exit;
LogMessage('Parsing search results...');
curPos := PosFrom('<a class="unstyled articleLink" href="/m/', HTML, curPos);
while curPos > 0 do begin
endPos := PosFrom('">', HTML, curPos);
URL := 'http://www.rottentomatoes.com'+Trim(Copy(HTML, curPos+38, endPos - curPos-38));
//curPos := PosFrom('">', HTML, curPos);
//endPos := PosFrom('</a>', HTML, curPos);
Title := TextBetween(HTML, '">', '</a>', True, curPos);
//curPos := PosFrom('<span class="movie_year">', HTML, curPos);
//endPos := PosFrom('</span> </div>', HTML, curPos);
Year := TextBetween(HTML, '<span class="movie_year">', '</span>', True, curPos);
AddSearchResult(Title+' '+Year, '', '', URL, '');
curPos := PosFrom('<a class="unstyled articleLink" href="/m/', HTML, curPos);
end;
end;
--- End code ---
search results now operates.
BTW:
--- Quote ---¿Do you want make a complete script or only one for your custom fields?
--- End quote ---
It can also be only one for custom fields, if you can, no problem.
VVV_Easy_Programing:
Attach you have a functional version:
VERSION: 0.3.0.0 (06/11/2016) -> VVV: Rebuild full script.
It retrieve the principal infos (you could complete the rest, I left hints in the code).
I don't now all the custom fields. I retrieved some ratings but I left hints and questions in the code.
Not poster or screenshot (if you have the https link I could retrieve it with curl and charge in PVdB by file but it's not easy to find in the page).
I see that RT is a slowly page, don't worry if the DOS windows stops a 5 or 6 seconds when you run in PVdB, the script waits (this was a big problem. now is solved).
--- Quote ---How to edit or connect now all files from curl_try to Rottentomatoes_ [HTTPS] in one executable file.
--- End quote ---
The script works directly in PVdB without need get out to DOS (curl_try was only a check).
To spread in internet, the best form to save the 4 needed files:
Scripts/Rottentomatoes_[HTTPS].psf,
Scripts/curl-ca-bundle.crt,
Scripts/curl.exe,
Scripts/libcurl.dll
is compressed in a zip file but the length is out of the limit of the forum. Now, the script checks that all files are present.
If you want to download the original ones: https://bintray.com/artifact/download/vszakats/generic/curl-7.50.2-win32-mingw.7z (The needed files were in "curl-7.50.2-win32-mingw\bin\" folder)
--- Quote ---Will now required for each movie in curl_try file manually enter the URL address of the movie, which is of course more movies lengthy work, or it will be automatically regulated.
--- End quote ---
Now the script have a automatical full search, but it must save in the 'http' (not S) base url. For back compatibility and for prevents PVdB crash. Do the same if you write it manually.
Finally, it's not difficult to adapt this script to another HTTPS page. All people are free to use it.
Ivek23:
A big thank you to the script, effort and work around it.
This script has one big mistake, that is not working properly as shown in the log file.
--- Code: ---(8.11.2016 8:46:09) PVD Version: 0.9.9.21
(8.11.2016 8:46:10) OS: Windows 7 Home Basic Edition
(8.11.2016 8:46:10) Plugin loaded: plainexp.dll 0.7.1.1
(8.11.2016 8:46:10) Plugin loaded: scriptint.dll 0.2.6.1
(8.11.2016 8:46:10) Plugin loaded: valueconvert.dll 0.1.0.2
(8.11.2016 8:46:10) Compiling script: Rottentomatoes_[HTTPS].psf
(8.11.2016 8:46:10) Script compiled successfully: Rottentomatoes_[HTTPS].psf
[Hint] (199:1): Variable 'ITEMSUBLIST' never used
(8.11.2016 8:46:10) Executing script binary
(8.11.2016 8:46:10) Script loaded: Rottentomatoes_[HTTPS].psf 0.2.0.3
(8.11.2016 8:46:10) Loading database: C:\Users\Ivo\Documents\Personal Video Database\movies.pvd
(8.11.2016 8:46:11) UpdateToolbar: 1
(8.11.2016 8:46:12) UpdateToolbar: 2
(8.11.2016 8:46:13) UpdateToolbar: 3
(8.11.2016 8:46:37) UpdateToolbar: 4
(8.11.2016 8:46:45) GET: http://www.videodb.info/upload/check.php
(8.11.2016 8:46:47) Update error: Socket Error # 10061
Connection refused.
(8.11.2016 8:46:52) UpdateToolbar: 5
(8.11.2016 8:47:03) Compiling script: Rottentomatoes_[HTTPS].psf
(8.11.2016 8:47:03) Script compiled successfully: Rottentomatoes_[HTTPS].psf
[Hint] (199:1): Variable 'ITEMSUBLIST' never used
(8.11.2016 8:47:03) Executing script binary
(8.11.2016 8:47:03) Logging in...
(8.11.2016 8:47:03) Function GetDownloadURL======================|
(8.11.2016 8:47:03) Global Var-Mode|0|
(8.11.2016 8:47:03) Global Var-DownloadURL||
(8.11.2016 8:47:03) Global Var-ScriptPath|C:\Program Files\Personal Video Database\Scripts\|
(8.11.2016 8:47:03) Waiting 1s for delete:C:\Program Files\Personal Video Database\Scripts\downpage.htm
(8.11.2016 8:47:04) Waiting 1s for delete:C:\Program Files\Personal Video Database\Scripts\downpage.htm
(8.11.2016 8:47:05) Waiting 1s for delete:C:\Program Files\Personal Video Database\Scripts\downpage.htm
(8.11.2016 8:47:06) Waiting 1s for delete:C:\Program Files\Personal Video Database\Scripts\downpage.htm
(8.11.2016 8:47:07) Waiting 1s for delete:C:\Program Files\Personal Video Database\Scripts\downpage.htm
(8.11.2016 8:47:08) Waiting 1s for delete:C:\Program Files\Personal Video Database\Scripts\downpage.htm
(8.11.2016 8:47:09) Waiting 1s for delete:C:\Program Files\Personal Video Database\Scripts\downpage.htm
(8.11.2016 8:47:11) Waiting 1s for delete:C:\Program Files\Personal Video Database\Scripts\downpage.htm
(8.11.2016 8:47:12) Waiting 1s for delete:C:\Program Files\Personal Video Database\Scripts\downpage.htm
(8.11.2016 8:47:13) Waiting 1s for delete:C:\Program Files\Personal Video Database\Scripts\downpage.htm
(8.11.2016 8:47:14) Waiting 1s for delete:C:\Program Files\Personal Video Database\Scripts\downpage.htm
(8.11.2016 8:47:15) Waiting 1s for delete:C:\Program Files\Personal Video Database\Scripts\downpage.htm
--- End code ---
This always happens when a URL (http or https or it is not. The error caused by this part of the code.
--- Code: --- //Delete the ancient dowloaded page file. Needed for wait to curl download.
While FileExists(ScriptPath+BASE_DOWNLOAD_FILE) do begin
FileExecute('cmd.exe', '/C del '+ScriptPath+BASE_DOWNLOAD_FILE);
LogMessage(' Waiting 1s for delete:'+ScriptPath+BASE_DOWNLOAD_FILE);
wait (1000);
end;
--- End code ---
If this part of the code is blocked, the script is working normally as shown in the log file.,
--- Code: ---?»?(8.11.2016 8:54:26) PVD Version: 0.9.9.21
(8.11.2016 8:54:27) OS: Windows 7 Home Basic Edition
(8.11.2016 8:54:27) Plugin loaded: plainexp.dll 0.7.1.1
(8.11.2016 8:54:27) Plugin loaded: scriptint.dll 0.2.6.1
(8.11.2016 8:54:27) Plugin loaded: valueconvert.dll 0.1.0.2
(8.11.2016 8:54:27) Compiling script: Rottentomatoes_[HTTPS].psf
(8.11.2016 8:54:27) Script compiled successfully: Rottentomatoes_[HTTPS].psf
[Hint] (199:1): Variable 'ITEMSUBLIST' never used
(8.11.2016 8:54:27) Executing script binary
(8.11.2016 8:54:27) Script loaded: Rottentomatoes_[HTTPS].psf 0.2.0.3
(8.11.2016 8:54:27) Loading database: C:\Users\Ivo\Documents\Personal Video Database\movies.pvd
(8.11.2016 8:54:28) UpdateToolbar: 1
(8.11.2016 8:54:30) UpdateToolbar: 2
(8.11.2016 8:54:30) UpdateToolbar: 3
(8.11.2016 8:54:58) Compiling script: Rottentomatoes_[HTTPS].psf
(8.11.2016 8:54:58) Script compiled successfully: Rottentomatoes_[HTTPS].psf
[Hint] (199:1): Variable 'ITEMSUBLIST' never used
(8.11.2016 8:54:58) Executing script binary
(8.11.2016 8:54:58) Logging in...
(8.11.2016 8:54:58) Function GetDownloadURL======================|
(8.11.2016 8:54:58) Global Var-Mode|0|
(8.11.2016 8:54:58) Global Var-DownloadURL||
(8.11.2016 8:54:58) Global Var-ScriptPath|C:\Program Files\Personal Video Database\Scripts\|
(8.11.2016 8:54:58) Some error deleting: C:\Program Files\Personal Video Database\Scripts\downpage.htm
(8.11.2016 8:54:58) Download with curl in file:|C:\Program Files\Personal Video Database\Scripts\downpage.htm| the information of:|https://www.rottentomatoes.com/m/godfather||
(8.11.2016 8:55:00) Return for parsing movie with stored information of:|https://www.rottentomatoes.com/m/godfather||
(8.11.2016 8:55:02) Now present file: C:\Program Files\Personal Video Database\Scripts\downpage.htm
(8.11.2016 8:55:02) Searching movie information for: Godfather
(8.11.2016 8:55:02) Function ParsePage======================|
(8.11.2016 8:55:02) Global Var-Mode|1|
(8.11.2016 8:55:02) Global Var-DownloadURL|https://www.rottentomatoes.com/m/godfather|
(8.11.2016 8:55:02) Global Var-ScriptPath|C:\Program Files\Personal Video Database\Scripts\|
(8.11.2016 8:55:02) Local Var-URL||
(8.11.2016 8:55:02) Parsing movie page
(8.11.2016 8:55:02) Get result url:http://www.rottentomatoes.com/m/godfather||
(8.11.2016 8:55:02) Get result title:The Godfather||
(8.11.2016 8:55:02) Get result year:1972||
(8.11.2016 8:55:02) Get result description:Popularly viewed as one of the best American films ever made, the multi-generational crime saga The Godfather is a touchstone of cinema: one of the most widely imitated, quoted, and lampooned movies of all time. Marlon Brando and Al Pacino star as Vito Corleone and his youngest son, Michael, respectively. It is the late 1940s in New York and Corleone is, in the parlance of organized crime, a "godfather" or "don," the head of a Mafia family. Michael, a free thinker who defied his father by enlisting in the Marines to fight in World War II, has returned a captain and a war hero. Having long ago rejected the family business, Michael shows up at the wedding of his sister, Connie (Talia Shire), with his non-Italian girlfriend, Kay (Diane Keaton), who learns for the first time about the family "business." A few months later at Christmas time, the don barely survives being shot by gunmen in the employ of a drug-trafficking rival whose request for aid from the Corleones' political connections was rejected. After saving his father from a second assassination attempt, Michael persuades his hotheaded eldest brother, Sonny (James Caan), and family advisors Tom Hagen (Robert Duvall) and Sal Tessio (Abe Vigoda) that he should be the one to exact revenge on the men responsible. After murdering a corrupt police captain and the drug trafficker, Michael hides out in Sicily while a gang war erupts at home. Falling in love with a local girl, Michael marries her, but she is later slain by Corleone enemies in an attempt on Michael's life. Sonny is also butchered, having been betrayed by Connie's husband. As Michael returns home and convinces Kay to marry him, his father recovers and makes peace with his rivals, realizing that another powerful don was pulling the strings behind the narcotics endeavor that began the gang warfare. Once Michael has been groomed as the new don, he leads the family to a new era of prosperity, then launches a campaign of murderous revenge against those who once tried to wipe out the Corleones, consolidating his family's power and completing his own moral downfall. Nominated for 11 Academy Awards and winning for Best Picture, Best Actor (Marlon Brando), and Best Adapted Screenplay, The Godfather was followed by a pair of sequels. ~ Karl Williams, Rovi||
(8.11.2016 8:55:02) Get result mpaa:R (N/A)||
(8.11.2016 8:55:02) Parse results List Genre:Drama||
(8.11.2016 8:55:02) Get results Genre:Drama||
(8.11.2016 8:55:02) Parse results List Directors:Francis Ford Coppola||
(8.11.2016 8:55:02) Get results Directors:Francis Ford Coppola||
(8.11.2016 8:55:02) Parse results List Writers:Francis Ford Coppola , Mario Puzo||
(8.11.2016 8:55:02) Get results Writers:Francis Ford Coppola||
(8.11.2016 8:55:02) Get results Writers:Mario Puzo||
(8.11.2016 8:55:02) Get result ĹĽTOMATOMETER All critics?:99||
(8.11.2016 8:55:02) Get result RT average rating All critics:8.8||
(8.11.2016 8:55:02) Get result ĹĽTOMATOMETER Top critics?:95||
(8.11.2016 8:55:02) Get result RT average rating Top critics:8.8||
(8.11.2016 8:55:02) Get result ĹĽAUDIENCE SCORE?:98||
(8.11.2016 8:55:02) Get result RT average rating audience:4.4/5||
(8.11.2016 8:55:02) Script end after retreived info of movie
(8.11.2016 8:55:14) UpdateToolbar: 4
(8.11.2016 8:55:14) UpdateToolbar: 5
--- End code ---
As you yourself have observed before, that you can not find the reason why this part of the code does not work.
Perhaps to this extent to make any corrections.
BTW:
Version of our script, which you mentioned before(version 0.3.0.0.) is not the same as written in the script(version 0.2.0.3).
Ivek23:
--- Quote ---
--- Quote ---How to edit or connect now all files from curl_try to Rottentomatoes_ [HTTPS] in one executable file.
--- End quote ---
The script works directly in PVdB without need get out to DOS (curl_try was only a check).
--- End quote ---
It is still necessary to manually in curl_try file to write the url address to search results or the URL for a particular movie to curl transferred information in downpage.htm file.
How do I arrange that, for example, when you run curl_try file to download data for search results, then it does not overwrite data for a particular movie in downpage.htm file but that these data are added. Now it is necessary to copy the data to search for prior information about the movie in downpage.htm file.
Whether such an option might be to add a script that would not be necessary prior to the transfer of data to the movie always restart the curl_try file.
Navigation
[0] Message Index
[#] Next page
[*] Previous page
Go to full version