Personal Video Database
English => Development => Scripts and Templates => Topic started by: rick.ca on March 16, 2009, 10:57:53 pm
-
The AllMovie site has been replaced by AllRovi. See AllRovi movie script (http://www.videodb.info/forum_en/index.php/topic,2487.msg12988.html#msg12988).
Attached is my modification of nostra's AllMovies.com script (http://www.videodb.info/forum_en/index.php?topic=895.msg2074#msg2074)to include additional data for custom fields. Essentially, I changed the script to gather as much data as possible from the left-hand column of the "Overview" page. At the beginning of the script, I've documented the additional custom fields necessary to accommodate this data (see below).
Warning: This script uses the standard Category and Tags fields. Ensure your Field overwrite settings are set correctly to avoid conflicts.
All of the 17 custom fields included are optional. Just add the ones you want to Preferences - Movies - Custom items. There is no need to modify the script for fields you choose not to use. The script refers to custom fields by name, so if you use a different name, you will have to change it in the script.
I've done this largely by copying and adapting parts of nostra's script without understanding exactly how or why it works. :-[ So suggestions or comments are welcome. Questions too—I love to share my confusion. ;D
Here's the documentation:
CUSTOM AND NON-STANDARD FIELD MAPPINGS
All custom fields are optional. All standard fields can be suppressed using
Preferences-Plugins-Overwrite fields. If custom names are changed in PVD, they
must also be changed where they occur in the body of this script. The type of
custom field used is also somewhat a matter of personal preference. A list which
is for viewing only will do just as well in a long text as a multiselect field.
ALLMOVIE CAPTION PVD FIELD FIELD TYPE
Cast Tab Cast memo
Production Credits Tab Production Credits memo
Review Tab Review memo
Run time Original duration number
Work rating AllMovie rating rating
Attributes Attributes multiselect list
MPAA + MPAA Reasons MPAA standard (IMDb is primary source)
Category Work type select list
Color type Color type select list
Types Types multiselect list
Flags Flags multiselect list
Keywords Keywords multiselect list
Themes Category standard (sole source)
Tones Tags standard (sole source)
Moods Moods multiselect list
From book (or story) Book short text
Set in Set in multiselect list
Box office Box office standard (IMDb is primary source)
Produced by Studio standard (IMDb is secondary source)
Release (or Premiere) Released short text
Released by Released by short text
(Date) Updated short text
Current version attached:
09/24/2010 – 0.1.2.2 – Modified ParseCast procedure to save entire Cast tab to custom memo field, as well as individual actors.
[attachment deleted by admin]
-
Now adds current date to Updated field.
The updated script is attached to my original message above.
-
Rick, this is great - thanks for sharing.
Just wondering, do you mainly use allmovies for your metadata? I was trialling a system with custom fields a while ago to attempt to maintain both IMDb and AMG fields in the same database (Used standard fields for IMDb and a whole bunch of custom fields for AMG - AMG Director, AMG Synopsis etc etc). It was getting a bit unwieldy.
There are a few situations where AMG is incomplete compared to IMDb or vice versa... what's your system?
-
Some feedback... works great! A few little niggly bits I've noticed so far...
1. Box Office doesn't format correctly in the standard field. Works fine as a "short text" custom entry
//Box office
AddCustomFieldValueByName('Box Office', HTMLValues(HTML,
'<span>Box Office</span>', '</table>',
'<li>', '</li>',
', ', EndPos));
2. Your MPAA + Reason formatting works okay, but if there is no reason listed, you get an output of "Rated M for ." etc
3. I added support for the "Region" field in AMG. It's being used more and more frequently on the site, but many older movies don't have it populated.
//Region
AddCustomFieldValueByName('Region', HTMLValues2(HTML,
'<span>Premiere</span>', '</table>',
'<a href="', '</a>',
', ', EndPos));
4. I seem to get a rating of 10 for every movie... must have a closer look.
Also, I'm not sure if it was you or Nostra who added the Reviews and Producion Credits functionality, but that's awesome - thanks :)
-
Rick, this is great - thanks for sharing.
You're welcome. It was painful, but I figured it was my turn. ;)
Just wondering, do you mainly use allmovies for your metadata?
I use both IMDb and allmovies together—usually run in a batch. IMDb goes first, because it is generally more complete. I let it handle all the people. You can see from my documentation where I use allmovie data to overwrite standard fields which may have been populated by the IMDb plugin. The rest goes into custom fields. As you probably know from doing this yourself, my approach in modifying the script was just to grab whatever I could from the stuff listed in the right-hand column of the page. I do actually have all those custom fields in my database, but only for the sake of being complete. The data elements I appreciate most are the Synopsis and Review (my original reason for going to allmovies), Tones (which I find gives a really good indication of how a movie feels), and perhaps Attributes. The rest is nice to have, but not critical.
My preferences and my skin design probably feed off each other. With that in mind, I'll attach some screenshots so you can see what I mean. We don't yet (as of 0.9.9.4) have the ability to change the placement of custom fields—they're still just dumped into the custom section of the skin. This is why I'm very deliberately placing some allmovie data into standard fields—so I can place them where I want them in my skin. Also, you may note most of the list data goes into Multiselect list fields, even thought they may look better as List or String fields. This is because I don't care so much about looking at them—I'd rather have them take up less space, and be used primarily as a means of grouping.
When I was doing this before, I suppose it got a bit unwieldy. Redoing the script, however, forced me to rethink what I was doing and why. Now it seems to work quite well. I do have to make sure my Overwrite field settings are correct—and stay that way. That can be nuisance when a plugin or script is used for more than one thing (e.g., movies and series). BTW, I think I have nostra partially committed to implementing my fix for that (http://www.videodb.info/forum_en/index.php?topic=963.msg2479#msg2479) in version 2.0. ;D
[attachment deleted by admin]
-
Thanks, that was really helpful... I agree that it's nice to know where the data came from. Up until now, I've been using IMDb for all standard fields, with AMG fields all belonging to custom fields with a "AMG " prefix. It's hard to get the best of both worlds... custom fields don't work too well with "People" fields...
A template to decide which order fields are displayed in would work really well, for both standard and custom fields I think...
-
A few little niggly bits I've noticed so far...
Thanks! My routine of downloading from two sources is concealing some bugs.
1. Box Office doesn't format correctly in the standard field. Works fine as a "short text" custom entry
Yes, I started with a custom field, but then decided I would prefer to use the standard. So I reverted to nostra's code. A simple dollar figure seems to work. But "$106,260,000/ Top grossing film of 1979" becomes "1,062,600,009" (!) Obviously the standard field is numeric. So I added the "+ 1" in the following:
//Box office
curPos := PosFrom('<span>Box office</span>', HTML, EndPos);
if curPos > 0 then begin
curPos := PosFrom('<li>', HTML, curPos) + 5;
EndPos := PosFrom('</li>', HTML, curPos);
TmpStr := Copy(HTML, curPos, EndPos - curPos);
curPos := Pos('/', TmpStr);
if curPos > 0 then
Delete(TmpStr, curPos, Length(TmpStr) - curPos + 1); <-----
curPos := Pos(':', TmpStr);
if curPos > 0 then
Delete(TmpStr, curPos, Length(TmpStr) - curPos + 1); <-----
AddFieldValueXML('money', TmpStr);
end;
I don't really understand the code (imagine a monkey banging on the keyboard :-[ ), so I'm not sure if that's an appropriate modification, although it seems to work for ones I can find. Allmovies doesn't seem to have much box office data—which is why I decided against an additional custom field. :-\
2. Your MPAA + Reason formatting works okay, but if there is no reason listed, you get an output of "Rated M for ." etc
Yes, just being lazy here. I only use this a backup to IMDb, which normally have exactly the same data (that's where the "Rated X for ..." format comes from). I couldn't be bothered fixing it for the rare case where IMDb would not have the data, and allmovie would have the rating but not the reason. Still, just fixing it would be easier than making excuses...
//MPAA Reasons
TmpStr1 := HTMLValues(HTML, '<span>MPAA Reasons</span>', '</table>', '<li>', '</li>', ', ', EndPos);
if (TmpStr1 = '') AND (TmpStr2 <> '') then
AddFieldValue(mfMPAA, 'Rated ' + TmpStr2);
if (TmpStr1 <> '') AND (TmpStr2 <> '') then
AddFieldValue(mfMPAA, 'Rated ' + TmpStr2 + ' ' + TmpStr1 + '.');
...well, maybe not. I think that does it, but I'm not sure. :-\
3. I added support for the "Region" field in AMG.
I think that the one that only occurred in 2% of my movies—so I deleted it. There are a number of such obscure items. The site design is very annoying. Tags like that are usually linked to a page with a definition of the term and/or a list of movies with the tag value, but there's no other way to access those pages. In other words, I have to know of a movie tagged with a region before I can find out anything about that tag. To enumerate a few examples...
854: From Play
855: From Poem By
856: From Poem
857: From Screenplay By
858: From Screenplay
860: From Story
861: Lampoons "Name" (?)
862: Location
862: Movie Budget
862: Recognizable Quote...
Sorry—thought I might find "Region" or something interesting if I kept going. I think they're on drugs, or just want to piss-off script writers. ;)
If only we had the entire list of tags and they appeared on the page in tag number order (they don't), that would be a more reliable way to reference these items, and perhaps aggregate some of them into something useful. For example, the "From" ones could be combined into one field populated with "From book ____", "From play ____", etc. )I combined Book and Story, but I couldn't find the rest.)
4. I seem to get a rating of 10 for every movie... must have a closer look.
Their rating tend to be generous, but that's not right. I don't know why this may cause you a problem, but note the script is using a custom field, rather than the usual (i.e., as in nostra's version) Other rating field.
I'm not sure if it was you or Nostra who added the Reviews and Production Credits functionality, but that's awesome
That would be nostra. I'm hoping you'll study his technique and use it to grab the "Similar Works" entries from the Overview page and save them as Connections. While it wouldn't be quite as useful, it would nevertheless be super-awesome! Besides, there's nothing left to grab, so this one's yours. ;D
-
Oh wow, I've never seen some of those fields up there! haha
Yeah, I'd like that extra info too... I wonder how it would be best presented though - maybe in a note/memo field. I'll have a look - but my approach so far has been similar to yours... look at the existing script and trying to adapt it without breaking it. I've broken it a few dozen times now :)
-
I'd like that extra info too...
I figured analyzing the tag URLs would be a good way to figure out what information is there and how it's organized. See attached bookmarks file (remove the .TXT and open in browser and/or import bookmarks). I gave up after 1,555 (tag numbers up to 2,200). Still no sign of Region. ::)
Note many of the URLs have no information other than the name of the tag. Others have a brief description, definition or instruction. Many are to lists of movies with the tag—so the link to those can be useful. You know, when you just have to have direct link to movies made in Albania... ;)
[attachment deleted by admin]
-
In an attachment there is a script which works both with www, and without www.
Applied same fix to this script (attached to first message above).
Thanks, Reset.
-
I have modified the script to conform to changes in the original provided with 0.9.9.5. It's attached to the first message above.
-
As mentioned here (http://www.videodb.info/forum_en/index.php?topic=1300.msg5164#msg5164), I've modified the script to save "Run time" to the custom field OrigDuration instead of the standard Duration field. I now no longer download to Duration, leaving this for the actual media length according to MediaInfo (i.e., the program's scan of the media file). The revised script is attached to the first message above.
BTW, the actual name of the field "Duration" is Length—that's how it must be referred to in a skin.
-
This modification just made my day! I love it. ;D ;D
Does it work with any version of the allmovies.com plugin or does it need updating, too?
-
Welcome, sunspot. Glad you like it.
There is no AllMovie plugin. This is a modification of the script written by nostra (and included in the installation package). Downloading/importing from any particular source is effected by a plugin or a script, not both.
-
Thank you for your answer! You seem to be a bedrock for this forum.
I obviously don't know enough to just ask a proper question. A follow-up one, if I may: So the script would need updating like any other, if the site allmovie changes? I ask to understand a little (little) bit better what's going on.
-
So the script would need updating like any other, if the site allmovie changes?
That's right. But, as you can see from this thread, it has required only a few minor changes. Please do let me know if you find any problems. It could be something has changed, but I haven't noticed because I'm no longer adding large numbers of new movies to my database.
-
Oh, that's something I can provide - heavy duty importing¨ ;D
And I will certainly try to give meaningful feedback. But, at the moment, it's just perfect! Don't see anything broken or so. Quite the contrary, I love how it replaces the imdb tags with more meaningful ones. Tone and Mood (where available) are superb, too. Thanks again. I think I will head now to the donation page, if you will excuse me. :)
-
Hi Rick,
This scripts looks very interesting and I'd like to get these extra fields too.
Unfortunately, (and i'm a bit embarassed here... :-[) I can't get pvd to "recognize" the script. I must be doing something wrong.
I have extracted it into the "script" folder in pvd's directory, and it does not show up in the "import" drop down in pvd.
I even renamed it so that it would replace the allmovie original script and again it failed to show up.
Can u help?
Thanks a million.
-
Sorry, I forgot to update the script posted here. I've done so now. The version is now...
SCRIPT_VERSION = '0.1.1.9'; //Converted to use Unicode strings effective 0.9.9.15
-
What and how is necessary to instal AMG Work ID in the script.
That would do just for myself, of course, if it is not too much input in the script.
Sorry for the dumb question.
-
I don't know. For other data in the left bar, the script takes advantage of the fact the target data is hyperlinked. That one is not, so a different technique is required. I'm sure it's not difficult for someone who understands how to do it, but that's not me.
Why do you need it? It's included in the URL.
-
But I have found this in the url,whipping post so I left it for others.
-
rick.ca
I have a question:
May I with your permission to publish the script AllMovie +modified,
I have organized my liking.
Works great for me.
-
Sure, but please post it in a new topic and give it a different name (maybe "IvexAllmovie" ;) ). We don't want it confused with my version or nostra's original.
-
OK,Thanks.
-
Hi Rick,
I am using your script to scrape the review from allmovie and I would also like to have the 'tones' added to my movies.
My problem is that I would like to keep my tags and add a similar 'Tones' field.
If I use the modification provided by Ivek23 I get the tones, but not as clickable items. I really would like to have them as clickable items (like tags are). Would you know how?
Thanks in advance.
NOTE: I have also just noticed that when using your script the rating is not captured, and when using the original one, it is. My overwrite options are the same for the two scripts.
D
-
I really would like to have them as clickable items (like tags are).
That's a characteristic unique to standard list fields. More generally, only standard fields can be used in a Simple search—which is what the link invokes.
I find Allmove Tones make much better use of this feature than IMDb Keywords. There are far too many keywords for a search of any one of them to be meaningful.
-
You're right, there are too many tags.
Still if I want to watch a movie say on Palestine, the "palestine" tag will be there, while the tone will not.
Is there a list of Allmovie tones somewhere?
I guess it's up to me to choose what I prefer to use, thanks for the info.
In terms of rating, I have noticed that If I want to add the Allmovie rating with your script I will have to add a custom field called "Allmovie rating". I would like to still use the original script way which is changing "additional rating" into "allmovie rating".
Could you tell me what I need to change?
-
Is there a list of Allmovie tones somewhere?
I don't recall where/how I got it, but...
Affectionate
Angry
Atmospheric
Austere
Autumnal
Biting
Bittersweet
Bleak
Bright
Campy
Cathartic
Cerebral
Chilly
Claustrophobic
Compassionate
Confrontational
Creepy
Cynical
Deadpan
Decadent
Deliberate
Disturbing
Downbeat
Drab
Dreamlike
Earnest
Earthy
Easygoing
Eerie
Elegant
Elegiac
Enigmatic
Fanciful
Flashy
Forceful
Frantic
Gentle
Gloomy
Goofy
Goth
Grim
Gritty
Gruesome
Hallucinatory
Harsh
Heartwarming
Humorous
Intimate
Ironic
Irreverent
Lavish
Light
Literate
Lurid
Lyrical
Macabre
Madcap
Matter-of-Fact
Meditative
Melancholy
Menacing
Merry
Moody
Nocturnal
Nostalgic
Ominous
Paranoid
Passionate
Poignant
Quirky
Racy
Raunchy
Reflective
Rousing
Satirical
Sensual
Sentimental
Sexual
Sexy
Silly
Slick
Somber
Springlike
Stirring
Stylish
Stylized
Summery
Sweeping
Sweet
Talky
Tearjerking
Tense
Understated
Upbeat
Visceral
Warm
Whimsical
Wintry
Wistful
Witty
Wry
Could you tell me what I need to change?
Not off-hand, but I'm sure you could figure it out by comparing the two scripts.
You may want to consider leaving it as is. Should you ever want to get a second rating using a plugin (rather than a script), using orating is the only option.
-
Yes but so far I have already added the Allmovie ratings with the other script (original one) so if I add it with your script I will double the info.
I have tried to paste the rating part of the original script into yours to no success.*
Anyway I will just run the original before running yours.
Thanks for the list of tones (and all your support) by the way.
-
I have tried to paste the rating part of the original script into yours to no success.
I don't know why not. It looks like the essential difference is just using...
AddFieldValue(mfRating, FloatToStr(StrToFloat(Copy(HTML, curPos, EndPos - curPos)) * 2));
instead of...
AddCustomFieldValueByName('AllMovie rating', FloatToStr(StrToFloat(Copy(HTML, curPos, EndPos - curPos)) * 2));
If you start the program with the -debug switch, the log will show any script errors.
-
This works great thank you :D
-
I've started importing the Allmovie reviews into a custom memo field, but there are no paragraph breaks like there are with the user reviews for the IMDb plugin. I checked 2 reviews
http://www.allmovie.com/work/twilight-420151/review
http://www.allmovie.com/work/obsessed-433716/review
and both have multiple paragraphs on the website but are one long paragraph when imported into the memo field. Any idea why this could be happening?
-
I don't know. It's always been that way, and I'm sure the original script is the same.
According to the Scripting Manual (http://www.videodb.info/help/hlp_scripting.html#string), the doLineBreaks parameter of the TextBetween function is to be "set to true if you want the <br> tag to be replaced with carriage return character." It is, but it doesn't seem to work. I don't understand these things, but I wonder if it's because this is XHTML and only the closing form of the tag—<br />—is used. In any case, the tags are not being converted, and the result of the function is to "return the resulting substring without tags"—so they just get stripped.
I would need nostra to fix the function, or suggest a workaround. I suppose it could be handled using different functions, but I'm unable figure it out. :'(
-
Hmm, since the IMDb plugin is a DLL and not a script, I presume there's no way to examine it to see how nostra did it for the IMDb user reviews. Have you noticed if any of the other Movie Information scripts are able to do the conversion successfully?
-
No and no. Hopefully, nostra can help.
-
You can use the StringReplace function to replace any type of Line Break tags and characters in proper ones (#13#10)
-
You under estimate the degree of my helplessness. :-[
I thought of that, but I can't figure out how to get the string without using TextBetween. If I do, the tags are stripped, and there's nothing to replace. :-\
-
Alright then:
1. Find the position of starting tag using the PosFrom or Pos function and save it to a variable
2. Find the position of ending tag using the PosFrom or Pos function and save it to another variable
3. Use the Copy function to retrieve the text between starting tag and ending tag
4. Use StringReplace function to remove unneeded substrings or replace substrings
-
I'm still not getting anywhere. Following is my attempt to do as you suggest...
procedure ParseReview(HTML : String);
var
curPos, endPos : Integer;
TmpStr, TmpStr1, TmpStr2 : String;
begin
curPos := Pos('<div id="results-table">', HTML);
endPos := curPos
if curPos < 1 then
Exit;
//Review
curPos := PosFrom('<td colspan="2">', HTML, curPos);
endPos := PosFrom('</p></td>', HTML, curPos);
TmpStr := Copy(HTML, curPos, EndPos - curPos);
TmpStr1 := StringReplace(TmpStr, ' -- ', '—', True, True, False);
TmpStr2 := StringReplace(TmpStr1, ' --- ', '—', True, True, False);
TmpStr := StringReplace(TmpStr2, '--', '—', True, True, False);
TmpStr1 := StringReplace(TmpStr, '<br />', #13#10, True, True, False);
TmpStr2 := RemoveTags(TmpStr1, True);
AddCustomFieldValueByName('Review', TmpStr2);
end;
The TmpStr1 := StringReplace(TmpStr, '<br />', #13#10, True, True, False); line does nothing. It seems the Copy function removes the <br /> tags, so there's nothing to replace. And even if there were something to replace, the #13#10 doesn't seem to be recognized by the function.
The result of the Copy function is a set of four CRLF between paragraphs. This, of course, results in three blank lines instead of one, as desired.
-
Post the whole script.
TmpStr1 := StringReplace(TmpStr, '<br />', #13#10, True, True, False);
This can only happen if the string does not contain "<br />"
It seems the Copy function removes the <br /> tags
No, it certainly does not remove anything
The result of the Copy function is a set of four CRLF between paragraphs
Then you can replace those with one line break.
-
No, it certainly does not remove anything
Maybe I misspoke. Let me put it this way: When I use only the Copy function to get the data, the <br /> tags that appeared to be in the source were not present (and apparently replaced by three CRLF's) in the imported text. No other tags "disappeared." Is this thing even a tag? It appears by itself, with no <br>. :-\
Then you can replace those with one line break.
Maybe whatever is going on with this <br /> is mute. Can you just show me how to replace multiple line breaks with one? StringReplace doesn't seem to do anything when I use #13#10 as OldPattern or NewPattern.
-
The problem was:
1. The function RemoveTags used in the parsing procedure does not accept normal line breaks (only HTML tags)
2. You had to replace #13 with #13#10
Fixed script is attached
[attachment deleted by admin]
-
Thanks! I think it's safe to say I couldn't have done that if my life depended on it. ;D
I've updated the first post of this topic to include an updated version. It's modified slightly—to reduce the blank lines between paragraphs from two to one. If anyone is wondering, the same change to the Synopsis/Description has not been make because all of those are a single paragraph.
-
I think it's safe to say I couldn't have done that if my life depended on it. Grin
Yes, it was a bit tricky as you needed an alternative RemoveTags function, but I think now you will be able to solve such problems with ease ;)
-
Update:
09/24/2010 – 0.1.2.2 – Modified ParseCast procedure to save entire Cast tab to custom memo field, as well as individual actors.
See top post. (http://www.videodb.info/forum_en/index.php?topic=1232.msg4393#msg4393)
-
Thanks,rick.ca for updates.
-
The AllMovie site has been replaced by AllRovi. See AllRovi movie script (http://www.videodb.info/forum_en/index.php/topic,2487.msg12988.html#msg12988).