Author Topic: [SOLVED] Multi threaded IMDB fetching?  (Read 29080 times)

0 Members and 1 Guest are viewing this topic.

Offline Happy2k

  • User
  • ***
  • Posts: 42
    • View Profile
[SOLVED] Multi threaded IMDB fetching?
« on: August 27, 2010, 01:06:54 am »
Is there any way to make the IMDB fetcing multi threaded?
Worst part is, if it encounters a movie its not sure about - everything stops. So you cant leave it over night, and just fix the unknown ones when you get up.

If there is a way to speed it up, id very much like to get a hint about it ;)

Else, a possible solution could be:
- Multi threadded IMDB fetching
- A "fix me"-list, for movies that are not recognized 100%

Besides that.. the most recent update seems to have helped on the recognition - its only few that are not automaticly recognized now!
« Last Edit: September 22, 2010, 01:00:54 am by rick.ca »

Offline rick.ca

  • Global Moderator
  • *****
  • Posts: 3241
  • "I'm willing to shoot you!"
    • View Profile
Re: Multi threaded IMDB fetching?
« Reply #1 on: August 27, 2010, 02:25:24 am »
Quote
If there is a way to speed it up, id very much like to get a hint about it

Silent mode.  ;)

Quote
A "fix me"-list, for movies that are not recognized 100%

Use a search to display the records not updated by the Silent mode update. Since all of those will require user input, you'll be kept reasonably busy while updating those with Silent mode off.

Quote
Multi threadded IMDB fetching

That's an interesting idea for other reasons (e.g., update speed), but I would be very surprised if IMDb did not ban IP's making multiple simultaneous requests. I'm actually surprised were able to get away with what we are doing.

Offline deazo

  • Older Power User
  • *****
  • Posts: 283
    • View Profile
Re: Multi threaded IMDB fetching?
« Reply #2 on: August 27, 2010, 09:01:19 am »

 Hi Rick,
 
 could you clarify "silent mode" for me please?
 I always thought it was taking the first result in the list of films found, hence that there was no way to return to the "not 100% match".

 But from what you wrote it sounds like it is just skipping the ones it is not sure about?
 Could you confirm please?

 Thanks

Offline rick.ca

  • Global Moderator
  • *****
  • Posts: 3241
  • "I'm willing to shoot you!"
    • View Profile
Re: Multi threaded IMDB fetching?
« Reply #3 on: August 27, 2010, 10:23:36 am »
Quote
But from what you wrote it sounds like it is just skipping the ones it is not sure about?

Right. It would make quite a mess of things if guessed. ;)

Offline Happy2k

  • User
  • ***
  • Posts: 42
    • View Profile
Re: Multi threaded IMDB fetching?
« Reply #4 on: August 27, 2010, 10:55:30 am »
Quote
If there is a way to speed it up, id very much like to get a hint about it

Silent mode.  ;)

Yea, but silent isnt faster, is it?

Quote
A "fix me"-list, for movies that are not recognized 100%

Use a search to display the records not updated by the Silent mode update. Since all of those will require user input, you'll be kept reasonably busy while updating those with Silent mode off.

That sure is a good possibility - gonna try it.


Quote
Multi threadded IMDB fetching

That's an interesting idea for other reasons (e.g., update speed), but I would be very surprised if IMDb did not ban IP's making multiple simultaneous requests. I'm actually surprised were able to get away with what we are doing.

Before using PVD, i was trying out different programs. Most of them had a very fast IMDb fetch. I dont think IMDb would care - ive never heard of IMDb ip banning someone.
I wouldnt mind being a ginniepig for testing out multi threaded IMDb fetching.
« Last Edit: August 27, 2010, 10:57:09 am by Happy2k »

Offline rick.ca

  • Global Moderator
  • *****
  • Posts: 3241
  • "I'm willing to shoot you!"
    • View Profile
Re: Multi threaded IMDB fetching?
« Reply #5 on: August 27, 2010, 11:33:24 am »
Quote
Yea, but silent isnt faster, is it?

In the context of your original question, it's days faster. ::)

Quote
I dont think IMDb would care - ive never heard of IMDb ip banning someone.

Perhaps not, but HTTP error 503 occurs regularly. They probably just mean the server is busy, but it could also mean the server is telling you it's busy because it's getting too many requests from you.

Offline deazo

  • Older Power User
  • *****
  • Posts: 283
    • View Profile
Re: Multi threaded IMDB fetching?
« Reply #6 on: August 27, 2010, 12:24:15 pm »

 Thanks for clarification Rick, I thought Silent Mode was taking the first movie found (!), that's why I was never using it.
 In terms of speed, I think the script is very fast or at least fast enough.
 Unless you're on 56K of course  ;D

Offline Happy2k

  • User
  • ***
  • Posts: 42
    • View Profile
Re: Multi threaded IMDB fetching?
« Reply #7 on: August 27, 2010, 02:10:39 pm »
Quote
Yea, but silent isnt faster, is it?

In the context of your original question, it's days faster. ::)

Quote
I dont think IMDb would care - ive never heard of IMDb ip banning someone.

Perhaps not, but HTTP error 503 occurs regularly. They probably just mean the server is busy, but it could also mean the server is telling you it's busy because it's getting too many requests from you.

I'd say its worth trying - if it isnt alot of work to get it working.
Again, i wouldnt mind testing it.



 Thanks for clarification Rick, I thought Silent Mode was taking the first movie found (!), that's why I was never using it.
 In terms of speed, I think the script is very fast or at least fast enough.
 Unless you're on 56K of course  ;D

Fast enough? Yesterday i added 700 movies. After 1½ hour, it had completed 270 movies.
Others seems to think so too: http://personal-video-database-portable.en.softonic.com/ (Cons: Database internet retrieval can be slow)
One more: http://www.softsea.com/review/IMDB-Movie-Explorer.html (quote: It is dog slow, but does the job)

Its working - which is the primary goal - but i'd just like to see it a bit faster. If that isnt possible, that's ok too :)

buah

  • Guest
Re: Multi threaded IMDB fetching?
« Reply #8 on: August 27, 2010, 03:16:10 pm »
I imported 5500 entries from IMDB in a day

Offline Happy2k

  • User
  • ***
  • Posts: 42
    • View Profile
Re: Multi threaded IMDB fetching?
« Reply #9 on: August 27, 2010, 04:38:31 pm »
I imported 5500 entries from IMDB in a day

What is your secret?! ;) I started PVD about 2½ hours ago, and it has now passed 500 entries - in silent mode.

Offline nostra

  • Administrator
  • *****
  • Posts: 2852
    • View Profile
    • Personal Video Database
Re: Multi threaded IMDB fetching?
« Reply #10 on: August 27, 2010, 05:40:01 pm »
Any multithreaded system is a lot of work and it is very difficult to make it work good with the current plugin system, so I do not see a quick solution here. I will still keep your suggestion in mind and will take a look at the download performance some time in future.
Gentlemen, you can’t fight in here! This is the War Room!

Offline Happy2k

  • User
  • ***
  • Posts: 42
    • View Profile
Re: Multi threaded IMDB fetching?
« Reply #11 on: August 27, 2010, 05:52:14 pm »
Any multithreaded system is a lot of work and it is very difficult to make it work good with the current plugin system, so I do not see a quick solution here. I will still keep your suggestion in mind and will take a look at the download performance some time in future.

Awesome :) I look forward to that.


Regarding the silent mode.. How do i see which movies it has skipped?

Offline nostra

  • Administrator
  • *****
  • Posts: 2852
    • View Profile
    • Personal Video Database
Re: Multi threaded IMDB fetching?
« Reply #12 on: August 27, 2010, 05:58:44 pm »
Search for movies without URL, for example.
Gentlemen, you can’t fight in here! This is the War Room!

Offline Happy2k

  • User
  • ***
  • Posts: 42
    • View Profile
Re: Multi threaded IMDB fetching?
« Reply #13 on: August 27, 2010, 07:02:57 pm »
Search for movies without URL, for example.

How do you do that? I can find the URL search in Advanced Search, but i cant see how im gonna search for no URL.

buah

  • Guest
Re: Multi threaded IMDB fetching?
« Reply #14 on: August 27, 2010, 07:09:20 pm »
I imported 5500 entries from IMDB in a day


What is your secret?! ;) I started PVD about 2½ hours ago, and it has now passed 500 entries - in silent mode.


2,5hrs x 10=25 hrs

500 x 10= 5000

;D

Offline nostra

  • Administrator
  • *****
  • Posts: 2852
    • View Profile
    • Personal Video Database
Re: Multi threaded IMDB fetching?
« Reply #15 on: August 27, 2010, 07:17:57 pm »
Quote
I can find the URL search in Advanced Search, but i cant see how im gonna search for no URL.

Click on the operator (=) and select IS NULL
Or select the containing operator and set keyword to imdb
Gentlemen, you can’t fight in here! This is the War Room!

Offline rick.ca

  • Global Moderator
  • *****
  • Posts: 3241
  • "I'm willing to shoot you!"
    • View Profile
Re: Multi threaded IMDB fetching?
« Reply #16 on: August 27, 2010, 07:57:54 pm »
I'd say its worth trying - if it isnt alot of work to get it working. Again, i wouldnt mind testing it.

It is a lot of work. It makes much more sense for you to do the testing, and then make the case for multi-threading. I'm not sure it will work, but you could install multiple copies of the program, and run them simultaneously using the -portable and -open=database switches so each uses it's own configuration and database.

Quote
Others seems to think so too: http://personal-video-database-portable.en.softonic.com/ (Cons: Database internet retrieval can be slow) One more: http://www.softsea.com/review/IMDB-Movie-Explorer.html (quote: It is dog slow, but does the job)

You're going to trust an outdated review by an amateur reviewer who tried the program once?! Slow compared to what? The second review is for another program. ::)

Offline Happy2k

  • User
  • ***
  • Posts: 42
    • View Profile
Re: Multi threaded IMDB fetching?
« Reply #17 on: August 27, 2010, 08:34:46 pm »
Quote
I can find the URL search in Advanced Search, but i cant see how im gonna search for no URL.

Click on the operator (=) and select IS NULL
Or select the containing operator and set keyword to imdb

Ahhhh there's actually a menu behind "containing" - awesome!

Dang.. 123 out of 712 have been skipped.


rick.ca -> Gonna try that when im re-doing the entire list! Worth trying at least.

If i had trusted only 1 review, i wouldn't have started using PVD ;) I always do alot more research than a single review.
The times where i trust one single review, is when its authored by me ;) (I do hardware reviews)

But i really like PVD - it was hard getting to work properly in the beginning, but the possiblities are endless.

Offline Happy2k

  • User
  • ***
  • Posts: 42
    • View Profile
Re: Multi threaded IMDB fetching?
« Reply #18 on: August 27, 2010, 08:42:22 pm »
I imported 5500 entries from IMDB in a day


What is your secret?! ;) I started PVD about 2½ hours ago, and it has now passed 500 entries - in silent mode.


2,5hrs x 10=25 hrs

500 x 10= 5000

;D

My math tells me two things: 5000 != 5500, and 25 hours != 24 hours ;)
And then there's the skipped ones. But it helps alot, when most can be done automaticly.

Offline nostra

  • Administrator
  • *****
  • Posts: 2852
    • View Profile
    • Personal Video Database
Re: Multi threaded IMDB fetching?
« Reply #19 on: August 27, 2010, 08:48:10 pm »
Quote
But i really like PVD - it was hard getting to work properly in the beginning, but the possiblities are endless.

Unfortunately it seems like many users have difficulties when starting using PVD. If you have suggestions how to make the process easier for beginners, then feel free to post in the Feature Suggestions board.
Gentlemen, you can’t fight in here! This is the War Room!