Personal Video Database

English => Feature Suggestions => Topic started by: CAD on August 14, 2010, 11:11:49 am

Title: File Scanner Improvements II
Post by: CAD on August 14, 2010, 11:11:49 am
Hi,

When scanning movies in from Hard drive, the "media label" is automatically populated with the volume of the hard drive.
I want to use the volume label as a reference to which hard drive a movie is on.
eg. avatar is on hard drive with volume label - movie001, the matrix is on hard drive with volume label - movie002 etc.

Would it be possible to have "scan folders for new movies/changed paths", also update the "Media Label" field with the Volume as part of the change path scan?
Title: Re: File Scanner Improvements II
Post by: nostra on August 14, 2010, 01:37:47 pm
Yes, it sounds logical.
Title: Re: File Scanner Improvements II
Post by: CAD on August 14, 2010, 02:05:47 pm
Cool,

I'll add it to your TODO list  ;D
Title: Re: File Scanner Improvements II
Post by: rick.ca on August 14, 2010, 08:55:48 pm
In a situation where a changed path is detected, it's possible the file is not the same as the original. For example, it may be one of a better quality replacing the original. So maybe the improvement should be a processing option to rescan files where the path has changed. That would ensure all the file information is correct. It would have to be optional, as the scanner is also used to update paths when entire folders or collections are moved.
Title: Re: File Scanner Improvements II
Post by: nostra on August 15, 2010, 12:48:15 am
So what exactly are you suggesting, Rick? I think the file Scanner already does what you describe and it will also update the media label in future.
Title: Re: File Scanner Improvements II
Post by: rick.ca on August 15, 2010, 02:24:07 am
If I were to obtain a file of a better quality than one I have, put it in a different folder, delete the original file, then run the scanner, it would detect a changed file path and update the path—without rescanning the file. (At least I assume so...Why else would CAD be asking for this?) If it did rescan the file, it would update Media label—but will also ensure all other file information is correct.

It's a minor thing, but from a design point-of-view, I think it's better to update things such as the file information from the same source (the file scanner) at the same time. Exceptions make data inconsistencies more likely. Or to put it another way, when I look at my file information data, I like to know it all came from the last scan of the file. And not that one field might have been changed by Scan folders and another might have been changed by a plugin. If an update is called for, rescan the file.
Title: Re: File Scanner Improvements II
Post by: CAD on August 15, 2010, 03:09:15 am
Quote
I think it's better to update things such as the file information from the same source

Rick are you saying, you would like file scanner to update, all fields related to media information when rescanning?
(not going to list all fields) but quality, duration, video codecs, Media label etc.

I think this a good idea. Adds a bit more processing time, but no different than adding a new movie.
Title: Re: File Scanner Improvements II
Post by: rick.ca on August 15, 2010, 03:44:28 am
Yes. As I said, it should be optional. The default state for me would be "on." If there were a handful of path changes in my scan results, I would not likely notice the file scan time. It takes about 1/2 second per file. If I moved 120 files to a different folder, I might want to turn it off to spare myself that agonizing minute of useless scanning. If I had moved my entire collection to a different drive, it would take a while (and a lot longer than the method you suggested). But for one-time things like that, I don't mind going for a coffee. ;)
Title: Re: File Scanner Improvements II
Post by: CAD on August 15, 2010, 08:59:20 am
One more thing, it should scan for path changes AND Volume changes separately.
I have movies stored on drive caddies (removable drive bays.)

Main movie directory is F:\dvd\moviename (year). The volume label of the drive is Movie001.
When this drive is full (and I have watched all movies on it) I will remove the drive and put it into storage (on a shelf most likely).
The new drive I put in will be mapped to F:\dvd\moviename (year) but will have a volume label of Movie002. Scanner folders will need to check for Media label changes as well as path changes to cover all contingencies.
Title: Re: File Scanner Improvements II
Post by: buah on August 15, 2010, 12:03:43 pm
Rick are you saying, you would like file scanner to update, all fields related to media information when rescanning?
(not going to list all fields) but quality, duration, video codecs, Media label etc.

Quote from: rick.ca
Yes

If I properly understood discussion, then choosing action "Add New Movie" instead of "Change File Path" from the context menu on a selected title in the File scanner, does exactly what are you suggesting as to be implemented?

I'm sorry if I misunderstood suggestion(s).
Title: Re: File Scanner Improvements II
Post by: CAD on August 15, 2010, 02:03:18 pm
Quote
then choosing action "Add New Movie" instead of "Change File Path" from the context menu on a selected title in the File scanner, does exactly what are you suggesting as to be implemented?

Not sure - haven't tried it.
What I'm suggesting is when you scan for HardDrive  path changes (under tools/scan folders for new movies-changed paths), it also checks for changed volumes (the name you give your hard drive). (it doesn't do this at the moment) This is all done locally - no need to download anything. I guess adding a new movie would detect its a duplicate and would download content as well.

hope that clears things up.
Title: Re: File Scanner Improvements II
Post by: buah on August 15, 2010, 02:35:16 pm
Try it, don't guess and hesitate, please. Simulate. Rename some other movie on some other hdd different by size of the one existing in your database. Scan that renamed movie. Right click in the scanner on it. Choose the option "Change action to "Add new movie"". Process it.

If it's good, rescan again, now with the original movie to get back the proper database record

I'm not sure what you meant by "duplicate" and "downloading content".

This action "purges" (overwrites) data about location and file information, and adds new data - those you and rick were asking for (media label, video, audio codec, path, all). The only thing is that you get additional screen-shots which you may want to erase afterward (or in a case of a "better rip" previously to erase screenshots of a "bad rip".

Again, maybe I comprehended something incorrectly.
Title: Re: File Scanner Improvements II
Post by: buah on August 15, 2010, 05:35:24 pm
P.S. Actually, the only option that doesn't exist at the moment regarding File scanner is the one I asked for (http://www.videodb.info/forum_en/index.php?topic=2055.0): to allow getting an additional database record regardless the existing title as the one in the database is recognized by File scanner
Title: Re: File Scanner Improvements II
Post by: CAD on August 16, 2010, 04:14:10 am
Hi buah.

The "scan folders for new movies/changed paths" options seems to be scanning for fields related to the media - a local change.
ie It scans the file(s) sitting on your hard drive for information for information about it.
 
Add movie requires searching /downloading content from internet.

The way I think change path should work is:
Program detects path has changed OR Media Label (Drive Volume)  and updates this information.
(optionally updates other fields related to the file media - such as video codec/audio codec etc).
Itr gets all fo this information from the actual file that is stored on hard drive.

Adding it as a new movie would require (re)downloading content from internet and comparing for changes.

Hope this clears up what i am trying to say.

PS - I will try your suggestions when I get a chance.
PPS - This issue is not related to your original post "File Scanner Improvements" Sorry to hijack the Subject title. :)
 
Title: Re: File Scanner Improvements II
Post by: rick.ca on August 16, 2010, 08:10:55 am
I've made a huge mistake. :'(

But first, the little one...

If I properly understood discussion, then choosing action "Add New Movie" instead of "Change File Path" from the context menu on a selected title in the File scanner, does exactly what are you suggesting as to be implemented?

OIC. Finally. Maybe. I always assumed "Add New Movie" meant "Add New Movie"—not "replace existing file with this file and rescan." ::)

After figuring this out, I decided to experiment further and see if I could improve my scanning routine. My mistake was forgetting I had learned long ago to leave well enough alone. The scanner works fine in a one-movie/one-file situation. A file is scanned and matched to a movie record. If the movie record has no file recorded, it must be a new addition. If the path is not the same, the movie's file must have been moved. But if there are multiple files for one movie, then things become more challenging.

Now, regrettably, I'll have to explain why I have multiple files for some movies. I tend to download, watch and delete. That's not an issue for PVD—it doesn't care if the file in the file path has been deleted. But it is for my media manager. It can only manage information for files that exist. Furthermore, it gets all it's video data from the PVD database, and the common file path is what links each record in the two databases. My workaround is to use a macro that creates a null file with the same name, moves that to a "viewed" folder, and then deletes the file. Both the PVD and media manager scanners detect and record the change, and the meta data for the dummy file is automatically updated. Strange, but all this works fine.

Things get more complicated, however, when I decide to keep a movie after viewing it. In order to have a record of viewing the movie in the media manager, I have to create the dummy file. Unfortunately, the scanner is not smart enough to propose adding the new dummy file to the existing movie file—it will only suggest replacing it. I can override this by changing the action to "Add New Movie." That will add another entry, and "Same movie" can be used to combine them. That, of course, is not at all practical—it's easier to just manually change the file path. Until recently, this has worked fine for me—probably because the number of movies I keep is small.

I recently started collecting trailers. I decided they'd be worthwhile having if they could be recorded in the PVD file path, and thereby integrated with my media manager. That much works very well. I use PVD to find the trailers, and as they're downloaded, I rename them by cutting and pasting the movie file name. At the same time, I cut and paste the trailer path name back into File path (before the movie path, so it would play first). I'll probably continue to do this, rather than rely on the scanner to add the trailers.

What didn't occur to me until I had a sizable collection of trailers, was this would "break" my scanning routine. Now, if I attempt to replace a viewed movie with a dummy file and rescan, the scanner is only able to suggest replacing the trailer (which I want to keep) with the dummy file.

I still haven't got to my huge mistake. Some testing suggested to me (incorrectly, it seems), that the scanner could handle multiple files if they were named in the same way multiple disc movies are. So I renamed all my trailers by appending "-1," movies with a "-2" and view log files with a "-3." Rescanning worked—after fixing lots of exceptions. But it made no difference when it came to moving/changing one of those files. So I tried moving them into the same folder and rescanning. It still didn't work. To add insult to injury, the scanner was unable to properly detect the changes when I moved the files back to their respective folders.

So after many hours of recovery, I'm back where I started. It seems I can't record trailers in File path and use the scanner to detect the change of the media files to dummies as movies are viewed. I'm sure I'm the only one with this particular issue, so maybe I'm expecting to much. But I find it disappointing the scanner does not compare each file detected to the files already associated with a movie, and propose actions based on that. Most of the time, the correct action would be to remove the file reference that no longer exists, and if a file previously referenced still exists, leave it and add the new file.

And as weak nod to staying on topic... If it worked this way, it would be possible to determine whether a file scan is necessary and, if so, how it should be done (i.e., multiple or single file scan). :-\
Title: Re: File Scanner Improvements II
Post by: buah on August 16, 2010, 08:55:30 am
While it was really hard for me to track your routine (:-\), I'll point that there are two options in the context menu for a file(s) scanned in a File scanner window:

1. Add new movie

2. Change action to "Add new movie"


I suggested to use the second option, because that option gives results you've asked for at the beginning of the topic. In addition, if I understood rick's most recent post, there is a checkbox "Add to existing files" in the window that opens after choosing second option, mentioned above. Maybe that could help with adding trailers or any other video file(s)? It works for me in cases when adding "Makin of" documentaries for a movie, for instance.
Title: Re: File Scanner Improvements II
Post by: nostra on August 16, 2010, 10:04:38 am
Hmm, it seems like the scanner will need a partial rewrite for the version 1 :-/
Title: Re: File Scanner Improvements II
Post by: rick.ca on August 16, 2010, 11:18:24 am
Quote
While it was really hard for me to track your routine...

They're mentioned in there somewhere. :P

In the case where the scan has found only one of two files...

1. allows whatever file(s) were missed by the scan to be added as a separate movie, which then has to be combined with the original using "same move;" not only is this a PITA, it's not likely to result in the files being in the correct order (i.e., trailer first).

2. will just result in the one file being recorded, and the file information will be updated.

So, fiddling with the scanner results is not a efficient as simply changing the results directly. And that's a lot safer, as in the ordinary course of things one would have to be very careful to make all the necessary adjustments to the scan results. Curiously, if the existing File path contents are deleted before, the scanner will correctly add all the existing files. That might be the workaround to use in my regular work flow. After viewing a movie, I'll mark it viewed, rate it, delete File path and replace the media file with a log file. Then (maybe at a later time) I'll run the scan.

I'll just have to move all my trailers from \Trailers to something that comes before \Movies. Then the scanner will add the files (including the logs in \Viewed) in the correct order. Unfortunately, because of this issue, I can't just rename the folder and rescan. ::) :'(

Quote
Hmm, it seems like the scanner will need a partial rewrite for the version 1 :-/

You mean it wasn't getting a complete rewrite anyway?! ;D

Maybe you already know what the problem is—I still have difficulty understanding exactly what it's doing. I wonder if there's any promise of an easy fix in my latest observation (highlighted above). If the scanner is able to correctly add all matching files when there's nothing in File path, then it shouldn't be confused by existing data. On the other hand, I don't think it is or should be quite that simple. For example, if a file has been moved, it doesn't seem right that the scanner should fail because of a regex issue. It should be able to match the file to the movie on the basis a file with the same name is currently recorded in that movie's File path. :-\
Title: Re: File Scanner Improvements II
Post by: CAD on August 16, 2010, 12:36:13 pm
Quote
Try it, don't guess and hesitate, please. Simulate.
Quote
choosing action "Add New Movie" instead of "Change File Path" from the context menu on a selected title in the File scanner,
Ok tried this. Didn't work.

The scanner needs to search that the  "Volume label" of the drive has changed as part of its detection. Not just the path.
I can have the same path but with a different volume label. ie when I swap drives out.

If I change path and then add movie - Yes it does update Media label (volume), but it also re-downloads everything from internet (not what I wanted and not necessary. This has already been done (and checked to be correct).
Selecting "Same movie" does not update Media label.

I do not get to choose anything if the path has not changed (but volume label has) to select add movie as an option.


Title: Re: File Scanner Improvements II
Post by: buah on August 16, 2010, 04:32:51 pm
It works, at least for me. See complete step-by-step process in pictures attached

[attachment deleted by admin]
Title: Re: File Scanner Improvements II
Post by: buah on August 16, 2010, 04:33:44 pm
Please see previous post first. These are pictures after updating file information.
IMDb information hadn't change, as you can see.

[attachment deleted by admin]
Title: Re: File Scanner Improvements II
Post by: CAD on August 17, 2010, 01:01:06 am
hi buah.
thanks for explaination.
Your process will only work (for me) if PVD detects a changed file path.
The core issue is at detection phase. PVD needs to be able to tell that the volume has been changed not just the path.
Once it has done this it can process as per your process

If you movie 20 movies and you move them from drive i: (with volume label of 15G) to drive h:(with volume label of 1TB) - PVD will detect that the path has changed and give you options to process these movies.

If you move movies from drive i: (with volume label of 15G) to c:\drive. Then swap 15G drive out with 1TB drive. Then copy movies from c: drive to 1TB drive. Now you have 1TB drive with movies on it, but path is still i: - PVD will not detect that path has changed. (it is still i: drive), but the volume label has changed from 15G to 1TB.
Title: Re: File Scanner Improvements II
Post by: nostra on August 17, 2010, 01:30:34 am
To be honest, CAD, I do not think such a functionality of detecting changed labels, but same paths will be available any time soon as I really can't imagine many users having such situations and you could just have changed the label (not switched hard drives) and the whole thing only to update one field (not a critical one)...
Title: Re: File Scanner Improvements II
Post by: buah on August 17, 2010, 01:54:50 am
Hi CAD.
Thanks for your patience.

Now I see that your original idea for File scanner was to be able to update volume change, regardless of path change, which wasn't clear to me before your most recent post, mostly because I don't have such a routine for storing movies (and it was obvious when I didn't know why someone would move movies so often:))

With my routine (discussed in some other topic) I can never be confused where my movie should be ("ID" story)

Although I don't need such an option, I agree it should work the way you suggested, of course.

Edit: nostra posted his post while I was typing mine, but I won't change it, because it complies with what nostra wrote (not many users...)
Title: Re: File Scanner Improvements II
Post by: rick.ca on August 17, 2010, 02:40:03 am
If you move movies from drive i: (with volume label of 15G) to c:\drive. Then swap 15G drive out with 1TB drive. Then copy movies from c: drive to 1TB drive. Now you have 1TB drive with movies on it, but path is still i: - PVD will not detect that path has changed. (it is still i: drive), but the volume label has changed from 15G to 1TB.

It's possible to reserve a drive letter for each drive used. I don't have need for it, but it seems USB Drive Letter Manager (http://www.uwe-sieber.de/usbdlm_e.html) is the tool to use.

But aside from that, you should be able to change those volume labels using Re-read file information. With 1TB attached, use Advance search to select Media label = 15G and re-scan. The movies with files on 1TB will have their Media label updated. The movies with files on 15G (only) will be unaffected because the files will not be found. I'm assuming, of course, the file scanner just skips these and doesn't report errors.
Title: Re: File Scanner Improvements II
Post by: CAD on August 17, 2010, 02:49:28 am
Hi buah,
glad you understand what I am getting at. :D

Quote
To be honest, CAD, I do not think such a functionality of detecting changed labels, but same paths will be available any time soon as I really can't imagine many users having such situations and you could just have changed the label (not switched hard drives) and the whole thing only to update one field (not a critical one)...

Hi Nostra,
I was going to say that users could rename the volume label, and paths would remain the same.
their databse would then have an incorrect volume label.

Also (as rick hinted) it would be great if all fields (that can be read from the stored media - eg video codecs etc) are updated as part the path chage detection. For me the Media label (volume label) is most important, but I can see how it would be useful to update other fields.

Quote
It's possible to reserve a drive letter for each drive used. I don't have need for it, but it seems USB Drive Letter Manager is the tool to use.
Yes - I can force pvd to recognise movies have changed path, but the change path scan does not update "media label field" which I would like to use to know which HD a particular movie is on.

Quote
you should be able to change those volume labels using Re-read file information
I will try this....when I find it (when i get home) may be easy solution.
Just tried it - and it updates volume label, should be a good work around.  :D
Title: Re: File Scanner Improvements II
Post by: buah on August 17, 2010, 06:54:21 am
It's possible to reserve a drive letter for each drive used.

This is exactly what I do, and what I intended to write about. I always assigns drives as follows:

C: Windows and Program Files drive
D: My Documents, Backup and pagefile.sys drive
E:-N: External and/or other internal mass storage drives
P:-T: USB stick drives
U:-Z: Optical and/or virtual drives.
 
Disc Management console does this job perfectly for me.

What I concluded was that you either used automatic assigning first available drive letter, or you purposely assigned same letter. If first, then you should reserve drive letter for each external HDD, and if second, you have a problem when both attached at the same time - one of them wouldn't recognized by Windows.

Quote from: CAD
Quote from: rick
It's possible to reserve a drive letter for each drive used.
Yes - I can force pvd to recognise movies have changed path, but the change path scan does not update "media label field" which I would like to use to know which HD a particular movie is on.

Now, I'm confused again. If you reserve different letter for each drive, it implies path changing, so scanning as I described earlier overwrites Media label field? But, this was rather rhetorical, you don't have to reply on this.
Title: Re: File Scanner Improvements II
Post by: CAD on August 17, 2010, 11:30:42 am
tested reread file information on my main database and seems to do the job with no ill effects.
A bit more convoluted but a good solution.

I use disk management console extensively - my partitions are mounted as sub-directories under c:

Quote
Now, I'm confused again. If you reserve different letter for each drive, it implies path changing, so scanning as I described earlier overwrites Media label field? But, this was rather rhetorical, you don't have to reply on this.

You are correct that selecting add movie will overwrite media label field, but it also redownloads everything (unless I missed something).
This thread got a bit sidetracked with our two needs.

My original request was for "scan new folders" to also update media label field (which I'm hoping Nostra will still consider) :)
Rick rightly suggested all these media related fields could be updated.(which I'm hoping Nostra will still consider) :)
A step further that would cover all contingencies (and got a bit lost in translation) would be to detect path change OR media label change and update media related fields  (which I'm hoping Nostra will still consider) :) - but he probably wont :(